Job description 

Responsibilities:

• Design and implement data integration, acquisition, cleansing, harmonization, and transformation processes to create curated high-quality datasets for the usage of vector stores/embeddings.
• Maintain scalable data processing pipelines and monitor technology trends and advancements in Generative AI and incorporate them to improve.
• Collaborate with Solution Architects, Data Scientists, Software Engineers and DevOps Engineers, Product Owners, researchers and business stakeholders on the cross-functional team.

Skills and Qualifications:

• 7+ years of experience in data engineering and 6 months of hands-on experience in Generative AI technologies like vector databases.
• Hands-on experiences on text processing tools (e.g. spaCy, NLTK, Word2vec, pyTorch) and data engineering tools (e.g. Hadoop, Airflow, Pandas)
• Experience with at least one cloud platform like Azure, AWS, GCP
• Hands-on experience in software development with one major programming language (e.g. Python) and experience in machine learning and Generative AI models
• Understand the difference, advantages, and disadvantages between the most common large language models (GPT, Llama)
• Hands-on experiences on information retrieval tools (e.g. Chroma DB, Pinecone, Vector, Elasticsearch or other vector stores)
• Good overview of re-usable frameworks and tools in the field of Generative AI (both commercial and open source)