Job description 

Inviting applications for the role of Senior Principal Consultant-Team Lead – Data Engineer!

Responsibilities
1. Design, develop, and maintain data pipelines and ETL processes on AWS using Lambda functions, S3 buckets, Aurora, and AWS RedShift with Spectrum.
2. Collaborate with stakeholders to understand data requirements and implement scalable solutions.
3. Experience working with batch orchestration tools such as Apache Airflow or equivalent, preferable Airflow
4. Hands-on Python experience including Batch scripting, data manipulation, distributable packages
5. Implement data quality frameworks, governance, and security measures to ensure data quality, integrity, and privacy.
6. Monitor and troubleshoot data pipelines to identify and resolve issues in a timely manner.
7. Experienced and proactive in end-to-end data validation
1. 6. Collaborate with cross-functional teams to integrate data from various sources and systems.
8. Perform performance tuning and optimization of data processing jobs.
9. Stay updated with the latest trends and technologies in data engineering and AWS services.
10. Focus Areas:
a. Designing and developing data pipelines using AWS Lambda, S3 buckets, Athena, Aurora and RedShift with Spectrum.
b. Build Data quality, governance, automation frameworks using python.
c. Integration of data from multiple sources and systems.
d. Keeping up with emerging technologies and best practices in data engineering and AWS services.

Qualifications we seek in you!
Minimum Qualifications
1. Bachelor's degree in computer science, information systems, or a related field.
2. Experience in data engineering or a similar role.
3. Demonstrated experience in designing and implementing data pipelines on AWS using Lambda, S3, and RedShift with spectrum.
4. Proficiency in SQL and data manipulation techniques.
5. Strong understanding of database systems and data warehousing concepts
6. Key Skill Sets:
1. Strong experience with AWS services such as Lambda, S3 buckets, Athena, and RedShift with Spectrum.
2. Proficiency in Python, SQL, and data manipulation techniques.
3. Experience with data validation, quality and related frameworks’ implementation, automation frameworks development.
4. Familiarity with data quality frameworks and best practices.
5. Strong problem-solving and troubleshooting skills.
6. Experience with version control systems and CI/CD pipelines for code deployment.
7. Ability to write clean, efficient, and reusable code.

Preferred Qualifications/skills will be plus:
1. Master's degree in computer science, information systems, or a related field.
2. AWS certification in data engineering or related field.
3. Experience with other AWS services such as Glue, StepFunctions, or EMR.
4. Experience Metadata tool and metadata management tool.
5. Familiarity with big data processing frameworks like Apache Spark.
6. Knowledge of machine learning concepts and frameworks.