JOB DESCRIPTION

Data Quality (DQ) Observability Engineer

• Familiarity with cloud platforms and containerization technologies (Data Bricks).
• Proficiency in scripting and automation using languages such as Python,
• Excellent problem-solving skills with the ability to analyse complex issues and provide efficient solutions.
• Strong communication skills and the ability to collaborate effectively across teams.
• Understanding of Agile and DevOps and GIT principles and their application in observability and monitoring contexts.
• Solid experience with instrumentation practices, including metrics, logging, and distributed tracing.
• E2E implementation. Right from requirement gathering and DQ design engaging with the Data Asset owners, development, and reporting.
• Development DQ rules with the help of Databricks
• Presentation of DQ results to the Data Asset Owners and other stakeholders and work with them to create an action plan to improve the DQ results.
• Putting in place specific Data Improvement Plans
• Scale up the DQ framework to all the data assets and more.
• Ensure systems run efficiently, monitoring system performance and manage support in demanding environment.
• Manage the configuration of existing toolset for data quality, KPI system and Reporting.
• Proficiency in implementing and maintaining observability tools is a plus

Qualifications and Experience Required

• Degree level education, preferably with overall 7-8 years of experience in SQL, Data warehousing and hands on Data Science
• Must be proficient and have experience in Data Science related technologies.
  • Strong Python/ PySpark and SQL development skills
  • Experience in Azure Databricks
  • Agile development methodology using DevOps and GIT repos.
• Experience with Big Data tools and Cloud environment, especially Azure is a big plus.
• Expertise in converting Functional requirements to technical solutions and present in the business context.
• Experience in working with Hub and Spoke model data lakes, preferably with Diageo context is a big plus.
• Keeping pace with IT trends, innovation and new developments in technology space
• Strong Communication Skills required.