Data Engineers - Databricks
Accenture
Brisbane, Queensland, Australia
As a Databricks Data Engineer, you will play a key role in designing, implementing, and maintaining data pipelines on the Databricks platform that support business and technology objectives. Your primary focus will be on building scalable and efficient data ingestion, transformation, and processing solutions to support our data-driven initiatives. You enjoy working both autonomously and as part of a team and have the confidence to influence and communicate with stakeholders at all levels, and to work in a fast-paced complex environment. About the role: You will be working on client projects with teams from across our Modern Data Platform and Applied Intelligence practice alongside our industry, functional and technology SMEs on some of our clients most challenging projects. From day one, you will be involved in the design and implementation of complex data solutions ranging from batch to streaming and event-driven architectures, across cloud, on-premise and hybrid client technology landscapes. We are looking for 3+ years of experience in data engineering in a customer or business facing capacity and experience in the following: Preferred • Ability to understand and articulate requirements to technical and non-technical audiences • Stakeholder management and communication skills, including prioritising, problem solving and interpersonal relationship building • Experiences with design, develop, and implement end-to-end data engineering solutions using Databricks for large-scale data processing and data integration projects. • Build and optimize data ingestion processes from various sources, ensuring data quality, reliability, and scalability. • Perform data transformation tasks, including data cleansing, aggregation, enrichment, and normalization, using Databricks and related technologies. • Monitor and troubleshoot data pipelines, identifying and resolving performance issues, data quality problems, and other technical challenges. • Implement best practices for data governance, data security, and data privacy within the Databricks environment. • Strong SQL, Python, PySpark knowledge • Collaborate with DevOps and infrastructure teams to optimize the performance and scalability of Databricks clusters and resources. • Provide guidance and mentorship to junior data engineers, fostering a culture of knowledge sharing and continuous learning within the team. Desirable • Knowledge in DataOps and experience in delivering CI/CD and DevOps capabilities in a data environment • Experience with advanced analytics and machine learning frameworks such as Apache Spark MLlib, TensorFlow, or PyTorch. • Proficiency in data visualization tools such as Tableau, Power BI, or Looker • Certification in Databricks Engineer Professional would be a plus
