Big Data Engineer

Apply Join for More Updates

You must Sign In before continuing to the company website to apply.

Description

Skill: Azure-ADF,Synapse, SQL,PySpark and any ETL Informatica/SSIS/Datastage

Must Have

•5+ years of IT experience in Datawarehouse

•Hands-on data experience on Cloud Technologies on Azure, Synapse, ADF, DataBricks, PySpark

•Prior Experience on any of the ETL Technologies like Informatica Power Centre, SSIS, DataStage

•Ability to understand Design, Source to target mapping (STTM) and create specifications documents

•Flexibility & willingness to work on non-cloud ETL technologies as per the project requirements, though main focus of this role is to work on cloud related projects

•Flexibility to operate from office location

•Able to mentor and guide junior resources, as needed

•Banking experience on RISK & Regulatory OR Commercial OR Credit Cards/Retail

Nice to Have

•Any relevant certifications

Set alert for similar jobsBig Data Engineer role in Chennai, India

Company

Hexaware Technologies

Job Posted

a year ago

Job Type

Full-time

WorkMode

On-site

Experience Level

3-7 years

Related Jobs

Data Integration Lead

Hexaware Technologies

Chennai, Tamil Nadu, India

Posted: a year ago

Description Data Integration Lead - Talend Developer Offshore   Responsibilities •  Leads the delivery processes of data extraction, transformation, and load from disparate sources into a form that is consumable by analytics processes, for projects with moderate complexity, using strong technical capabilities and sense of database performance •  Designs, develops and produces data models of relatively high complexity, leveraging a sound understanding of data modelling standards to suggest the right model depending on the requirement •  Batch Processing - Capability to design an efficient way of processing      high volumes of data where a group of transactions is collected over a period  •  Data Integration (Sourcing, Storage and Migration) - Capability to design and implement models, capabilities, and solutions to manage data within the enterprise (structured and unstructured, data archiving principles, data warehousing, data sourcing, etc.).  This includes the data models, storage requirements and migration of data from one system to another •  Data Quality, Profiling and Cleansing - Capability to review (profile) a     data set to establish its quality against a defined set of parameters and to highlight data where corrective action (cleansing) is required to     remediate the data  •  Stream Systems - Capability to discover, integrate, and ingest all available data from the machines that produce it, as fast as it is produced, in any  format, and at any quality •  Excellent interpersonal skills to build network with variety of department across business to understand data and deliver business value and may interface and communicate with program teams, management and stakeholders as required to deliver small to medium-sized projects •  Understand the difference between on-prem and cloud-based data integration technologies.   The Role offers •  Opportunity to join a global team to do meaningful work that contributes to global strategy and individual development  •  An outstanding opportunity to re-imagine, redesign, and apply technology to add value to the business and operations •  Gives an opportunity to showcase candidates’ strong analytical skills and problem-solving ability •  Learning & Growth opportunities in cloud and Big data engineering spaces   Essential Skills   •  6+ years’ experience in developing large scale data pipelines in a cloud/on-prem environment. •  Highly Proficient in any or more of market leading ETL tools like Informatica, DataStage, SSIS, Talend, etc., •  Deep knowledge in Data warehouse/Data Mart architecture and modelling •  Define and develop data ingest, validation, and transform pipelines. •  Deep knowledge of distributed data processing and storage •  Deep knowledge of working with structured, unstructured, and semi structured data •  Working experience needed with ETL/ELT patterns •  Extensive experience in the application of analytics, insights and data mining to commercial “real-world” problems •   Technical experience in any one programming language preferably, Java,.Net or Python   Essential Qualification •  BE/Btech in Computer Science, Engineering or relevant field  

Big Data Lead

Hexaware Technologies

Pune, Maharashtra, India

+2 more

Posted: a year ago

Description   Senior ADF Engineer Work timing: 12pm to 9.30pm IST As a Senior Azure ADF Engineer, you will be responsible for designing, developing, and maintaining data pipelines using Azure Data Factory (ADF). You will work closely with customers to understand their business requirements and translate them into scalable, performant, and secure data integration solutions. You will be responsible for ensuring that the data pipelines integrate seamlessly with other Azure services, such as Azure Blob Storage, Azure Synapse Analytics , SQL Database, and Azure Data Lake Storage. Key Responsibilities: - Design, develop, and maintain data pipelines using Azure Data Factory (ADF). - Work closely with customers to understand their business requirements and translate them into scalable, performant, and secure data integration solutions. - Collaborate with other architects, engineers, and technical teams to ensure that data pipelines are aligned with overall solution architecture and best practices. - Optimize data pipelines for performance, scalability, and cost efficiency. - Develop and maintain technical documentation, including pipeline diagrams, data flow diagrams, and code documentation. - Participate in data-related aspects of pre-sales activities, including solution design, proposal development, and customer presentations. - Stay current with emerging technologies and industry trends related to data integration and management in Azure, and provide recommendations for improving existing solutions and implementing new ones.   : - Bachelor's or master's degree in computer science, engineering, or a related field. - At least 7 years of experience in data engineering, including at least 3 years of experience with Azure Data Factory. - Strong knowledge of Azure Data Factory, including data ingestion, transformation, and orchestration. - Experience with data integration and ETL processes and writing SQL queries and SQL/Scripts. - Experience with Azure Data lake and building pipelines for batch and CDC (change data capture). - Experience with implementing ETL frameworks. - Strong understanding of data security, including encryption, access control, and auditing. - Excellent communication and presentation skills, with the ability to effectively communicate complex technical concepts to both technical and non-technical audiences. - Strong problem-solving and analytical skills, with the ability to identify and resolve complex technical issues. Good to have Qualifications : - Microsoft Certified: Azure Data Engineer Associate certification. - Experience with other Azure data services, such as Azure Synapse Analytics and Azure Databricks. - Experience with other data integration tools, such as Talend. - Experience with programming languages, such as SQL, Powershell.

Data Integration Lead

Hexaware Technologies

Bengaluru, Karnataka, India

+2 more

Posted: a year ago

Description   Responsibilities •  Leads the delivery processes of data extraction, transformation, and load        from disparate sources into a form that is consumable by analytics                processes, for projects with moderate complexity, using strong technical        capabilities and sense of database performance •  Designs, develops and produces data models of relatively high                        complexity, leveraging a sound understanding of data modelling                    standards to suggest the right model depending on the requirement •  Batch Processing - Capability to design an efficient way of processing      high volumes of data where a group of transactions is collected over a          period  •  Data Integration (Sourcing, Storage and Migration) - Capability to design      and implement models, capabilities, and solutions to manage data within      the enterprise (structured and unstructured, data archiving principles,            data warehousing, data sourcing, etc.).  This includes the data models,          storage requirements and migration of data from one system to another •  Data Quality, Profiling and Cleansing - Capability to review (profile) a     data set to establish its quality against a defined set of parameters and to     highlight data where corrective action (cleansing) is required to     remediate the data  •  Stream Systems - Capability to discover, integrate, and ingest all available     data from the machines that produce it, as fast as it is produced, in any         format, and at any quality •  Excellent interpersonal skills to build network with variety of department       across business to understand data and deliver business value and may         interface and communicate with program teams, management and               stakeholders as required to deliver small to medium-sized projects •  Understand the difference between on-prem and cloud-based data               integration technologies. The Role offers •  Opportunity to join a global team to do meaningful work that contributes      to global strategy and individual development  •  An outstanding opportunity to re-imagine, redesign, and apply                      technology to add value to the business and operations •  Gives an opportunity to showcase candidates’ strong analytical skills and        problem-solving ability •  Learning & Growth opportunities in cloud and Big data engineering              spaces Essential Skills   •  6+ years’ experience in developing large scale data pipelines in a                   cloud/on-prem environment. •  Highly Proficient in any or more of market leading ETL tools like                      Informatica, DataStage, SSIS, Talend, etc., •  Deep knowledge in Data warehouse/Data Mart architecture and                    modelling •  Define and develop data ingest, validation, and transform pipelines. •  Deep knowledge of distributed data processing and storage •  Deep knowledge of working with structured, unstructured, and semi             structured data •  Working experience needed with ETL/ELT patterns •  Extensive experience in the application of analytics, insights and data              mining to commercial “real-world” problems •   Technical experience in any one programming language preferably, Java,       .Net or Python Essential Qualification •  BE/Btech in Computer Science, Engineering or relevant field JD: • Strong working knowledge of IICS - Informatica Cloud, Informatica designer transformations like Source Qualifier , Dynamic and Static Lookups , connected and Unconnected lookups , Expression , Filter , Router , Joiner , Normalizer and Update Strategy transformation. • Solid hands-on development experience in Informatica PowerCenter - usage of reusable transformations, aggregates, lookups, caches, performance tuning, joiners, rank, router, update strategy etc. • Strong Knowledge on Snowflake  • Experience in working in Data Warehousing - Must have knowledge of Data Warehousing concepts - SCD1, SCD2 etc. • Troubleshoot issues and identify bottlenecks in existing data workflows. • Provides performance tuning insight and create reusable objects and templates. • Should be strong in migrating object process from Lower Environment to higher Environment. • Should be strong in scheduling process of the workflows, tasks and mappings. • Basic understanding of Informatica Administration • Strong development skills in SQL. Having knowledge on AWS, HVR would be an added advantage • Should have good communication and customer interaction skills

Data Integration Lead

Hexaware Technologies

Bengaluru, Karnataka, India

+3 more

Posted: a year ago

Description Job Description 1. 6+ years’ experience in developing large scale data pipelines in a cloud/on-prem environment. 2. Highly Proficient in any or more of market leading ETL or Application Integration tools like Preferred (Informatica Intelligent Cloud Services),Informatica, SSIS, Talend, etc., 3. Deep knowledge on Data Cleansing, Data Profiling and Data Integrations using the Data Rules with API Integration (thorough knowledge on using Connectors, Process Objects or variable mechanism, PostMan. 4. Thorough knowledge on SQL Server Queries and Procedures 5. Deep knowledge in Data warehouse/Data Mart architecture and modelling 6. Define and develop data ingestion, validation, and transform pipelines. 7. Deep knowledge of distributed data processing and storage 8. Deep knowledge of working with structured, unstructured, and semi structured data 9. Working experience needed with ETL/ELT patterns 10. Extensive experience in the application of analytics, insights and data mining to commercial “real-world” problems 11. Technical experience in any one programming language preferably, Java, .Net or Python

Cloud Data Architect

Hexaware Technologies

Chennai, Tamil Nadu, India

+2 more

Posted: a year ago

Description   Responsibilities: Demonstrate knowledge of cloud architecture and implementation features (OS, multi-tenancy, virtualization, orchestration, scalability). Act as a Subject Matter Expert to the organization for cloud/On-prem end-to-end architecture, including/ any one of AWS, GCP, Azure and future providers, networking, provisioning, and management. Communicate and provide thought leadership in Data strategy, Technologies, engineering to business and IT leadership teams and stakeholders. Providing a vital function bridging the gap between business and technology. In partnership with Product Owner(s), Stakeholders and other Subject Matter Experts, and work with technologies such as Cloud data warehousing, Data lakes, Analytical platforms and solutions.  Should be able to define modern data eco system, considering current and futuristic data needs. Work closely with all business areas to capture BI (Business Intelligence, Data Analytics and Reporting) requirements based on business objectives, initiatives, challenges and questions. Translate user stories into wireframe designs of the dashboards. Provide BI / reporting on all the identified KPI areas specified in the KPI requirements document by creating, developing and maintaining BI reports in visualization tools such as Microsoft Power BI. The dashboards will be updated and maintained by the various business teams (self service).  Should be able to cover astatic design aspects in report/dashboard design. Provide advice on technical aspects of BI development and integration, including the operational and maintenance aspects of systems under development, and proposed system recovery procedures. Ensure that relevant technical strategies, policies, standards, and practices are applied correctly. Support the Reporting & Analytics team to respond positively to proposed data initiatives, arising from the Information Governance Group, and provide the necessary support and expertise to deliver such initiatives. Provide ongoing support and implementation of change for the ETL solution interfacing between the applications. Apply enterprise data management practices and principles in ways of working. Identify patterns, trends and insight through use of the appropriate tools. Identify data quality issues through data profiling, analysis and stakeholder engagement. Analyze data to provide insights for internal and external stakeholders  on improving the performance. The Role offers: Opportunity to join a global team to do meaningful work that contributes to global strategy and individual development.  An outstanding opportunity to re-imagine, redesign, and apply technology to add value to the business and operations. Essential Skills: Hands-on-experience in delivering business intelligence solutions. Hands-on-experience with OLTP and OLAP database models. Expertise in end to end (data base + ETL /pipeline + Visualization Reporting) on one of the Cloud or Cloud Agnostic Azure: Synapse, ADF, HD Insights. Expertise in areas of data governance, advanced analytics on premise platform understanding covering one or more of the following skills Teradata, Cloudera, Netezza, Informatica, DataStage, SSIS, BODS, SAS, Business Objects, Cognos, MicroStrategy, WebFocus, Crystal. Programming: Write computer programs and analyze large datasets to uncover answers to complex problems.  Fluent in relational database concepts and flat file processing concepts. Must be knowledgeable in software development lifecycles/methodologies i.e. agile. Data storytelling: Communicate actionable insights using data, often for a non-technical audience. Business intuition: Connect with stakeholders to gain a full understanding of the problems they are looking to solve. Analytical thinking. Find analytical solutions to abstract business issues. Critical thinking: Apply objective analysis of facts before concluding. Interpersonal skills: Communicate across a diverse audience across all levels of an organization. Has strong presentation and collaboration skills and can communicate all aspects of the job requirements, including the creation of formal documentation. Strong problem solving, time management and organizational skills. Database Technology: SQL Server, Netezza, Hadoop, Cloudera. ETL Technology: SSIS, DataStage, Talend, CRON scripting, Perl. BI Technology: SSRS, SSAS, Tableau, MicroStrategy. Cloud SaaS:  Snowflake, Databricks, Matillion, HVR, etc. Familiarity in Data Engineering tools, data ops tools in On-prem and Cloud eco system. Good to have certifications in Togaf, PMP, CSM. Essential Qualification: BE or Btech in Computer Science, Engineering, or relevant field.

Senior Software Engineer - Big Data

Freshworks

Chennai, Tamil Nadu, India

Posted: a year ago

Job Description The primary responsibilities of the role include: Design and develop a real-time data pipeline for Data ingestion for real-time business usecases Develop complex and efficient functions to transform raw data sources into powerful, reliable components of our data lake Grow our analytics capabilities with faster, more reliable data pipelines, and better tools, handling petabytes of data every day. Brainstorm and create new platforms features, which can help in our quest to make data available to cluster users in all shapes and forms, with low latency and horizontal scalability. Make changes to our data platform, refactoring/redesigning as needed and diagnosing any problems across the entire technical stack. Think outside the box with to implement solutions with new components and various emerging technologies in AWS, and Open Source for successful execution of various projects Optimize and improve existing features or data processes for performance and stability. Write unit tests and support continuous integration. Be obsessed with quality and ensure minimal production downtimes. Mentor peers, share information and knowledge, and help build a great team. Monitor job performances, file system/disk-space management, cluster and database connectivity, log files, management of backup/security, and troubleshoot various user issues. Collaborate with cross-functional and business teams Qualifications We are looking for a candidate with proven experience in Big Data Engineering role with hands-on expertise in Apache SparkTM (Scala or PySpark Preferred) and associated performance optimization Advanced working Knowledge in SQL and working familiarity with a variety of databases. Working knowledge of various API interfaces for Bulk or Stream-based data extraction and load processes is a must Experience building and deploying a range of data engineering pipelines into production, including using automation best practices for CI/CD Experience performing root cause analysis on all data and processes to answer specific questions and identify opportunities for improvement. Build processes supporting data transformation, data structures, metadata, dependency and workload management. A successful history of manipulating, processing and extracting value from large disconnected datasets. Working knowledge of Kafka, Spark, stream processing, and scalable 'big data' data stores. Experience with cloud solutions on top of AWS Good to have ML-ops Knowledge Preferred Experience: 3-5 Years 