Title: Lead Software Engineer - Big Data

Grade Level: 10

Location: Hyderabad, INDIA

Job Description:

S&P Global Ratings is looking for a Big Data Engineer to join Ingestion Pipelines Engineering team within Data Services group, a team of data and technology professionals who define and execute the strategic data roadmap for S&P Global Ratings. The successful candidate will participate in the design and build of S&P Ratings cutting edge Ingestion pipelines solutions.

The Team

You will be an expert contributor and part of the Rating Organization’s Data Services Product Engineering Team. This team, who has a broad and expert knowledge on Ratings organization’s critical data domains, technology stacks and architectural patterns, fosters knowledge sharing and collaboration that results in a unified strategy. All Data Services team members provide leadership, innovation, timely delivery, and the ability to articulate business value. Be a part of a unique opportunity to build and evolve S&P Ratings next gen analytics platform.

Our Hiring Manager Says

If you are an individual that brings demonstrated experience of delivering big data projects as a data engineer, this is an excellent opportunity. I am looking for someone with sound technical knowledge, needs to be hands-on, worked on transformational initiatives, and can drive results.

Responsibilities:                                       

• Design & Build ingestion pipelines using ELT and schema on read in Databricks delta lake
• Design & Develop “Transformations” aspects using ELT framework to modernize the Ingestion pipelines and build data transformations at scale
• Work closely with other data teams & Data Science team and participate in development of Ingestion pipelines
• Provide technical expertise in the areas of design and implementation of Ratings Integrated Data Facility with modern AWS cloud technologies such as S3, Hive, Databricks, Scala, Python and Spark
• Build and maintain a data environment for speed, accuracy, consistency and ‘up’ time
• Ensure data governance principles adopted, data quality checks and data lineage implemented in each hop of the data
• Partner with the data teams, enterprise architecture organization to ensure best use of standards for the key data domains and use cases
• Be in tune with emerging trends Big data and cloud technologies and participate in evaluation of new technologies
• Ensure compliance through the adoption of enterprise standards and promotion of best practice / guiding principles aligned with organization standards

Experience & Qualifications:

• BE, MCA or MS degree in Computer Science or Information Technology
• 6 years of experience as data engineer at an innovative organization
• 5 years of hands-on experience in implementing data lake systems using AWS/Azure cloud technologies such as Hive, Spark, Java/ Scala/ Python etc.
• Good to have expertise in building application using Kafka, Golden Gate etc. for building ingestion pipeline for Bulk and incremental data loads.
• 3+ years of expertise in handling stakeholder management, people management, risk management,
• Should be in position to articulate and convert requirements into solution.
• Experience With Machine Learning Libraries and Frameworks (TensorFlow, MLlib) is an added advantage
• Experience with development frameworks as well as data and integration technologies such as Informatica, Python, Scala
• Expert knowledge of Agile approaches to software development and able to put key Agile principles into practice to deliver solutions incrementally.
• Monitors industry trends and directions; develops and presents substantive technical recommendations to senior management
• Excellent analytical thinking, interpersonal, oral, and written communication skills with strong ability to influence both IT and business partners
• Ability to prioritize and manage work to critical project timelines in a fast-paced environment
• Financial services industry experience