Senior Data Engineer

Responsibilities

• Design, develop, and maintain secure, scalable, and robust data pipelines using Azure Databricks, Python, and Apache Spark
• Extract, transform, and load (ETL) data from legacy platforms like Teradata and IBM DataStage into modern cloud-based environments
• Orchestrate and schedule complex data workflows and ETL processes using tools such as Apache Airflow
• Optimize data pipeline performance, troubleshoot bottlenecks, and ensure data quality and integrity
• Collaborate with cross-functional teams to translate business requirements into efficient engineering solutions
• Provide technical guidance and mentoring to junior engineers within the team
• Leverage enterprise-approved AI tools to streamline workflows and automate tasks

Requirements

• Bachelor's or Master's degree in Computer Science, Information Technology, or a related field
• 3+ years of professional experience in data engineering, database management, or software engineering
• 3+ years of experience designing and developing data pipelines using Apache Spark (PySpark or Scala) and Python
• 2+ years of experience building and optimizing modern cloud data solutions using Azure Databricks
• 2+ years of experience migrating or integrating legacy ETL platforms (e.g., IBM DataStage) and databases (e.g., Teradata)
• 1+ years of experience with workflow scheduling and orchestration tools (e.g., Apache Airflow, Autosys)

Preferred Qualifications

• Experience building and orchestrating data integrations using Azure Data Factory
• Hands-on experience utilizing generative AI tools (such as GitHub Copilot) to accelerate coding and documentation
• Experience writing and optimizing Unix Shell or Perl scripts
• Knowledge of healthcare domain datasets, including Customer, Member, Claims, and Provider information

Benefits

• Comprehensive health plans and coverage to support wellness
• Financial benefits including retirement planning support
• Resources for emotional well-being and stress management
• Opportunities for continuous professional training and career development

About the Company

Optum is a global organization that delivers care, aided by technology, to help millions of people live healthier lives. We connect people with the care, pharmacy benefits, data, and resources they need to feel their best through innovation and advanced data engineering.