Senior Data Engineer

Responsibilities

• Design and implement robust, production-grade pipelines using Python, Spark SQL, and Airflow
• Lead efforts to canonicalize raw healthcare data into internal models
• Own the full lifecycle of core pipelines from ingestion to validated datasets
• Onboard new customers by integrating raw data into internal pipelines
• Build resilient, idempotent transformation logic with data quality checks
• Refactor and scale existing pipelines to meet growing business needs
• Tune Spark jobs and optimize distributed processing performance
• Implement schema enforcement and versioning
• Monitor pipeline health and participate in on-call rotations
• Build and enhance streaming pipelines using Kafka or SQS

Requirements

• 6+ years of experience as a Data Engineer building production-grade pipelines
• Strong expertise in Python, Spark SQL, and Airflow
• Experience processing large-scale file-based datasets (CSV, Parquet, JSON)
• Experience mapping and standardizing raw external data into canonical models
• Familiarity with AWS or other cloud platforms and distributed compute
• Experience onboarding new customers and integrating non-standard data formats
• Strong written and verbal communication skills
• Experience working with large-scale or messy datasets

Preferred Qualifications

• Familiarity with healthcare data (837, 835, EHR, UB04, claims normalization)
• Experience building streaming pipelines using Kafka or SQS

Benefits

• Work from anywhere in the US
• Full Medical/Dental/Vision for employees and families
• Unlimited FTO
• Competitive salary and equity
• 401(k) with employer match

About the Company

Machinify is a leading healthcare intelligence company that delivers value, transparency, and efficiency to health plan clients through an AI-powered platform.