Senior Data Engineer

Responsibilities

• Architect, design, and implement high-volume batch and real-time data pipelines using PySpark, SparkSQL, and Databricks
• Build end-to-end ingestion frameworks integrating with Snowflake, AWS services, and vendor APIs
• Develop data modeling frameworks including star and snowflake schemas
• Lead technical solution design for health plan clients across multi-account AWS environments
• Implement security automation including RBAC, encryption, and HIPAA/SOC 2 compliance
• Establish data engineering best practices including CI/CD, automated testing, and observability
• Conduct performance profiling and optimize compute costs across Databricks and Snowflake
• Mentor junior engineers through technical reviews and coaching

Requirements

• Bachelor’s degree in Computer Science or a related technical field
• 7+ years of experience in data engineering with large-scale distributed systems
• Expert proficiency in Python, SQL, and PySpark
• Experience building production-grade ETL/ELT pipelines using Databricks or Airflow
• Experience with dbt, Kafka, Delta Lake, and event-driven architectures
• Strong knowledge of AWS data ecosystem (S3, SQS, Lambda, Glue, IAM)
• Proficiency with Terraform and CI/CD pipelines
• Experience with cloud data warehouses like Snowflake, BigQuery, or Redshift
• Knowledge of U.S. healthcare data domains (claims, eligibility, provider datasets)

Preferred Qualifications

• Experience in large-scale healthcare or payer data environments

Benefits

• Competitive Leave & Benefits
• Comprehensive health coverage
• Equity for every employee
• Growth-focused environment

About the Company

Abacus Insights is transforming how data works for health plans by making healthcare data usable to improve outcomes, reduce waste, and deliver better experiences for members and providers.