Senior Data Engineer

Responsibilities

• Design, build, and maintain ETL/ELT pipelines in Databricks to ingest, clean, and transform data
• Construct gold layer tables in Lakehouse architecture for ML training and real-time APIs
• Monitor data quality, lineage, and reliability using Databricks best practices
• Collaborate with AI/ML teams to structure data for natural language prompts and semantic retrieval
• Build data interfaces and agent tools to interact with structured data and AI agents
• Design and implement serverless APIs via AWS Lambda to expose gold tables to frontend applications
• Implement provisioning, deployment, security, and monitoring frameworks for scaling data pipelines

Requirements

• 3+ years of experience as a Data Engineer in an agile, distributed team
• Proven expertise with Databricks, including orchestration and medallion architecture
• Proficiency in Spark or Scala for data wrangling
• Experience with CI/CD best practices and test-driven development
• Familiarity with MLOps / AIOps lifecycles

Preferred Qualifications

• Familiarity with AWS Lambda (Node.js/TypeScript) and API Gateway
• Knowledge of API design principles (RESTful or GraphQL)
• Exposure to React-based frontend architecture
• Experience with A/B testing, experiment, and inference logging

About the Company

CME is a multinational technology consulting firm helping companies scale operations, achieve innovation, and transform business models through customized end-to-end tech solutions.