Senior Data Engineer

Responsibilities

• Design, develop, and maintain scalable ETL/data pipeline frameworks
• Build and manage data ingestion pipelines from multiple structured and unstructured sources
• Develop and orchestrate workflows using Apache Airflow
• Implement modular and microservices-based architecture for Python data platforms
• Write efficient and complex SQL queries for data transformation and analysis
• Design and implement REST APIs with proper validation and error handling
• Monitor, troubleshoot, and optimize pipeline performance
• Analyze logs and system performance using tools like Splunk
• Collaborate with cross-functional teams to ensure data quality and reliability

Requirements

• 5+ years of experience in data engineering
• Strong programming experience in Python
• Hands-on experience with Apache Airflow (DAG creation, scheduling, monitoring)
• Solid understanding of ETL concepts and pipeline design
• Experience working with RDBMS (e.g., Snowflake, PostgreSQL, MySQL)
• Strong proficiency in writing complex SQL queries
• Experience in building data ingestion frameworks
• Knowledge of REST API development and integration
• Familiarity with microservices architecture and modular design patterns
• Experience with Splunk queries/log analysis
• Experience with Data Catalog platforms (e.g., Open-metadata, Alation, Collibra, DataHub)
• Experience with Informatica CDGC

Preferred Qualifications

• Exposure to AWS services (S3, Opensearch, ALB, K8S)
• Experience with data governance or metadata management tools