Data Engineer

Responsibilities

• Design, build, and optimize scalable data pipelines in Apache NiFi to automate ingestion, cleansing, and enrichment
• Architect and manage datasets on HDFS to support high-volume, fault-tolerant storage
• Develop distributed processing workflows in PySpark and Hive for large-scale transformations
• Maintain and tune Postgres databases for high availability and performance
• Partner with Data Scientists and BI engineers to deliver clean, reliable datasets
• Implement monitoring, logging, and alerting frameworks to track pipeline health
• Enforce standards for data quality, lineage, and security
• Develop and maintain technical documentation for pipelines and data models

Requirements

• Bachelor’s degree in Computer Science, Information Technology, or related field
• Proven experience as a Data Engineer with expertise in HDFS, Apache NiFi, Hive, PySpark, Postgres, Python, and SQL
• Strong background in ETL/ELT design and distributed processing
• Experience with on-premises big data ecosystems
• Solid debugging, optimization, and performance tuning skills
• Ability to work in agile environments

Preferred Qualifications

• Familiarity with data governance frameworks and lineage tracking
• Knowledge of security standards and encryption in on-premises environments
• Prior experience with Media Mix Modeling (MMM/MMO) or marketing analytics
• Exposure to workflow schedulers like Airflow or Oozie
• Proficiency in developing automation scripts in Python for CI/CD