Senior Data Engineer

Responsibilities

• Design, develop, and optimize distributed data pipelines using PySpark
• Work with Apache tools such as Hadoop, Hive, HDFS, and others for large-scale data ingestion, transformation, and processing
• Ensure the performance, reliability, and scalability of ETL workflows in production environments
• Collaborate with stakeholders to gather requirements and deliver scalable data solutions
• Implement robust data quality checks and lineage tracking for auditability and transparency
• Handle data integration from diverse structured and unstructured sources
• Utilize Apache NiFi for automated data flow orchestration
• Write clean and maintainable code primarily in Python, with working knowledge of Java
• Participate in architectural discussions and performance tuning initiatives

Requirements

• 5–7 years of experience in data engineering roles
• Expertise in PySpark for distributed computing and data transformation
• Strong understanding of Apache ecosystem (Hadoop, Hive, Spark, HDFS)
• Knowledge of ETL principles, data modeling, and data warehousing concepts
• Experience working with large-scale datasets and optimizing performance
• Hands-on proficiency with SQL and exposure to NoSQL databases
• Solid coding skills in Python, with working knowledge of Java
• Experience with version control (Git) and working in CI/CD environments