Data Engineer

About the Company

At Algotale, we believe in the transformative potential of data to reshape industries, drive innovation, and create unparalleled value. Established in 2020, we specialize in IT services, staffing, and consulting, empowering our clients to unlock the full potential of their data through analytics, AI, and cloud solutions.

Responsibilities

• Design, develop, and maintain scalable and efficient data pipelines using PySpark
• Build and manage workflows using Apache Airflow for scheduling and orchestration
• Work extensively with SQL for data extraction, transformation, and analysis
• Develop and maintain data solutions on Cloudera Hadoop ecosystem (HDFS, Hive, Impala)
• Optimize data processing jobs for performance and scalability
• Collaborate with data scientists, analysts, and business teams to deliver data solutions
• Ensure data quality, integrity, and security across data pipelines
• Troubleshoot and resolve data-related issues in a timely manner
• Implement best practices for data engineering, including version control and CI/CD

Requirements

• 4+ years of relevant experience in data engineering or big data technologies
• Strong hands-on experience with PySpark and distributed data processing
• Proficiency in Apache Airflow for workflow orchestration
• Advanced knowledge of SQL
• Experience with Cloudera ecosystem (Hadoop, HDFS, Hive, Impala)
• Understanding of data warehousing concepts and ETL processes
• Experience working with large-scale structured and unstructured datasets
• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field

Preferred Qualifications

• Experience with cloud platforms (AWS, Azure, or GCP)
• Knowledge of Kafka or other streaming technologies
• Familiarity with CI/CD pipelines and DevOps practices
• Experience with Python for data processing beyond PySpark