Lead Data Platform Engineer

Responsibilities

• Manage and optimize data platforms, specifically Databricks or Palantir.
• Ensure high availability, security, and performance of all data systems.
• Provide valuable insights regarding data platform usage and efficiency.
• Optimize computing and storage resources for large-scale data processing.
• Design and maintain system libraries in Python used for ETL pipelines and platform governance.
• Enhance and tune existing ETL processes to improve performance, scalability, and reliability.

Requirements

• Minimum 10 years of experience in IT or Data roles.
• Minimum 5 years of experience as a Data Platform Engineer or Data Engineer.
• Bachelor's degree in IT or a related field.
• Expertise in cloud infrastructure and services within Azure or AWS (storage, networking, compute).
• Deep proficiency in either Palantir or Databricks.
• Strong programming skills in Python and PySpark for distributed computing and ETL development.
• Expertise in writing and optimizing SQL queries (PostgreSQL, MySQL, Oracle, or Snowflake).
• Experience with data warehousing concepts and dimensional modelling.
• Proficiency with Git for version control and codebase management.
• Experience with data pipeline monitoring and data quality frameworks.

Preferred Qualifications

• Experience with containerization and orchestration tools such as Docker and Kubernetes.
• Proficiency with Infrastructure as Code (IaC) using Terraform.
• Understanding of the Investment Data domain.

About the Company

Luxoft is a global provider of high-end software engineering services, helping leading enterprises accelerate their digital transformation through advanced technology solutions.