W
Posted a day ago
Staff Data Platform Engineer
WayveStaff Data Platform Engineer
Requirements
Large-scale data transfer pipeline experience, Workflow orchestration (Flyte, Airflow, or Prefect), Cloud platform expertise (Azure preferred), Proficiency in Python, Proficiency in Terraform, Cloud-to-cloud networking experience
Skills
PythonTerraformAzure
About the role
About the Company
Founded in 2017, Wayve is the leading developer of Embodied AI technology. Our advanced AI software and foundation models enable vehicles to perceive, understand, and navigate any complex environment, enhancing the usability and safety of automated driving systems.
Responsibilities
- Set technical direction for the Data Transfer Hub, making trade-offs across reliability, throughput, cost, and observability
- Lead the technical design and delivery of the Data Transfer Hub project to ingest petabytes of sensor data daily
- Design and build parallelised, distributed data transfer pipelines using Flyte, Kafka, and Azure
- Build and operate a globally distributed hub-and-spoke data transfer model
- Write infrastructure-as-code in Terraform and build pipeline logic in Python
- Drive cloud-to-cloud and cross-cloud networking solutions
Requirements
- Proven experience building and operating large-scale data transfer pipelines at multi-TB or PB scale
- Experience orchestrating parallelised transfer jobs (Flyte preferred, or Airflow/Prefect)
- Experience building reliable, observable data systems with retry strategies and data integrity checks
- Strong cloud platform skills, preferably Azure, with knowledge of AWS or GCP
- Proficiency in Python and Terraform
- Experience with cloud-to-cloud networking and large-scale object storage transfer
- High degree of autonomy and a product-oriented mindset
Preferred Qualifications
- Experience with Kafka for event-driven transfer observability
- Multi-cloud experience across Azure, AWS, and GCP
- Experience working directly with external data partners
- Background in autonomous vehicles, robotics, or data-intensive domains
ScoutJobs Agent
Get matches like this delivered daily
Sign up free — we'll pull jobs that fit your CV from across the web and rank them for you.
Get started — it's freeStaff Data Platform Engineer
Wayve · London
