Join Wayve’s Ingestion Systems team to build and optimize petabyte-scale data pipelines for autonomous driving. Debug, enhance, and scale high-volume Spark-based workflows, ensuring reliable data ingestion for model training, evaluation, and annotation. Requires deep expertise in distributed systems, Python, and production-grade ETL pipelines.
Key Highlights
Build and optimize petabyte-scale data ingestion pipelines for autonomous driving using Apache Spark
Debug and improve high-throughput, resilient systems with multi-step workflows and complex dependencies
Work with real-world driving data for annotation, model training, and evaluation in a fast-moving AI platform
Key Responsibilities
Debug and unblock failing ingestion pipelines, stalled jobs, and processing queues in petabyte-scale systems
Investigate and resolve corrupt, malformed, or unexpected data from diverse third-party sources
Optimize Spark jobs and batch-processing pipelines for throughput, reliability, and compute efficiency
Collaborate with engineers to prioritize critical datasets and reduce operational toil
Improve monitoring, retries, and failure recovery across multi-step ingestion workflows
Technical Skills Required
Apache Spark
Python
Distributed Data Processing
Discover our full range of relocation jobs with comprehensive support packages to help you relocate and settle in your new location.
Benefits & Perks
Market-benchmarked annual salaries
Meaningful equity with long-term ownership
Hybrid working model with flexible core hours
Relocation support and visa sponsorship where applicable
Comprehensive benefits including health insurance, dental, enhanced parental leave, retirement, and wellbeing support
Nice to Have
Experience with Airflow, Flyte, Databricks, Delta Lake
Proficiency in Scala or Java
Knowledge of complex orchestration systems with queues, retries, and priority data flows
Job Description
Before the detail, here's the challenge you'd help us solve.We build the embodied intelligence that moves real vehicles safely, and the ecosystem a billion machines will run on in the future. Very few people in AI can say this. Every role here, whatever the team, plugs into that.Here’s what this particular role covers.Our Ingestion Systems team sits within Wayve’s AI Platform organisation and builds the data foundations behind our self-driving technology. We process vast amounts of real-world driving data so it can be used for annotation, model training and evaluation. Operating at petabyte scale, the team focuses on making these critical data flows reliable, efficient and easy to operate.
Want the full job description?
Read the complete details on LinkedIn, the original posting.
Continue on LinkedIn
This is a short excerpt. All rights to the full description belong to its original publisher.
