Lead Data Engineer
  
Location: London (Office Based)
Salary: DOE
  
The Role
We’re looking for a Lead Data Engineer to build and own the data infrastructure powering an AI-driven platform.
  
As the first senior data hire, you’ll define the data architecture, tooling and engineering standards from the ground up. You’ll work closely with AI, backend and product teams to build reliable systems capable of processing large volumes of operational data.
  
This is a highly hands-on role with significant technical ownership and the opportunity to help shape the future data engineering team.
  
Key Responsibilities

• Architect and build scalable data pipelines and infrastructure.
• Design data ingestion, transformation and storage systems for both product and AI workloads.
• Develop and manage batch and real-time data pipelines.
• Build and optimise vector search, retrieval and ML data workflows.
• Ensure data reliability, security, governance and quality.
• Implement monitoring, observability and data quality frameworks.
• Optimise large-scale datasets and query performance.
• Work closely with AI and backend engineers to support training, inference and product features.
• Shape the long-term data architecture and technical strategy.
• Help build and mentor the future data engineering team.

  
What We’re Looking For

• 7+ years’ experience in data engineering or backend engineering.
• Strong experience building scalable data pipelines and distributed systems.
• Strong Python development experience.
• Excellent knowledge of relational databases, particularly PostgreSQL.
• Experience with NoSQL and vector databases.
• Strong understanding of data modelling, storage architecture and performance optimisation.
• Experience with technologies such as Spark, Airflow, Kafka and Elasticsearch/OpenSearch.
• Experience with MongoDB, Qdrant, Milvus or pgvector.
• Familiarity with Pandas and/or Polars.
• Experience building scalable backend systems.

  
Desirable:

• Experience with AI/ML platforms and data workflows.
• JavaScript/Node.js experience.
• Stream processing and event-driven architectures.
• Cloud experience across GCP, AWS or Azure.
• Experience in high-growth or early-stage technology businesses.

  
Please Note: Rayner Personnel – Property Recruitment are acting as a recruitment agency with regards to this position. By submitting your application, you are consenting to Rayner Personnel processing and storing your data for the purposes of your job search.
 
We receive a great many applications for our vacancies and try to respond to as many as we can as quickly as possible. If you have not heard from a member of the team within 48 working hours, please assume that on this occasion your application has not been successful, but please feel that you may apply for other roles we are advertising in the future.