Lead Data Engineer

We are seeking an experienced Lead Data Engineer with deep expertise in Python, Databricks and Spark to lead the delivery of large-scale data engineering initiatives.

This is a unique opportunity for a technical leader who enjoys combining hands-on engineering with project delivery, team leadership and stakeholder engagement in a complex enterprise environment.

Client Details

Our client is a leading international financial institution with a long-established presence across major global markets. Serving a diverse client base that includes corporates, financial institutions and investors, the organisation delivers a broad range of banking, financing and capital markets services.

With significant investment in digital transformation and data-led innovation, the organisation is modernising its technology estate and expanding its enterprise data capabilities. Data plays a critical role in supporting business operations, regulatory obligations, analytics and strategic decision-making, making this an exciting opportunity to contribute to large-scale, business-critical data initiatives within a complex global environment.

Description

\n
\n
• Lead the end-to-end delivery of data engineering projects, ensuring successful outcomes against business objectives, timelines and quality standards
\n
• Design, develop and optimise scalable data pipelines using Python, Databricks and PySpark
\n
• Build and enhance ETL/ELT workflows, reusable frameworks and modern data engineering solutions
\n
• Provide technical leadership and architectural guidance across data platform initiatives
\n
• Design and govern Lakehouse architectures leveraging Databricks and Delta Lake
\n
• Manage sprint planning, backlog prioritisation and delivery tracking within Agile teams
\n
• Identify and mitigate delivery risks, dependencies and technical challenges
\n
• Troubleshoot complex data, platform and performance-related issues
\n
• Drive engineering best practices across coding standards, testing, performance optimisation and code reviews
\n
• Act as a key interface between business stakeholders and engineering teams, translating requirements into scalable technical solutions
\n
• Collaborate with architects, product owners and senior stakeholders to shape delivery roadmaps and technical direction
\n
• Lead, mentor and develop a team of data engineers, fostering a culture of ownership and engineering excellence
\n
• Ensure data quality, governance, reliability and security across data platforms
\n
• Implement monitoring, logging and alerting capabilities to support operational excellence
\n
• Champion modern engineering practices, including CI/CD, DevOps and automation
\n
• Support production environments and drive continuous improvement across data solutions
\n
\n

Profile

You will bring:

\n
• 10+ years' experience in data or software engineering
\n
• Proven experience as a Lead Data Engineer, Technical Lead, Engineering Lead, or similar hands-on leadership role
\n
• Strong hands-on expertise in Python
\n
• Extensive experience with Databricks, including workflows, notebooks and Delta Lake
\n
• Strong experience with Apache Spark / PySpark
\n
• Experience building and optimising enterprise-scale ETL pipelines
\n
• Strong understanding of modern data architecture and Lakehouse concepts
\n
• Advanced SQL and distributed data systems expertise
\n
• Experience with cloud platforms such as AWS, Azure or GCP
\n
• Experience working within Agile delivery environments
\n
• Excellent stakeholder management and communication skills
\n
• Ability to balance technical leadership with hands-on delivery responsibilities
\n
• Experience delivering data engineering solutions within Banking, Financial Services, Capital Markets, Insurance, or other highly regulated enterprise environments would be highly advantageous
\n

Desirable experience includes:

\n
• Databricks certifications
\n
• Kafka or Structured Streaming
\n
• CI/CD and DevOps practices
\n
• Airflow or Databricks Workflows
\n
• Delta Lake optimisation techniques
\n
• Docker and Kubernetes
\n
• Exposure to machine learning pipelines
\n

Job Offer

\n
• Competitive day rate of £650-880 per day Inside IR35 (Umbrella)
\n
• Initial 6-month contract
\n
• Hybrid working arrangement - 2/3 days onsite
\n
• Opportunity to lead strategic data transformation initiatives
\n
• Access to modern cloud and data engineering technologies
\n
• Significant influence over technical direction and delivery
\n
• Collaborative, high-performing engineering environment
\n