Senior Data Engineer & Technical Delivery Lead (Python / Databricks)

Senior Data Engineer & Technical Delivery Lead (Python / Databricks)

Location: London (Hybrid – 2-3 days onsite)

Contract: Initial 6 months

We're currently supporting a major Investment Bank that's expanding its Data Engineering function and is looking to hire multiple contractors across Senior Data Engineer and Technical Delivery Lead level.

Whether you're a highly technical Senior Data Engineer looking to remain hands-on, or an experienced Technical Lead who enjoys combining delivery leadership with coding, we'd be keen to speak with you.

You'll be joining a large-scale cloud data transformation programme, helping design, build and optimise enterprise data platforms using Python, Databricks and Spark within a modern Lakehouse environment.

Key Responsibilities

\n
• Design, develop and maintain scalable data pipelines using Python and Databricks.
\n
• Build, optimise and support enterprise ETL/ELT workflows using Apache Spark and Delta Lake.
\n
• Design and develop robust data models and Lakehouse architectures.
\n
• Implement and manage data workflows within the Databricks ecosystem.
\n
• Ensure high levels of data quality, governance and pipeline reliability.
\n
• Optimise performance across large-scale distributed data processing platforms.
\n
• Collaborate with architects, product owners, analysts and engineering teams to deliver high-quality data solutions.
\n
• Implement monitoring, logging and alerting across critical data platforms.
\n
• Drive engineering best practice through code reviews, mentoring and technical collaboration.
\n
• For Technical Delivery Lead opportunities, provide technical leadership, architectural oversight and end-to-end delivery of complex data engineering initiatives.
\n

Required Skills & Experience

\n
• Strong commercial experience developing data engineering solutions using Python.
\n
• Extensive hands-on experience with Databricks, including Workflows, Notebooks and Delta Lake.
\n
• Strong experience with Apache Spark / PySpark.
\n
• Proven experience building scalable ETL/ELT pipelines.
\n
• Strong SQL and data modelling skills.
\n
• Experience designing modern Lakehouse architectures.
\n
• Experience working with cloud platforms including AWS, Azure or GCP.
\n
• Experience with modern data warehousing technologies such as Snowflake, Redshift or BigQuery.
\n
• Experience with orchestration tools such as Airflow or Databricks Workflows.
\n
• Previous experience working within Agile delivery environments.
\n

Desirable Experience

\n
• Databricks certifications.
\n
• Kafka or Structured Streaming.
\n
• CI/CD and DevOps practices.
\n
• Docker and Kubernetes.
\n
• Exposure to Machine Learning pipelines.
\n
• Experience within Financial Services or other highly regulated environments.
\n

About You

You'll be an experienced Data Engineer or Technical Lead with a passion for building modern, scalable data platforms. Comfortable working in a collaborative Agile environment, you'll enjoy solving complex technical problems and delivering high-quality engineering solutions. If you're applying at Technical Delivery Lead level, you'll also bring strong leadership, stakeholder management and mentoring experience whilst remaining hands-on technically.