Lead Azure Databricks Platform Engineer / Architect

Lead Azure Databricks Platform Engineer / ArchitectHands-on Platform Engineering | Serverless | FinOps | POSIT/RStudio Migration

6 Month contract

Inside IR35 - 500 a day

London/Hybrid

We are seeking a highly experienced, hands-on Azure Databricks Platform Engineer / Architect to enhance and optimise an enterprise Data Platform. The role combines architecture with direct implementation: the successful candidate must be able to configure, develop, troubleshoot and optimise Azure Databricks rather than operate only at design or governance level. The role is centred on three outcomes: enabling and optimising Databricks Serverless, strengthening FinOps and platform controls, and enhancing the Databricks Discovery Zone to support workloads currently delivered through POSIT/RStudio.

\n
• Assess existing workloads and determine suitability for Serverless, classic, job or interactive compute based on duration, utilisation, SLA, concurrency, performance and cost.
\n
• Enable and configure Serverless for appropriate jobs, SQL workloads, notebooks, analytical processing and data pipelines.
\n
• Establish workload-placement guidance, including when Serverless is not economical for predictable, heavy or continuously running workloads.
\n
• Implement compute policies, autoscaling, quotas, budget controls and operational guardrails.
\n
• Measure cost and performance outcomes, identify idle or oversized compute, and recommend optimisation actions.
\n

\n
• Define and embed a practical FinOps operating model covering ownership, accountability, projects, environments, teams, applications and cost centres.
\n
• Implement mandatory tagging and integrate validation into CI/CD so non-compliant resources are prevented from being provisioned.
\n
• Provide granular cost attribution by workspace, project, application, workload, job and team/user where technically appropriate.
\n
• Implement budget policies, thresholds, proactive alerts and usage reporting to prevent uncontrolled spend.
\n
• Use platform usage and billing data to identify idle compute, inefficient workloads, unnecessary storage/data movement and cost anomalies.
\n

\n
• Enhance the Databricks Discovery Zone to support migration from POSIT/RStudio
\n
• Enable application deployment, secure API integrations, external data ingestion, LLM integration, scheduling, BI connectivity, local IDE-based development and operational reporting.
\n
• Define reusable onboarding and migration patterns that reduce technology sprawl while improving security, supportability and delivery speed.
\n

\n
• Design and build reliable ingestion and transformation pipelines using Python, PySpark, SQL and Delta Lake.
\n
• Implement full and incremental ingestion, CDC where appropriate, schema evolution, reconciliation, error handling and data quality controls.
\n
• Design reusable integration patterns for REST APIs, SaaS platforms, databases, files, object storage, document repositories, enterprise applications and public/external data providers.
\n
• Implement secure authentication and credential handling for external and internal integrations.
\n
• Build end-to-end data flows from source through governed ingestion and curated layers to BI, ML or application consumption.
\n

Deep hands-on Azure Databricks implementation and troubleshootingDatabricks Serverless and compute/workload optimisationAzure identity, networking, security, secrets, monitoring and private connectivityDatabricks SQL, Delta Lake and performance optimisationPython, PySpark and SQLJobs/workflows, incremental processing, CDC and data qualityREST/API and external data integration patternsFinOps, cost attribution, tagging, budgets, monitoring and operational support

\n
• Significant experience delivering enterprise Azure Databricks platforms in production environments.
\n
• Demonstrable ability to move between architecture, implementation, debugging and optimisation without depending entirely on specialist engineering teams.
\n
• Strong understanding of platform security, data governance, operational support and controlled delivery in regulated or complex enterprises.
\n
• Experience working collaboratively with data engineers, data scientists, architects, security teams, platform teams and business stakeholders.
\n
• Clear communication skills and the ability to document standards, patterns, decisions and operational guidance.
\n

\n
• POSIT/RStudio migration or consolidation experience.
\n
• Migration of analytical/data science workloads (convert and migrate R development/Libraries to Databricks).
\n
• AI/ML, LLM integration, model lifecycle, RAG/vector retrieval or model-serving experience.
\n
• Large-scale enterprise platform transformation and regulated-industry experience.
\n
• Strong cost optimisation and FinOps delivery experience across Azure and Databricks.
\n