Lead Azure Databricks Platform Engineer / Architect

Lead Azure Databricks Platform Engineer / ArchitectHands-on Platform Engineering | Serverless | FinOps | POSIT/RStudio Migration6 Month contractInside IR35 - 500 a dayLondon/HybridRole PurposeWe are seeking a highly experienced, hands-on Azure Databricks Platform Engineer / Architect to enhance and optimise an enterprise Data Platform. The role combines architecture with direct implementation: the successful candidate must be able to configure, develop, troubleshoot and optimise Azure Databricks rather than operate only at design or governance level.The role is centred on three outcomes: enabling and optimising Databricks Serverless, strengthening FinOps and platform controls, and enhancing the Databricks Discovery Zone to support workloads currently delivered through POSIT/RStudio.Key Responsibilities1. Databricks Serverless Enablement and OptimisationAssess existing workloads and determine suitability for Serverless, classic, job or interactive compute based on duration, utilisation, SLA, concurrency, performance and cost.Enable and configure Serverless for appropriate jobs, SQL workloads, notebooks, analytical processing and data pipelines.Establish workload-placement guidance, including when Serverless is not economical for predictable, heavy or continuously running workloads.Implement compute policies, autoscaling, quotas, budget controls and operational guardrails.Measure cost and performance outcomes, identify idle or oversized compute, and recommend optimisation actions.2. FinOps and Enterprise Platform ControlsDefine and embed a practical FinOps operating model covering ownership, accountability, projects, environments, teams, applications and cost centres.Implement mandatory tagging and integrate validation into CI/CD so non-compliant resources are prevented from being provisioned.Provide granular cost attribution by workspace, project, application, workload, job and team/user where technically appropriate.Implement budget policies, thresholds, proactive alerts and usage reporting to prevent uncontrolled spend.Use platform usage and billing data to identify idle compute, inefficient workloads, unnecessary storage/data movement and cost anomalies.3. Databricks Discovery Zone and POSIT/RStudio MigrationEnhance the Databricks Discovery Zone to support migration from POSIT/RStudioEnable application deployment, secure API integrations, external data ingestion, LLM integration, scheduling, BI connectivity, local IDE-based development and operational reporting.Define reusable onboarding and migration patterns that reduce technology sprawl while improving security, supportability and delivery speed.4. Data Engineering and IntegrationDesign and build reliable ingestion and transformation pipelines using Python, PySpark, SQL and Delta Lake.Implement full and incremental ingestion, CDC where appropriate, schema evolution, reconciliation, error handling and data quality controls.Design reusable integration patterns for REST APIs, SaaS platforms, databases, files, object storage, document repositories, enterprise applications and public/external data providers.Implement secure authentication and credential handling for external and internal integrations.Build end-to-end data flows from source through governed ingestion and curated layers to BI, ML or application consumption.Required Hands-on Technical SkillsDeep hands-on Azure Databricks implementation and troubleshootingDatabricks Serverless and compute/workload optimisationAzure identity, networking, security, secrets, monitoring and private connectivityDatabricks SQL, Delta Lake and performance optimisationPython, PySpark and SQLJobs/workflows, incremental processing, CDC and data qualityREST/API and external data integration patternsFinOps, cost attribution, tagging, budgets, monitoring and operational supportExperience and Candidate ProfileSignificant experience delivering enterprise Azure Databricks platforms in production environments.Demonstrable ability to move between architecture, implementation, debugging and optimisation without depending entirely on specialist engineering teams.Strong understanding of platform security, data governance, operational support and controlled delivery in regulated or complex enterprises.Experience working collaboratively with data engineers, data scientists, architects, security teams, platform teams and business stakeholders.Clear communication skills and the ability to document standards, patterns, decisions and operational guidance.Highly Desirable but not MandatoryPOSIT/RStudio migration or consolidation experience.Migration of analytical/data science workloads (convert and migrate R development/Libraries to Databricks).AI/ML, LLM integration, model lifecycle, RAG/vector retrieval or model-serving experience.Large-scale enterprise platform transformation and regulated-industry experience.Strong cost optimisation and FinOps delivery experience across Azure and Databricks.TPBN1_UKTJ