• Design, build, and maintain scalable data pipelines using Databricks Notebooks, Jobs, and Workflows for both batch and streaming data.
• Optimise Spark and Delta Lake performance through efficient cluster configuration, adaptive query execution, and caching strategies.
• Conduct performance testing and cluster tuning to ensure cost-efficient, high-performing workloads.
• Implement data quality, lineage tracking, and access control policies aligned with Databricks Unity Catalogue and governance best practices.
• Develop PySpark applications for ETL, data transformation, and analytics, following modular and reusable design principles.
• Create and manage Delta Lake tables with ACID compliance, schema evolution, and time travel for versioned data management.
• Integrate Databricks solutions with Azure services such as Azure Data Lake Storage, Key Vault, and Azure Functions.

• Proven experience with Databricks, PySpark, and Delta Lake.
• Strong understanding of workflow orchestration, performance optimisation, and data governance.
• Hands-on experience with Azure cloud services.
• Ability to work in a fast-paced environment and deliver high-quality solutions.
• SC Cleared candidates

If you're interested in this role, click 'apply now' to forward an up-to-date copy of your CV, or call us now.

If this job isn't quite right for you, but you are looking for a new position, please contact us for a confidential discussion about your career.