Principal Data Engineer (Databricks)

Role Overview

Reporting to the Data & Analytics Lead, the Principal Data Engineer is a hands-on role, combining the building and optimisation of production data pipelines with the technical leadership needed to raise engineering standards and develop the people around them. The role functions as a key interface between a) business stakeholders and department heads, b) the wider data and analytics team, and c) owners of source systems and data in Finance and IT.

Key Responsibilities

• Designing and building robust, production-grade data pipelines on Databricks, making appropriate use of current platform capabilities such as Spark Declarative Pipelines (SDP)
• Working “under the hood” to optimise the platform for performance and cost — tuning compute, jobs and queries, managing storage and table layout, and keeping platform spend under control
• Defining and maintaining consistent business metrics and a semantic ...