Senior Data Engineer

• Rich knowledge of Databricks and recent hands-on experience implementing a Databricks lakehouse at enterprise scale.
• Advanced Python development skills.
• Good knowledge of SQL.
• Experience working using a modern DevOps approach. You should not just have written data code but also the CI/CD pipelines necessary to test and deploy that code in a professional environment.
• A robust understanding of core data engineering topics – ETL vs ELT, structured and unstructured data, data quality and data governance.
• Ability to contribute to all aspects of a solution – design, infrastructure, development, testing and maintenance.
• The ability to design and advocate for technical solutions to business problems.
• Effective collaboration with technical and non-technical team members through agile ceremonies – roadmap planning, feature workshops, backlog elaboration, code review.
• Understanding of cloud technology and significant hands-on experience working with it. You must understand how cloud-native solutions must be built differently from traditional ones.
• Track record of taking initiative and delivering projects end-to-end; clear evidence of being self-driven and motivated
• Immense curiosity, high energy and desire to go the extra mile to make a difference.

Desirable Skills

Beyond the required skills we are open to individuals of diverse talents. Experience with additional technologies, data science knowledge or a business background are all valued. We want to know how your unique abilities can contribute to our team.

Our Technology

We operate in a diverse technical landscape and are looking for flexible engineers who can adapt to and use many different tools. We would not expect any engineer to be familiar with the entire tech stack – no engineer can. Instead, we seek people with a good understanding of data structures and algorithms and the ability to apply this knowledge in learning new tools.

We have an existing data landscape involving both MS SQL Server and Oracle databases. We use SSIS for ETL. We will also need to source data from our integration layer – webMethods.

If some of that sounds like it needs bringing up to date, we know! We hope to start building a modern cloud data platform to realise the full value of our data. The technologies we use for that platform are still to be decided. It will be based on Databricks with Azure Data Factory for ELT. Other parts of our business have had success using Prefect for orchestration and dbt for transformation and we will reuse where it is sensible.

We prepare data for use by analysts working with a variety of tools – Tableau, PowerBI, and even Excel.

We take a DevOps approach and strive for continuous integration / continuous deployment. We use Azure Pipelines to deploy our code. We deploy infrastructure the same way using Terraform and Docker.

Our technology landscape is not fixed. Our engineering and architecture teams drive the technology we adopt.