Lead Data Engineer

Role Purpose

As a Lead Data Engineer at Prevalent AI, you will lead the design and delivery of advanced data engineering projects for our clients. You will write code, review the team's code, and stay accountable for how the solution performs in production.

The work involves large data volumes, complex joins and transformations, and tight processing windows. Clients usually come to us when their pipelines are slow, expensive or difficult to change. Working out why, and then fixing it, is a large part of the role.

This role suits engineers who know Spark internals well enough to read the logs and explain what is going wrong, who ask the right questions about the data before they write code, and who can run a small team to a delivery date.

Key Accountabilities

• Lead the design and delivery of client data engineering projects, from understanding the requirement through to production.

• Design pipelines that meet their SLA and continue to do so as data volumes grow.

• Diagnose failures and performance problems in large distributed jobs using logs and metrics.

• Set the data layout and storage strategy so jobs read only the data they need.

• Build reusable, configuration-driven components that cut development effort across the team.

• Review the team's code and agree the standards the team works to.