Illumina—Senior Data Engineer

Job description

Description: What if the work you did every day could impact the lives of people you know? Or all of humanity? At Illumina, we are expanding access to genomic technology to realize health equity for billions of people around the world. Our efforts enable life-changing discoveries that are transforming human health through the early detection and diagnosis of diseases and new treatment options for patients. Working at Illumina means being part of something bigger than yourself. Every person, in every role, has the opportunity to make a difference. Surrounded by extraordinary people, inspiring leaders, and world changing projects, you will do more and become more than you ever thought possible. Senior Data Engineer Role Overview The Databricks Platform Engineer is the technical owner of our Databricks Lakehouse platform — the shared foundation on which enterprise data products, analytics, reporting and AI/ML workloads are built. This is a hands-on individual-contributor role responsible for the reliability, scalability, security, automation, governance and cost efficiency of the Databricks platform. The successful candidate will have production experience operating Databricks as an enterprise platform, not just building on it. They will work AI-natively and set that standard for the team. You will own the Data Platforms domain within a cross-skilled Data & Analytics Platforms engineering team. You will establish platform engineering standards, design and build infrastructure automation, create reusable reference architectures and engineering patterns, and ensure the platform can be operated reliably at enterprise scale. You will also mentor engineers and help build platform engineering capability within the India team. Key Responsibilities Platform ownership - Own workspace- and account-level architecture: Unity Catalog, compute policies, Lakeflow Jobs and shared compute infrastructure, identity and service principals, secrets management, audit logging, system tables, storage integration, platform upgrades and environment promotion across development, test and production. - Design, build and review infrastructure-as-code, CI/CD and observability capabilities for the platform, establishing reusable standards and automation for the wider engineering team. - Own platform reliability: availability, capacity, incident management, root-cause analysis, and DR/BCP; define and meet platform SLOs for shared platform services. - Run Databricks FinOps: consumption monitoring, cost optimisation, budget and anomaly management, and chargeback/showback with transparent unit costs for consuming teams. - Implement platform security and governance controls: identity federation/SCIM, service principals, network controls, secrets, RBAC/ABAC, governed tags, row/column security and masking, PII handling, lineage and audit. - Enable and operate the platform capabilities required for AI/ML workloads — MLflow, Model Serving, vector search and GenAI infrastructure patterns — partnering with analytics and AI teams on workload architecture, governance and cost controls. - Build and maintain reference implementations and paved-road patterns (PySpark, SQL, Delta Lake, Auto Loader) that show engineering teams how to use the platform; enable consuming teams rather than owning delivery of their data products. - Treat the platform as an internal product: understand consuming-team needs, define paved-road capabilities and self-service patterns, track adoption and reliability, and continuously reduce developer friction. - Own the platform roadmap: upgrades, new capabilities, deprecations. Engineering standards - Contribute to the team engineering handbook and architecture decisions; lead code reviews on platform and foundation work. - Take part in architecture and engineering forums, communicating trade-offs clearly to peers and global stakeholders. - Maintain reusable patterns and automation used across the team. - Establish AI-native engineering practice for the team: how AI assistants are used for infrastructure code, test generation, runbooks, code review and troubleshooting, and where human review is mandatory. Mentoring and collaboration - Mentor engineers through onboarding, pairing and code review. - Provide secondary coverage for an adjacent platform area as part of the team's cross-skilling model. Required Qualifications - 3–5 years of professional data platform or data engineering experience, including at least 2 years administering and operating Databricks in production as an enterprise platform (account and workspace administration, Unity Catalog, compute policies, environment promotion), not only developing pipelines on it. - Demonstrated ability to design, build and review infrastructure-as-code (Terraform or equivalent) and CI/CD for data platforms (GitHub Actions, AWS CodePipeline or similar). - Strong Python and PySpark; advanced SQL. - Experience running platform operations: monitoring and alerting, incident management, runbooks, capacity and performance management. - Experience with Databricks cost management and FinOps practices. - Platform security and compliance experience: identity, service principals, secrets, network controls, audit logging, RBAC and PII handling, with an understanding of controls such as SOX. - Solid understanding of distributed systems and system design for large-scale data processing. - Strong AWS fundamentals: S3, IAM, VPC and networking, KMS, and how Databricks integrates with them. - Demonstrated effective use of AI-assisted engineering tools, or a strong ability and willingness to adopt them, across coding, IaC, testing, documentation and troubleshooting, with sound judgement around verification, security and human review. Specific tools are not a screening criterion. - Strong written and verbal communication skills; able to work effectively with global stakeholders and lead technical discussion. - Bachelor's degree in Computer Science, Information Systems, Engineering or a related field, or equivalent demonstrable experience. Preferred Qualifications - Databricks Certified Data Engineer Professional and/or AWS Certified Solutions Architect. - Experience operating platforms in compliance-driven environments (SOX, GxP / 21 CFR Part 11 or similar): change control, audit evidence, access reviews, segregation of duties. - Experience supporting, onboarding or mentoring other engineers. - Experience taking over or migrating an existing platform estate. - Experience working in a global delivery model with distributed stakeholders. We are a company deeply rooted in belonging, promoting an inclusive environment where employees feel valued and empowered to contribute to our mission. Built on a strong foundation, Illumina has always prioritized openness, collaboration, and seeking alternative perspectives to propel innovation in genomics. We are proud to confirm a zero-net gap in pay, regardless of gender, ethnicity, or race. We also have several Employee Resource Groups (ERG) that deliver career development experiences, increase cultural awareness, and offer opportunities to engage in social responsibility. We are proud to be an equal opportunity employer committed to providing employment opportunity regardless of sex, race, creed, color, gender, religion, marital status, domestic partner status, age, national origin or ancestry, physical or mental disability, medical condition, sexual orientation, pregnancy, military or veteran status, citizenship status, and genetic information. Illumina conducts background checks on applicants for whom a conditional offer of employment has been made. Qualified applicants with arrest or conviction records will be considered for employment in accordance with applicable local, state, and federal laws. Background check results may potentially result in the withdrawal of a conditional offer of employment. The background check process and any decisions made as a result shall be made in accordance with all applicable local, state, and federal laws. Illumina prohibits the use of generative artificial intelligence (AI) in the application and interview process. If you require accommodation to complete the application or interview process, please contact accommodations@illumina.com. To learn more, visit: https://www.dol.gov/ofccp/regs/compliance/posters/pdf/eeopost.pdf. The position will be posted until a final candidate is selected or the requisition has a sufficient number of qualified applicants. This role is not eligible for visa sponsorship.