Infosys—Databricks Administration

Job description

Description: • Experience: 2+ years in data platform engineering, particularly focusing on Databricks/Spark environments. • 2+ years in AI/LLM/Agentic platform administration (enterprise-grade). • Databricks Expertise: Strong, hands-on experience with Databricks Lakehouse architecture, Unity Catalog, Delta Live Tables (DLT), and Serverless SQL. • Technical Proficiency: Proficiency in Python, PySpark, and Spark SQL. • Cloud Platforms: Deep understanding of one or more major cloud providers (AWS, Azure, GCP) and associated services. • Own administration of Databricks accounts and multi-workspace environments across SDLC. • Define and enforce platform standards for clusters, jobs, SQL warehouses, runtimes, and workspace configurations. • Lead Unity Catalog governance strategy including catalogs, schemas, permissions, and external locations. • Design and oversee enterprise monitoring, alerting, compliance dashboards, and operational SLOs. • Lead high-severity incident management, root-cause analysis, and preventive remediation. • Establish and maintain security, audit, and regulatory compliance controls. • Drive cost governance, capacity forecasting, and performance optimization across workloads. • Define and lead Infrastructure-as-Code and automation strategy using Terraform, APIs, and CI/CD pipelines. • Standardize CI/CD, Databricks Asset Bundles, and DevOps operating models. • Partner with cloud platforms, security, and data engineering teams on architecture and governance decisions. • Own platform documentation, onboarding standards, and the operational maturity roadmap. • Coach and mentor junior and mid-level Databricks administrators and act as senior technical escalation point. • Administer Databricks accounts and workspaces across development, test, and production environments. • Configure and manage clusters, jobs, SQL warehouses, runtimes, repos, and workspace settings. • Implement monitoring, alerting, dashboards, and operational runbooks. • Provide Tier 2/3 platform support, incident resolution, and root-cause analysis. • Enforce security and governance using Unity Catalog, RBAC, service principals, and least-privilege access. • Manage secrets, audit logging, and compliance reporting. • Administer Unity Catalog objects, external locations, and cloud storage integrations. Requirements: We are seeking a hands-on Databricks Administrator to manage and support the Databricks platform for enterprise data and analytics. This role leads platform governance, security posture, automation strategy, compliance, reliability, and cost management while defining standards to support large-scale analytics and AI/ML workloads. Responsibilities: • Experience: 2+ years in data platform engineering, particularly focusing on Databricks/Spark environments. • 2+ years in AI/LLM/Agentic platform administration (enterprise-grade). • Databricks Expertise: Strong, hands-on experience with Databricks Lakehouse architecture, Unity Catalog, Delta Live Tables (DLT), and Serverless SQL. • Technical Proficiency: Proficiency in Python, PySpark, and Spark SQL. • Cloud Platforms: Deep understanding of one or more major cloud providers (AWS, Azure, GCP) and associated services. • Own administration of Databricks accounts and multi-workspace environments across SDLC. • Define and enforce platform standards for clusters, jobs, SQL warehouses, runtimes, and workspace configurations. • Lead Unity Catalog governance strategy including catalogs, schemas, permissions, and external locations. • Design and oversee enterprise monitoring, alerting, compliance dashboards, and operational SLOs. • Lead high-severity incident management, root-cause analysis, and preventive remediation. • Establish and maintain security, audit, and regulatory compliance controls. • Drive cost governance, capacity forecasting, and performance optimization across workloads. • Define and lead Infrastructure-as-Code and automation strategy using Terraform, APIs, and CI/CD pipelines. • Standardize CI/CD, Databricks Asset Bundles, and DevOps operating models. • Partner with cloud platforms, security, and data engineering teams on architecture and governance decisions. • Own platform documentation, onboarding standards, and the operational maturity roadmap. • Coach and mentor junior and mid-level Databricks administrators and act as senior technical escalation point. • Administer Databricks accounts and workspaces across development, test, and production environments. • Configure and manage clusters, jobs, SQL warehouses, runtimes, repos, and workspace settings. • Implement monitoring, alerting, dashboards, and operational runbooks. • Provide Tier 2/3 platform support, incident resolution, and root-cause analysis. • Enforce security and governance using Unity Catalog, RBAC, service principals, and least-privilege access. • Manage secrets, audit logging, and compliance reporting. • Administer Unity Catalog objects, external locations, and cloud storage integrations. Skills: 2–5 years of experience in Databricks Administration / Big Data Platforms. Strong hands-on experience with Databricks (Workspace, Clusters, Jobs, Notebooks). Experience with Apache Spark (PySpark/Scala) and distributed data processing. Knowledge of cloud platforms (Azure preferred, AWS/GCP also acceptable). Experience with data storage systems like ADLS, S3, Blob Storage. Good understanding of data engineering concepts, ETL/ELT pipelines. Familiarity with security best practices (IAM roles, data encryption, private endpoints). Proficiency in SQL and scripting languages (Python, Shell, etc.). Experience in performance tuning and cost optimization for clusters and workloads.