Infosys—Azure Administration

Job description

Description: • Administer and manage Azure Databricks workspaces across Development, QA, and Production environments. • Configure and manage interactive and job clusters, including autoscaling, auto termination, cluster pools, and policies. • Perform environment setup, workspace configuration, and platform upgrades. • Ensure high availability and operational stability of Azure Databricks services. • Integrate Azure Databricks with Azure Data Lake Storage Gen2 (ADLS), Azure SQL, Synapse, Event Hub, and other Azure services. • Manage VNet injected Databricks workspaces, private endpoints, and network security configurations. • Work with Azure networking teams to ensure secure and optimized connectivity. • Manage users, groups, and access controls using Azure Active Directory (Entra ID). • Implement data governance using Unity Catalog, including catalog, schema, table, and column level security. • Configure secure authentication using AAD SSO, Managed Identities, and Service Principals. • Manage secrets and credentials using Azure Key Vault integration. • Ensure compliance with enterprise security, audit, and regulatory requirements. • Monitor Azure Databricks usage and implement cost optimization strategies, including: o Cluster right sizing o Autoscaling and auto termination o Instance pool configuration • Analyze workload patterns and recommend performance tuning best practices. • Provide usage and cost reports to stakeholders and FinOps teams. • Deploy and manage Azure Databricks infrastructure using Infrastructure as Code tools such as Terraform or ARM templates. • Support CI/CD pipelines for Databricks notebooks, jobs, libraries, and workspace artifacts using Azure DevOps or GitHub. • Automate operational tasks to reduce manual effort and improve reliability. • Set up monitoring and alerting using Azure Monitor, Log Analytics, and Databricks metrics. • Troubleshoot cluster failures, job issues, workspace access problems, and performance bottlenecks. • Perform root cause analysis and implement preventive measures. • Provide Level 2/Level 3 support within a 24×7 operational model. • Create and maintain platform documentation, runbooks, SOPs, and architecture diagrams. • Collaborate with data engineers, data scientists, cloud engineers, and security teams. • Participate in platform reviews and continuous improvement initiatives. Requirements: We are looking for an experienced Azure Databricks Administrator to manage, operate, and optimize Azure Databricks platforms within an enterprise Azure ecosystem. The role is responsible for ensuring high availability, security, performance, and cost optimization of Azure Databricks environments while supporting data engineering, analytics, and AI/ML workloads. The ideal candidate will have strong hands on experience with Azure Databricks, Azure infrastructure, security, and governance, and will work closely with cloud, data, and business teams to deliver a stable and scalable data platform. Responsibilities: • Administer and manage Azure Databricks workspaces across Development, QA, and Production environments. • Configure and manage interactive and job clusters, including autoscaling, auto termination, cluster pools, and policies. • Perform environment setup, workspace configuration, and platform upgrades. • Ensure high availability and operational stability of Azure Databricks services. • Integrate Azure Databricks with Azure Data Lake Storage Gen2 (ADLS), Azure SQL, Synapse, Event Hub, and other Azure services. • Manage VNet injected Databricks workspaces, private endpoints, and network security configurations. • Work with Azure networking teams to ensure secure and optimized connectivity. • Manage users, groups, and access controls using Azure Active Directory (Entra ID). • Implement data governance using Unity Catalog, including catalog, schema, table, and column level security. • Configure secure authentication using AAD SSO, Managed Identities, and Service Principals. • Manage secrets and credentials using Azure Key Vault integration. • Ensure compliance with enterprise security, audit, and regulatory requirements. • Monitor Azure Databricks usage and implement cost optimization strategies, including: o Cluster right sizing o Autoscaling and auto termination o Instance pool configuration • Analyze workload patterns and recommend performance tuning best practices. • Provide usage and cost reports to stakeholders and FinOps teams. • Deploy and manage Azure Databricks infrastructure using Infrastructure as Code tools such as Terraform or ARM templates. • Support CI/CD pipelines for Databricks notebooks, jobs, libraries, and workspace artifacts using Azure DevOps or GitHub. • Automate operational tasks to reduce manual effort and improve reliability. • Set up monitoring and alerting using Azure Monitor, Log Analytics, and Databricks metrics. • Troubleshoot cluster failures, job issues, workspace access problems, and performance bottlenecks. • Perform root cause analysis and implement preventive measures. • Provide Level 2/Level 3 support within a 24×7 operational model. • Create and maintain platform documentation, runbooks, SOPs, and architecture diagrams. • Collaborate with data engineers, data scientists, cloud engineers, and security teams. • Participate in platform reviews and continuous improvement initiatives. Skills: Mandatory Technical Skills • 2+ years of experience in data platform or cloud administration, with strong focus on Azure Databricks. • Hands on experience with Azure services including ADLS Gen2, Azure Networking, Azure Monitor, and Azure AD. • Strong understanding of Apache Spark architecture and performance tuning. • Experience managing secure, enterprise scale Azure Databricks environments. • Proficiency in Python, PowerShell, or Bash scripting. • Hands on experience with Terraform or ARM templates. ________________________________________ Preferred / Good to Have • Experience with Unity Catalog, data lineage, and governance frameworks. • Familiarity with Delta Lake, job orchestration, and workload isolation. • Experience with CI/CD pipelines using Azure DevOps or GitHub Actions. • Understanding of FinOps practices in Azure. • Certifications: o Azure Databricks Certified Associate/Professional o Microsoft Azure Administrator / Solutions Architect ________________________________________ Soft Skills • Strong analytical and troubleshooting capabilities. • Excellent communication skills with technical and business stakeholders. • Ability to work independently in a fast paced, 24×7 support environment. • Proactive, detail oriented, and focused on operational excellence. ________________________________________ Nice to Have • Experience supporting large enterprise or multi workspace Azure Databricks platforms. • Exposure to onsite offshore delivery models and global teams. • Experience working in regulated environments (Healthcare).