Birlasoft—Azure Databricks Developer

Job description

Description: Area(s) of responsibility Key Responsibilities 1. Data Engineering Execution - Build and maintain ingestion frameworks (ADF / Databricks / Spark) - Implement Bronze -> Silver transformations aligned to architecture - Architect data quality checks, schema validation, and contract rules - Build and optimize robust, high-throughput ELT/ETL pipelines, enabling ingestion, transformation, and curation of structured, semi structured, and unstructured data. - Integrate data from multiple on premise and cloud based systems, APIs, and third-party sources. - Implement complex transformations using PySpark, ensuring performance efficiency and code modularity. - Build orchestration workflows in ADF, including pipelines, triggers, linked services, integration runtimes, and parameterized datasets. - Familiar with using Databricks Genie. - Build: Dimensional models (star/snowflake schemas), Fact tables, dimensions, surrogate keys, SCD handling - Translate Silver datasets into: Analytics-ready models, Consistent KPI definitions and business logic - Ensure: Consistency across domains (common dimensions, conformed models), Reusability and scalability of models 2. Databricks & PySpark Engineering - Develop scalable transformation scripts using PySpark on Databricks, applying advanced optimizations like caching, partitioning, and Delta Lake capabilities. - Implement Delta Lake features—ACID transactions, schema enforcement, schema evolution, and time travel—across the data lifecycle. - Perform performance tuning, handling bottlenecks related to cluster configuration, shuffle operations, joins, and parallelization. - Collaborate with platform teams to manage Databricks clusters, jobs, notebooks, and CI/CD integrations.