Senior PySpark Developer

Key Responsibilities Design and develop scalable data pipelines using PySpark and Apache Spark. Build and optimize ETL/ELT frameworks for processing large volumes of structured and unstructured data. Develop high-performance Spark applications using Spark SQL, DataFrames, and RDDs. Work closely with data architects and business teams to define data requirements. Implement data quality, validation, and governance processes. Optimize Spark jobs for performance, memory management, and scalability. Integrate data from various sources including databases, APIs, cloud storage, and streaming platforms. Support production deployments and resolve performance bottlenecks. Mentor junior developers and participate in code reviews.

Key Responsibilities Design and develop scalable data pipelines using PySpark and Apache Spark. Build and optimize ETL/ELT frameworks for processing large volumes of structured and unstructured data. Develop high-performance Spark applications using Spark SQL, DataFrames, and RDDs. Work closely with data architects and business teams to define data requirements. Implement data quality, validation, and governance processes. Optimize Spark jobs for performance, memory management, and scalability. Integrate data from various sources including databases, APIs, cloud storage, and streaming platforms. Support production deployments and resolve performance bottlenecks. Mentor junior developers and participate in code reviews.