We are seeking an experienced and forward-thinking Senior Big Data Engineer (7+ Years) to join our technology team. In this role, you will be responsible for designing, developing, and optimizing high-performance, large-scale data processing pipelines and regulatory data platforms using Apache Spark, Java/Scala, Spring Boot, and Python/PySpark. The ideal candidate brings deep expertise in distributed data computing, complex Spark SQL query optimization, data analysis, and modern CI/CD DevOps workflows. You will actively leverage GenAI engineering tools (such as GitHub Copilot and Devin AI) to accelerate delivery, drive code quality, and translate complex business requirements into robust High-Level (HLD) and Low-Level Designs (LLD). Key Responsibilities Architect & Develop Data Solutions: Design, implement, and maintain scalable, high-throughput batch and stream data pipelines using Apache Spark, Java/Scala, and PySpark. End-to-End System Design: Analyze business and regulatory requirements to create comprehensive High-Level Designs (HLD), Low-Level Designs (LLD), data flow diagrams, and interface specifications. Data Engineering & Analytics: Perform in-depth data analysis across the data lifecycle—including data sourcing, schema evolution, transformation logic, quality validation, and actionable insight generation. Spark SQL Optimization: Write, tune, and maintain complex Spark SQL queries, partitioned datasets, and transformations for optimal performance over large enterprise datasets. Microservices & API Integration: Build enterprise-grade services and data ingestion microservices using Spring and Spring Boot. DevOps & CI/CD Delivery: Implement automated build, test, and deployment workflows using Git/GitHub, Jenkins / Lightspeed, and Harness. AI-Augmented Engineering: Leverage modern GenAI developer tooling (e.g., GitHub Copilot, Devin AI) to accelerate prototyping, code review, unit testing, and documentation. Cross-Functional Collaboration: Partner closely with Product Owners, Data Analysts, Business Teams, and Quality Assurance to ensure end-to-end alignment and timely delivery. Candidate Qualifications & Skill Set 1. Core Technical Skills (Must-Have) Experience:5-9 years of hands-on software and data engineering experience in enterprise environments. Distributed Computing & Big Data: Deep practical expertise in Apache Spark architecture (RDDs, DataFrames, Datasets, execution plans, memory management, partitioning, and performance tuning). Primary Programming Languages: Advanced proficiency in Java and/or Scala, with strong object-oriented and functional programming paradigms. Frameworks: Extensive hands-on experience building scalable applications using Java and Apache Spark. Scripting & Python/PySpark: Strong proficiency in Python with practical experience writing data transformations using PySpark. Data Querying & SQL: Expert-level proficiency in SQL / Spark SQL, window functions, complex joins, indexing, and query plan execution analysis (EXPLAIN). Data Analysis & Processing: Proven ability to source, cleanse, enrich, reconcile, and validate large datasets to deliver high-integrity reporting and analytical outputs. 2. Architecture & DevOps Solution Architecture: Proven track record of drafting clear, maintainable HLD and LLD documents, technical specifications, and data modeling artifacts. Version Control & CI/CD: Hands-on experience with GitHub / Bitbucket, build automation with Jenkins / Lightspeed, and modern continuous delivery platforms like Harness. 3. AI Tools & Modern Engineering Practices AI-Assisted Development: Experience utilizing AI productivity tools such as GitHub Copilot or Devin AI for test generation, code refactoring, and rapid prototyping. Engineering Excellence: Strong commitment to test-driven development (TDD), automated unit/integration testing, SonarQube quality gates, and secure coding standards. 4. Preferred / Good-to-Have Business Intelligence & Visualization: Experience with Tableau or similar BI visualization tools for dashboarding and reporting validation. Domain Knowledge: Prior exposure to Banking, Financial Services, Liquidity Reporting, Regulatory Data (e.g., NDTL/LCR), or Risk Management. 5. Soft Skills & Professional Attributes Exceptional Communication: Outstanding verbal and written communication skills; adept at explaining technical concepts to both technical peers and non-technical business stakeholders. Ownership & Problem Solving: Proactive attitude, strong analytical reasoning, and the ability to drive initiatives from inception to production deployment. ------------------------------------------------------ Job Family Group: Technology------------------------------------------------------ Job Family: Applications Development------------------------------------------------------ Time Type: Full time------------------------------------------------------ Most Relevant Skills Please see the requirements listed above.------------------------------------------------------ Other Relevant Skills For complementary skills, please see above and/or contact the recruiter.------------------------------------------------------ Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law.
