Moody's—Senior QA AI Engineer

Job description

Description: At Moody's, we unite the brightest minds to turn today’s risks into tomorrow’s opportunities. We do this by striving to create an inclusive environment where everyone feels welcome to be who they are—with the freedom to exchange ideas, think innovatively, and listen to each other and customers in meaningful ways. Moody’s is transforming how the world sees risk. As a global leader in ratings and integrated risk assessment, we’re advancing AI to move from insight to action—enabling intelligence that not only understands complexity but responds to it. We decode risk to unlock opportunity, helping our clients navigate uncertainty with clarity, speed, and confidence. If you are excited about this opportunity but do not meet every single requirement, please apply! You still may be a great fit for this role or other open roles. We are seeking candidates who model our values: invest in every relationship, lead with curiosity, champion diverse perspectives, turn inputs into actions, and uphold trust through integrity. Skills and Competencies - 5-7 years of experience in software quality engineering with hands-on expertise testing Java-based backend services and RESTful application programming interfaces - Proficiency with test automation frameworks such as Selenium, TestNG, JUnit, RestAssured, or Cucumber for user interface and application programming interface testing - Strong understanding of relational databases including PostgreSQL and Microsoft SQL Server to perform data validation and verify data integrity across test scenarios - Hands-on experience testing Anthropic Claude integrated features, including large language model output validation, semantic evaluation techniques, hallucination detection, and streaming application programming interface behavior - Familiarity with Model Context Protocol and agentic artificial intelligence testing patterns, including tool-use accuracy, context management, and deterministic versus non-deterministic output validation - Experience with artificial intelligence security testing, including prompt injection testing, jailbreak validation, sensitive data leakage checks, and application reliability and performance assessments - Strong understanding of software design principles and microservices architectures, with the ability to develop effective test strategies and identify system-level edge cases - Deep expertise in artificial intelligence, with a track record of implementing advanced AI solutions to drive strategic transformation and operational efficiency. Strong experience using AI tools to lead innovation initiatives. Demonstrated leadership in managing AI-related risks, ensuring ethical governance, and fostering a culture of responsible AI adoption across the organization. Education - Bachelor or Master of Science in Computer Science, Mathematics, Physics, or a related field Responsibilities Lead quality engineering and artificial intelligence testing initiatives to ensure reliable, scalable, and secure software solutions across cloud-based platforms. - Design, develop, and maintain automated test suites for Java-based backend services and RESTful application programming interfaces to ensure high software quality across releases - Own the test strategy for Anthropic Claude integrated features, including large language model output validation, regression baselines, tool-call accuracy verification, context management validation, and end-to-end integration coverage - Build and maintain continuous integration integrated evaluation frameworks for non-deterministic artificial intelligence outputs, including rubric-based scoring, automated prompt regression suites, and behavioral drift detection - Collaborate with product owners, software engineers, and machine learning engineers to define testable requirements, review acceptance criteria, and ensure high-quality delivery across agile releases - Perform database validation testing against PostgreSQL and Microsoft SQL Server environments to verify data integrity, transformation logic, and backend workflow accuracy - Conduct artificial intelligence specific security testing, including prompt injection, adversarial input validation, jailbreak testing, and sensitive data leakage assessments aligned with Moody’s governance standards - Define and enforce synthetic or masked test data standards for artificial intelligence feature testing to ensure appropriate handling of confidential and regulated information - Monitor and report quality, performance, and artificial intelligence observability metrics including token consumption, streaming latency, tool-call success rates, model error rates, and reliability indicators - Integrate automated testing into continuous integration and continuous delivery pipelines, ensuring quality gates and deployment controls are effective and actionable - Conduct test code reviews, mentor quality engineering team members, perform exploratory testing, and drive continuous improvement of testing practices and processes - Support performance and load testing of cloud-native software-as-a-service platforms on Amazon Web Services and Microsoft Azure to validate scalability, availability, and reliability requirements - Maintain clear test documentation, traceability matrices, defect analysis reports, and quality metrics to ensure transparency and alignment across stakeholders About the Team At Moody’s, we unite the brightest minds to turn today’s risks into tomorrow’s opportunities. As part of Moody’s Insurance Solutions, the Climate on Demand team develops innovative software, advanced analytics, and artificial intelligence enabled capabilities that help organizations understand and manage climate and natural catastrophe risks. You will contribute to building impactful solutions that support informed decision-making, strengthen resilience for customers around the world, and advance the adoption of responsible artificial intelligence across our products and services. Moody’s is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, disability, protected veteran status, sexual orientation, gender expression, gender identity or any other characteristic protected by law. Candidates for Moody's Corporation may be asked to disclose securities holdings pursuant to Moody’s Policy for Securities Trading and the requirements of the position. Employment is contingent upon compliance with the Policy, including remediation of positions in those holdings as necessary.