Manager II, Program Management, AI Model Evaluation, International Seller Growth - ISG

Join the Seller AI team where you'll lead benchmarking and evaluation of AI models that enhance the seller experience across Amazon's global marketplace. You'll manage a team dedicated to validating, testing, and improving Artificial Intelligence (AI) and Large Language Models (LLMs) that power innovative seller tools. This role combines strategic leadership with hands-on technical oversight, requiring exceptional communication skills, team management, and stakeholder engagement.

In this position, you'll drive the development and implementation of comprehensive benchmarking methodologies to evaluate AI model performance across accuracy, robustness, bias, and reliability metrics. Your expertise will be crucial in translating technical findings into actionable insights that improve Seller Assistant's performance and contribute to the growth of Amazon's seller community worldwide.

Key job responsibilities

1. Lead LLM based automated evaluations program, Plan and execute benchmarking exercises for AI models, defining test plans, metrics, and acceptance criteria across accuracy, robustness, bias, and reliability dimensions. 2. Lead a team responsible for validating data based on specific annotation guidelines, ensuring accuracy and quality while escalating potential regulatory risks 3. Prepare comprehensive audit and benchmarking reports, including error ratings, root-cause analysis, and recommendations for senior stakeholders 4. Drive process efficiencies and explore automation opportunities to enhance the productivity of data generation initiatives 5. Mentor team members and help develop their skills while managing overall schedules, proactively mitigating risks, and keeping project scope under control.