Principal AI Architect - M365 IC3 Team (Intelligent Conversation and Communications Cloud)

We are looking for a Principal AI Architect to help define, build, and scale the evaluation systems that shape the future of AI products. This role sits at the intersection of engineering, applied science, product architecture, and AI evaluation. The ideal candidate has deep technical judgment, strong systems thinking, hands-on experience evaluating LLMs, and the ability to translate emerging AI capabilities into reliable product experiences. 

This role is especially important for agentic AI systems, where product behavior is often nondeterministic, context-dependent, and difficult to evaluate with traditional testing alone. The person in this role will help teams build a deep understanding of how agents behave, where they succeed, where they fail, which gaps matter most, and where those gaps should be addressed: in prompts, tools, orchestration, retrieval, ranking, product UX, safety systems, or core code. 

Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.

• Define the technical vision and architecture for AI, LLM, RAG, and agent evaluation systems across the product lifecycle.
• Design evaluation frameworks that assess product quality before implementation is complete, during development, at launch, and post-ship.
• Build an understanding of how nondeterministic agentic systems behave across tasks, users, contexts, tools, and product surfaces.
• Identify behavioral gaps, failure modes, model limitations, retrieval issues, orchestration defects, and product-quality risks before they become customer-impacting issues.
• Determine where issues should be addressed across the system: model behavior, prompts, tool use, search and retrieval, ranking, grounding, orchestration, UX, policy, telemetry, or product code.
• Partner with engineering, applied science, and data science teams to bring ML, DS, LLM, RAG, and agent evaluation methods directly into product codebases and development workflows.
• Integrate evals into build pipelines, release gates, experimentation systems, and engineering workflows so evaluation becomes a standard part of how products are built and shipped.
• Make evaluation results easy to access, interpret, and act on through dashboards, scorecards, quality reports, and product-health views.
• Build systems that connect product telemetry, offline evaluation, human judgment, automated evals, experimentation, RAG quality, agent behavior, and customer-quality signals.
• Understand and evaluate enterprise search, RAG, grounding, indexing, ranking, permissions, freshness, and relevance systems for products such as Copilot.
• Translate ambiguous product goals into measurable evaluation strategies, success criteria, timelines, and technical plans.
• Drive architecture decisions across components, services, data pipelines, model interfaces, search systems, retrieval layers, evaluation harnesses, dashboards, and reporting systems.
• Work with product leaders to prioritize evaluation investments and align them with product milestones and release decisions.
• Mentor senior engineers and applied scientists on building reliable, scalable, and reusable evaluation infrastructure.
• Stay current with LLM evaluation methods, agentic systems, RAG evaluation, benchmark design, prompt/model behavior, experimentation, and responsible AI practices.
• In this role, you will help evaluate products before the code is fully ready, before launch, and after they ship.
• You will work across product, engineering, applied science, and data science teams to bring rigorous LLM, RAG, agent, and AI evaluation practices into the product lifecycle.
• You will help IC3 and partner teams understand whether AI systems are working as intended, where they fail, how they improve, and what it takes to ship them responsibly at scale. 

• Bachelor's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 6+ years related experience (e.g., statistics, predictive analytics, research)
  • OR Master's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 4+ years related experience (e.g., statistics, predictive analytics, research)
  • OR Doctorate in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 3+ years related experience (e.g., statistics, predictive analytics, research)
  • OR equivalent experience.