Set scientific strategy for customer-grounded Copilot quality. Translate customer feedback, product signals, and incidents into evaluation and post-training priorities. Design representative evaluation sets, rubrics, graders, calibration methods, reward signals, and reusable learning assets. Diagnose failures across retrieval, grounding, tools, orchestration, reasoning, generation, and evaluation. Connect offline metrics with online customer outcomes, ensure rigor and reproducibility, and mentor scientists and engineers across organizational boundaries.
Overview
Responsibilities
Qualifications
CADET (Customer and Analytics Driven Evals Team) is building a customer-grounded quality system for Copilot. Our mission is to rapidly identify the customer scenarios that matter most, represent them faithfully in evaluation and learning assets, run quality gates continuously, and turn every important failure into reusable product and model improvements.
We bring together DSAT and other product signals, deep customer engagements to create representative eval sets. Operating in a fast-paced environment, we connect customer grounded quality issues with quality teams to advance Copilot quality and product innovation.
We are looking for a Principal Applied Scientist to work directly with enterprise customers and Copilot teams, translating high-value workflows, expected outcomes, and recurring pain points into trusted evaluation and learning signals. You will set the scientific direction for customer-grounded quality: define what “good” means, assess whether eval portfolios represent real needs, diagnose model and agent failures, and convert evidence into reusable evals, RLEs, reward signals, and post-training priorities. The ideal candidate combines scientific depth with product judgment and can turn ambiguous customer problems into rigorous, scalable methods in partnership with applied researchers and ML engineers.
Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.
Responsibilities
- Set the scientific strategy for customer-grounded quality across priority Copilot intents, defining what good means and the tradeoffs across various quality and safety attributes
- Translate user research, enterprise customer feedback, DSAT, and production incidents into evaluation and post-training priorities, then lead cross-team creation of reusable evaluation, regression, RLE, and post-training assets for the highest-value workflows and failure patterns.
- Develop methods to assess evaluation-set representativeness, coverage, freshness, discrimination, grader reliability, and alignment with production outcomes, identifying material gaps, drift, and emerging loss patterns.
- Design behavior and task evaluations, rubrics, graders, and calibration methods that translate qualitative customer expectations into measurable release-over-release quality.
- Establish methods to attribute quality losses across grounding, retrieval, tools, orchestration, model reasoning, response generation, and evaluation, linking offline movement with online signals such as DSAT, task completion, retries, abandonment, and escalation.
- Set a high bar for scientific rigor, reproducibility, documentation, and interpretation of results while mentoring scientists and engineers and influencing evaluation and post-training strategy across organizational boundaries.
Qualifications
Required Qualifications:
- Bachelor's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 6+ years related experience (e.g., statistics, predictive analytics, research)
- OR Master's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 4+ years related experience (e.g., statistics, predictive analytics, research)
- OR Doctorate in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 3+ years related experience (e.g., statistics, predictive analytics, research)
- OR equivalent experience.
Preferred: Qualifications:
- Advanced degree in computer science, machine learning, statistics, applied mathematics, or a related quantitative field, or equivalent practical experience.
- Significant experience applying machine learning, natural language processing, information retrieval, reinforcement learning, experimentation, or evaluation methods to complex production systems.
- Experience designing evaluations, metrics, experiments, datasets, graders, or reward functions for AI or agentic systems.
- Hands-on ability to inspect model outputs, identify behavioral patterns, and translate qualitative judgments into testable hypotheses and measurable evaluation criteria.
- Solid understanding of statistical inference, sampling, measurement validity, bias, uncertainty, and experimental design.
- Solid written and verbal communication skills, with a demonstrated ability to drive results across organizational boundaries by aligning science, engineering, product, and platform teams around shared quality goals, explicit ownership boundaries, and measurable outcomes.
- Experience with large language models, copilots, agents, tool use, retrieval-augmented generation, or enterprise grounding.
- Experience running or partnering on RLHF, direct preference optimization, instruction tuning, fine-tuning, or human-preference data programs.
- Experience connecting offline metrics with online product behavior and customer outcomes.
- Experience working directly with enterprise customers or translating qualitative research and customer signals into scientific assets.
- Publication record, patents, or demonstrated industry impact in relevant applied research areas.
#cadets
Applied Sciences IC5 - The typical base pay range for this role across the U.S. is USD $142,800 - $274,800 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $188,000 - $304,200 per year.
Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here:
https://careers.microsoft.com/us/en/us-corporate-pay
This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled.
Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.
Microsoft Redmond, Washington, USA Office
1 Microsoft Way, Redmond, WA, United States, 98052
Microsoft Bellevue, Washington, USA Office
Bellevue, United States
Microsoft Issaquah, Washington, USA Office
Issaquah, United States
Microsoft Kirkland, Washington, USA Office
Kirkland, United States
Microsoft Mercer Island, Washington, USA Office
Mercer Island, United States
Microsoft Sammamish, Washington, USA Office
Sammamish, United States
Microsoft Seattle, Washington, USA Office
Seattle, United States
Similar Jobs
Cloud • Computer Vision • Information Technology • Sales • Security • Cybersecurity
Architects, deploys, and operates scalable data platform infrastructure supporting analytics. Responsibilities include administering Airflow and Superset, managing Terraform infrastructure, building CI/CD pipelines, implementing security and compliance controls, developing dbt data models, cataloging assets in OpenMetadata, automating with Python, and mentoring engineers. The role requires extensive infrastructure and data engineering experience, cloud expertise, container orchestration, and eligibility for CJIS clearance.
Top Skills:
Ai TechnologiesApache AirflowApache SupersetAWSAzureCi/CdDbtDockerGCPIamKubernetesMachine Learning PipelinesOciOpenmetadataPythonSQLTerraform
Cloud • Computer Vision • Information Technology • Sales • Security • Cybersecurity
Architect and build production AI security systems spanning endpoint sensors and cloud-scale infrastructure. Lead autonomous cyber defense capabilities, including detection, threat hunting, response, adversary simulation, secure agent execution, and model evaluation. Establish technical direction, benchmarks, and safety practices while mentoring senior engineers and researchers. Carry frontier AI research into the Falcon platform and represent the company through publications, open source, standards, and industry collaboration.
Top Skills:
Adversarial Machine LearningAgentic ArchitecturesCC++Cloud SecurityCyber RangesDistributed SystemsEndpoint SecurityGoGpu InferenceKernel-Mode TelemetryLarge Language Models (Llms)PythonReinforcement LearningReward ModelingRustSandboxed ExecutionSglangSIEMStreaming Data InfrastructureSynthetic DataTensorrt-LlmTritonUser-Mode TelemetryVllm
Cloud • Computer Vision • Information Technology • Sales • Security • Cybersecurity
Own the technical strategy and architecture for CrowdStrike’s global authentication and authorization platform. Design highly scalable, low-latency, multi-tenant identity systems supporting MSSP models, cross-region operations, fine-grained access controls, and five-nines availability. Lead initiatives involving passwordless authentication, continuous verification, adaptive access, Zero Trust, policy engines, and modern identity standards. Mentor senior engineers and influence product, security, and engineering stakeholders while driving long-term technical vision.
Top Skills:
AbacAWSAzureCedarFido2GCPGoGoogle ZanzibarJavaKubernetesOauth 2.0OidcOpaPythonRbacRebacSAMLWebauthn
What you need to know about the Seattle Tech Scene
Home to tech titans like Microsoft and Amazon, Seattle punches far above its weight in innovation. But its surrounding mountains, sprinkled with world-famous hiking trails and climbing routes, make the city a destination for outdoorsy types as well. Established as a logging town before shifting to shipbuilding and logistics, the Emerald City is now known for its contributions to aerospace, software, biotech and cloud computing. And its status as a thriving tech ecosystem is attracting out-of-town companies looking to establish new tech and engineering hubs.
Key Facts About Seattle Tech
- Number of Tech Workers: 287,000; 13% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Amazon, Microsoft, Meta, Google
- Key Industries: Artificial intelligence, cloud computing, software, biotechnology, game development
- Funding Landscape: $3.1 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Madrona, Fuse, Tola, Maveron
- Research Centers and Universities: University of Washington, Seattle University, Seattle Pacific University, Allen Institute for Brain Science, Bill & Melinda Gates Foundation, Seattle Children’s Research Institute
