Sr Lead Machine Learning Engineer

Upwork Inc.’s (Nasdaq: UPWK) family of companies connects businesses with global, AI-enabled talent across every contingent work type including freelance, fractional, and payrolled. This portfolio includes the Upwork Marketplace, which connects businesses with on-demand access to highly skilled talent across the globe, and Lifted, which provides a purpose-built solution for enterprise organizations to source, contract, manage, and pay talent across the full spectrum of contingent work. From Fortune 100 enterprises to entrepreneurs, businesses rely on Upwork Inc. to find and hire expert talent, leverage AI-powered work solutions, and drive business transformation. With access to professionals spanning more than 10,000 skills across AI & machine learning, software development, sales & marketing, customer support, finance & accounting, and more, the Upwork family of companies enables businesses of all sizes to scale, innovate, and transform their workforces for the age of AI and beyond.

Since its founding, Upwork Inc. has facilitated more than $30 billion in total transactions and services as it fulfills its purpose to create opportunity in every era of work. Learn more about the Upwork Marketplace at Upwork.com and follow us on LinkedIn, Facebook, Instagram, TikTok, and X; and learn more about Lifted at Go-Lifted and follow on LinkedIn.

We're looking for a Sr Lead MLE/Applied Scientist to define how success is measured for AI agents performing real-world tasks. This role is at the forefront of building trust and quality into agentic systems by crafting rigorous, reproducible evaluation frameworks that shape what we ship. You’ll work cross-functionally to evaluate human+AI collaboration, assess outcomes beyond accuracy metrics, and uncover what’s truly working for freelancers and clients. Join us in revolutionizing agent evaluation and making a measurable impact on AI systems that power the future of work.

• Design and implement comprehensive evaluation frameworks that reflect real-world task success for agentic systems, with a focus on human+AI collaboration outcomes
• Build benchmarking pipelines that capture nuanced success indicators including trust calibration, intervention frequency, and agent handoff quality
• Lead development of observability tools and instrumentation for analyzing agent behavior in production
• Translate complex qualitative and quantitative signals into actionable insights that inform model iteration and product prioritization
• Collaborate with researchers, engineers, and product teams to align evaluation methodologies with business and user goals
• Own benchmarking infrastructure that enables reproducible, scalable evaluation across AI initiatives
• Champion rigorous experimental design and statistical analysis across teams to ensure consistent and meaningful measurement standards

• Proven experience designing evaluation systems for agentic or LLM-based AI, ideally in complex, interactive or open-ended environments
• Deep expertise in statistical experimentation, benchmark creation, and human-AI interaction assessment
• Fluency in building data pipelines and tooling using Python, SQL, and distributed data processing frameworks
• Demonstrated ability to influence product and model roadmaps through evaluation insights and performance measurement
• Adaptive-level proficiency in integrating AI tools into technical workflows for analysis, experimentation, and observability refinement

At Upwork, you’ll shape the future of work for a global, remote-first workforce, creating economic opportunities for professionals worldwide. While we have a physical office in Palo Alto, we currently hire full-time employees in 21 U.S. states, making it easier than ever to join our mission from wherever you call home.

Our culture is built on trust, risk-taking, customer focus, and excellence, all in service of our core mission: to create economic opportunities so people have better lives. We embrace authenticity and inclusion, encouraging everyone to bring their whole selves to work. Personal and professional growth is a priority here, supported through development programs, mentorship, and our Upwork Belonging Communities.

We’re proud to offer benefits that go beyond the basics, including comprehensive medical coverage for you and your family, unlimited PTO, a 401(k) plan with matching, 12 weeks of paid parental leave, and an Employee Stock Purchase Plan. Visit our Life at Upwork page to learn more about our values, working principles, and the overall employee experience.

Ready to help shape the future of work? Check out our Careers page and follow us on LinkedIn, Facebook, Instagram, TikTok, and X to learn more about life at Upwork.

Upwork is an Equal Opportunity Employer committed to recruiting and retaining a diverse and inclusive workforce. We do not discriminate based on race, religion, color, national origin, gender (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender identity, gender expression, age, status as a protected veteran, status as an individual with a disability, genetic information, or other legally protected characteristics under federal, state, or local law.

Please note that a criminal background check may be required once a conditional job offer is made. Qualified applicants with arrest or conviction records will be considered in accordance with applicable law, including the California Fair Chance Act and local Fair Chance ordinances.