Evaluate AI systems on personalized, real-world life tasks. Use extensive LLM experience to assess whether outputs are useful, realistic, safe, complete, and tailored to context, preferences, constraints, and tradeoffs. Provide detailed written judgments across areas such as food, health, productivity, careers, and learning, contributing to improvements in AI assistants and personal workflows. Expected commitment is 20–40 hours per week.
This role is for one of our clients
Compensation: $50 - $200 per hour
A leading AI research organization is seeking advanced LLM power users to evaluate how well AI systems handle personalized, real-world life tasks.
This role is for people who use AI tools heavily in their personal lives and can clearly judge whether an AI response is useful, personalized, realistic, and successful.
Requirements
Who We’re Looking For
Strong candidates have:
- Heavy personal usage of LLM products
- Experience using AI for multi-step tasks, planning, research, decision-making, or personal workflows
- Familiarity with tools such as ChatGPT, Claude, Gemini, Perplexity, Cursor, Windsurf, Codex, or other AI agents
- Ability to explain what makes an AI output good, bad, incomplete, unsafe, or unrealistic
- Strong written judgment and attention to detail
Why This Work Matters
LLMs are quickly becoming personal assistants for everyday decisions, but truly useful AI needs to do more than produce generic advice. It needs to understand context, preferences, constraints, tradeoffs, and what success looks like in real life. Your evaluations will help improve how AI systems support people with practical, high-context tasks across food, health, productivity, careers, and learning. This work directly contributes to making AI assistants more personalized, trustworthy, and useful for real-world personal workflows.
Engagement Details
- Expected commitment: 20-40 hours/week
We consider all qualified applicants without regard to legally protected characteristics and provide reasonable accommodations upon request.
Similar Jobs
Cloud • Information Technology • Productivity • Security • Software • App development • Automation
Principal Software Engineer leading frontend and full-stack architecture for Atlassian’s Growth Platform. The role defines technical strategy for messaging, personalization, notifications, and experimentation systems; designs scalable microservices, APIs, data models, and React/TypeScript components; drives performance, reliability, security, accessibility, observability, and CI/CD standards; and influences multiple teams through architecture reviews, cross-functional partnership, and mentorship.
Top Skills:
APIsAutomated TestingCi/CdClient SdksClient-Side State ManagementCore Web VitalsCSSData ModelingDistributed SystemsDistributed TracingHTML5Javascript Es6+MicroservicesObservabilityReactTelemetryTypescript
Cloud • Information Technology • Productivity • Security • Software • App development • Automation
Provides organization-wide technical leadership for Atlassian’s Dev AI platforms and autonomous developer initiatives. Defines multi-year architectures for agent runtimes, orchestration engines, execution sandboxes, memory systems, and context retrieval. Drives distributed-system architecture, reliability, security, responsible AI governance, evaluation, and operational standards. Leads cross-organizational technical initiatives, resolves architectural ambiguity, mentors senior engineers, partners with product and engineering leaders, and remains hands-on with strategic prototypes and architectural validation.
Top Skills:
Agent OrchestrationAi Evaluation FrameworksArtificial IntelligenceAtlassian Teamwork GraphAutonomous Ai AgentsBitbucketCode IntelligenceCode SynthesisDistributed SystemsJIRAResponsible AiRetrieval SystemsRuntime SandboxesSemantic SearchTelemetry
Cloud • Information Technology • Productivity • Security • Software • App development • Automation
Lead enterprise customer engagements to adopt and operationalize Atlassian’s Rovo AI platform. Facilitate discovery, design and configure AI agents and connectors, guide deployments from sandbox to production, and drive measurable adoption outcomes. Own engagement playbooks, enable internal teams, provide product feedback, and collaborate with sales, customer success, product, and engineering teams. The role involves supporting complex enterprise use cases, mentoring without direct authority, and up to 15% domestic and international travel.
Top Skills:
Agentic WorkflowsAi AgentsAtlassianConfluenceGitGoogle DriveJIRALlm-Powered ArchitecturesPrompt EngineeringRovoRovo StudioSharepointSlack
What you need to know about the Seattle Tech Scene
Home to tech titans like Microsoft and Amazon, Seattle punches far above its weight in innovation. But its surrounding mountains, sprinkled with world-famous hiking trails and climbing routes, make the city a destination for outdoorsy types as well. Established as a logging town before shifting to shipbuilding and logistics, the Emerald City is now known for its contributions to aerospace, software, biotech and cloud computing. And its status as a thriving tech ecosystem is attracting out-of-town companies looking to establish new tech and engineering hubs.
Key Facts About Seattle Tech
- Number of Tech Workers: 287,000; 13% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Amazon, Microsoft, Meta, Google
- Key Industries: Artificial intelligence, cloud computing, software, biotechnology, game development
- Funding Landscape: $3.1 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Madrona, Fuse, Tola, Maveron
- Research Centers and Universities: University of Washington, Seattle University, Seattle Pacific University, Allen Institute for Brain Science, Bill & Melinda Gates Foundation, Seattle Children’s Research Institute
