Appen Logo

Appen

Urdu LLM Evaluator

Sorry, this job was removed Sorry, this job was removed at 12:08 p.m. (PST) on Thursday, May 22, 2025
Be an Early Applicant
Remote
Hiring Remotely in United States
Remote
Hiring Remotely in United States

Similar Jobs

2 Hours Ago
Easy Apply
Remote
Hybrid
8 Locations
Easy Apply
Entry level
Entry level
Greentech • Hardware • Internet of Things • Machine Learning • Software • Business Intelligence • Agriculture
Halter invites candidates to join a talent pool for future roles across various fields, emphasizing personal growth and an inclusive workplace.
Top Skills: CreativeDataDigital ProductFirmwareHardwareMachine LearningMarketingProductionSoftwareSupply ChainSystemsTradesman
3 Hours Ago
Remote
Hybrid
New York, NY, USA
150K-195K Annually
Senior level
150K-195K Annually
Senior level
Artificial Intelligence • Big Data • Cloud • Information Technology • Software • Big Data Analytics • Automation
The Strategic Enterprise Account Executive will manage named accounts, drive enterprise sales, engage C-level executives, develop strategies, and ensure client success.
Top Skills: ApmMarketingSales Engineering
3 Hours Ago
Remote
Hybrid
Waltham, MA, USA
Junior
Junior
Artificial Intelligence • Big Data • Cloud • Information Technology • Software • Big Data Analytics • Automation
As a Sr. Technical Specialist, you will provide technical support, troubleshoot applications, and work with various technologies to ensure customer success.
Top Skills: .NetDjangoFlaskIbm Db2JavaMs SqlOraclePrometheusPythonSaaSSnmp
Join Project Spearmint, a multilingual AI response evaluation project reviewing large language model (LLM) outputs in different languages, focused on either Tone or Fluency. Native-level fluency in a target language, along with strong English comprehension, is required.

As an evaluator, you will review short, pre-segmented datasets and assess model-generated replies based on specific quality dimensions. Your input will help validate evaluation frameworks and establish baseline quality metrics for future model development.

Key Responsibilities:
- Evaluate model replies in your native language based on either Tone or Fluency.
- Assess the overall quality, correctness, and naturalness of responses.
- Read the user prompt and two model replies, then rate each using a five-point scale.
- Provide brief rationales for any extreme ratings.

Project Breakdown:

Batch 1 – Tone: Determine whether replies are helpful, insightful, engaging, and fair. Flag formality mismatches, condescension, bias, or other tonal issues.

Batch 2 – Fluency: Assess grammatical accuracy, clarity, coherence, and natural flow.


This is a project-based opportunity with CrowdGen, where you will join the CrowdGen Community as an Independent Contractor. If selected, you will receive an email from CrowdGen regarding the creation of an account using your application email address. You will need to log in to this account, reset your password, complete the setup requirements, and proceed with your application for this role.

Make an impact on the future of AI – apply today and contribute from the comfort of your home.

Appen Kirkland, Washington, USA Office

12131 113th Ave NE, Kirkland, WA, United States, 98304

What you need to know about the Seattle Tech Scene

Home to tech titans like Microsoft and Amazon, Seattle punches far above its weight in innovation. But its surrounding mountains, sprinkled with world-famous hiking trails and climbing routes, make the city a destination for outdoorsy types as well. Established as a logging town before shifting to shipbuilding and logistics, the Emerald City is now known for its contributions to aerospace, software, biotech and cloud computing. And its status as a thriving tech ecosystem is attracting out-of-town companies looking to establish new tech and engineering hubs.

Key Facts About Seattle Tech

  • Number of Tech Workers: 287,000; 13% of overall workforce (2024 CompTIA survey)
  • Major Tech Employers: Amazon, Microsoft, Meta, Google
  • Key Industries: Artificial intelligence, cloud computing, software, biotechnology, game development
  • Funding Landscape: $3.1 billion in venture capital funding in 2024 (Pitchbook)
  • Notable Investors: Madrona, Fuse, Tola, Maveron
  • Research Centers and Universities: University of Washington, Seattle University, Seattle Pacific University, Allen Institute for Brain Science, Bill & Melinda Gates Foundation, Seattle Children’s Research Institute
By clicking Apply you agree to share your profile information with the hiring company.

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account