Content Evaluator (100% Remote)

Our Customer is a leading global collaboration platform that's transforming the way people work together, from the smallest business to the largest enterprise. With more than 500 million registered users across more than 180 countries, their mission is to unleash the world’s creative energy by designing a more enlightened way of working. Headquartered in San Francisco, CA, they have more than 12 offices around the world.

Our Customer is seeking a Content Evaluator on a contract basis to help support our Customer's business needs.

responsible for assessing how relevant a given document, image, or video is to a particular user query. This involves analyzing user intent, rating content relevance on a scale of 1–5, and providing qualitative feedback on system-generated answers. The role requires research, critical thinking, and attention to detail to ensure accurate evaluations that improve overall search and assistant performance.

This role is 100% remote.

Responsibilities:

• Evaluate the relevance of documents, images, or videos for specific user queries by rating them on a 1–5 scale (from not relevant to highly relevant).
• Analyze and interpret user intent to determine whether the query is informational, navigational, or transactional.
• Review query-document pairs, including context such as query details and timing, to ensure accurate evaluation.
• Assess the quality of assistant-generated answers for correctness, descriptiveness, and appropriateness (e.g., identifying when the system apologizes for lack of information).
• Conduct light research to validate the correctness of assistant responses and terminology used.
• Formulate test queries in search systems (including KQL for Elastic Search indexes) to better understand expected results.
• Document ratings, findings, and feedback according to defined rubrics and evaluation guidelines.
• Collaborate with team members to ensure consistency and quality across evaluations.

Skills and Qualifications:

• Strong understanding of English (both written and verbal).
• Ability to formulate effective search queries or ChatGPT-like prompts.
• Critical thinking skills and the ability to evaluate user intent.
• Strong attention to detail and consistency in applying evaluation rubrics.

Preferred:

• Prior experience in relevance evaluation, labeling, or information retrieval tasks.
• Basic ability to script tasks in Python to automate repetitive processes (not required but beneficial).
• Familiarity with search engines, Elastic Search, or AI-driven assistant tools.
• Experience with annotation platforms, labeling tools, or similar rating systems.

We offer a competitive salary range for this position. Most candidates who join our team are hired at the median of this range, ensuring fair and equitable compensation based on experience and qualifications.

Contractor benefits are available through our 3rd Party Employer of Record (Available upon completion of waiting period for eligible engagements)

Benefits include: Medical, Dental, Vision, 401k.

Please note: In order to to create a safe, productive work environment, our client is requiring all contractors who plan to be onsite to be fully vaccinated according to the CDC guidelines. At the point of offer, contractors will be required to attest their vaccination status.

An Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, or protected veteran status and will not be discriminated against on the basis of disability