Annotation Judge (Remote - San Francisco, CA)

Our Customer is a tech real-estate marketplace company with a portfolio of the largest and most vibrant real estate and home-related brands on the web and mobile. The company’s brands focus on all stages of the home lifecycle: renting, buying, selling, financing and home improvement. They are committed to empowering consumers with unparalleled data, inspiration and knowledge around homes, and connecting them with the right local professionals to help.

We are seeking an Annotation Judge on a contract basis to support our Customer’s business needs. This role is 100% remote in San Francisco, CA.

Responsibilities:

• Fine-tune and calibrate AI/ML processes that support verbal interactions between consumers and real estate professionals.
• Collaborate with Operations Managers, ML Engineers, Data Scientists, Software Developers, and Natural Language experts to improve AI model performance.
• Review and evaluate annotated data to ensure accuracy, quality, and alignment with project requirements.
• Review AI-generated chatbot responses for accuracy, humanlike behavior, and conversational quality.
• Evaluate automatically annotated conversation elements and search result relevance based on complex user queries.
• Provide feedback and guidance to engineering teams to improve annotation quality and model accuracy.
• Collaborate with cross-functional teams to optimize machine learning model performance using annotated datasets.
• Develop and maintain annotation guidelines, workflows, and best practices.
• Create and maintain Ground Truth datasets for training and validation purposes.
• Provide examples and input for training content development.
• Identify gaps in training materials and recommend improvements.
• Support scaling efforts by defining clear annotation and quality standards usable across global teams.
• Apply strong attention to detail and language understanding to improve AI and NLP solutions.

Skills and Qualifications:

• Bachelor’s degree in English, Linguistics, or a related field.
• Minimum of 3+ years of active experience as a licensed real estate agent or broker.
• Strong understanding of the real estate transaction process.
• Demonstrated interest and curiosity in Artificial Intelligence and AI-generated content evaluation.
• Experience reviewing or evaluating AI-generated content.
• Experience working on Natural Language Processing (NLP) projects.
• Experience with transcription, localization, or translation-focused NLP initiatives.
• Strong analytical and attention-to-detail skills.
• Ability to work collaboratively in cross-functional team environments.
• Strong written and verbal communication skills.
• Ability to identify language subtleties and conversational nuances.
• Experience working with annotation tools or software used in NLP projects is a plus.

Preferred Qualifications:

• Experience with copywriting, UX writing, or content development aligned with voice and brand guidelines.
• Experience with machine-generated speech evaluation, search results evaluation, or search query categorization.
• Experience managing or following complex operational processes independently and collaboratively.

We offer a competitive salary range for this position. Most candidates who join our team are hired at the median of this range, ensuring fair and equitable compensation based on experience and qualifications.

Contractor benefits are available through our 3rd Party Employer of Record (Available upon completion of waiting period for eligible engagements) Benefits include: Medical, Dental, Vision, 401k.

An Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, or protected veteran status and will not be discriminated against on the basis of disability.