HiringCafe
📋 External Recruiting Agencies

The entity, Union Rebellion, is identified as a recruiting agency hiring on behalf of other companies.

This company was flagged and excluded from default search results. Proceed with caution.

Save Job
Posted 2mo ago

Software Engineering, Data Science & Systems Design Experts (AI Model Evaluation)

@ Union Rebellion
View All Jobs
United States or United Kingdom or Canada
$60-$100/hrRemoteMultiple Commitments Available
Responsibilities:Evaluate AI-generated, Annotate responses, Test code
Requirements Summary:3+ years in software engineering or data science; strong proficiency in multiple programming languages; independent problem-solver; experience with LLMs in coding workflows; detail-oriented with ability to critique technical reasoning.
Technical Tools Mentioned:Python, JavaScript, C++, Java, SQL, Go, Rust
Save
Mark Applied
Hide Job
Report & Hide
Job Description

Role Overview

As a Technical AI Evaluator, you will assess and improve AI-generated responses to coding and software engineering tasks. Your work will directly impact how accurately and effectively AI systems solve problems, write code, and communicate technical concepts.

This is a hands-on, intellectually rigorous role suited for professionals with strong coding, analytical, and problem-solving skills.

Key Responsibilities

  • Evaluate AI-generated responses for accuracy, reasoning, clarity, and completeness

  • Execute and test code to validate correctness and performance

  • Identify bugs, inefficiencies, and logical flaws in model outputs

  • Annotate responses with detailed feedback and improvement suggestions

  • Assess code quality, readability, and adherence to best practices

  • Fact-check technical claims using reliable and authoritative sources

  • Apply structured evaluation frameworks and guidelines consistently

What We’re Looking For

  • Bachelor’s, Master’s, or PhD in Computer Science or a related field

  • 3+ years of professional experience in software engineering, data science, or related roles

  • Strong proficiency in at least two programming languages (e.g., Python, JavaScript, C++, Java, SQL, Go, Rust, etc.)

  • Ability to independently solve medium to hard algorithmic problems

  • Experience working with or alongside LLMs in coding workflows

  • Strong attention to detail and ability to evaluate complex technical reasoning

Nice to Have

  • Experience with AI model evaluation, RLHF, or data annotation

  • Background in competitive programming

  • Contributions to open-source projects (merged pull requests preferred)

  • Experience reviewing production-level code

  • Ability to explain complex technical concepts clearly to non-experts

What Success Looks Like

  • You consistently identify subtle bugs, flawed logic, and edge cases

  • Your feedback improves the accuracy and clarity of AI-generated code

  • You produce structured, reproducible evaluation outputs

  • AI systems become more reliable for real-world engineering tasks

Compensation & Terms

  • Independent contractor engagement

  • Fully remote with flexible working hours (open to US and non-US candidates)

  • Full-time or part-time contract options available

  • Weekly payments based on services rendered

Equal Opportunity

We consider all qualified applicants without regard to legally protected characteristics and provide reasonable accommodations upon request.