📝 AI Trajectory Annotator and Reviewer, are you there?

Hey there! We're looking for an AI Trajectory Annotator and Reviewer for a US-based AI research company building custom reinforcement learning environments and coding agent benchmarks. If you are methodical and detail-oriented with Python or Go and Linux, keep reading!

🌎 100% Remote

💵 Salary in USD

📍 LATAM (Brazil, Argentina, or Colombia) - Reporting to US, EST

🕒 Contract, hourly

🇬🇧 Advanced English

About our client 🤓

You will be working for our client, a US-based AI research company that builds custom reinforcement learning environments and benchmarks for advanced AI coding agents. They run large-scale evaluation pipelines where agents work autonomously for hundreds of steps inside real open-source codebases, and they need precise human judgment to assess the quality of those runs. This is a contract role with flexible scheduling.

About Athyna 🙋🏻

Athyna is a Global Talent Platform, helping align world-class talent with the best employment opportunities across the globe.

We believe in the future of work. That's why we are a 100% globally distributed team across 5 continents. We believe in people, and we value freedom, integrity, quality, and sustainability.

We're proudly carbon neutral as an organization. We're very keen on social impact - we truly care about everyone on our team, and we want to make the world a better place for all of us.

In your day-to-day, you will 👇🏼

Read and annotate AI agent trajectories step by step, evaluating each command against the actual state of the environment
Review existing annotations from AI tooling or other annotators, confirming, correcting, or rejecting each step with clear technical reasoning
Identify the precise point in a run where the agent first went off course, not just where it visibly failed
Distinguish between incorrect agent decisions and merely inefficient ones, as both are scored differently on the platform
Flag cases where the agent attempted to game the evaluation rather than solve the task genuinely
Apply human judgment throughout the entire annotation process, with no AI tools used for any decision-making step

Skills you have 🚀

2+ years of hands-on experience in software engineering, DevOps, or site reliability, with real debugging in real codebases
Strong proficiency in Python or Go, plus the ability to read and reason about unfamiliar languages under time pressure
Solid Linux knowledge: logs, running processes, build failures, and containerized environments
Everyday comfort with Git, diffs, pull requests, and issue tracking
Ability to debug effectively with no test suite and no error message pointing directly to the cause
Capacity to hold context across long agent runs where a decision at step 70 may depend on what happened at step 10
Clear written English: every annotation judgment requires a technical explanation that another engineer can independently verify

Skills that will make you stand out 💥

Experience with Rust, C, or C++
Prior AI evaluation work such as RLHF, supervised fine-tuning, model evaluation, or red teaming
Familiarity with Docker and containerized test environments
Background in concurrency, asynchronous code, or distributed systems