Research Scientist (Remote/US/LATAM)

This is a research role at heart: you decide what a good evaluation is , design the benchmarks that prove it, and defend the methodology under lab scrutiny. You'll build frontier-grade evaluation packages across reasoning, coding, agents, tool use, and multi-modal — grounded in expert-verified truth, validated against multiple models, and QC'd to survive buyer-side review.

Responsibilities

● Evaluation research. Turn eval targets into original benchmark designs. Own the hard measurement questions: construct validity, item discrimination, headroom, reliability, contamination, and capability elicitation. Push toward evals that stay informative as models improve.

● Benchmark development. Build evaluation packages with subject-matter experts, each with expert-verified ground truth, multi-model headroom results, and rigorous QC (calibration layers, severity-weighted rubrics, deterministic verifiers).

● Experts. Recruit, calibrate, and review a pool across coding, agentic/tool-use, and STEM/reasoning. Be the final arbiter of correctness and frontier difficulty.

● Lab relationships. Be a technical point of contact for labs, with CEO support. Understand what they're trying to measure and translate it into an evaluation design.

● Delivery & dissemination. Turn lab requests into winning sample packages and own pilots end to end. Where the work generalizes, help turn it into public benchmarks and papers: we support publishing at venues like NeurIPS Datasets & Benchmarks, ICLR, and ACL.

What we're looking for

● Research background in ML evaluation or benchmarking (a track record of published or open benchmarks, eval/measurement research, or equivalent hands-on work that labs have relied on).

● Deep LLM/frontier-model benchmarking expertise, with real strength in code-model and agentic evaluation.

● Fluency with the measurement problem itself: construct validity, psychometrics, rubrics, pass rates, headroom, contamination, and what makes a task genuinely discriminate a model.

● Interest in the safety side of evaluation, capability elicitation, robustness, and measuring the things that are hardest to measure honestly.

● Proven ability to hold a team or expert pool to a rigorous standard.

● Comfort with the full research loop: framing the question, running the study, and writing it up.

● Fluent English; Spanish a plus.