Hiring | AI Engineer – Agent Evaluation Specialist | C2C/W2 | 14+ Years Experience | Remote
At TMS, LLC · Los Angeles, CA, United States
- Data and ML
- Contract
Early applicant Posted 23 hours ago. Radar emails jobs like this within hours of us finding them. The free weekly email can take up to a week.
See how well you match Match me reads your CV and picks the jobs that suit you best.
Get jobs like this within hours Radar emails you new jobs open to you within hours of us finding them, before most people see them.
Where you can work from
Open to the United States
“Los Angeles, CA, United States”
From the location.
“Location: Los Angeles, CA (Hybrid)”
From the job description.
Report a mistake
- Posted
- 23 hours ago
- First seen by us
- 6 November 2026
- Last checked
- 17 hours ago
Job description
From TMS, LLC’s job post, as they wrote it.
Job Description
Role: AI Engineer – Agent Evaluation Specialist
Location: Los Angeles, CA (Hybrid)
Duration: 3+ Months (Possible Extension)
Primary Focus: Hands-on Agent Evaluation
Specialisation: Agentic AI, AWS Bedrock AgentCore, Claude Agent SDK, Langfuse, AI Observability & Evaluation
We are looking for an AI Engineer with strong, practical experience in evaluating LLM-powered agents, rather than someone focused primarily on building or developing agents.
Read the full description
Key Requirements (Must-Have):
* LLM-as-a-Judge: Hands-on experience designing, implementing, and running LLM-based evaluation frameworks to assess agent response quality, accuracy, relevance, and reliability.
* Golden Dataset Development: Experience collaborating with Business Analysts and stakeholders to create, validate, and maintain golden datasets that serve as benchmarks for agent evaluation.
* Langfuse: Practical experience instrumenting AI agents using Langfuse, including tracing agent execution, capturing evaluation scores, monitoring performance, and troubleshooting failures.
* Agent Evaluation: Strong understanding of end-to-end agent evaluation, including testing agent behavior, tool usage, reasoning workflows, and response quality.
Critical Screening Criteria:
Candidates must demonstrate actual implementation experience with agent evaluation frameworks, not just theoretical knowledge or general GenAI/LLM development.
Agent evaluation is the core requirement and will be the primary selection criterion.
Additional Information
All your information will be kept confidential according to EEO guidelines.
Hiring | AI Engineer – Agent Evaluation Specialist | C2C/W2 | 14+ Years Experience | RemoteTMS, LLC
Apply on TMS, LLC (opens in a new tab)Get jobs like this by email
Once a week, we email the new remote jobs open where you live, each with the line that proves it. Every email has a one-click unsubscribe.
