AI Evaluation Engineer
At Raydar · United States
Salary: US$200,000–300,000 a year
- Engineering
- Full-time
See how well you match Match me reads your CV and picks the jobs that suit you best.
Where you can work from
Open to Canada and the United States
“United States”
From the location.
“Remote (US or Canada)”
From the job description.
Report a mistake
- Posted
- 15 October 2026
- First seen by us
- 16 October 2026
- Last checked
- 5 hours ago
Job description
From Raydar’s job post, as they wrote it.
About the company
Our client is a software company.
The role
Raydar is recruiting for this role on behalf of our client. Lead the creation of a measurement approach for an AI system that carries out technical design work. Draw on both hands-on engineering judgment and evaluation know-how, and collaborate with domain experts and other teams to convert findings into clear improvement priorities.
What you'll do
Read the full description
- Establish a new assessment framework that defines quality for design work where several answers can be valid.
- Collaborate with domain experts, external users and internal teams to see how seasoned engineers operate and where the system struggles.
- Reproduce realistic design tasks and typical failure cases in repeatable test setups that developers can run again.
- Create and tune automated graders alongside human review processes, and confirm that the graders agree with expert opinion.
- Convert assessment results into a ranked list of improvements to the system's reasoning.
Requirements
What we're looking for
- About five or more years of professional experience in hardware engineering, AI, or a blend of both, with depth in one and practical familiarity with the other.
- Substantial hands-on background assessing AI systems, including authoring test tasks, building automated graders and benchmarking them against human review.
- Solid grasp of how autonomous, agent-style AI systems behave.
- Grounding in a hardware engineering field, enough to tell sound design from weak design.
- Comfort working directly with external users, specialists, partners and technical colleagues across functions.
- Self-assurance to guide work among many stakeholders in a role with high ownership.
- Based in the US or Canada, and able to travel to an office for one week every 6 weeks.
Bonus points
- Prior time at a robotics or physical AI company.
- Experience building things from zero in an early-stage startup.
Benefits
Compensation and benefits
- Base salary: USD 200,000 to 300,000 per year
- Equity
Location and work model
- Remote (US or Canada)
- Full-time
Get jobs like this by email
Once a week, we email the new remote jobs open where you live, each with the line that proves it. Every email has a one-click unsubscribe.
