AI Evaluation & QA Engineer
- Full-time
- Mid level
- Remote (worldwide)
Remote AI Evaluation & QA Engineer at MacroCoderz: test LLM systems, build evaluation harnesses and catch hallucinations and regressions before release.
Role at a glance
| Employment type | Full-time |
|---|---|
| Level | Mid level |
| Experience | 2+ years |
| Location | Remote (worldwide) |
| Team | AI engineering |
| Education | Degree in a technical field or equivalent QA experience |
| Posted | 10 October 2026 |
About the role
Testing an AI system is different from testing traditional software: the same input can produce different outputs, and a wrong answer can look perfectly confident. As an AI Evaluation & QA Engineer you build the harnesses that make quality measurable.
You define what a good answer looks like, build test sets with domain experts, automate scoring with rules and LLM judges, and block releases that make things worse.
What you will do
- Design evaluation strategies for RAG systems, agents and LLM features.
- Build and maintain automated evaluation harnesses in CI.
- Create test datasets, including adversarial and edge-case inputs.
- Calibrate LLM-as-judge scoring against human review.
- Track quality across releases and report regressions with clear evidence.
What we are looking for
- 2+ years in software QA, test automation or data quality.
- Comfortable writing test code in Python.
- Understands why non-deterministic systems need statistical, not single-run, testing.
- Has worked with or studied LLM evaluation - hallucination, groundedness, relevance.
Nice to have
- Experience with Ragas, DeepEval, promptfoo or a similar evaluation tool.
- Red-teaming or AI safety testing.
- Statistics background.
Skills and tools
- Python
- Test automation
- LLM evaluation
- LLM-as-judge
- CI/CD
- Red-teaming
Who this role suits
QA and test automation engineers who want to move into AI - the testing mindset transfers directly.
The work behind this role
AI Evaluation & QA Engineer: common questions
Yes. The AI Evaluation & QA Engineer role at MacroCoderz is fully remote and open to applicants worldwide. It is a full-time, mid level position.
2+ years (mid level). Education: Degree in a technical field or equivalent QA experience. Who it suits: QA and test automation engineers who want to move into AI - the testing mindset transfers directly.
Yes - it is one of the most direct routes into AI. Test design, automation and a habit of looking for failure all transfer. The new parts are statistical testing of non-deterministic outputs and scoring answers with rules and LLM judges, which we teach on the job.
Use the application form on this page. It asks for your contact details, city, LinkedIn profile, expected salary, how you heard about the role, a short description of two or three projects you have worked on, and your CV as a PDF or Word file. An engineer reviews every application and emails shortlisted applicants to arrange a call.