Staff Engineer, AI Evals

The Opportunity

At Sema4.ai, we’re building an Enterprise AI Agent platform that fundamentally changes how knowledge work gets done—by enabling people and AI agents to collaborate in durable, trustworthy ways.

As a Staff Engineer, AI Evals, you’ll design and own the evaluation systems that determine whether our agents are actually good: correct, reliable, efficient, and improving over time. You’ll build the measurement backbone that guides model choice, agent design, product decisions, and customer trust.

This is an early, high‑impact role. You’ll be defining how we measure success for AI agents in production, where ambiguity is real, and ground truth can be messy. We’re looking for an engineer who brings rigor, judgment, and strong opinions about what “good” looks like, and who know how to operationalize it.

Who You Are

AI Systems & Evaluation Expert

You understand that AI systems are only as good as the way they’re measured. You’ve worked with LLMs and agentic systems in production and have seen how offline benchmarks, synthetic data, and human judgment can all fail in different ways. You know how to design evaluations that are meaningful, repeatable, and decision‑useful, not just theoretically impressive.

In‑Depth Technologist

You stay close to research and industry practice in evaluation, alignment, and reliability. You understand where automated metrics work, where they break down, and how to combine them with human review, golden datasets, and production signals. You bring creativity to building evaluation sets and scenarios, and in sourcing (or synthesizing) the data you need.

What You’ll Do

Build and Own the Evaluation Platform

Design, build, and operate Sema4.ai’s core evaluation infrastructure for LLMs and agents: offline benchmarks, regression tests, task‑level metrics, and production feedback loops. These systems will directly inform product launches, model upgrades, and customer requirements.

Define “Good” for Agents in Production

Work closely with agent, product, and field engineering teams to translate fuzzy goals around correctness, reliability, usefulness into concrete, measurable signals. You’ll help define success criteria for new capabilities and ensure we can detect regressions before customers do.

Tackle Ambiguous, High‑Leverage Problems

• How to evaluate long‑running, multi‑step agents
• How to balance automated scoring with human judgment
• How to measure improvement when tasks evolve
• How to compare models under cost and latency constraints

Influence Technical and Product Direction

Use evaluation results to guide architectural decisions, model selection, and roadmap tradeoffs. You’ll participate in design reviews, set technical standards for eval rigor, mentor other engineers, and help interview senior technical candidates.

What You Bring

• 7+ years of software engineering experience, including 2+ years building AI/ML systems in production
• Deep experience with backend systems in Python, including data pipelines, observability, and reliability
• Hands‑on experience evaluating LLM‑based systems (agents, retrieval, tool use, workflows, etc.)
• Strong intuition for metrics, experimentation, and failure analysis in non‑deterministic systems
• Strong communication skills: whether you’re talking to colleagues, customers, or machines, you communicate clearly, concisely, and collaboratively
• A high‑ownership mindset: you care deeply about the integrity of the systems you build and the decisions they inform