✦NeurIPS 2026 · Expo Panel

Agents in Production: Evals, Harnesses, and the Reinvention of Recommendation

Agents are live. Most fail quietly. How do we make them reliable, measurable, and efficient at scale?

Sun, Dec 6, 2026 6:00 – 7:00 PM Sydney time Parkside 2 · ICC Sydney

That's Saturday, December 5, 11 PM – midnight Pacific · Sunday 2 – 3 AM Eastern · Sunday 7 – 8 AM London.

A robot agent ranking user profiles and passing the best match forward
Why now

Three problems, one proving ground

Agents fail quietly

Errors compound across steps, cost grows per tool call, and offline benchmarks rarely predict production.

Evaluate the trajectory

Not single outputs. Harnesses and simulators that reflect production; SFT and RL on traces; cost-aware routing.

RecSys as the test case

When systems understand goals and take actions, recommendation shifts from predicting clicks to achieving objectives.

?

Will agents replace the retrieve → rank → recommend stack, or rebuild it?

People

Panelists Tentative

Discussion arc

Three segments. Each opens with a live poll.

1
12 min

Agents in the Wild

  • What blocks deployment: reliability, compounding errors, tool misuse, cost, or safety?
  • What is solved in research but unsolved in production?
Submit a question for this segment
2
18 min

Evals, Harnesses & Optimization

  • How do we evaluate trajectories when success isn't a single label?
  • Can simulated users close the offline–online gap?
  • What moves the needle: base models, SFT/RL on traces, or scaffolding?
Submit a question for this segment
3
20 min

RecSys as the Proving Ground

  • Where do agents sit in the retrieval and ranking stack?
  • Can they optimize long-term goals and stay measurable?
  • From recommending items to executing actions?
Poll: "Within 3 years, ranking models will be a tool agents call, not the product."
Submit a question for this segment

Final 10 minutes: your questions, top-voted on Slido.

Q&A

Ask a question. Upvote the best.

Open Slido ↗Event code #2469026. The Q&A opens here shortly before the panel.