Repository navigation
Mapping Response (messages/agent/context_variables) to a portable EvalPort ResultSet #107
Description
Quick, low-priority note — not expecting active maintenance here, but figured I'd leave it since the connection point is pretty clean.
EvalPort (https://github.com/adhabnr-ux/evalport) is an open JSON-Schema spec for portable LLM eval documents (test suites, test cases, result sets), so eval data isn't locked to one framework's format. Full disclosure: I've also been working with the OpenAI Agents SDK team on native to_openeval()/from_openeval() support in openai/openai-python (PR #3619, still open/in review, not merged) — this is a separate, much smaller idea specific to this repo's own Response type, not a rehash of that PR.
Looking at swarm/types.py and swarm/core.py: Swarm.run(agent, messages, ...) returns a Response(messages=history[init_len:], agent=active_agent, context_variables=context_variables). response.agent after a run is genuinely useful signal — it's literally which agent ended up handling the conversation after any handoffs, which is exactly what you'd want to regression-test for a triage/routing setup (e.g. did the "Weather Agent" handoff still land on the right specialist agent for a given input).
A small converter turning a Response into an EvalPort Result would make that testable as structured data instead of eyeballing REPL output:
# swarm/types.py: Response(messages, agent, context_variables)
def swarm_response_to_result(response, test_case_id: str, expected_agent: str | None = None) -> dict:
final_text = next(
(m["content"] for m in reversed(response.messages) if m.get("role") == "assistant" and m.get("content")),
"",
)
handoff_ok = expected_agent is None or (response.agent and response.agent.name == expected_agent)
return {
"test_case_id": test_case_id,
"actual_output": final_text,
"grader_results": [{
"grader_id": "handoff_target",
"type": "custom",
"score": 1.0 if handoff_ok else 0.0,
"passed": handoff_ok,
"reason": f"ended on agent={response.agent.name if response.agent else None}, expected={expected_agent}",
}],
"passed": handoff_ok,
"metadata": {"context_variables": response.context_variables},
}Running a small suite of (messages, expected_agent) cases through Swarm.run() and collecting these into an EvalPort ResultSet (spec/schemas/resultset.json) would let handoff behavior be compared across Agent.instructions/functions changes, or even across model swaps — as plain diffable JSON rather than reading Response.messages by hand.
Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md
Schema: https://github.com/adhabnr-ux/evalport/blob/main/spec/schemas/resultset.json
No pressure either way — this is genuinely a "leave it here in case it's ever useful" note, not expecting a response given this repo's educational/maintenance status. Thanks for open-sourcing swarm; it's been a clean reference for a lot of what came after it.
Activity
Parker-Fawcett commentedon Aug 29, 2026
Interesting approach - portable eval formats are important. One thing worth considering: the spec that defines what "correct" looks like should be locked before evaluation runs, not generated alongside it. Otherwise the eval inherits whatever assumptions the generator made. I explored this in rebuild-dossier where the spec is mechanically enforced before tests execute. Paper: https://arxiv.org/abs/2608.23616
adhabnr-ux commentedon Aug 31, 2026
Good point, and it's worth being precise about what the TestCase/Grader/Result split actually buys here, versus what it doesn't.
Structurally: a
Graderis a standalone object, referenced by ID from aTestCase, and aResultSetis validated against the suite those IDs live in — everytest_case_idin a ResultSet must correspond to a test case in the source suite, and every grader ID a TestCase references must exist in the suite'sgradersarray (or be inline on the TestCase itself). So aResultnever carries an ad-hoc, self-serving definition of "correct" alongside its own score — grading criteria live in the input document, not the output one. That handles one version of your concern: whoever produces the Result can't quietly redefine what correctness means for that row while producing it.But that's not the same guarantee you're describing, and I don't want to oversell it. Nothing in the format today enforces that the Grader was authored or finalized before the run, chronologically. If the same process (or the same model call) writes the TestCase, the Grader, and then executes both, schema validation passes cleanly even though the grading spec inherited exactly the generator's assumptions — which is the failure mode you're flagging. The closest thing EvalPort has is optional Sigstore signing over a suite's published bytes, which proves a Grader wasn't altered after signing — that's tamper-evidence, not a guarantee about when it was written relative to the run, and it's opt-in besides. So "mechanically enforced before tests execute," as you put it, is a stronger property than anything the spec currently gives you. EvalPort can carry the artifact of a locked spec; it doesn't itself lock one.
That's a real gap rather than a solved problem, and if you wanted to write it up as a proposal (grader provenance / lock-before-run attestation, something a runner could check rather than just a schema shape), it'd be a legitimate spec question — a few of these design gaps have shipped as concrete changes from outside proposals rather than internal ones. Given swarm's own maintenance status though, I'll leave that as an offer rather than push anything against this repo specifically — thanks for engaging with it seriously.