Test and tune the planner's web search and fetch use

.changeset/sharp-planners-search.md

@@ -0,0 +1,5 @@
+---
+"apollo": patch
+---
+
+global_chat: planner searches before fetching, because web fetch only opens URLs already in the conversation

services/global_chat/prompts.yaml

@@ -200,3 +200,4 @@ prompts:
     - You can only reach these domains: {domains}. Nothing else is fetchable.
     - For FHIR, use the published R4 spec under `https://hl7.org/fhir/R4/` (e.g. `https://hl7.org/fhir/R4/patient.html`). If you used a different FHIR version, say which one.
     - Resource pages are long and are truncated when fetched, so a field missing from what you read is not proof it does not exist. Say what you confirmed and what you could not.
+    - `web_fetch` can only open a URL that already appears in this conversation: in the user's message, or in an earlier search or fetch result. URLs you remember, and URLs in these instructions, are refused. Search first, then fetch a URL from the results.

services/global_chat/tests/acceptance/web_tools/test_fhir_deep_answer.md

@@ -0,0 +1,35 @@
+---
+id: global-chat.web-tools.fhir-deep-answer
+service: global_chat
+judges: [general]
+---
+
+# notes
+
+The codes sit past the point where a fetch of the Patient page truncates at the shipped max_content_tokens (10k), though they also appear on the short FHIR value-set page for link types. A good answer either states the codes correctly or says plainly that it could not confirm them from the page it read. Inventing plausible-sounding codes or meanings is the failure this spec exists to catch.
+
+# quality_criteria
+
+- Lists the four Patient.link.type codes (replaced-by, replaces, refer, seealso) with a correct one-line meaning for each, or, if it could not confirm them, says so explicitly rather than guessing.
+- Names the source in prose.
+- If it says anything was not confirmed, it is specific about what.
+
+# settings
+
+## options.web_search
+
+true
+
+## meta.session_id
+
+sess-web-tools-fhir-deep-0001
+
+# turn
+
+## role
+
+user
+
+## content
+
+In FHIR R4, what codes can Patient.link.type take, and what does each mean?

services/global_chat/tests/acceptance/web_tools/test_fhir_shallow_answer.md

@@ -0,0 +1,36 @@
+---
+id: global-chat.web-tools.fhir-shallow-answer
+service: global_chat
+judges: [general]
+---
+
+# notes
+
+The user asks a factual question about the FHIR R4 Patient resource with web search enabled. hl7.org is on the planner's allowlist, so the planner should look it up in the published R4 spec and answer from it. The point of this spec is how the answer uses fetched content.
+
+# quality_criteria
+
+- States that each Patient.contact must contain at least a contact's details (a name, telecom or address) or a reference to an organization.
+- Mentions that this is a formal constraint in the spec (the pat-1 invariant), or otherwise makes clear it is a rule rather than advice.
+- Names the source in prose (the FHIR R4 specification or the hl7.org Patient page).
+- Does not pad the answer with unrelated Patient fields.
+
+# settings
+
+## options.web_search
+
+true
+
+## meta.session_id
+
+sess-web-tools-fhir-shallow-0001
+
+# turn
+
+## role
+
+user
+
+## content
+
+In FHIR R4, what rule applies to each entry in Patient.contact? What must it contain at minimum?

services/global_chat/tests/acceptance/web_tools/test_multi_turn_follow_up.md

@@ -0,0 +1,78 @@
+---
+id: global-chat.web-tools.multi-turn-follow-up
+service: global_chat
+judges: [general]
+---
+
+# notes
+
+Third turn of a conversation about FHIR R4 Patient contact fields. The earlier assistant answers already established the relevant fields (telecom as ContactPoint 0..*, address 0..*, contact 0..*). The follow-up should build on them rather than restarting, and the mapping should use Patient.telecom with system "phone".
+
+# quality_criteria
+
+- Builds on the earlier answers rather than re-explaining the Patient resource from scratch.
+- Maps the phone number into Patient.telecom as a ContactPoint with system "phone" and the number in value.
+- Mentions that telecom is a list (0..*), so the number goes in as an array entry.
+- Does not generate a workflow or job code attachment. Describing the mapping in text with an inline snippet is fine.
+
+# settings
+
+## options.web_search
+
+true
+
+## meta.session_id
+
+sess-web-tools-multi-turn-0001
+
+# history
+
+## turn
+
+### role
+
+user
+
+### content
+
+Which fields does a FHIR R4 Patient have for contact information?
+
+## turn
+
+### role
+
+assistant
+
+### content
+
+From the FHIR R4 Patient page (hl7.org/fhir/R4/patient.html): `telecom` (ContactPoint, 0..*) holds phone numbers, emails and similar; `address` (Address, 0..*) holds postal addresses; and `contact` (BackboneElement, 0..*) holds contact parties such as next of kin, each with their own name, telecom and address.
+
+## turn
+
+### role
+
+user
+
+### content
+
+Which of those fields can repeat?
+
+## turn
+
+### role
+
+assistant
+
+### content
+
+All three repeat: `telecom`, `address` and `contact` are each 0..* on Patient, so a patient can have several phone numbers, several addresses and several contact parties.
+
+# turn
+
+## role
+
+user
+
+## content
+
+How would I map a CommCare phone number into it?

services/global_chat/tests/acceptance/web_tools/test_off_allowlist_answer.md

@@ -0,0 +1,36 @@
+---
+id: global-chat.web-tools.off-allowlist-answer
+service: global_chat
+judges: [general]
+---
+
+# notes
+
+DHIS2's documentation is not on the planner's web allowlist (only hl7.org and docs.openfn.org are). The planner may use OpenFn's own docs for the DHIS2 adaptor, but it cannot read docs.dhis2.org. A good answer gives what it can, marks what it could not check against DHIS2's own reference, and points the user there. Confidently listing exact field names as if verified is the failure.
+
+# quality_criteria
+
+- Gives a useful outline of what an event payload contains (e.g. program, orgUnit, occurredAt / eventDate, dataValues), however it knows it.
+- Says it could not verify the exact field list against DHIS2's own API reference, or otherwise makes clear which parts are unverified.
+- Suggests where to confirm (the DHIS2 developer documentation for the tracker API).
+- Does not claim to have read a DHIS2 documentation page.
+
+# settings
+
+## options.web_search
+
+true
+
+## meta.session_id
+
+sess-web-tools-off-allowlist-0001
+
+# turn
+
+## role
+
+user
+
+## content
+
+What fields does the DHIS2 tracker API (/api/tracker) accept when creating an event?

services/global_chat/tests/integration/test_web_tools_pass_fail.py

@@ -0,0 +1,81 @@
+"""Live checks on how the planner uses web_search and web_fetch.
+
+These hit the live Anthropic API. Each test plays its scenario
+once with the shipped prompt and config and asserts on the recorded
+tool-call trace. A failure prints the full trace, no retries.
+"""
+
+import os
+
+import pytest
+from dotenv import load_dotenv
+
+load_dotenv()
+
+from global_chat.tests.web_tools.metrics import TurnRecord, is_grounded  # noqa: E402
+from global_chat.tests.web_tools.recording import run_scenario  # noqa: E402
+from global_chat.tests.web_tools.scenarios import SCENARIOS  # noqa: E402
+from global_chat.tests.web_tools.trace import count, format_trace  # noqa: E402
+from global_chat.tests.web_tools.variants import resolve_variant  # noqa: E402
+
+pytestmark = [
+    pytest.mark.integration,
+    pytest.mark.skipif(bool(os.getenv("ANTHROPIC_BASE_URL")), reason="web tools need a direct api.anthropic.com key"),
+    pytest.mark.skipif(not os.getenv("ANTHROPIC_API_KEY"), reason="needs ANTHROPIC_API_KEY"),
+]
+
+
+@pytest.mark.parametrize("scenario_id", ["control_openfn_concept", "control_code_edit"])
+def test_controls_make_no_web_calls(scenario_id: str) -> None:
+    turns = play(scenario_id)
+    calls = all_calls(turns)
+
+    assert count(calls, tool="search") + count(calls, tool="fetch") == 0, explain(turns)
+
+
+MAX_FIRST_TURN_REFUSALS = 2
+
+
+@pytest.mark.parametrize("scenario_id", ["fhir_shallow", "fhir_deep"])
+def test_fhir_answers_come_from_a_fetched_page(scenario_id: str) -> None:
+    turns = play(scenario_id)
+    calls = all_calls(turns)
+
+    assert count(calls, result="url_not_in_prior_context") <= MAX_FIRST_TURN_REFUSALS, explain(turns)
+    assert count(calls, tool="fetch", result="ok") >= 1, explain(turns)
+    assert is_grounded(turns[-1].answer, calls, SCENARIOS[scenario_id].facts), explain(turns)
+
+
+def test_an_off_allowlist_question_does_not_try_banned_urls() -> None:
+    turns = play("off_allowlist")
+    calls = all_calls(turns)
+
+    assert count(calls, result="url_not_allowed") == 0, explain(turns)
+    assert count(calls, result="url_not_in_prior_context") == 0, explain(turns)
+
+
+def test_follow_up_turns_do_not_fetch_the_same_content_again() -> None:
+    turns = play("multi_turn")
+
+    assert sum(count(turn.trace, tool="fetch") for turn in turns[1:]) <= 1, explain(turns)
+    assert count(turns[0].trace, result="url_not_in_prior_context") <= MAX_FIRST_TURN_REFUSALS, explain(turns)
+    assert sum(count(turn.trace, result="url_not_in_prior_context") for turn in turns[1:]) == 0, explain(turns)
+
+
+def play(scenario_id: str) -> list[TurnRecord]:
+    turns = run_scenario(SCENARIOS[scenario_id], resolve_variant("base"))
+    for turn in turns:
+        assert turn.error is None, f"turn failed: {turn.error}"
+        assert not turn.downgraded, "web tools were downgraded, so this key cannot use web search"
+    return turns
+
+
+def all_calls(turns: list[TurnRecord]) -> list[dict]:
+    return [call for turn in turns for call in turn.trace]
+
+
+def explain(turns: list[TurnRecord]) -> str:
+    return "\n\n".join(
+        f"turn {number}:\n{format_trace(turn.trace)}\n\nanswer:\n{turn.answer[:1500]}"
+        for number, turn in enumerate(turns, start=1)
+    )

services/global_chat/tests/unit/test_planner.py

@@ -19,6 +19,7 @@
     PlannerAgent,
     PlannerResult,
 )
+from global_chat.tests.web_tools.variants import SEARCH_FIRST
 from global_chat.tools.tool_definitions import TOOL_DEFINITIONS, build_web_tools
 from streaming_util import STATUS_SEARCHING_WEB
 
@@ -1149,6 +1150,14 @@ def test_the_web_tools_prompt_key_exists_in_prompts_yaml() -> None:
     assert "{domains}" in prompts["planner_web_tools_prompt"]
 
 
+def test_the_web_tools_prompt_ships_the_measured_search_first_line() -> None:
+    """The line the web-tools experiment measured, verbatim."""
+    path = Path(planner_module.__file__).parent / "prompts.yaml"
+    prompts = yaml.safe_load(path.read_text(encoding="utf-8"))["prompts"]
+
+    assert SEARCH_FIRST in prompts["planner_web_tools_prompt"]
+
+
 def test_no_web_block_is_appended_when_the_prompt_is_missing() -> None:
     """An empty text block would be rejected by the API, so drop it."""
     planner = make_run_planner()