Repository navigation

Test and tune the planner's web search and fetch use - #712

Open

IZO-Ong wants to merge 17 commits into

OpenFn:mainfrom

IZO-Ong:search-test

Commits on Oct 8, 2026
- test: add web tool trace builder for planner behaviour tests
  IZO-Ong
  committed
  0b6c268
- test: add web tools experiment metrics and winner selection
  IZO-Ong
  committed
  37b6adb
- test: add recording planner and variants for web tools experiment
  IZO-Ong
  committed
  5e8ba66
- test: add calibration to estimate cost and token size
  IZO-Ong
  committed
  03b1a24
- test: pair code execution results in the web tool trace
  IZO-Ong
  committed
  a52caf8
- test: add web tools behaviour scenarios
  IZO-Ong
  committed
  4b1cf04
- test: add staged web tools experiment runner
  IZO-Ong
  committed
  b312eff
- test: add strict web tools behaviour suite for the planner
  IZO-Ong
  committed
  6924306
- test: report how often the truncated source page was fetched
  IZO-Ong
  committed
  07a9dab
- test: add judged acceptance specs for web tool answers
  IZO-Ong
  committed
  9181f11
- test: let variants remove prompt text, add 1d experiment variant
  IZO-Ong
  committed
  c8c6472
- feat: tell planner to search before fetching
  IZO-Ong
  committed
  3be01a6
- test: drop 1a from the fingerprint test
  IZO-Ong
  committed
  daa0911
- test: allow the first remembered URL refusal in the strict test suite
  IZO-Ong
  committed
  e405392
- test: pair every server tool result in the web tool trace
  IZO-Ong
  committed
  d622d59
- test: stop failed variants winning and fold curly apostrophes in web tool metrics
  IZO-Ong
  committed
  45fc6ed
- test: key the web tools run cache by prompt and scenario, retry failed runs
  IZO-Ong
  committed
  b0b8f8f

Navigation Menu

Repository navigation

Test and tune the planner's web search and fetch use - #712

Test and tune the planner's web search and fetch use#712
IZO-Ong wants to merge 17 commits into
OpenFn:mainfrom
IZO-Ong:search-test

Commits on Oct 8, 2026

test: add web tool trace builder for planner behaviour tests

test: add web tools experiment metrics and winner selection

test: add recording planner and variants for web tools experiment

test: add calibration to estimate cost and token size

test: pair code execution results in the web tool trace

test: add web tools behaviour scenarios

test: add staged web tools experiment runner

test: add strict web tools behaviour suite for the planner

test: report how often the truncated source page was fetched

test: add judged acceptance specs for web tool answers

test: let variants remove prompt text, add 1d experiment variant

feat: tell planner to search before fetching

test: drop 1a from the fingerprint test

test: allow the first remembered URL refusal in the strict test suite

test: pair every server tool result in the web tool trace

test: stop failed variants winning and fold curly apostrophes in web tool metrics

test: key the web tools run cache by prompt and scenario, retry failed runs