Skip to content

tools/outcrawl: add Outcrawl search and scrape tools - #1568

Open
heydryft wants to merge 1 commit into
tmc:mainfrom
heydryft:feat/outcrawl-tools
Open

heydryft wants to merge 1 commit into
tmc:mainfrom
heydryft:feat/outcrawl-tools

Conversation

@heydryft

@heydryft heydryft commented Nov 9, 2026

Copy link
Copy Markdown

Thanks for langchaingo: the tools.Tool interface makes it easy to drop a new search backend into an agent with a few lines.

Summary

Adds tools/outcrawl, two tools for the agent executor next to serpapi, duckduckgo and metaphor: a web search tool and a scrape tool. Outcrawl is web infrastructure for AI agents. Its search API (POST https://api.outcrawl.ai/v1/search) returns ranked hits with title, URL and snippet, and its scrape API (POST https://api.outcrawl.ai/v1/scrape) returns the main content of a page as markdown, including pages that render with JavaScript. The free plan includes 1,000 credits a month. Nothing changes for existing users.

Changes

  • tools/outcrawl/outcrawl.go: SearchTool (NewSearch, name OutcrawlSearch) and ScrapeTool (NewScrape, name OutcrawlScrape), both implementing tools.Tool with the same CallbacksHandler handling as the serpapi tool. The API key comes from OUTCRAWL_API_KEY or WithAPIKey; without one New* returns ErrMissingToken. Requests use httputil.DefaultClient unless WithHTTPClient is set. Search formats hits as a numbered list (title, URL, snippet); scrape sends {url, formats: ["markdown"], onlyMainContent} and returns the markdown. API errors surface the status, code and message; empty scrape content returns ErrNoResult.
  • tools/outcrawl/options.go: WithAPIKey, WithHTTPClient, WithBaseURL, WithLimit (search hits), WithOnlyMainContent (scrape, default true).
  • tools/outcrawl/doc.go: package docs with an agent example.
  • tools/outcrawl/outcrawl_test.go: tests against an httptest server (request path, auth header and payload, result formatting, optional limit, API error, empty markdown, non-JSON error, missing key, key from env). No live key and no recordings needed.
  • docs/parity_matrix.md: one row under Tools.

Testing

  • go test -count=1 -v ./tools/outcrawl: 9 tests passed.
  • go test -count=1 ./tools/...: all packages pass.
  • go vet ./tools/outcrawl and golangci-lint run ./tools/outcrawl/...: 0 issues.
  • Live call with a real key through the new code: NewSearch(WithLimit(2)).Call(ctx, "langchaingo agents tools") returned two formatted hits, and NewScrape().Call(ctx, "https://example.com") returned 1,342 characters of markdown.

Disclosure: I work on Outcrawl.

This PR was prepared with AI assistance (omp with Claude): the code, tests, docs line and this description were written with it, and the tests, lint and the live call above were run as part of preparing it.

PR Checklist

  • Read the Contributing documentation.
  • Read the Code of conduct documentation.
  • Name your Pull Request title clearly, concisely, and prefixed with the name of the primarily affected package you changed according to Good commit messages.
  • Check that there isn't already a PR that solves the problem the same way to avoid creating a duplicate.
  • Provide a description in this PR that addresses what the PR is solving.
  • Describes the source of new concepts: the layout follows tools/serpapi.
  • References existing implementations as appropriate.
  • Contains test coverage for new functions.
  • Passes all golangci-lint checks.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant