Skip to content

test(examples): LLM-mocked tests for guides and python (#3601)#3979

Draft
Photon079 wants to merge 3 commits into
topoteretes:devfrom
Photon079:feat/mocked-examples-guides-python
Draft

test(examples): LLM-mocked tests for guides and python (#3601)#3979
Photon079 wants to merge 3 commits into
topoteretes:devfrom
Photon079:feat/mocked-examples-guides-python

Conversation

@Photon079

Copy link
Copy Markdown

Description

Part of #3601. Adds LLM-mocked tests for examples/guides/ (14 scripts) and examples/python/ (1 script), so every guide and python example runs end to end in CI with no API key, no network, and deterministic output.

Built on the mocking harness from #3958. This branch is based on that PR, so until it merges the diff also shows its harness files (cognee/tests/utils/example_mock_llm.py, cognee/tests/utils/example_runner.py, cognee/tests/examples/conftest.py). Please review only cognee/tests/examples/{guides,python}/. I will rebase to a clean diff once #3958 lands.

What's in this PR

  • cognee/tests/examples/guides/test_guides.py: one test per script in examples/guides/ (14).
  • cognee/tests/examples/python/test_python.py: examples/python/ (1 script; memory_provenance_demo, references_example, and schema_inventory_demo moved to examples/demos/ and are covered by the demos batch).

Each test loads the real on-disk example via import_example(...) and awaits its entrypoint under the shared isolated_example_env fixture (mocked LLM + embeddings, per-test tmp_path isolation), asserting it runs to completion with no exception. No example logic is copied into the tests.

How it works

  • The harness (Feat/mocked examples test suite #3958) patches the single choke point LLMGateway.acreate_structured_output and uses cognee's built-in MOCK_EMBEDDING=true, so every structured LLM call gets a deterministic response and the real tokenizer is preserved.
  • isolated_example_env points cognee's data/system roots at a per-test tmp_path and prunes before and after, so tests do not share graph/vector/relational state.
  • A failing example (import error, signature drift, runtime exception) fails its test loudly.

Verification

Keyless run (no LLM_API_KEY, no network): 12 passed, 1 skipped, 2 xfailed.

env -u LLM_API_KEY -u OPENAI_API_KEY LOG_LEVEL=CRITICAL \
  python -m pytest cognee/tests/examples/guides/ cognee/tests/examples/python/ -p no:logging -q
image

Acceptance Criteria (progress toward #3601)

  • Every script under examples/guides/ and examples/python/ has a corresponding mocked test.
  • The tests run with no real API key and are deterministic.
  • Tests fail loudly when an example breaks.
  • Every script under all of examples/ (follow-up folder PRs).
  • CI wiring (follow-up PR).

Scope & honest notes

  • 1 skipped: guides/s3_storage talks to a real s3:// bucket (requires_aws).
  • 2 xfailed, both harness follow-ups I will raise on Feat/mocked examples test suite #3958:
    • guides/custom_graph_model: the mocked KnowledgeGraph leaves the example's optional used_in: List[Field] = None field as None, and cognee re-validates the extracted graph against the full model, which requires a list. Fix belongs in build_mock_response: populate list-typed fields with [].
    • guides/consolidate_entity_descriptions: cognee's consolidate memify pipeline raises KeyError('id') on the minimal mocked graph. Needs a richer default mock graph.

Related: #3601 (issue), #3958 (harness).

Type of Change

  • Other: test coverage

Pre-submission Checklist

  • I have tested my changes thoroughly before submitting this PR
  • This PR contains minimal changes necessary to address the issue/feature
  • My code follows the project's coding standards and style guidelines
  • I have added tests that prove my feature works
  • I have added necessary documentation
  • All new and existing tests pass
  • I have searched existing PRs to ensure this change hasn't been submitted already
  • I have linked any relevant issues in the description
  • My commits have clear and descriptive messages

DCO Affirmation

I affirm that all code in every commit of this pull request conforms to the terms of the Topoteretes Developer Certificate of Origin.

kr3shna and others added 2 commits July 8, 2026 10:30
Part of topoteretes#3601. Adds a reusable LLM/embedding mocking harness that patches the
single choke point LLMGateway.acreate_structured_output (Instructor + BAML) and
uses cognee's built-in MOCK_EMBEDDING flag, plus 3 canary tests that run
examples end-to-end with no API key or network.

Signed-off-by: kr3shna <krishna2shende@gmail.com>
Signed-off-by: Anish <anish79u@gmail.com>
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Hello @Photon079, thank you for submitting a PR! We will respond as soon as possible.

kr3shna added a commit to kr3shna/cognee that referenced this pull request Jul 9, 2026
…dback

Part of topoteretes#3958. Addresses conversion feedback from topoteretes#3979:
- build_mock_response: [] for list-typed fields with None defaults
- Richer KnowledgeGraph defaults for memify/consolidate paths
- Multimedia mocks return objects with .text / .choices
- JSON-shaped str mock for agent/skill completion paths
- import_example/invoke_example_main: neutralize sys.argv, sandbox CWD

Signed-off-by: kr3shna <krishna2shende@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants