test(examples): LLM-mocked tests for guides and python (#3601)#3979
Draft
Photon079 wants to merge 3 commits into
Draft
test(examples): LLM-mocked tests for guides and python (#3601)#3979Photon079 wants to merge 3 commits into
Photon079 wants to merge 3 commits into
Conversation
Part of topoteretes#3601. Adds a reusable LLM/embedding mocking harness that patches the single choke point LLMGateway.acreate_structured_output (Instructor + BAML) and uses cognee's built-in MOCK_EMBEDDING flag, plus 3 canary tests that run examples end-to-end with no API key or network. Signed-off-by: kr3shna <krishna2shende@gmail.com>
Signed-off-by: Anish <anish79u@gmail.com>
Contributor
|
Hello @Photon079, thank you for submitting a PR! We will respond as soon as possible. |
This was referenced Jul 8, 2026
kr3shna
added a commit
to kr3shna/cognee
that referenced
this pull request
Jul 9, 2026
…dback Part of topoteretes#3958. Addresses conversion feedback from topoteretes#3979: - build_mock_response: [] for list-typed fields with None defaults - Richer KnowledgeGraph defaults for memify/consolidate paths - Multimedia mocks return objects with .text / .choices - JSON-shaped str mock for agent/skill completion paths - import_example/invoke_example_main: neutralize sys.argv, sandbox CWD Signed-off-by: kr3shna <krishna2shende@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Part of #3601. Adds LLM-mocked tests for
examples/guides/(14 scripts) andexamples/python/(1 script), so every guide and python example runs end to end in CI with no API key, no network, and deterministic output.Built on the mocking harness from #3958. This branch is based on that PR, so until it merges the diff also shows its harness files (
cognee/tests/utils/example_mock_llm.py,cognee/tests/utils/example_runner.py,cognee/tests/examples/conftest.py). Please review onlycognee/tests/examples/{guides,python}/. I will rebase to a clean diff once #3958 lands.What's in this PR
cognee/tests/examples/guides/test_guides.py: one test per script inexamples/guides/(14).cognee/tests/examples/python/test_python.py:examples/python/(1 script;memory_provenance_demo,references_example, andschema_inventory_demomoved toexamples/demos/and are covered by the demos batch).Each test loads the real on-disk example via
import_example(...)and awaits its entrypoint under the sharedisolated_example_envfixture (mocked LLM + embeddings, per-test tmp_path isolation), asserting it runs to completion with no exception. No example logic is copied into the tests.How it works
LLMGateway.acreate_structured_outputand uses cognee's built-inMOCK_EMBEDDING=true, so every structured LLM call gets a deterministic response and the real tokenizer is preserved.isolated_example_envpoints cognee's data/system roots at a per-test tmp_path and prunes before and after, so tests do not share graph/vector/relational state.Verification
Keyless run (no
LLM_API_KEY, no network): 12 passed, 1 skipped, 2 xfailed.Acceptance Criteria (progress toward #3601)
examples/guides/andexamples/python/has a corresponding mocked test.examples/(follow-up folder PRs).Scope & honest notes
guides/s3_storagetalks to a reals3://bucket (requires_aws).guides/custom_graph_model: the mocked KnowledgeGraph leaves the example's optionalused_in: List[Field] = Nonefield as None, and cognee re-validates the extracted graph against the full model, which requires a list. Fix belongs inbuild_mock_response: populate list-typed fields with[].guides/consolidate_entity_descriptions: cognee's consolidate memify pipeline raisesKeyError('id')on the minimal mocked graph. Needs a richer default mock graph.Related: #3601 (issue), #3958 (harness).
Type of Change
Pre-submission Checklist
DCO Affirmation
I affirm that all code in every commit of this pull request conforms to the terms of the Topoteretes Developer Certificate of Origin.