Skip to content

Test integrity: tests that pass without testing #8

Description

@Ugbot

The suite is large and mostly green (770 tests pass in the main module), but three mechanisms let it report green on untested code.

  1. Fixture-missing skips hide the flagship parity proof. The cross-runtime parity tests resolve examples/pipelines/banking.yaml via a CWD-relative path guarded by assumeTrue. Under the documented build invocation the path misses and they silently skip — measured: 3/3 and 5/6 skipped. Re-run with the correct working directory, all 21 parity/replay/CEP tests pass — so the parity claim is real but unverified by default. The same fail-open pattern exists in the Pekko and Clojure parity tests.
  2. H2 stands in for Postgres, which is precisely why the invalid-SQL bug in Correctness bugs found by audit #5 shipped. Needs Testcontainers.
  3. *IT.java files are unrunnable in every profile — no failsafe plugin, and surefires filename filter excludes them before the tag filter applies. Net effect: all Redis/Fluss integration tests have zero executed coverage, and no IT exists at all for Postgres, pgvector, Qdrant or Milvus (~2,300 LOC of untested vector-store drivers).

Also worth a sweep: a set of tests assert nothing meaningful — one re-declares the parser regexes locally instead of importing the class under test, one constructs a function and never invokes it, and the config test never sets an env var despite the class existing to resolve them.


Roadmap epic. Detailed findings, file:line citations and sub-tasks live in the AI-grind tracker (project AGS), which is the source of truth for this work. Local dev — build, test, tracking, profiling — is managed with AI-grind.


Tracked as AGS-8 in the devtools-mcp tracker.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions