Skip to content

fix(testing): support list_append, if_not_exists, and count in memory backend - #571

Open
vicolby wants to merge 3 commits into
ferrumio:mainfrom
vicolby:fm/pydynox-memory-backend-gaps
Open

vicolby wants to merge 3 commits into
ferrumio:mainfrom
vicolby:fm/pydynox-memory-backend-gaps

Conversation

@vicolby

@vicolby vicolby commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Intent

Fix two gaps in the pydynox in-memory testing backend, found while fixing the docs examples for #325: (1) python/pydynox/testing/memory.py _apply_update_expression treats SET x = list_append(...) as a silent no-op, so docs/examples/atomic/testing_atomic.py::test_atomic_append fails; (2) MemoryClient has no count(), so docs/examples/scan/testing_scan.py::test_count_items fails with AttributeError.

Follow-up on PR #571, as a separate commit: SonarCloud reports one new issue, rule python:S5778 ("Refactor this exception test to have only one invocation possibly throwing an exception") at tests/unit/test_memory_backend.py:356 in test_atomic_list_append_on_missing_attribute_raises. Build the atomic operation list (Tagged.tags.append(["new"]), Tagged.notes.append(["note"])) before the pytest.raises block so only tagged.sync_update(atomic=...) is called inside it.

What Changed

  • The in-memory backend's SET handling now evaluates list_append(...) and plain if_not_exists(attr, :value) operands (nested calls work too), where before they were silently ignored. It follows DynamoDB rules: list_append on a value that is not a list, a path to a missing attribute, an undefined :value, or an unknown function name all raise ValidationException. Updates are applied to a copy of the item, so a rejected update expression leaves the stored item unchanged.
  • MemoryClient now has count() and sync_count(), built on scan(), with support for filter expressions, index_name, and consistent_read. The memory backend support table in docs/guides/testing.md now lists count().
  • Added unit tests in tests/unit/test_memory_backend.py for append and prepend, nested if_not_exists/list_append, the error cases, the unchanged-item-on-error case, and count with and without filters. In the missing-attribute test, the atomic operations are now built before pytest.raises, so only sync_update runs inside it.

🤖 Generated with Claude Code

Risk Assessment

✅ Low: The change is confined to the in-memory testing backend. It matches the real DynamoDBClient count signature and return shape, and the expressions that pydynox atomic operations emit for list_append and if_not_exists. Updates are now built on a copy and stored only when they succeed, which makes a rejected update leave the stored item unchanged, as DynamoDB does.

Testing

I built the extension, then ran the two docs examples. With the base memory.py both fail exactly as reported: the list_append is a silent no-op (['initial'] vs ['initial','new_tag']) and count raises AttributeError. With this change both pass. An end-user script then drove the sync and async Model APIs through append, prepend, if_not_exists, and count, plus adversarial cases. Appending to a missing attribute or a non-list raises ValidationException, and the stored item stays unchanged. I also ran the same operations against a throwaway LocalStack DynamoDB container, and every result matched the memory backend. The 9 new unit tests pass, including the Sonar-refactored one, which now calls only sync_update inside pytest.raises. The full memory-backend test file and all docs testing_*.py examples also pass. I removed transient caches and build outputs, and the worktree is clean.

  • Live validation: ✅ go - 9 of 9 scenarios driven live against the product
Scenario Result Live Evidence
Docs example test_atomic_append: appending to a list via Counter.tags.append under pydynox_memory_backend persists ['initial','new_tag'] (fails at base, passes at head) ✅ pass live docs-examples-before.txt (AssertionError ['initial'] at base) and docs-examples-after.txt (PASSED)
Docs example test_count_items: User.count() under pydynox_memory_backend returns 3 (AttributeError at base, passes at head) ✅ pass live docs-examples-before.txt (AttributeError: 'MemoryClient' object has no attribute 'count') and docs-examples-after.txt (PASSED)
Sync and async parity: sync_update/update with append, prepend, empty append, and if_not_exists produce the expected stored values ✅ pass live drive_memory_backend.txt
count/sync_count on an empty table, a seeded table, with a matching filter, and with a no-match filter return 0/N/filtered/0, and the metrics carry items_count ✅ pass live drive_memory_backend.txt
Adversarial: an update that appends to a missing list (alongside another SET) raises ValidationException and leaves the stored item fully unchanged ✅ pass live drive_memory_backend.txt (status stays 'before', notes None); unit test test_atomic_list_append_on_missing_attribute_raises
Adversarial: list_append on a non-list (string) attribute raises ValidationException and the item is unchanged ✅ pass live drive_memory_backend.txt
The memory backend's results match real DynamoDB (LocalStack 4.4.0) for append/prepend, if_not_exists, both error cases, and count ✅ pass live parity_localstack.txt (ALL MATCH)
SonarCloud S5778 follow-up: test_atomic_list_append_on_missing_attribute_raises builds the atomic list before pytest.raises, calls only sync_update inside it, and passes ✅ pass live tests/unit/test_memory_backend.py:355-359 and unit-new-tests.txt
No regression in existing memory-backend behavior after update_item switched to updating a copy of the stored item ✅ pass live regression-memory-and-docs.txt (test_memory_backend.py and all docs testing_*.py examples pass)
Evidence: Docs examples before the fix (reproduces both reported failures)
============================= test session starts ==============================
=================================== FAILURES ===================================
E       AssertionError: assert ['initial'] == ['initial', 'new_tag']
E         
E         Right contains one more item: 'new_tag'
E         Use -v to get more diff
E       AttributeError: 'MemoryClient' object has no attribute 'count'
=========================== short test summary info ============================
FAILED docs/examples/atomic/testing_atomic.py::test_atomic_append - Assertion...
FAILED docs/examples/scan/testing_scan.py::test_count_items - AttributeError:...
============================== 2 failed in 0.10s ===============================
Evidence: Docs examples after the fix (all 8 pass incl. test_atomic_append, test_count_items)
============================= test session starts ==============================
platform darwin -- Python 3.15.0, pytest-9.0.3, pluggy-1.6.0 -- ~/.no-mistakes/worktrees/bdd84ca597e4/01M6T2C0T8PY57WR6W55Z8NEF9/.venv/bin/python
codspeed: 4.2.0 (disabled, mode: walltime, callgraph: enabled, timer_resolution: 41.7ns)
benchmark: 5.2.3 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
hypothesis profile 'default'
rootdir: ~/.no-mistakes/worktrees/bdd84ca597e4/01M6T2C0T8PY57WR6W55Z8NEF9
configfile: pyproject.toml
plugins: memray-1.8.0, benchmark-5.2.3, hypothesis-6.150.2, codspeed-4.2.0, asyncio-1.3.0, pydynox-1.8.1
asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function
collecting ... collected 8 items

docs/examples/atomic/testing_atomic.py::test_atomic_increment PASSED     [ 12%]
docs/examples/atomic/testing_atomic.py::test_atomic_decrement PASSED     [ 25%]
docs/examples/atomic/testing_atomic.py::test_atomic_append PASSED        [ 37%]
docs/examples/atomic/testing_atomic.py::test_atomic_set PASSED           [ 50%]
docs/examples/scan/testing_scan.py::test_scan_all_items PASSED           [ 62%]
docs/examples/scan/testing_scan.py::test_scan_with_filter PASSED         [ 75%]
docs/examples/scan/testing_scan.py::test_scan_with_numeric_filter PASSED [ 87%]
docs/examples/scan/testing_scan.py::test_count_items PASSED              [100%]

============================== 8 passed in 0.43s ===============================
Evidence: End-user drive of MemoryBackend (sync and async, edge and error cases)
== sync API ==
count on empty table: 0
after append: ['a', 'b', 'c']
after prepend: ['z', 'a', 'b', 'c']
after empty append: ['z', 'a', 'b', 'c']
if_not_exists on missing hits: 5
if_not_exists on present hits: 5
add after if_not_exists: 6
append to missing attr raised: 'ValidationException: The provided expression refers to an attribute that does not exist in the item'
status after rejected update (unchanged?): 'before'
notes after rejected update: None
sync_count total: 3
sync_count status==inactive: 1
sync_count no match: 0
count metrics items_count: 3
== async API ==
async count seeded: 2
async count filtered: 1
async append: ['x', 'y']
async append to empty list: ['first']
append to string attr raised: 'ValidationException: Invalid UpdateExpression: Incorrect operand type for operator or function; operator or function: list_append'
status after rejected append: 'active'
Evidence: LocalStack DynamoDB vs MemoryBackend parity table

scenario | LocalStack | MemoryBackend | match count empty | 0 | 0 | yes append+prepend | ['z','a','b'] | ['z','a','b'] | yes if_not_exists twice | 5 | 5 | yes append missing | ValidationException | ValidationException | yes status after rejected | 'active' | 'active' | yes append to string | ValidationException | ValidationException | yes count total | 2 | 2 | yes count filtered | 1 | 1 | yes ALL MATCH

scenario                 | LocalStack DynamoDB                      | MemoryBackend                            | match
count empty              | 0                                        | 0                                        | yes
append+prepend           | ['z', 'a', 'b']                          | ['z', 'a', 'b']                          | yes
if_not_exists twice      | 5                                        | 5                                        | yes
append missing           | 'ValidationException'                    | 'ValidationException'                    | yes
status after rejected    | 'active'                                 | 'active'                                 | yes
append to string         | 'ValidationException'                    | 'ValidationException'                    | yes
count total              | 2                                        | 2                                        | yes
count filtered           | 1                                        | 1                                        | yes
ALL MATCH
Evidence: New unit tests run
tests/unit/test_memory_backend.py::test_atomic_if_not_exists_sets_only_missing_attribute PASSED [ 11%]
tests/unit/test_memory_backend.py::test_atomic_list_append_and_prepend PASSED [ 22%]
tests/unit/test_memory_backend.py::test_async_atomic_list_append PASSED  [ 33%]
tests/unit/test_memory_backend.py::test_atomic_list_append_on_missing_attribute_raises PASSED [ 44%]
tests/unit/test_memory_backend.py::test_list_append_with_if_not_exists_starts_missing_list PASSED [ 55%]
tests/unit/test_memory_backend.py::test_list_append_rejects_non_list_operand PASSED [ 66%]
tests/unit/test_memory_backend.py::test_sync_count_with_and_without_filter PASSED [ 77%]
tests/unit/test_memory_backend.py::test_async_count PASSED               [ 88%]
tests/unit/test_memory_backend.py::test_client_count_accepts_dynamodb_client_arguments PASSED [100%]
======================= 9 passed, 55 deselected in 0.13s =======================
Evidence: Drive script source
"""End-user drive of the pydynox MemoryBackend for list_append, if_not_exists, and count."""
import asyncio

from pydynox import Model, ModelConfig
from pydynox.attributes import ListAttribute, NumberAttribute, StringAttribute
from pydynox.testing import MemoryBackend


class Tagged(Model):
    model_config = ModelConfig(table="tagged")
    pk = StringAttribute(partition_key=True)
    tags = ListAttribute()
    notes = ListAttribute()
    status = StringAttribute(default="active")
    hits = NumberAttribute()


def show(label, value):
    print(f"{label}: {value!r}")


def sync_scenarios():
    print("== sync API ==")
    with MemoryBackend():
        show("count on empty table", Tagged.sync_count()[0])

        t = Tagged(pk="A", tags=["a"])
        t.sync_save()
        t.sync_update(atomic=[Tagged.tags.append(["b", "c"])])
        show("after append", Tagged.sync_get(pk="A").tags)
        t.sync_update(atomic=[Tagged.tags.prepend(["z"])])
        show("after prepend", Tagged.sync_get(pk="A").tags)
        t.sync_update(atomic=[Tagged.tags.append([])])
        show("after empty append", Tagged.sync_get(pk="A").tags)

        # if_not_exists on missing then on present attribute
        t.sync_update(atomic=[Tagged.hits.if_not_exists(5)])
        show("if_not_exists on missing hits", Tagged.sync_get(pk="A").hits)
        t.sync_update(atomic=[Tagged.hits.if_not_exists(99)])
        show("if_not_exists on present hits", Tagged.sync_get(pk="A").hits)
        t.sync_update(atomic=[Tagged.hits.add(1)])
        show("add after if_not_exists", Tagged.sync_get(pk="A").hits)

        # Adversarial: append to a missing list must raise, and the whole update must not apply
        t.sync_update(atomic=[Tagged.status.set("before")])
        ops = [Tagged.status.set("SHOULD_NOT_APPLY"), Tagged.notes.append(["n"])]
        try:
            t.sync_update(atomic=ops)
            show("append to missing attr", "NO ERROR (bug)")
        except Exception as exc:
            show("append to missing attr raised", f"{type(exc).__name__}: {exc}")
        after = Tagged.sync_get(pk="A")
        show("status after rejected update (unchanged?)", after.status)
        show("notes after rejected update", after.notes)

        Tagged(pk="B", status="inactive").sync_save()
        Tagged(pk="C").sync_save()
        show("sync_count total", Tagged.sync_count()[0])
        show("sync_count status==inactive", Tagged.sync_count(filter_condition=Tagged.status == "inactive")[0])
        show("sync_count no match", Tagged.sync_count(filter_condition=Tagged.status == "nope")[0])
        cnt, metrics = Tagged.sync_count()
        show("count metrics items_count", getattr(metrics, "items_count", None))


async def async_scenarios():
    print("== async API ==")
    with MemoryBackend(seed={"tagged": [{"pk": "S1", "tags": ["x"], "status": "active"},
                                        {"pk": "S2", "tags": [], "status": "inactive"}]}):
        show("async count seeded", (await Tagged.count())[0])
        show("async count filtered", (await Tagged.count(filter_condition=Tagged.status == "active"))[0])
        s1 = await Tagged.get(pk="S1")
        await s1.update(atomic=[Tagged.tags.append(["y"])])
        show("async append", (await Tagged.get(pk="S1")).tags)
        s2 = await Tagged.get(pk="S2")
        await s2.update(atomic=[Tagged.tags.append(["first"])])
        show("async append to empty list", (await Tagged.get(pk="S2")).tags)
        # Adversarial: list_append against a non-list attribute
        try:
            await s1.update(atomic=[Tagged.status.append(["x"])])
            show("append to string attr", "NO ERROR (bug)")
        except Exception as exc:
            show("append to string attr raised", f"{type(exc).__name__}: {exc}")
        show("status after rejected append", (await Tagged.get(pk="S1")).status)


sync_scenarios()
asyncio.run(async_scenarios())
Evidence: Parity script source
"""Run the same model operations against LocalStack DynamoDB and the MemoryBackend; compare."""
from pydynox import DynamoDBClient, Model, ModelConfig, set_default_client
from pydynox.attributes import ListAttribute, NumberAttribute, StringAttribute
from pydynox.testing import MemoryBackend
from testcontainers.localstack import LocalStackContainer


class Tagged(Model):
    model_config = ModelConfig(table="tagged")
    pk = StringAttribute(partition_key=True)
    tags = ListAttribute()
    notes = ListAttribute()
    status = StringAttribute(default="active")
    hits = NumberAttribute()


def run():
    out = []
    rec = lambda k, v: out.append((k, v))
    rec("count empty", Tagged.sync_count()[0])
    t = Tagged(pk="A", tags=["a"])
    t.sync_save()
    t.sync_update(atomic=[Tagged.tags.append(["b"])])
    t.sync_update(atomic=[Tagged.tags.prepend(["z"])])
    rec("append+prepend", Tagged.sync_get(pk="A").tags)
    t.sync_update(atomic=[Tagged.hits.if_not_exists(5)])
    t.sync_update(atomic=[Tagged.hits.if_not_exists(99)])
    rec("if_not_exists twice", Tagged.sync_get(pk="A").hits)
    try:
        t.sync_update(atomic=[Tagged.status.set("X"), Tagged.notes.append(["n"])])
        rec("append missing", "no error")
    except Exception as exc:
        rec("append missing", type(exc).__name__)
    rec("status after rejected", Tagged.sync_get(pk="A").status)
    try:
        t.sync_update(atomic=[Tagged.status.append(["x"])])
        rec("append to string", "no error")
    except Exception as exc:
        rec("append to string", type(exc).__name__)
    Tagged(pk="B", status="inactive").sync_save()
    rec("count total", Tagged.sync_count()[0])
    rec("count filtered", Tagged.sync_count(filter_condition=Tagged.status == "inactive")[0])
    return out


c = LocalStackContainer(image="localstack/localstack:4.4.0").with_services("dynamodb")
c.start()
try:
    client = DynamoDBClient(region="us-east-1", endpoint_url=c.get_url(), access_key="testing", secret_key="testing")
    client.sync_create_table("tagged", partition_key=("pk", "S"), wait=True)
    set_default_client(client)
    real = run()
finally:
    c.stop()

with MemoryBackend():
    mem = run()

ok = True
print(f"{'scenario':24} | {'LocalStack DynamoDB':40} | {'MemoryBackend':40} | match")
for (k, r), (_, m) in zip(real, mem):
    match = r == m
    ok &= match
    print(f"{k:24} | {r!r:40} | {m!r:40} | {'yes' if match else 'NO'}")
print("ALL MATCH" if ok else "MISMATCH")

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

  • Live validation: ✅ go - 9 of 9 scenarios driven live against the product
Scenario Result Live Evidence
Docs example test_atomic_append: appending to a list via Counter.tags.append under pydynox_memory_backend persists ['initial','new_tag'] (fails at base, passes at head) ✅ pass live docs-examples-before.txt (AssertionError ['initial'] at base) and docs-examples-after.txt (PASSED)
Docs example test_count_items: User.count() under pydynox_memory_backend returns 3 (AttributeError at base, passes at head) ✅ pass live docs-examples-before.txt (AttributeError: 'MemoryClient' object has no attribute 'count') and docs-examples-after.txt (PASSED)
Sync and async parity: sync_update/update with append, prepend, empty append, and if_not_exists produce the expected stored values ✅ pass live drive_memory_backend.txt
count/sync_count on an empty table, a seeded table, with a matching filter, and with a no-match filter return 0/N/filtered/0, and the metrics carry items_count ✅ pass live drive_memory_backend.txt
Adversarial: an update that appends to a missing list (alongside another SET) raises ValidationException and leaves the stored item fully unchanged ✅ pass live drive_memory_backend.txt (status stays 'before', notes None); unit test test_atomic_list_append_on_missing_attribute_raises
Adversarial: list_append on a non-list (string) attribute raises ValidationException and the item is unchanged ✅ pass live drive_memory_backend.txt
The memory backend's results match real DynamoDB (LocalStack 4.4.0) for append/prepend, if_not_exists, both error cases, and count ✅ pass live parity_localstack.txt (ALL MATCH)
SonarCloud S5778 follow-up: test_atomic_list_append_on_missing_attribute_raises builds the atomic list before pytest.raises, calls only sync_update inside it, and passes ✅ pass live tests/unit/test_memory_backend.py:355-359 and unit-new-tests.txt
No regression in existing memory-backend behavior after update_item switched to updating a copy of the stored item ✅ pass live regression-memory-and-docs.txt (test_memory_backend.py and all docs testing_*.py examples pass)
  • uv sync && uv run --with maturin maturin develop (built the Rust extension into the project venv)
  • uv run pytest docs/examples/atomic/testing_atomic.py docs/examples/scan/testing_scan.py -v at HEAD
  • Same two failing docs tests rerun with base-commit python/pydynox/testing/memory.py swapped in, to reproduce the original failures, then restored
  • End-user script drive_memory_backend.py: sync and async Model API under MemoryBackend() covering append, prepend, empty append, if_not_exists on missing/present, count on empty/seeded/filtered tables, and rejected updates (missing list, non-list target) checked to leave the stored item unchanged
  • Parity script parity_localstack.py: the same operations against real LocalStack 4.4.0 DynamoDB (testcontainers, disposable container) and against MemoryBackend, results compared side by side
  • uv run pytest tests/unit/test_memory_backend.py -k <9 new tests> including test_atomic_list_append_on_missing_attribute_raises
  • Read tests/unit/test_memory_backend.py:346-365 to confirm only sync_update is inside pytest.raises
  • uv run pytest tests/unit/test_memory_backend.py docs/examples/**/testing_*.py -q (targeted regression for the changed update_item path)
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

… backend

The memory backend ignored SET x = list_append(...) and SET x = if_not_exists(x, :v),
so atomic append, prepend, and if_not_exists updates were silent no-ops.
SET operands are now evaluated like DynamoDB: list_append concatenates in
argument order, if_not_exists falls back when the path is missing, and a
missing path or a non-list operand raises ValidationException. Updates are
applied to a copy so a rejected expression leaves the stored item unchanged.

MemoryClient also gains count() and sync_count() with the same arguments
and return shape as DynamoDBClient, so Model.count() works in tests.
@vicolby
vicolby marked this pull request as draft October 31, 2026 06:06
@codecov

codecov Bot commented Oct 31, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.64286% with 3 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
python/pydynox/testing/memory.py 94.64% 3 Missing ⚠️

📢 Thoughts on this report? Let us know!

@vicolby
vicolby marked this pull request as ready for review October 31, 2026 06:32
@vicolby
vicolby marked this pull request as draft October 31, 2026 06:35
@vicolby
vicolby marked this pull request as ready for review November 7, 2026 03:46
Build the atomic operation list before the pytest.raises block so only
sync_update can raise inside it (SonarCloud python:S5778).
@sonarqubecloud

sonarqubecloud Bot commented Nov 7, 2026

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant