Skip to content

Commit 2a2a3d5

Browse files
committed
Merge PR #26 (audriB backend hardening) into current main
PR #26 was opened against 361df8d, before #27 landed, and had gone un-mergeable. Its SQLite/packaging fixes are still needed: run against current main, 22 of its 24 new tests fail. Its lint work, however, is superseded -- #27 solved the same problem the other way. Kept (all of the hardening): - LIKE/regexp metacharacter escaping in hasfield/isa and contains_string/depends_on; get_docs IN-clause chunking. - _do_add_doc atomicity (branch checked before any INSERT, rollback plus fields-cache restore on every failure path) and OnDuplicate. - binarydoc_matfid fails loudly instead of using an uninitialized Fileobj. - example_schema JSON shipped in the wheel, with the wheel-smoke CI job. - doc2sql class_name-first superclass resolution and list-of-dict expansion. - isa diamond-parity fixtures; symmetry missing-artifact skips become failures. Dropped (superseded by #27): - The [tool.ruff.lint] select = ["E4","E7","E9","F"] narrowing. #27 fixed all 56 findings and pinned ruff==0.16.5/black==26.5.1 instead, which is the better fix and is already on main. The dependency bounds and package-data from the same commit are kept. - The import-sort commit, subsumed by #27's repo-wide I001 pass. - The Document()/Document('base') constructor fix: main already builds both via the database_documents definition path. Only the narrow guard that converts a descriptor-list 'base' to a defaults dict survives, on the legacy database_schema fallback path, where base.schema.json still stores 'base' as a list. Conflict resolutions: - database.add_docs takes both sides: main's validate=True/validate_docs pass and the PR's OnDuplicate member check. - sqlitedb._do_add_doc takes the PR's handler. Main's FOREIGN KEY -> ValueError translation is subsumed by the up-front branch existence check, and its swallowing of duplicate IntegrityErrors is exactly what OnDuplicate replaces. - document.py keeps main's _reset_file_info and _merge_superclasses alongside the PR's _utc_timestamp and _field_descriptors_to_defaults. - base.datestamp is now ISO-8601 milliseconds + 'Z', per base.schema.json's own default ('2018-12-05T18:36:47.241Z') and DID-matlab's UTCLeapSeconds output. Main's _utcnow() in document.py was a DTZ003 lint rename that preserved the space-separated, tz-less form; it is gone. file.py's unrelated _utcnow stays. - test_add_docs_atomicity passes validate=False: it exercises _do_add_doc's transaction handling with synthetic stub documents (class 'thing', short non-UID ids) that main's new schema validation rejects before _do_add_doc is reached. Verified on the merged tree with the pinned toolchain: ruff 0.16.5 and black 26.5.1 both clean over src/ and tests/; pytest 139 passed, 1 skipped, 1 xfailed; wheel built, installed into a clean venv, and Document('demoA') / Document('base') constructed from outside the repo. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4rNahWHpwiLTUZv3DskN8
2 parents 0014b42 + 619d2c9 commit 2a2a3d5

17 files changed

Lines changed: 1116 additions & 85 deletions

‎.github/workflows/python-package.yml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,3 +56,28 @@ jobs:
5656
5757
- name: Test with pytest
5858
run: pytest
59+
60+
wheel-smoke:
61+
# A pytest run imports from the source tree, so it cannot catch missing
62+
# package DATA (example_schema/*.json) in the built distribution. Build a
63+
# wheel, install it into a clean environment, and construct a document from
64+
# OUTSIDE the repo so a broken package-data config fails here.
65+
runs-on: ubuntu-latest
66+
steps:
67+
- uses: actions/checkout@v4
68+
69+
- name: Set up Python 3.12
70+
uses: actions/setup-python@v5
71+
with:
72+
python-version: "3.12"
73+
74+
- name: Build wheel
75+
run: |
76+
python -m pip install --upgrade pip build
77+
python -m build --wheel --outdir dist
78+
79+
- name: Install wheel and construct a document from outside the repo
80+
run: |
81+
python -m pip install dist/*.whl
82+
cd "$RUNNER_TEMP"
83+
python -c "from did.document import Document; Document('demoA'); Document('base'); print('wheel smoke OK')"

‎.github/workflows/symmetry.yml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,14 @@ jobs:
8080
run: pytest -m make_artifacts -v
8181

8282
- name: "Step 2: Python readArtifact tests"
83+
# Artifacts are produced by earlier steps in THIS job (MATLAB
84+
# makeArtifacts in Step 1, Python makeArtifacts above), so a missing
85+
# artifact dir/file here is a real failure, not a reason to skip green.
86+
# This env var makes the readArtifact test fail hard on absence instead
87+
# of skipping (skips exit 0 and let the badge go green having compared
88+
# nothing across languages).
89+
env:
90+
DID_SYMMETRY_REQUIRE_ARTIFACTS: "1"
8391
run: pytest -m read_artifacts -v
8492

8593
# --- Step 3: MATLAB readArtifacts ---

‎pyproject.toml‎

Lines changed: 16 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -11,11 +11,11 @@ authors = [
1111
{ name = "Author", email = "vanhoosr@brandeis.edu" },
1212
]
1313
dependencies = [
14-
"numpy",
15-
"pandas",
16-
"portalocker",
17-
"networkx",
18-
"matplotlib",
14+
"numpy>=1.20",
15+
"pandas>=2.0,<4",
16+
"portalocker>=2.0",
17+
"networkx>=2.6",
18+
"matplotlib>=3.5",
1919
]
2020
requires-python = ">=3.10"
2121

@@ -40,3 +40,14 @@ where = ["src"]
4040

4141
[tool.setuptools.package-dir]
4242
"" = "src"
43+
44+
# Ship the example schema/document JSON in the built distribution. Without this
45+
# a pip-installed `did` has no example_schema/ under site-packages, so
46+
# PathConstants.DEFPATH points at a nonexistent directory and Document('demoA')
47+
# raises FileNotFoundError. example_schema/ is a data directory (no __init__.py),
48+
# so packages.find does not discover it and its JSON must be declared here.
49+
[tool.setuptools.package-data]
50+
"did" = ["example_schema/**/*.json"]
51+
52+
[tool.setuptools]
53+
include-package-data = true

‎src/did/database.py‎

Lines changed: 21 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -93,16 +93,35 @@ def validate_docs(self, document_objs):
9393

9494
_validate_docs(document_objs, all_ids, debug=self.debug)
9595

96-
def add_docs(self, document_objs, branch_id=None, validate=True, **kwargs):
96+
#: Allowed values for the ``OnDuplicate`` option, mirroring DID-matlab's
97+
#: ``did.database.add_docs`` ``OnDuplicate {mustBeMember(...)} = 'error'``.
98+
_ON_DUPLICATE_CHOICES = ("ignore", "warn", "error")
99+
100+
def add_docs(
101+
self,
102+
document_objs,
103+
branch_id=None,
104+
validate=True,
105+
OnDuplicate="error",
106+
**kwargs,
107+
):
97108
if branch_id is None:
98109
branch_id = self.current_branch_id
99110

111+
# Reject invalid OnDuplicate up front (mirror MATLAB mustBeMember) so a
112+
# typo cannot silently fall through to default behaviour.
113+
if str(OnDuplicate).lower() not in self._ON_DUPLICATE_CHOICES:
114+
raise ValueError(
115+
"OnDuplicate must be one of "
116+
f"{self._ON_DUPLICATE_CHOICES}; got {OnDuplicate!r}."
117+
)
118+
100119
# Ensure all input docs pass schema validation (unless requested not to)
101120
if validate:
102121
self.validate_docs(document_objs)
103122

104123
for doc in document_objs:
105-
self._do_add_doc(doc, branch_id, **kwargs)
124+
self._do_add_doc(doc, branch_id, OnDuplicate=OnDuplicate, **kwargs)
106125

107126
# ... other document-related methods would follow ...
108127

‎src/did/document.py‎

Lines changed: 44 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,21 @@
66
from .common import PathConstants
77

88

9-
def _utcnow():
10-
"""Naive UTC timestamp, identical to the deprecated datetime.utcnow()."""
11-
return datetime.now(timezone.utc).replace(tzinfo=None)
9+
def _utc_timestamp():
10+
"""Return the current UTC time as an ISO-8601 millisecond string with 'Z'.
11+
12+
Ported from NDI-python (ndi.fun.timestamp) with the leap-second guard.
13+
``str(datetime.utcnow())`` previously emitted a space-separated,
14+
timezone-less string ('2026-07-20 22:36:19.611068') that diverges from the
15+
DID-matlab UTCLeapSeconds ISO-8601 output and from base.schema.json's own
16+
default ('2018-12-05T18:36:47.241Z'); a JS ``new Date()`` parses the
17+
tz-less form as LOCAL time. Emit '%Y-%m-%dT%H:%M:%S.%fZ' truncated to
18+
milliseconds, clamping the (theoretical) leap-second :60 to :59.999.
19+
"""
20+
now = datetime.now(timezone.utc)
21+
ts = now.strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3]
22+
ts = ts.replace(":60.", ":59.999")
23+
return ts + "Z"
1224

1325

1426
class Document:
@@ -18,7 +30,7 @@ def __init__(self, document_type="base", **kwargs):
1830
else:
1931
self.document_properties = self.read_blank_definition(document_type)
2032
self.document_properties["base"]["id"] = ido.IDO.unique_id()
21-
self.document_properties["base"]["datestamp"] = str(_utcnow())
33+
self.document_properties["base"]["datestamp"] = _utc_timestamp()
2234

2335
for key, value in kwargs.items():
2436
path = key.split(".")
@@ -165,9 +177,22 @@ def read_blank_definition(json_file_location_string):
165177
if os.path.exists(filepath):
166178
with open(filepath, "r") as f:
167179
data = json.load(f)
180+
# Ensure the 'base' key is a dict the constructor can stamp
181+
# base.id / base.datestamp onto. base.schema.json stores 'base'
182+
# as a LIST of field descriptors (unlike demoA.schema.json,
183+
# which has no top-level 'base'), so Document('base') /
184+
# Document() used to raise "TypeError: list indices must be
185+
# integers". Convert that descriptor list to a {name:
186+
# default_value} defaults dict. (This is the narrow constructor
187+
# fix only; the broader build-from-database_documents rework is
188+
# tracked separately.)
168189
if "base" not in data:
169190
data["base"] = {}
170-
return Document._normalize_to_document_class(data)
191+
elif isinstance(data["base"], list):
192+
data["base"] = Document._field_descriptors_to_defaults(data["base"])
193+
# Convert flat classname/superclasses to document_class format
194+
data = Document._normalize_to_document_class(data)
195+
return data
171196

172197
# Fallback for base
173198
if json_file_location_string == "base":
@@ -263,6 +288,20 @@ def _merge_superclasses(data):
263288
data["document_class"] = class_props
264289
return data
265290

291+
@staticmethod
292+
def _field_descriptors_to_defaults(descriptors):
293+
"""Convert a schema field-descriptor list to a {name: default_value} dict.
294+
295+
Each descriptor is a dict with at least a 'name' and (usually) a
296+
'default_value'. Used to turn base.schema.json's 'base' list into a
297+
blank base group the constructor can stamp id/datestamp onto.
298+
"""
299+
defaults = {}
300+
for field in descriptors:
301+
if isinstance(field, dict) and "name" in field:
302+
defaults[field["name"]] = field.get("default_value", "")
303+
return defaults
304+
266305
@staticmethod
267306
def _normalize_to_document_class(data):
268307
"""Convert flat schema format to MATLAB-compatible document_class format."""

‎src/did/implementations/binarydoc_matfid.py‎

Lines changed: 23 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,14 @@
22
from ..file import Fileobj
33

44

5-
class BinaryDocMatfid(BinaryDoc, Fileobj):
5+
class BinaryDocMatfid(Fileobj, BinaryDoc):
6+
# NOTE: base order is (Fileobj, BinaryDoc). Previously it was
7+
# (BinaryDoc, Fileobj): BinaryDoc.__init__ is a bare ``pass`` that never
8+
# calls super().__init__(), so Fileobj.__init__ never ran (fullpathfilename
9+
# / permission / fid were unset) and every ``super().f*`` call resolved to
10+
# BinaryDoc's abstract no-op stub, so fopen() silently returned None without
11+
# opening anything. With Fileobj first in the MRO, __init__ initializes the
12+
# file object and the f* methods delegate to real implementations.
613
def __init__(self, key="", doc_unique_id="", **kwargs):
714
super().__init__(**kwargs)
815
self.key = key
@@ -31,11 +38,22 @@ def feof(self):
3138
return super().feof()
3239

3340
def fwrite(self, data, precision=None, skip=0):
34-
# The precision and skip parameters would need to be handled
35-
# using Python's struct module for a full implementation.
41+
# precision/skip (MATLAB struct-format writes) are not implemented yet.
42+
# Fail loudly rather than silently ignoring them and writing raw bytes.
43+
if precision is not None or skip:
44+
raise NotImplementedError(
45+
"BinaryDocMatfid.fwrite does not yet support precision/skip; "
46+
"only raw byte writes are supported."
47+
)
3648
return super().fwrite(data)
3749

3850
def fread(self, count=-1, precision=None, skip=0):
39-
# The precision and skip parameters would need to be handled
40-
# using Python's struct module for a full implementation.
51+
# precision/skip (MATLAB struct-format reads) are not implemented yet.
52+
# Raise rather than returning None so a caller can't mistake an
53+
# unimplemented typed read for an empty file.
54+
if precision is not None or skip:
55+
raise NotImplementedError(
56+
"BinaryDocMatfid.fread does not yet support precision/skip; "
57+
"only raw byte reads are supported."
58+
)
4159
return super().fread(count)

‎src/did/implementations/doc2sql.py‎

Lines changed: 55 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import json
12
import re
23

34

@@ -49,6 +50,14 @@ def _get_superclass_str(doc_props):
4950
MATLAB produces comma-space separated, sorted unique superclass names.
5051
For MATLAB-style definitions like "$PATH/base.json", strip path and extension.
5152
For DID-python style ["base", "demoA"], use directly.
53+
54+
Schema-v2 (DID-schema V_delta / V_epsilon) names a superclass directly with a
55+
bare ``{"class_name": ...}`` object. Read ``class_name`` first but UNION in the
56+
``definition``-derived name rather than short-circuiting, so a mixed-shape entry
57+
never silently narrows the superclass set. This mirrors the reference contract
58+
in ndi-cloud-node ``api/src/dal/class_lineage.ts`` (``computeClassLineage``) and
59+
NDI-python ``ndi.document.doc_superclass``, keeping the SQL ``meta.superclass``
60+
column (and both isa paths that read it) consistent across all three stacks.
5261
"""
5362
# DID-python schema format: top-level 'superclasses' list of strings
5463
if "superclasses" in doc_props and isinstance(
@@ -65,12 +74,16 @@ def _get_superclass_str(doc_props):
6574
for sc in superclasses:
6675
if isinstance(sc, str):
6776
names.append(sc)
68-
elif isinstance(sc, dict) and "definition" in sc:
69-
# MATLAB-style: extract name from definition path
70-
defn = sc["definition"]
71-
name = re.sub(r".+/", "", defn)
72-
name = re.sub(r"\..+$", "", name)
73-
names.append(name)
77+
elif isinstance(sc, dict):
78+
# class_name-first, UNION with the definition-derived name.
79+
if sc.get("class_name"):
80+
names.append(sc["class_name"])
81+
if sc.get("definition"):
82+
# MATLAB-style: extract name from definition path
83+
defn = sc["definition"]
84+
name = re.sub(r".+/", "", defn)
85+
name = re.sub(r"\..+$", "", name)
86+
names.append(name)
7487
names = sorted(set(names))
7588
return ", ".join(names)
7689

@@ -82,11 +95,15 @@ def _get_superclass_str(doc_props):
8295
if isinstance(superclasses, list):
8396
names = []
8497
for sc in superclasses:
85-
if isinstance(sc, dict) and "definition" in sc:
86-
defn = sc["definition"]
87-
name = re.sub(r".+/", "", defn)
88-
name = re.sub(r"\..+$", "", name)
89-
names.append(name)
98+
if isinstance(sc, dict):
99+
# class_name-first, UNION with the definition-derived name.
100+
if sc.get("class_name"):
101+
names.append(sc["class_name"])
102+
if sc.get("definition"):
103+
defn = sc["definition"]
104+
name = re.sub(r".+/", "", defn)
105+
name = re.sub(r"\..+$", "", name)
106+
names.append(name)
90107
elif isinstance(sc, str):
91108
names.append(sc)
92109
names = sorted(set(names))
@@ -115,15 +132,39 @@ def _serialize_depends_on(doc_props):
115132

116133

117134
def _flatten_dict(d, prefix=""):
118-
"""Flatten a nested dict using ___ separator for nested keys (matching MATLAB's getMetaTableFrom)."""
135+
"""Flatten a nested dict using ___ separator for nested keys.
136+
137+
Matches DID-matlab getMetaTableFrom/recurseFields:
138+
139+
* nested dicts recurse with a ``___`` separator;
140+
* a list of dicts (a struct array) expands into one column per element,
141+
each element's flattened column suffixed ``_<1-based idx>`` — but ONLY
142+
when the list has more than one element (a single-element list is
143+
unsuffixed, mirroring doc2sql.m's ``numElements > 1`` rule);
144+
* any residual scalar/mixed list is JSON-encoded (not ``str()``-repr'd) so
145+
the stored value is valid JSON parseable by MATLAB/JS consumers instead
146+
of a Python repr with single quotes.
147+
148+
Previously every list was ``str(value)``'d into one column, so per-element
149+
fields were unsearchable and the stored string disagreed with what
150+
DID-matlab writes for the same document.
151+
"""
119152
items = []
120153
for key, value in d.items():
121154
col_name = f"{prefix}___{key}" if prefix else key
122155
if isinstance(value, dict):
123156
items.extend(_flatten_dict(value, col_name))
124157
elif isinstance(value, list):
125-
# Convert lists to string representation
126-
items.append((col_name, str(value)))
158+
if value and all(isinstance(el, dict) for el in value):
159+
# Struct array: expand per element.
160+
multi = len(value) > 1
161+
for idx, el in enumerate(value, start=1):
162+
for sub_name, sub_val in _flatten_dict(el, col_name):
163+
name = f"{sub_name}_{idx}" if multi else sub_name
164+
items.append((name, sub_val))
165+
else:
166+
# Scalar / empty / mixed list: JSON-encode.
167+
items.append((col_name, json.dumps(value)))
127168
else:
128169
items.append((col_name, value))
129170
return items

0 commit comments

Comments
 (0)