ericprud/SWObjects
Folders and files
| Name | Name | Last commit date | ||
|---|---|---|---|---|
Repository files navigation
Hi, welcome to SWObjects and all that. You probably want to:
cmake -B build
cmake --build build -j8 # tools land in bin/, e.g. bin/sparql
ctest --test-dir build # run the test suites
Feature selection that used to live in the CONFIG file is now done with
cmake options (see the options block at the top of CMakeLists.txt):
cmake -B build -DSWOBJ_XML_PARSER=LIBXML2 -DSWOBJ_SQL_MYSQL=ON
Some test suites (test_Turtle, test_Trig, test_RDFXML, test_SPARQL11)
expect the W3C test data unpacked under tests/ ; fetch it with:
sh tests/fetch-test-suites.sh
CI (.github/workflows/ci.yml) runs the same steps on Linux and macOS.
# Debian instructions
you'll need these packages:
- cmake
- flex (>= 2.6) and libfl-dev (FlexLexer.h)
- bison (>= 3.2)
- uuid-dev
- libexpat1-dev (or libxml2-dev with -DSWOBJ_XML_PARSER=LIBXML2)
- libmysqlclient-dev (only with -DSWOBJ_SQL_MYSQL=ON)
- libboost-dev
- libboost-log-dev
- libboost-regex-dev
- libboost-test-dev
- libboost-program-options-dev
- libboost-iostreams-dev
- libboost-thread-dev
- libboost-filesystem-dev
On macOS: brew install cmake boost bison flex expat
(cmake prefers Homebrew's bison/flex over the older /usr/bin ones.)
# ShEx
lib/ShExCParser.ypp and lib/ShExSchema.{hpp,cpp} implement ShEx 2.1
(ShExC syntax including IMPORT, ABSTRACT and EXTENDS; validation by
interval-based matching over SORBE-normalized triple expressions, after
jena-shex). EXTERNAL shapes resolve through ShEx::ExternalResolver
(e.g. against a .shextern schema). lib/ShExShapeMap.{hpp,cpp} parses
fixed and query shape maps (node@shape, node@!shape, START,
{FOCUS <p> _} patterns) and the shexTest JSON map format.
tests/test_ShEx.cpp runs the shexSpec/shexTest suite (schemas,
negativeSyntax, negativeStructure and validation manifests);
tests/fetch-test-suites.sh clones the test data.
bin/sparql runs shex-webapp-style validation manifests (JSON or YAML;
see lib/ShExManifest.hpp and tests/ShExManifest/):
bin/sparql --shex-manifest tests/ShExManifest/manifest.yaml
bin/sparql --shex-manifest m.json --shex-run '2-5,0'
--shex-run selects 0-based entries: ranges like "1", "2-5", "5-2"
(descending) or "*" (everything), comma-separated.
lib/ShExMap.{hpp,cpp} implements the http://shex.io/extensions/Map/#
semantic-action language for mapping between two schemas: validating
against a source schema whose constraints carry %Map:{ var %} (or
regex(...)/hashmap(...) lift functions) collects variable bindings,
and materialization walks a target schema emitting the bound (or
lowered) values. Manifest entries drive it with outputSchema[URL],
outputShape, createRoot, staticVars and optional expectedBindingsURL /
outputDataURL comparisons (see tests/ShExMap/, ported from shex.js's
packages/extension-map examples):
bin/sparql --shex-manifest tests/ShExMap/manifest.yaml
--shex-endpoint validates a shape map against a *live* SPARQL protocol
endpoint instead of an in-memory graph: node neighborhoods are fetched
and cached as validation visits them (lib/BNodeResolver.hpp's
RemoteGraphProvider wired in as Validator::NeighborhoodSource), so
response blank nodes crossing the round trips this needs are told-bnode-
safe (see the algae3 repo's doc/told-bnodes.md). Fixed associations and
ONE query pattern are both supported — the pattern's FOCUS candidates
are fetched with a single additional round trip
(lib/ShExShapeMap.hpp's PatternMatcher abstracts this from the existing
local-graph matching, now LocalPatternMatcher). Demo (from tests/, with
bin/sparql itself serving the evidence data):
../bin/sparql --serve http://127.0.0.1:8099/sparql \
-d Algae3/data/evidence.ttl -d Algae3/data/evidence-groups.ttl
../bin/sparql --shex-endpoint http://127.0.0.1:8099/sparql \
--shex-schema schema.shex \
--shex-map 'evgrp:PeerReviewedCohorts@<GroupShape>'
# or validate every node matching a pattern in one go:
../bin/sparql --shex-endpoint http://127.0.0.1:8099/sparql \
--shex-schema schema.shex --shex-map '{FOCUS ex:member _}@<GroupShape>'
Node-side prefixes in --shex-map borrow the schema's own PREFIX
declarations (there's no separate "data document" to carry its own).
"@!shape" asserts the node should NOT conform (exit code counts
mismatches, matching --shex-manifest's PASS/FAIL convention).
# Algae 3
lib/Algae3Parser.ypp, lib/Algae3Scanner.lpp and lib/Algae3.{hpp,cpp}
implement Algae 3, a pipeline-of-actions query language (a modernized
Algae2); bin/algae3 is the CLI. The language itself — data model,
actions, pattern combinators, the two evaluation modes — is specified
in the proposal repo, https://github.com/ericprud/algae3 (doc/).
Run the examples from tests/ (load paths are relative):
cd tests
../bin/algae3 Algae3/evidence-dnf.topdown.a3 # the flagship (below)
../bin/algae3 --mode bottomup Algae3/evidence-dnf.bottomup.a3
../bin/algae3 --proofs Algae3/rule.a3 # per-row provenance
../bin/algae3 --algebra Algae3/connectives.a3 # print the algebra tree
../bin/algae3 --debug Algae3/negation-scoped-topdown.a3 # REPL debugger
# step/next/into/where/print/bindings/watch/try/mode/break/run/reset
../bin/algae3 --serve 8080 Algae3/evidence-dnf.topdown.a3
# then open SPARQL-explorer.html (repo root): paste a raw
# .rq and sparql-to-a3.js compiles it to either mode and
# steps the evaluation visually
tests/test_Algae3.cpp is the suite (golden results plus differential
topdown-vs-bottomup runs); tests/Algae3/regen-compiled.mjs regenerates
the compiled .a3 forms from the .rq sources via sparql-to-a3.js.
The flagship query, tests/Algae3/evidence-dnf.rq, is a DNF rule engine
abstracted from a production query: a rule fires when ALL selectors
pass, a selector when ANY disjunct passes, a disjunct when ALL its
conjuncts pass — universal quantification by counting, at two levels,
with the proof log assembled by nested GROUP_CONCATs:
SELECT (STRAFTER(STR(?rule), "#") AS ?ruleHash) ?message ?evidences ?log
FROM <data/evidence-rules.ttl>
FROM <data/evidence-groups.ttl>
FROM <data/evidence.ttl>
WHERE { {
?rule a ex:Rule ; ex:action [ ex:message ?message ] .
{ SELECT ?rule (COUNT(?selector) AS ?selectorsPassed) … (…AS ?log) …
WHERE {
?rule ex:selector ?selector .
{ SELECT ?evidence ?selector … WHERE {
?selector ex:or ?disjunct
{ SELECT ?evidence ?disjunct (COUNT(?conjunct) AS ?conjunctsPassed) …
WHERE {
?disjunct ex:and ?conjunct .
{ # conjunct kinds: group membership (via bnode
# memberships!), attribute comparison, type test —
# each with positive and MINUS-based negative polarity
…
} } GROUP BY ?evidence ?disjunct }
{ SELECT ?disjunct (COUNT(*) AS ?conjunctCount) WHERE {
?disjunct ex:and ?conjunct . } GROUP BY ?disjunct }
FILTER (?conjunctsPassed = ?conjunctCount) # ALL conjuncts
} } GROUP BY ?rule } }
{ SELECT ?rule (COUNT(*) AS ?selectorCount) WHERE {
?rule ex:selector ?selector . } GROUP BY ?rule }
FILTER (?selectorCount = ?selectorsPassed) # ALL selectors
} }
ORDER BY ?message ?evidences ?ruleHash
(abridged — see the file for the runnable whole). Expected: two rows,
"confound-sensitive" firing on OBS-2214 and "underpowered" on
OBS-2214 + RNC-0859.1, each with a log like
confound(confound_selector0(OBS-2214, …or0(…and0∧…)∨…or1(…))).
Both compiled forms agree across both evaluation modes.
# Told bnodes (BNodeResolver)
lib/BNodeResolver.{hpp,cpp} emulates "told bnodes" (rejected in SPARQL
1.0): executing over a SPARQL endpoint, each response blank node is
identified by a minimal uniqueness-verified fragment of in/out arcs
(found by instrumenting the query with interrogation round trips,
recursing through neighbor bnodes) and interned to a stable proxy that
later queries can mention. RemoteGraphProvider caches fetched triple
patterns with subsumption and feeds both ShEx validation over an
endpoint (Validator::NeighborhoodSource) and remote Algae 3 execution
(Engine::remote). The algorithm is written up in the algae3 repo,
doc/told-bnodes.md.
Run it:
ctest --test-dir build -R BNodeResolver # or:
cd tests && ../build/test_BNodeResolver
The suite runs everything against a client that freshly relabels every
response bnode, so nothing can pass by label luck. Its flagship case
executes evidence-dnf.topdown.a3 (the query above, whose group
memberships are anonymous bnodes) purely through such a client — no
local load — and requires the exact local golden results, proof logs
included; a second copy of that case drives an in-process HTTP
endpoint through the real protocol client.
`attach` in an .a3 script runs against a live endpoint (bin/algae3
installs an HTTPSPARQLClient factory; plain http — the ASIO agent
doesn't speak TLS). Demo with bin/sparql serving the evidence data
(from tests/):
../bin/sparql --serve http://127.0.0.1:8098/sparql \
-d Algae3/data/evidence.ttl \
-d Algae3/data/evidence-groups.ttl \
-d Algae3/data/evidence-rules.ttl &
sed 's|^load .*|attach <http://127.0.0.1:8098/sparql> ep|' \
Algae3/evidence-dnf.topdown.a3 | awk '!/^attach/ || !seen++' > /tmp/attach.a3
../bin/algae3 /tmp/attach.a3 # == the local results, proof logs included
Endpoint data should use absolute IRIs: relative ones resolve
differently between data load and query parse, so they cannot survive
the protocol round trip (the evidence fixtures carry explicit absolute
bases for exactly this reason).
# More info
To learn more about SWObjects, checkout some tutorials given at SWAT4LS in 2010 and 2011:
2010: http://www.w3.org/2010/Talks/1208-egp-swobjects/
2011: https://sites.google.com/site/swobjectstutorial/documentation