Updates

.github/workflows/TestingCI.yml

@@ -74,5 +74,58 @@ jobs:
           restore-keys: ${{ runner.os }}-cargo-
       - name: Build
         run: cargo build --verbose
+      # Serialized like the Linux jobs. The REPL tests spawn the binary as a
+      # subprocess and several tests share fixtures under tests/data/, so
+      # running them in parallel here made macOS and Windows flaky in a way
+      # Linux never showed.
       - name: Run tests
-        run: cargo test --verbose
+        run: cargo test --verbose
+        env:
+          RUST_TEST_THREADS: 1
+
+  lint:
+    name: Format and Clippy
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install Rust toolchain
+        uses: dtolnay/rust-toolchain@stable
+        with:
+          components: rustfmt, clippy
+      - name: Cache dependencies
+        uses: actions/cache@v4
+        with:
+          path: |
+            ~/.cargo/registry
+            ~/.cargo/git
+            target
+          key: ${{ runner.os }}-cargo-lint-${{ hashFiles('**/Cargo.lock') }}
+          restore-keys: ${{ runner.os }}-cargo-lint-
+      - name: Check formatting
+        run: cargo fmt --all -- --check
+      - name: Clippy
+        run: cargo clippy --all-targets -- -D warnings
+
+  msrv:
+    name: Minimum Supported Rust Version
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      # Pinned to the rust-version declared in Cargo.toml. That field claimed
+      # 1.70 for a long time while the dependency tree had moved well past it,
+      # because nothing ever checked.
+      - name: Install Rust toolchain
+        uses: dtolnay/rust-toolchain@1.88
+      - name: Cache dependencies
+        uses: actions/cache@v4
+        with:
+          path: |
+            ~/.cargo/registry
+            ~/.cargo/git
+            target
+          key: ${{ runner.os }}-cargo-msrv-${{ hashFiles('**/Cargo.lock') }}
+          restore-keys: ${{ runner.os }}-cargo-msrv-
+      # Library and binaries only: rust-version is a promise to consumers, and
+      # dev-dependencies are not part of it.
+      - name: Check build at MSRV
+        run: cargo check --verbose

CLAUDE.md

@@ -50,10 +50,22 @@ SQL execution uses a bytecode VM inspired by SQLite's architecture:
 
 ### VM Module (`src/vm/`)
 
-- `bytecode.rs`: Defines bytecode instruction set
-- `compiler.rs`: Compiles SQL AST to bytecode
+- `bytecode.rs`: Bytecode instruction set and register/program types
+- `compiler.rs`: SQL AST to bytecode. Holds `code_expr`, the single expression
+  compiler (the analogue of SQLite's `sqlite3ExprCode`) used from the SELECT
+  list, WHERE, HAVING, ORDER BY and SET clauses alike, plus `NameCtx`, which
+  carries the FROM sources a name can resolve against
+- `compiler_aggregate.rs`: aggregates, GROUP BY, HAVING
+- `compiler_join.rs`: joins
+- `compiler_dml.rs`: INSERT / UPDATE / DELETE
+- `compiler_ddl.rs`: CREATE / DROP / ALTER / TRUNCATE
+- `compiler_window.rs`: window functions
+- `ast_compat.rs`: thin accessors normalizing sqlparser AST shapes
 - `engine.rs`: Executes bytecode instructions
 
+Each statement is compiled and executed separately, so a statement observes the
+effects of the ones before it.
+
 ### File Handling
 
 - **`file_handler.rs`**: Manages loading files into database tables
@@ -62,9 +74,10 @@ SQL execution uses a bytecode VM inspired by SQLite's architecture:
 
 ### SQL Features
 
-- `aggregate.rs`: COUNT, SUM, AVG, MIN, MAX functions
-- `join.rs`: Cross join and INNER JOIN implementations
-- `string_functions.rs`: UPPER, LOWER, TRIM, SUBSTR, REPLACE
+Expression compilation, including all scalar and string functions, lives in
+`src/vm/compiler.rs` behind `code_expr`. Aggregates are compiled in
+`src/vm/compiler_aggregate.rs` and executed by the `AggStep`/`AggFinal`
+opcodes; `src/aggregate.rs` retains only the function-name lookup.
 
 ### Safe Writeback Model
 
@@ -79,4 +92,15 @@ Tests are in `tests/` organized by functionality:
 - `helpers/`: Test utilities
 - `common.rs`: Shared test helpers including REPL script runner
 
+Two suites carry the correctness contract:
+
+- `tests/golden/`: exact-output characterization tests. They assert the
+  COMPLETE stdout -- every row, in order -- via `assert_query`. The rest of the
+  suite uses `predicate::str::contains`, which is substring matching: a test
+  expecting `"Engineering,3"` passes even when extra wrong rows are emitted and
+  regardless of row order. Prefer `assert_query` for new tests.
+- `tests/defects/`: `#[ignore]`d tests encoding CORRECT behaviour for known
+  defects. Run with `cargo test --test mod defects -- --ignored`. Remove the
+  `#[ignore]` when a defect is fixed; the list only shrinks.
+
 Tests use `assert_cmd` for CLI testing and `tempfile` for temporary test data.

Cargo.toml

@@ -10,13 +10,24 @@ homepage = "https://github.com/jgarzik/sqawk"
 readme = "README.md"
 keywords = ["csv", "tsv", "sql", "delimited", "awk"]
 categories = ["command-line-utilities", "text-processing", "parser-implementations"]
-# Minimum version of Rust required
-rust-version = "1.70.0"
-exclude = ["tests/", "doc/", ".github/", "CLAUDE.md"]
+# Minimum supported Rust version.
+#
+# Verified by building with that toolchain, not inferred. The binding
+# constraints are transitive: `home` (via rustyline) and `psm` (via
+# sqlparser -> recursive -> stacker) both require 1.88.
+rust-version = "1.88"
+exclude = [
+    "tests/",
+    "doc/",
+    ".github/",
+    "CLAUDE.md",
+    "generated-icon.png",
+    ".replit",
+]
 
 [dependencies]
 clap = { version = "4", features = ["derive"] }
-sqlparser = "0.36"
+sqlparser = "0.62"
 csv = "1"
 anyhow = "1"
 regex = "1"

README.md

@@ -12,6 +12,11 @@ Sqawk is an SQL-based command-line tool for processing delimiter-separated files
 - **Joins** - INNER, LEFT, RIGHT, FULL OUTER, and CROSS joins with ON conditions
 - **Aggregates** - COUNT, SUM, AVG, MIN, MAX with GROUP BY support
 - **Functions** - String (UPPER, LOWER, SUBSTR, REPLACE, etc.), math (ABS, ROUND, etc.), date/time
+- **Subqueries** - Scalar, `IN (SELECT ...)`, and `EXISTS`, including correlated
+- **Set Operations** - UNION, UNION ALL, INTERSECT, EXCEPT
+- **Window Functions** - ROW_NUMBER, RANK, DENSE_RANK, LAG, LEAD, and aggregates with `OVER (PARTITION BY ... ORDER BY ...)`
+- **DDL** - CREATE TABLE, CREATE TABLE AS SELECT, DROP, ALTER TABLE ADD COLUMN, TRUNCATE
+- **Expressions** - CASE, CAST, COALESCE, NULLIF, BETWEEN, IN, LIKE/ILIKE, `||`, arithmetic
 - **File Formats** - CSV, TSV, and custom delimiters; headerless files via `--tabledef`
 - **Safe by Default** - Files unchanged unless `--write` flag is specified
 - **Interactive REPL** - Explore data interactively with `-i` flag
@@ -38,6 +43,29 @@ sqawk -s "SELECT department, AVG(salary) FROM employees GROUP BY department" emp
 sqawk -s "UPDATE data SET status = 'archived' WHERE year < 2020" data.csv --write
 ```
 
+## `tsq` - test data generator
+
+`cargo install sqawk` also installs `tsq`, which generates deterministic
+multi-table CSV data plus a corpus of SQL queries for exercising sqawk.
+
+```sh
+tsq --seed 42 --rows 1000 --output-dir /tmp/sqawk-test
+sqawk -s "SELECT * FROM customers LIMIT 10" /tmp/sqawk-test/data/customers.csv
+```
+
+It writes `data/` (customers, products, orders, order_items, reviews, with
+realistic foreign-key relationships), `queries/` (numbered `.sql` files
+covering selects, joins, aggregates, subqueries and window functions),
+`verify/run_verification.sh`, and a `metadata.json` recording the seed and row
+counts. The same seed always produces the same data.
+
+| Option | Meaning |
+| --- | --- |
+| `-s`, `--seed` | Seed for reproducible generation; a random one is printed if omitted |
+| `-r`, `--rows` | Base customer row count; other tables scale proportionally (default 100000) |
+| `-o`, `--output-dir` | Where to write the generated tree (required) |
+| `-v`, `--verbose` | Show generation progress |
+
 ## Documentation
 
 - [User Guide](doc/user_guide.md) - Installation, CLI options, and examples