Refactor/drop replaces - #3
Merged
Merged
Conversation
frrist
force-pushed
the
refactor/drop-replaces
branch
from
July 31, 2026 18:45
4677ccf to
4e87ca9
Compare
The Hilt S3 RPC client, sigv4, s3perm, zapucan, and the /s3/* wire contract (named error sentinels, Operation/ClassifyRequest) moved to libforge, next to the commands/s3 bindings they are built from. Hilt keeps the server side: the Authorizer and bucket services, stores, vault, plus the hilt-internal AdminClient and Sprue UploadClient. This is the first half of removing ingot's replace directives: with the contract in libforge, ingot no longer needs to import the hilt module at all. hilt/go.mod still pins the pre-move libforge; it is bumped when the libforge change lands (until then, builds resolve via the workspace). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…stead ingot's hilt surface (client, sigv4, s3perm, auth/bucket wire contract) now lives in libforge, so the require + replace of the hilt module goes away. This is half of what makes the ingot library consumable outside the repo: a downstream 'go get' no longer resolves a nonexistent hilt@v0.0.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
itest imports smelt's stack harness, and go.mod cannot mark a dependency as test-only — so as long as itest lived in the library module, ingot required the unpublished smelt module and needed a replace directive to resolve it. itest/ is now its own leaf module. Nothing imports a test module, so its two replace directives (ingot, smelt) are harmless — they never appear in any consumer's build. ingot's library go.mod is left with no intra-repo requires at all: with the hilt surface consumed from libforge (previous commit), both replace directives are gone and the library resolves like any published module. The itest module is added to go.work, and 'make itest' now enters itest/. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With ingot's replace directives gone, no Dockerfile needs a sibling module in its build context. Revert the repo-root-context machinery from the container work: piri/hilt/sprue Dockerfiles go back to their service-directory form, ingot's drops the hilt/smelt manifest and source copies, publish-ghcr.yml builds each service with context <service>/, and ingot no longer rebuilds on hilt or smelt changes. The root .dockerignore is gone (nothing builds from the root); each service's own .dockerignore applies again, now also ignoring goreleaser's dist/. smelt's BuildImage helpers revert to their one-directory-per-service shape: BuildPiriImage(t, "../piri") builds piri from its own directory, same as the out-of-repo services. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All six modules (hilt, ingot, ingot/itest, piri, smelt, sprue) now require libforge v0.0.0-20260731172903-928cf2a21b7e — the fil-forge/libforge#52 head — so the whole repo agrees on one libforge again, and hilt/ingot resolve the extracted client/contract packages without any workspace. hilt and ingot container images build standalone for the first time since the extraction. The bump crosses libforge's breaking change #50 (blob-release Cause), which this commit adopts: - sprue: /blob/release now carries Cause — the task link of the client /blob/remove it translates — and ships the remove invocation in the request container so the node can verify the release's origin. The remove handler threads both through forwardBlobRelease. - piri: release-handler tests construct wire-valid arguments (Cause set, following the RandomCID precedent the allocate tests use). The node-side verification #50 calls for (reject UnknownCause/InvalidCause) is not yet implemented — piri still accepts releases without checking the cause chain. After PR#52 merges, re-pin to the merged commit (or the next libforge tag) so go.mod references a commit on main rather than a PR head. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The itest suite never tested ingot-the-module — it boots the whole Forge stack and drives it through ingot's S3 API, with the working-tree ingot binary mounted by the harness. It is a system suite, and it now lives with the system: the S3-client glue (former ingot/testing) becomes smelt/pkg/s3glue, and the conformance partition + scenarios (former ingot/itest) become smelt/tests/s3, inside smelt's own module next to where the compat suites are headed. This deletes the nested ingot/itest module and with it the last replace directives in the repository. Both of its imports (pkg/stack, the glue) are now same-module, so there is nothing left to resolve by path: every go.mod in the repo is resolvable by any consumer, and in-repo HEAD-vs-HEAD testing is preserved because tests and harness share a module and the service under test arrives as a working-tree binary, not an import. Cost, accepted deliberately: smelt's module graph gains aws-sdk-go-v2 and the versitygw fork (go.mod cannot mark test-only deps). smelt's consumers are themselves test suites, so graph leanness buys them nothing. ingot's 'make itest' forwards to smelt's new 'make test-s3'; docs updated to the new division: out-of-repo services keep behavior tests in their repos (pseudo-version imports of pkg/stack), in-repo system suites live in smelt/tests/. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two tiers over one commit, mirroring the old per-repo split:
unit per-module, path-filtered, GOWORK=off, CGO_ENABLED=0 (piri adds
-tags=skiff). Each filter is just the service's own subtree — no
module imports another, so nothing else can break its compile.
stack one run per PR: smelt's e2e smoke suite, then the S3 gateway system
suite (smelt/tests/s3), which compiles the working tree's ingot and
bind-mounts it over the published image. Sequential in one job: the
suites sweep smeltery-* containers, so two stacks must never share
a Docker host. Container logs are dumped and uploaded on failure.
Also fixes a latent path bug from the itest move: localIngotBinary built
from '..' (correct when the suite lived at ingot/itest, wrong from
smelt/tests/s3). Verified end-to-end by running TestForgeNativeProvision
against a real stack boot: working-tree binary mounted, hilt tenant
provisioned, 512KiB round-trip through sprue/piri.
The stack job pulls ghcr.io/fil-forge/forge/<svc>:main for the sibling
services, so it needs the container-build changes merged to main (which
publishes those tags) before it can go green on a PR.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
frrist
force-pushed
the
refactor/drop-replaces
branch
from
July 31, 2026 19:18
4e87ca9 to
28f9272
Compare
…exists The only committed snapshot (3-piri-filesystem-sqlite) captures a sqlite-backed piri, and piri's curio PDP pipeline now refuses sqlite — the fixture can never boot healthy again. Skip with the unskip condition stated: capture and commit a postgres-backed snapshot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each e2e subtest boots a complete Forge stack. Without -parallel 1 Go runs them GOMAXPROCS-wide, and two simultaneous stack boots overwhelm the 2-vCPU runner: a container exec misses its readiness handshake and compose fails with runc's 'procReady not received'. Same guard ci/stack's ci.yml already carries; SMELT_E2E_MAX_PARALLEL stays as the escape hatch for beefier hosts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The six modules had drifted across five different go directives (1.24.0 to 1.26.4), which just failed PR CI in a subtle way: the stack job installs the toolchain from smelt/go.mod (1.25.7), setup-go pins GOTOOLCHAIN=local, and the S3 suite's in-test 'go build ./cmd/ingot' then refuses — ingot requires >= 1.26.4 and auto-download is disabled. All modules now declare go 1.26.5 (the latest release, and what go.work and the shared builder image already use), so whichever go.mod a workflow reads, the installed toolchain satisfies every module. Same rationale as the dependency unification: one commit should hold one view of the toolchain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The stack tier's whole point — every service at this commit — surfaced a drift the floating-baseline runs had been masking: hilt now classifies multipart operations distinctly and requires s3:AbortMultipartUpload, s3:ListMultipartUploadParts and s3:ListBucketMultipartUploads, but the test tenant's access key list predated them. Hilt rejected those operations as OperationNotPermitted before bucket/argument validation, so pass-table cases expecting NoSuchBucket / InvalidArgument saw AccessDenied. Adds the three permissions and a lockstep guard (TestHiltPermissionsCoverS3Perm): every listed permission must be one s3perm recognizes, and every classified operation's required permission must be in the list — so the key can never silently lag the permission model again. Verified against a live stack: the ListMultipartUploads, AbortMultipartUpload and CompleteMultipartUpload categories all pass. Also bumps sprue's (branch-local, soon-deleted) Dockerfile builder to golang:1.26 — the go 1.26.5 unification made the 1.25 base unable to compile. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Peeja
pushed a commit
to fil-forge/forge-2
that referenced
this pull request
Sep 8, 2026
… (fil-forge#3) (#4) (fil-forge#5) (fil-forge#6) (fil-forge#8) (#9) (#10) (#11) (#12) This PR introduces a new `pkg/store` layer with both in-memory and Postgres-backed implementations for core entities (provider, tenant, bucket, access key, delegation), plus embedded goose migrations and test utilities to run Postgres-backed store tests via testcontainers. **Changes:** - Add store interfaces + in-memory and Postgres implementations for provider/tenant/bucket/accesskey/delegation. - Add embedded goose migrations and a migrations runner to initialize Postgres schema. - Add test utilities and cross-implementation tests (memory + Postgres) for each store. --- Adds a new Go client for interacting with the upload service via UCAN RPC invocations, along with structured logging helpers to make invocation arguments/metadata easier to inspect in logs. **Changes:** - Introduce `pkg/client.UploadClient` with `RegisterCustomer` (`/customer/add`) and `ProvisionSpace` (`/provider/add`) operations. - Add `pkg/lib/zapucan` helpers for logging UCAN invocations and decoding CBOR-encoded IPLD maps into zap fields. - Bump `github.com/fil-forge/libforge` dependency version. --- Adds initial scaffolding for running hilt as a service: a Cobra-based `hilt serve` command that loads config via Viper and starts an Echo HTTP server wired together with Uber Fx. **Changes:** - Introduce Fx modules for config surfacing, zap logging, and Echo server lifecycle management. - Add Viper-based config loading with defaults, env var support, and pflag binding. - Add `cmd/main.go` CLI entrypoint plus basic Makefile/.gitignore and new dependencies (Echo, Cobra, Viper, Fx). --- Adds initial Tenant + Access Key management API scaffolding and wires store backends (memory/postgres) into the Fx app based on configuration, including new config types/flags and Postgres migration hooks. **Changes:** - Introduces API route stubs and request/response types for tenant + access key endpoints, and auto-registers routes onto the Echo server via an Fx group. - Adds storage backend selection (memory vs postgres) in `AppModule`, plus Fx modules to provide the corresponding store implementations. - Expands configuration to include `storage.*` and `storage.postgres.*` settings and flag bindings; standardizes PG unique-violation handling using `pgerrcode`. --- Adds a configurable “partner key” authentication mechanism to gate access to the Tenant API routes, wiring the key through config/flags and enforcing it via Echo middleware. **Changes:** - Introduces `AuthConfig` (`auth.partner_key`) and exposes it through the fx config module. - Wraps all Tenant API routes in partner-key bearer auth middleware while keeping `/` and `/health` open. - Adds new Echo middleware + unit tests for partner-key bearer authentication. --- Adds a KMS-agnostic `vault.Vault` interface plus in-memory and HashiCorp Vault (KV v2) implementations, and wires vault selection/config into the fx app and CLI so Hilt can persist private key material in a configurable backend. **Changes:** - Introduce `pkg/vault.Vault` interface with `ErrNotFound`, plus an in-memory implementation and a HashiCorp Vault (KV v2) implementation (including AppRole auth helper). - Add config schema + viper defaults + CLI flags for selecting/configuring the vault backend, and wire backend selection into the fx app graph. - Add tests for the vault interface contract and HashiCorp AppRole login using a testcontainers Vault dev container. --- Implements tenant provisioning via `PUT /tenants/{tenantId}` by introducing an external tenant identifier, wiring a did:plc directory client, generating/storing tenant keys in Vault, and persisting tenant records. **Changes:** - Added `external_id` to tenant storage model and introduced `GetByExternalID` for idempotent provisioning. - Added PLC directory configuration + fx wiring to provide a `plc.DirectoryClient`. - Implemented `ProvisionTenant` API handler with accompanying tests (memory stores + httptest PLC server). --- Implements the remaining tenant management endpoints by wiring real store-backed handlers and adding the store capabilities required to support tenant deletion cascades (buckets, access keys, delegations, vault secrets, and did:plc deactivation). **Changes:** - Implement `GET /tenants/:tenantId`, `POST /tenants/:tenantId/status`, and `DELETE /tenants/:tenantId` handlers (with did:plc deactivation + cascade cleanup). - Extend tenant/bucket/access-key stores with `Delete` and “list by tenant” capabilities, including pagination for buckets. - Add/extend unit tests for the new store methods and HTTP handlers; bump dependencies to newer `ucantone` and `dag-json-gen`. --- This PR implements the Management API access-key endpoints, including key material storage in Vault and issuing tenant→access-key UCAN delegations based on requested S3 permissions. It also extends the store layer to support access-key expiry and efficient bucket lookups by ID for response rendering. **Changes:** - Implement Create/List/Get/Delete access-key handlers, including Vault secret storage and delegation issuance/revocation. - Add `expires_at` support for access keys across schema, store implementations, and tests. - Extend bucket listing to support filtering by explicit bucket IDs via new `bucket.ListOption`/`bucket.ListConfig`. --- Adds initial scaffolding for Hilt’s UCAN-based RPC surface (used by Ingot for S3 tenant management), including an identity-backed UCAN server mounted on the existing Echo HTTP server. **Changes:** - Introduces UCAN RPC handler stubs and wires them into an `fx` module that builds a `ucantone` HTTP server. - Adds a configurable service identity (PEM-backed or ephemeral) and exposes a public `did:web` DID document endpoint. - Extends configuration/CLI flags to support identity settings.
Peeja
pushed a commit
to fil-forge/forge-2
that referenced
this pull request
Sep 8, 2026
- Link directly to the relevant RFC sections for easy checking: RFC 3394 §2.2.3.1 (default IV) and §4 (test vectors), plus RFC 3394/5649 references in the package doc. - Document that only the base RFC 3394 mode is supported (single fixed initial value), not the RFC 5649 padded variant / Alternative Initial Value, which is what would allow non-block-aligned key-data lengths. - Name the RFC 3394 round count as a `rounds` constant rather than a literal 6. No behavior change; aeskw and wrap tests still pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fzupq5mPYzBGqkbUGabJyP
Peeja
pushed a commit
to fil-forge/forge-2
that referenced
this pull request
Sep 8, 2026
Review fact-check on PR fil-forge#3: - The Alternative Initial Value is introduced by RFC 3394 itself (§2.2.3.2), which anticipates IVs for key data that "may not always be a multiple of 64 bits" and defers concrete definitions to future work; RFC 5649 is one such, defining an AIV with a message-length indicator for padded wrapping. Correct the package doc, which had credited RFC 5649 with introducing the AIV. - Add a comment noting the nested loop performs rounds*n = 6n block encryptions total, matching the RFC's single-index t = 1..6n form (answering a review question about the loop bound; behavior unchanged, RFC 3394 §4 vectors pass). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fzupq5mPYzBGqkbUGabJyP
Peeja
pushed a commit
to fil-forge/forge-2
that referenced
this pull request
Sep 8, 2026
Per review on PR fil-forge#3: both this package and aeskw implement "wrapping", so the bare name `wrap` was ambiguous. Rename fee/wrap -> fee/ecdhkw ("ECDH key wrap", parallel to aeskw = "AES key wrap"); wrap.go -> ecdhkw.go and wrap_test.go -> ecdhkw_test.go. The exported API (Wrap, Unwrap, Wrapped, AlgorithmECDHESA256KW) is unchanged. Also apply the no-leading-colon error convention here, matching aeskw: messages carry the bare package name as a prefix (e.g. "ecdhkw nil recipient public key"), reserving ": %w" for genuine error wrapping. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fzupq5mPYzBGqkbUGabJyP
Peeja
added a commit
to fil-forge/forge-2
that referenced
this pull request
Sep 8, 2026
…256kw-tenant-recipient-wrapunwrap-over-x25519 fee/ecdhkw: ECDH-ES+A256KW tenant recipient wrap/unwrap over X25519 (FIL-471)
Peeja
pushed a commit
to fil-forge/forge-2
that referenced
this pull request
Sep 8, 2026
* wip: migrating to ucan1 service * fix: generate ID for guppy and configure in container * smoll handoff doc * fix: guppy data dir * feat: go.work-based local dev (SMELT_WORKSPACE) replacing SMELT_DEV hack Build services from local sibling checkouts via a Go workspace + binary injection, replacing the branch-local SMELT_DEV / compose.dev.yml / forked Dockerfile approach (which also silently ignored local libforge edits). - pkg/workspace: detect services from the active go.work use-list, compile each from local source (static linux/amd64), and render a compose override that bind-mounts the binaries over the published images. libforge in the use-list forces all services to rebuild. - pkg/stack: WithWorkspaceBinaries() + WithServiceBinary(); the piri-only binary override is generalized to all services. - cmd/smelt: `workspace build` subcommand; Makefile SMELT_WORKSPACE=1 parity for make up/build/fresh via generated/compose/workspace.override.yml. - tests/e2e: opt into workspace binaries when SMELT_WORKSPACE=1. - Remove compose.dev.yml, dockerfiles/, and the generator's dev-mode code. - docs: add docs/DEVELOPING.md; update CLAUDE.md/README/EXTENDING * fix: generate ucan1 proofs via ucantone/libforge; pin workspace build toolchain The Go test stack generated delegation proofs with go-ucanto (ucan 0.9), but the ucan1 services decode them with ucantone, so the delegator failed at startup with "unmarshaling delegation envelope CBOR: cbor input should be of type array" (while `make` worked, since it shells out to `delegator gen`). - pkg/stack/proofs.go: mirror delegator/cmd/gen.go — build delegations with libforge command primitives (claim.Cache / egress.Track), subject = issuer's own DID, and serialize with ucantone delegation.Encode. Drop the per-node piri->upload proofs (obsolete in ucan1; provider registration no longer reads proof files). Adds ucantone + libforge, drops go-ucanto. - pkg/workspace: build workspace binaries with the test's own toolchain (runtime.GOROOT()/bin/go) rather than bare PATH `go`, so a stale system go on an IDE run-config PATH (e.g. GoLand's /usr/bin/go 1.18) can't break the build. - tests/e2e/smoke_test.go: run against workspace-built binaries unconditionally. * feat: switch mkdelegation to ucantool * regenerate snapshot * address feedback * make linter happy --------- Co-authored-by: Alan Shaw <alan138@gmail.com>
Peeja
pushed a commit
to fil-forge/forge-2
that referenced
this pull request
Sep 8, 2026
This is part of reverting the change to send allocate and accept delegations from the client. We decided this should in fact be made with the storage provider as the subject and that the upload service should be pre-authorized by the storage provider to invoke. This PR updates the delegator to expect proofs to be provided by Piri so that the upload service can invoke `/blob/allocate`, `/blob/accept`, `/blob/replica/allocate` and `/pdp/info`. It also adds validation that the required delegations were provided.
Peeja
pushed a commit
to fil-forge/forge-2
that referenced
this pull request
Sep 8, 2026
…il-forge#3) * refactor: migrate signing-service to ucantone + libforge (UCAN 1.0) Server side - pkg/server/server.go is now ~30 lines: ucantone server.NewHTTP + bindexec.NewHandler wrappers for the five capabilities. The old go-ucanto Option chain + WithServiceMethod / Provide pattern is gone. - Handlers (pkg/server/handlers/{access_grant,sign}.go) move to bindexec.HandlerFunc[A, O]. Args come from req.Task().Arguments(); subject validation is "task.Subject() != id.DID()". The /access/grant handler issues a delegation per requested capability (UCAN 1.0 delegations are single-command) and attaches the signed envelopes via res.SetMetadata(container.New(WithDelegations(...))). - pkg/server/handlers/sign.go translates eip712.AuthSignature (fixed- size address/hash fields) into libforge sign.AuthSignature (raw []byte fields) for the receipt OK. Client side - pkg/client/client.go uses ucantone client.HTTPClient and libforge capability bindings. Each method takes proofs []ucan.Delegation so callers can attach a /access/grant delegation; their CIDs ride on the invocation, the envelopes ride on the execution request. - pkg/client/client_test.go wires a ucantone server as an http.Round Tripper for the client and exercises the full encode/decode path without listening on a port. Inprocess + types - pkg/types.SigningService picks up the new proofs []ucan.Delegation parameter; in-process implementation accepts and ignores it (authorization is out-of-band). - pkg/config switches the service signer to ucantone ed25519/signer/ did Parse + Wrap. - main.go wires the ucantone HTTPServer into Echo via echo.WrapHandler; the legacy POST / handler path that streamed UCAN responses through the go-ucanto transport is gone. * rest of the changes for piri * deps: update ucantone and libforge * fix: pass 32-byte seed to ucantone ed25519.FromRaw SignerFromEd25519PEMFile parses a PKCS#8 PEM into a crypto/ed25519 PrivateKey, which is 64 bytes (seed||pub). ucantone's ed25519.FromRaw expects exactly 32 bytes (the seed). Without .Seed(), startup fails with "loading service key from PEM file: invalid length: 64 wanted: 32". * needs did resolution * update to latest libforge * update latest ucantone * feat: make did:web resolution configurable - supporting secure and insecure resolution * refactor: remove unused log var
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Allows all modules to co-exist without replace statements by moving the shared seam of hilt and ingot to fil-forge/libforge#52 while also extracting integration tests from ingot to smelt - allowing repos to remain independent.