Sync piri, hilt, ingot, sprue and smelt to the polyrepo mains - #6
Draft
Peeja wants to merge 8 commits into
Draft
Conversation
Replaces hilt/ with the upstream tree at fil-forge/hilt f842aba (2026-09-04), transformed the way the original import was: module path github.com/fil-forge/hilt -> github.com/fil-forge/forge/hilt, and the per-repo .github/, Dockerfile and Dockerfile.release dropped (CI and images come from the shared .github/workflows and docker/Dockerfile). Kept from this repo: .dockerignore and .goreleaser.yaml (upstream has neither; the shared image build and release.yml need them), the Makefile's image/image-dev targets (merged with upstream's new itest target), and AGENTS.md's monorepo notes (merged with upstream's edits). Dropped from this repo: the edits PR #3 made to point hilt at libforge's feat/hilt-s3-client branch (sigv4, s3perm, client/hilt, commands/s3/*) and to delete the in-tree originals. Upstream still carries those packages in-tree and builds against libforge main, which never took that branch, so hilt returns to the upstream shape here; the shared copies arrive in the consolidation PR instead. Everything else is upstream's: hilt/ differs from the transformed upstream tree only in the four kept files. To reproduce: git -C <fil-forge/hilt> archive f842aba | tar -x -C tree rm -rf tree/.github tree/Dockerfile tree/Dockerfile.release grep -rlI github.com/fil-forge/hilt tree | xargs perl -pi -e \ 's#github\.com/fil-forge/hilt(?![-A-Za-z0-9_.])#github.com/fil-forge/forge/hilt#g' diff -r tree hilt Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CGAGAib517Ae1kg8SCdcEt
Replaces ingot/ with the upstream tree at fil-forge/ingot 59fcb5e (2026-08-29), transformed the way the original import was: module path github.com/fil-forge/ingot -> github.com/fil-forge/forge/ingot, and the per-repo .github/, Dockerfile and Dockerfile.release dropped (CI and images come from the shared .github/workflows and docker/Dockerfile). Kept from this repo, merged with upstream's own edits: .dockerignore, .goreleaser.yaml, the Makefile's image/image-dev targets, and CLAUDE.md's monorepo notes. Dropped from this repo: PR #3's edits pointing ingot at libforge's feat/hilt-s3-client branch (client/hilt, sigv4, s3perm, commands/s3/*). Upstream imports those from fil-forge/hilt directly and builds against libforge main, which never took that branch; ingot returns to the upstream shape here and the shared copies arrive in the consolidation PR. Two upstream directories return that this repo had relocated: ingot/itest/ (moved to smelt/tests/s3/ by PR #5, and grown upstream by five suites since) and ingot/testing/ (moved to smelt/pkg/s3glue/). The smelt copies stay as they are, so both exist until the consolidation PR reconciles them; ingot's itest/ is behind the itest build tag and does not run under go test ./... Everything else is upstream's: ingot/ differs from the transformed upstream tree only in the four kept files. To reproduce: git -C <fil-forge/ingot> archive 59fcb5e | tar -x -C tree rm -rf tree/.github tree/Dockerfile tree/Dockerfile.release grep -rlI github.com/fil-forge/ingot tree | xargs perl -pi -e \ 's#github\.com/fil-forge/ingot(?![-A-Za-z0-9_.])#github.com/fil-forge/forge/ingot#g' diff -r tree ingot Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CGAGAib517Ae1kg8SCdcEt
Replaces sprue/ with the upstream tree at fil-forge/sprue 5954487 (2026-08-28), transformed the way the original import was: module path github.com/fil-forge/sprue -> github.com/fil-forge/forge/sprue, and the per-repo .github/, Dockerfile and Dockerfile.release dropped (CI and images come from the shared .github/workflows and docker/Dockerfile). Kept from this repo, merged with upstream's own edits: .dockerignore, .goreleaser.yaml and the Makefile's image/image-dev targets. Dropped from this repo: PR #3's edits to pkg/piriclient/client.go and pkg/service/handlers/blob_remove.go, which tracked libforge's feat/hilt-s3-client branch. Upstream carries the same /blob/release Cause change in a further-evolved form against libforge main, so the upstream files are taken as they are. Everything else is upstream's: sprue/ differs from the transformed upstream tree only in the three kept files. To reproduce: git -C <fil-forge/sprue> archive 5954487 | tar -x -C tree rm -rf tree/.github tree/Dockerfile tree/Dockerfile.release grep -rlI github.com/fil-forge/sprue tree | xargs perl -pi -e \ 's#github\.com/fil-forge/sprue(?![-A-Za-z0-9_.])#github.com/fil-forge/forge/sprue#g' diff -r tree sprue Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CGAGAib517Ae1kg8SCdcEt
Replaces piri/ with the upstream tree at fil-forge/piri 804b1f0 (2026-09-04), transformed the way the original import was: module path github.com/fil-forge/piri -> github.com/fil-forge/forge/piri, and the per-repo .github/, Dockerfile and Dockerfile.release dropped (CI and images come from the shared .github/workflows and docker/Dockerfile; piri's docs workflow already lives at .github/workflows/docs-site.yml). Kept from this repo, merged with upstream's own edits: .dockerignore, .goreleaser.yaml, the Makefile's image/image-dev targets, and the monorepo notes in AGENTS.md and CLAUDE.md (upstream rewrote CLAUDE.md wholesale; its text is taken, with the image-build note re-applied). Dropped from this repo: PR #3's three-line edit to pkg/ucanhandlers/ucanfxtest/rpc/blob_release_test.go, which tracked libforge's feat/hilt-s3-client branch; upstream has the equivalent change against libforge main. Everything else is upstream's: piri/ differs from the transformed upstream tree only in the five kept files. To reproduce: git -C <fil-forge/piri> archive 804b1f0 | tar -x -C tree rm -rf tree/.github tree/Dockerfile tree/Dockerfile.release grep -rlI github.com/fil-forge/piri tree | xargs perl -pi -e \ 's#github\.com/fil-forge/piri(?![-A-Za-z0-9_.])#github.com/fil-forge/forge/piri#g' diff -r tree piri Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CGAGAib517Ae1kg8SCdcEt
Replaces smelt/ with the upstream tree at fil-forge/smelt 6ae0dea (2026-09-04), transformed the way the original import was: module paths github.com/fil-forge/smelt -> github.com/fil-forge/forge/smelt (and the nested systems/stress-tester module likewise) and the per-repo .github/ dropped. smelt is where this repo's own work lives, so much more is kept here than for the other services, merged with upstream's edits where both touched a file: the compat suite (tests/compat/), the S3 integration suite and its client glue that PR #5 moved here from ingot (tests/s3/, pkg/s3glue/), the workspace builder's build-tags support (pkg/workspace: piri needs -tags skiff; upstream's alsoBinIn field and its new service entries — signing-service, indexer, delegator, guppy, hilt, ingot, swarf — are taken alongside it), the stack SDK's image/binary plumbing (pkg/stack), the compose and env tweaks under systems/, and the notes in CLAUDE.md and docs/. Two upstream changes are taken over this repo's: the e2e smoke matrix (upstream runs filesystem, s3, postgres and s3_and_postgres; PR #3 had dropped the first two without saying why) and everything in pkg/generate and pkg/manifest. go.mod: upstream's, with two pins this repo needs on top. libforge stays on the feat/hilt-s3-client head (928cf2a) — the only module here that does — because tests/s3 imports libforge/s3perm, which exists nowhere else; and github.com/fil-forge/versitygw is pinned to 7a65883, the revision whose go.mod carries the fork's module path (an unpinned resolve lands on an older revision that still declares itself as github.com/versity/versitygw and fails to load). Both pins go away when the consolidation PR brings s3perm and the S3 wire contract into this repo. To reproduce the upstream side: git -C <fil-forge/smelt> archive 6ae0dea | tar -x -C tree rm -rf tree/.github grep -rlI github.com/fil-forge/smelt tree | xargs perl -pi -e \ 's#github\.com/fil-forge/smelt(?![-A-Za-z0-9_.])#github.com/fil-forge/forge/smelt#g' diff -r tree smelt # differs only in the kept paths listed above Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CGAGAib517Ae1kg8SCdcEt
hilt, ingot, sprue and piri now declare go 1.27.0, so the workspace file and the shared builder image have to follow: go.work moves from 1.26.5 to 1.27.0 (a workspace's go line must be at least that of every module it uses) and docker/Dockerfile builds on golang:1.27-bookworm. smelt is still on 1.25.9 and builds unchanged under both. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CGAGAib517Ae1kg8SCdcEt
TestExpiredEntryIsLiveCheckedBeforeJanitor minted a delegation expiring at ucan.Now()+1 and added it to the cache. Expiry has one-second granularity and Add drops anything whose remaining lifetime is not positive, so if the wall clock crossed a second boundary between minting and Add the entry was never indexed and the first assertion failed with "expected 1, actual 0". That is what happened on the first CI run of this branch (run 33911230776); locally the test passed 60 times in a row because the window is a few milliseconds wide. Minting two seconds out cannot be reached by Add, and the sleep moves to 2.5s so the entry is still past expiry when the test checks it. The test is otherwise upstream's (fil-forge/ingot@59fcb5e); this is the one place this branch's ingot differs from it, and it is worth carrying upstream. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CGAGAib517Ae1kg8SCdcEt
The forge repository moved ingot's Docker-backed system suite to smelt/tests/s3 (and ingot/testing to smelt/pkg/s3glue) in 8d55284: the suite validates the whole stack through ingot's S3 API, so it lives with the harness, and ingot's library go.mod no longer needs the smelt module. The sync of ingot to fil-forge/ingot@59fcb5e in 32f181b took upstream's tree as-is, which resurrected ingot/itest and ingot/testing while leaving forge's moved copies at their import-point content. That left the stack job running a stale suite against a much newer ingot. The first stack run on this branch (33913565946) failed 125 of the S3 suite's cases: 110 XFail rows "unexpectedly passed" — the ratchet's signal that the surface they cover has landed (checksums, conditional reads and writes, ListParts, ListMultipartUploads, and the teardown-blocked rows released by hilt's blob.Remove) — and PutObject/racey_success failed its bucket-delete teardown because upstream now runs it under the versioned teardown. This applies the move to the live content instead. Each file in smelt/tests/s3 and smelt/pkg/s3glue is a three-way merge of forge's copy (ours) with ingot/itest at 59fcb5e (theirs) over the pre-move copy at 96a672e (base), with the move's transformation — package s3test, the s3glue import, smelt's in-repo import path — applied to the live side. Every file merged cleanly except stack_test.go, resolved two ways: the SMELT_STACK_PREBUILT / workspace gating stays (upstream added INGOT_ITEST_HILT_{IMAGE,BINARY}, the same class of escape hatch that gating replaced), and upstream's comment on the multipart permissions joins forge's TestHiltPermissionsCoverS3Perm guard. Five files are new from upstream: the object-lock, tagging and versioning partitions, the deferred-multipart scenario and the delete scenario. ingot/itest and ingot/testing go again; ingot's Makefile gets forge's forwarding itest target back, and its docs point at smelt/tests/s3. smelt gains go-fee, which the merged suite imports, at ingot's pin, and go-multihash as a direct dependency. ingot's go.mod loses the smelt module and everything only it pulled in — itest was its sole importer. forge_multipart_deferred_test.go is gofmt'd (import order). Verified GOWORK=off: smelt and ingot build, vet and tidy clean; the tagged suite compiles (go vet -tags itest ./tests/s3/...); ingot's unit tests pass. The suite itself needs Docker and runs in the stack job. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CGAGAib517Ae1kg8SCdcEt
This was referenced Sep 4, 2026
Peeja
pushed a commit
to fil-forge/forge-2
that referenced
this pull request
Sep 8, 2026
…eration (fil-forge#6) * refactor: route tests through PDP path via in-memory fakes Adds pdpfake — in-memory types.PieceAPI and commp.Calculator fakes plus an fx module that exposes them as pdp.PDP — and switches the three existing fxtest suites to wire them in alongside app.UCANModule. Today the tests are the only consumers of the non-PDP code path: every fxtest setup omits app.PDPModule, leaving pdp.PDP nil in the handler graph, and tests seed bytes via svc.Blobs().Store().Put. Subsequent commits will delete that path entirely; routing the tests through PDP first lets the deletion remain a pure-removal commit. No production code changes. * refactor: drop non-PDP code paths Deletes the bimodal seams: the six `if s.PDP() == nil` branches in handlers (allocate, accept, replica/transfer, pdp_info), the `if params.API != nil` adapter switch in retrieval wiring, and the `optional:"true"` tags on PDP fields in the storage and replicator fx providers. With the non-PDP path gone, the byte-storage surface on the blobs service is no longer reachable: removes Store, Presigner and Access from the blobs.Blobs interface; deletes pkg/service/blobs/server.go (the GET/PUT /blob/:blob handler), pkg/access and pkg/presigner packages, and pkg/fx/presigner. blobs.Blobs now exposes only Allocations and Acceptances. Also retires the deprecated blobstore.PDPStore alias. * refactor: handlers take per-handler Deps structs Replaces the fat view interfaces (AllocateService, AcceptService, TransferService, BlobAllocateService, BlobAcceptService, ReplicaAllocateService, PDPInfoService, AccessGrantService) with per-handler `*Deps` structs populated by fx, each declaring its own narrow consumer-defined interfaces next to the handler: - AllocationStore, PieceAllocator alongside Allocate - AcceptanceStore, PieceReader alongside Accept - PieceResolver alongside pdp/info The Deps structs use fx.In, so wiring stays declarative — pkg/fx/storage gains a handful of identity providers that bind concrete production types (allocationstore.AllocationStore, types.PieceAPI) to the narrow handler-side interfaces, and a NewUploadConnection provider that extracts the upload client.Connection from AppConfig as a top-level bean. The replicator's adapter struct goes away; the Service now constructs a TransferDeps value once and reuses it for the queue handler and the failure receipt. transferBlobFromSource builds an AcceptDeps subset from the TransferDeps to call into blobhandler.Accept. Tests stop populating `var svc storage.Service` and instead populate the narrow types they actually assert against. The umbrella interfaces still exist as composition (storageServiceWrapper) but no longer satisfy any view interface — commit 4 deletes them. * refactor: delete umbrella service interfaces Removes the umbrellas that the per-handler Deps refactor made obsolete: - storage.Service (and the storageServiceWrapper that satisfied it) - pdp.PDP (and TODO_PDP_Impl, ProvideTODOPDPImplInterface) - blobs.Blobs (and the entire pkg/service/blobs + pkg/fx/blobs) The replicator constructor now takes the narrow concrete deps (types.PieceAPI, commp.Calculator, acceptancestore.AcceptanceStore) directly instead of the pdp.PDP and blobs.Blobs umbrellas. The PDP fx module provides types.PieceReaderAPI via fx.As so retrieval's non-optional API field resolves without going through TODO_PDP_Impl. pdpfake's fx module no longer manufactures a pdp.PDP wrapper — it provides types.PieceAPI / types.PieceReaderAPI / commp.Calculator directly, mirroring the production wiring. * test: add direct unit tests for blob handlers Adds table-driven unit tests against Allocate and Accept that operate on the per-handler Deps structs directly, with in-memory implementations (allocationstore + acceptancestore backed by go-datastore MapDatastore, plus pdpfake). The full suite for pkg/service/storage/handlers/blob runs in ~17ms versus ~7s for the fxtest-based integration tests in pkg/service/storage, with coverage at 72.6%. Allocate coverage focuses on the decision tree: - no prior allocation, blob not received → size = requested, address set - allocation exists, blob not received → size = 0, address set - allocation exists, blob received → size = 0, address nil - allocation in another space, blob received → size = requested, address nil - unsupported hash → error Accept coverage: - piece absent → "piece not found" error - piece held → returns Claim + PDP invocation, persists acceptance with pdp/accept promise, persists location claim, calls publisher exactly once Transfer unit tests are intentionally deferred — the handler is mostly HTTP/UCAN glue best covered by the existing fxtest integration tests. * refactor: remove BlobGetter and the retrieval umbrella Applies the same Deps-struct pattern to retrieval that we used for storage. The shape was identical: a fat retrieval.Service umbrella (ID + Allocations + Blobs() blobstore.BlobGetter), a RetrievalService wrapper, and per-handler view interfaces (BlobRetrievalService, SpaceContentRetrievalService) that handlers reached through to call .Blobs().Get(). With those gone, the only thing keeping blobstore.BlobGetter alive was a BlobGetterAdapter wrapping types.PieceReaderAPI to satisfy it. - spacecontent.Retrieve now takes types.PieceReaderAPI directly and calls pieces.Read returning *types.PieceReader. Range errors still propagate as blobstore.RangeNotSatisfiableError through the production piece.StoreReader chain, so the test assertions remain valid. - WithBlobRetrieveMethod and WithSpaceContentRetrieveMethod take new BlobRetrieveDeps / SpaceContentRetrieveDeps structs (fx.In). - pkg/service/retrieval/{interface,service}.go deleted (umbrella + wrapper). - pkg/pdp/store/adapter deleted (the BlobGetterAdapter was its only purpose). - pkg/fx/retrieval/provider.go deleted — the retrieval module had no remaining providers; pkg/fx/retrieval/ucan.Module still wires the UCAN handlers. - blobstore.BlobGetter removed; Get is folded directly into Blobstore. - fx.As(new(blobstore.BlobGetter)) annotations dropped from the memory/s3/filesystem store providers. - Retrieval tests construct piece.NewStoreReader(blobstore...) to produce a PieceReaderAPI from the existing in-memory blobstore — using the production read path keeps RangeNotSatisfiableError assertions accurate. * fix: remove failing test that wrongly depended on old storacha infra * refactor: drop claims/publisher umbrellas and dead Server scaffolds Removes the last bag-style interfaces and a couple of dead scaffolds that were leftover after the storage/retrieval refactor: - claims.Claims (2-getter umbrella over ClaimStore + Publisher). Handlers that used to chain `deps.Claims.Store().Put(...)` and `deps.Claims.Publisher().Publish(...)` now take the two narrow deps directly in their Deps structs. - The whole pkg/service/claims/{interface.go,service.go} pair goes with it (only server.go survives — the /claim/:claim route registrar). - publisher.Publisher.Store() — the interface method had zero callers outside a nil-returning test stub. The PublisherStore is already a top-level fx bean. Removed from the interface, the impl method, and the corresponding field on PublisherService. - pkg/service/storage/server.go `Server` struct and pkg/service/retrieval/server.go `Server` struct were never instantiated. Only their NewHandler peers are used by fx. - pkg/store/claimstore deleted — the package was a single file aliasing delegationstore.DelegationStore with an author-acknowledged "// TODO a glorified type alias, remove this" comment. All callers switched to delegationstore.DelegationStore directly. replicator.New's constructor and the matching fx Params struct now take ClaimStore + Publisher fields instead of the umbrella. * refactor: inline fx modules into their impl packages Moves the parallel pkg/fx/ tree's service-mirror modules into the packages they describe. Each impl package now owns its own fx wiring as fx.go, eliminating the maintenance tax of keeping two parallel trees in sync (which already failed once — pkg/fx/blobs lingered after the impl side was deleted, and we caught pkg/fx/retrieval in the BlobGetter removal). The fx framework is no longer hidden behind a parallel directory; the colocation makes deletions and renames atomic. Moves: - pkg/fx/claims → pkg/service/claims/fx.go - pkg/fx/replicator → pkg/service/replicator/fx.go - pkg/fx/publisher → pkg/service/publisher/fx.go - pkg/fx/proofs → pkg/service/proofs/fx.go - pkg/fx/principalresolver → pkg/principalresolver/fx.go - pkg/fx/wallet → pkg/wallet/fx.go - pkg/fx/storage/ → pkg/service/storage/fx.go - pkg/fx/storage/ucan/{,handlers} → merged into pkg/service/storage/ucan/fx.go - pkg/fx/retrieval/ucan/{,handlers} → merged into pkg/service/retrieval/ucan/fx.go Stays in pkg/fx/ (legitimately framework-only — no impl-package home): app, echo, root, database, store, pdp, scheduler, identity, claimvalidation. The storage/ucan and retrieval/ucan merges fold the former "outer" provider (Handler + Module) and the "inner" handlers Module (per-ability fx.Annotate calls) into a single Module per package. To avoid a parent→child cycle in the storage tree (where pkg/service/storage/fx.go would need pkg/service/storage/ucan.PieceResolver while pkg/service/storage/ucan/fx.go needs storage.NewHandler), the echo handler that wraps the UCAN server moved into the ucan subpackage as an unexported newEchoHandler, and pkg/service/{storage,retrieval}/server.go were deleted. No behavior change. The fx graph composition is identical; the only visible difference is import paths in pkg/fx/app/{ucan,common,pdp}.go. * refactor: supply narrow config sub-structs; collapse replicator constructor Treats the AppConfig-as-context-bag pattern as the same smell that bag interfaces were. Services no longer take app.AppConfig and reach inside for the one or two fields they need — instead, CommonModules supplies each meaningful sub-config struct via fx.Supply, and consumers request just what they actually use. CommonModules additions (pkg/fx/app/common.go): fx.Supply(cfg.UCANService.Services) // ExternalServicesConfig fx.Supply(cfg.UCANService.Services.Upload) // UploadServiceConfig fx.Supply(cfg.UCANService.Services.Indexer) // IndexingServiceConfig fx.Supply(cfg.UCANService.Services.Publisher) // PublisherServiceConfig fx.Supply(cfg.UCANService.Services.EgressTracker) Consumer narrowing: - identity.ProvideIdentity AppConfig → IdentityConfig - principalresolver.NewFx AppConfig → UCANServiceConfig - publisher.NewFx AppConfig → PublisherServiceConfig + IndexingServiceConfig - retrieval/ucan.NewServerHandler Params.Config → Upload UploadServiceConfig - storage handler Deps types (AccessGrantDeps, TransferDeps) replace `UploadConn client.Connection` with `Upload app.UploadServiceConfig`; handler bodies read `.Connection` from the struct. The top-level `client.Connection` provider (storage.NewUploadConnection) is deleted — it existed only to lift one config field into the fx graph as a bare primitive. With the source struct supplied directly, the lift isn't needed. With that done, replicator's two-step constructor (positional New, fx-wrapper NewFx) has no remaining justification. Its only callers were inside the fx wrapper, and its config dep is now a single UploadServiceConfig field. Collapsed into a single fx.In Params-based constructor. Publisher keeps its two-constructor shape: publisher.New(id, store, addr, opts...) is the variadic-options API that publisher_test.go exercises directly with custom options like WithLogLevel. publisher.NewFx wraps it for fx callers — that wrapping is the real config-translation layer and earns its keep. * refactor: replace cliutil.ReadPrivateKeyFromPEM with library method * refactor: trim dead nil checks and narrow remaining AppConfig consumers Cleanup pass after the audit. Two categories of cuts. Dead nil checks. After the claims/publisher umbrella refactor, both blob.Accept and replica.createLocationAssertion unconditionally set their pdp/accept invocation field — the nil checks at three call sites were unreachable: - pkg/service/storage/ucan/blob_accept.go:49 (resp.PDP) - pkg/service/storage/handlers/replica/transfer.go:222 (acceptResp.PDP) - pkg/service/storage/handlers/replica/transfer.go:241 (pdpAcceptInv) Dead provider. pkg/service/egresstracker/fx.go had a ProvideConsolidationStore function that was commented out of fx.Provide (line 35: //ProvideConsolidationStore). Nothing referenced it; consolidation stores come from pkg/fx/store/{memory, filesystem,s3} providers instead. Deleted the function and its leveldb imports. Five remaining AppConfig consumers narrowed to their actual config slice: - ProvideEthClient(cfg AppConfig) → ProvideEthClient(cfg PDPServiceConfig) - ProvideLotusClient(cfg AppConfig) → ProvideLotusClient(cfg PDPServiceConfig) - StartEchoServer(cfg AppConfig, ...) → StartEchoServer(cfg ServerConfig, ...) - ProvideReceiptsClient(cfg AppConfig) → ProvideReceiptsClient(cfg EgressTrackerServiceConfig) - NewEgressTrackerService(... cfg AppConfig) → ServerConfig + EgressTrackerServiceConfig Only CommonModules still takes app.AppConfig — and that's correct, it's the assembly point that explodes the whole thing into the supply tree. * chore: gofmt three files and tidy go.mod CI's go-check job flagged three files as not gofmt-ed and required the aws-sdk-go-v2 dependencies (no longer used after the s3 module cleanup in this branch) to be removed from go.mod via go mod tidy.
Peeja
pushed a commit
to fil-forge/forge-2
that referenced
this pull request
Sep 8, 2026
… (fil-forge#3) (#4) (fil-forge#5) (fil-forge#6) (fil-forge#8) (#9) (#10) (#11) (#12) This PR introduces a new `pkg/store` layer with both in-memory and Postgres-backed implementations for core entities (provider, tenant, bucket, access key, delegation), plus embedded goose migrations and test utilities to run Postgres-backed store tests via testcontainers. **Changes:** - Add store interfaces + in-memory and Postgres implementations for provider/tenant/bucket/accesskey/delegation. - Add embedded goose migrations and a migrations runner to initialize Postgres schema. - Add test utilities and cross-implementation tests (memory + Postgres) for each store. --- Adds a new Go client for interacting with the upload service via UCAN RPC invocations, along with structured logging helpers to make invocation arguments/metadata easier to inspect in logs. **Changes:** - Introduce `pkg/client.UploadClient` with `RegisterCustomer` (`/customer/add`) and `ProvisionSpace` (`/provider/add`) operations. - Add `pkg/lib/zapucan` helpers for logging UCAN invocations and decoding CBOR-encoded IPLD maps into zap fields. - Bump `github.com/fil-forge/libforge` dependency version. --- Adds initial scaffolding for running hilt as a service: a Cobra-based `hilt serve` command that loads config via Viper and starts an Echo HTTP server wired together with Uber Fx. **Changes:** - Introduce Fx modules for config surfacing, zap logging, and Echo server lifecycle management. - Add Viper-based config loading with defaults, env var support, and pflag binding. - Add `cmd/main.go` CLI entrypoint plus basic Makefile/.gitignore and new dependencies (Echo, Cobra, Viper, Fx). --- Adds initial Tenant + Access Key management API scaffolding and wires store backends (memory/postgres) into the Fx app based on configuration, including new config types/flags and Postgres migration hooks. **Changes:** - Introduces API route stubs and request/response types for tenant + access key endpoints, and auto-registers routes onto the Echo server via an Fx group. - Adds storage backend selection (memory vs postgres) in `AppModule`, plus Fx modules to provide the corresponding store implementations. - Expands configuration to include `storage.*` and `storage.postgres.*` settings and flag bindings; standardizes PG unique-violation handling using `pgerrcode`. --- Adds a configurable “partner key” authentication mechanism to gate access to the Tenant API routes, wiring the key through config/flags and enforcing it via Echo middleware. **Changes:** - Introduces `AuthConfig` (`auth.partner_key`) and exposes it through the fx config module. - Wraps all Tenant API routes in partner-key bearer auth middleware while keeping `/` and `/health` open. - Adds new Echo middleware + unit tests for partner-key bearer authentication. --- Adds a KMS-agnostic `vault.Vault` interface plus in-memory and HashiCorp Vault (KV v2) implementations, and wires vault selection/config into the fx app and CLI so Hilt can persist private key material in a configurable backend. **Changes:** - Introduce `pkg/vault.Vault` interface with `ErrNotFound`, plus an in-memory implementation and a HashiCorp Vault (KV v2) implementation (including AppRole auth helper). - Add config schema + viper defaults + CLI flags for selecting/configuring the vault backend, and wire backend selection into the fx app graph. - Add tests for the vault interface contract and HashiCorp AppRole login using a testcontainers Vault dev container. --- Implements tenant provisioning via `PUT /tenants/{tenantId}` by introducing an external tenant identifier, wiring a did:plc directory client, generating/storing tenant keys in Vault, and persisting tenant records. **Changes:** - Added `external_id` to tenant storage model and introduced `GetByExternalID` for idempotent provisioning. - Added PLC directory configuration + fx wiring to provide a `plc.DirectoryClient`. - Implemented `ProvisionTenant` API handler with accompanying tests (memory stores + httptest PLC server). --- Implements the remaining tenant management endpoints by wiring real store-backed handlers and adding the store capabilities required to support tenant deletion cascades (buckets, access keys, delegations, vault secrets, and did:plc deactivation). **Changes:** - Implement `GET /tenants/:tenantId`, `POST /tenants/:tenantId/status`, and `DELETE /tenants/:tenantId` handlers (with did:plc deactivation + cascade cleanup). - Extend tenant/bucket/access-key stores with `Delete` and “list by tenant” capabilities, including pagination for buckets. - Add/extend unit tests for the new store methods and HTTP handlers; bump dependencies to newer `ucantone` and `dag-json-gen`. --- This PR implements the Management API access-key endpoints, including key material storage in Vault and issuing tenant→access-key UCAN delegations based on requested S3 permissions. It also extends the store layer to support access-key expiry and efficient bucket lookups by ID for response rendering. **Changes:** - Implement Create/List/Get/Delete access-key handlers, including Vault secret storage and delegation issuance/revocation. - Add `expires_at` support for access keys across schema, store implementations, and tests. - Extend bucket listing to support filtering by explicit bucket IDs via new `bucket.ListOption`/`bucket.ListConfig`. --- Adds initial scaffolding for Hilt’s UCAN-based RPC surface (used by Ingot for S3 tenant management), including an identity-backed UCAN server mounted on the existing Echo HTTP server. **Changes:** - Introduces UCAN RPC handler stubs and wires them into an `fx` module that builds a `ucantone` HTTP server. - Adds a configurable service identity (PEM-backed or ephemeral) and exposes a public `did:web` DID document endpoint. - Extends configuration/CLI flags to support identity settings.
Peeja
pushed a commit
to fil-forge/forge-2
that referenced
this pull request
Sep 8, 2026
Peeja
pushed a commit
to fil-forge/forge-2
that referenced
this pull request
Sep 8, 2026
Upgrades to the new ucantone version and deals with the fallout. refs fil-forge/smelt#7
Peeja
pushed a commit
to fil-forge/forge-2
that referenced
this pull request
Sep 8, 2026
…oyment feat: forge production deployment
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[From Claude:]
First of two stacked PRs (the second is #7). This one only catches the monorepo up: forge's five service directories are five-week-old snapshots (imported 2026-07-30) and the per-service repos have kept moving, so before consolidating anything the services here are brought to their current upstream
mains. Nothing is consolidated yet — that is #7, which stacks on this branch.What each commit is
One commit per service, one for the toolchain, and two fixes that the first CI runs of this branch asked for:
fa085b7fil-forge/hilt@f842aba(2026-09-04)32f181bfil-forge/ingot@59fcb5e(2026-08-29)ccf722bfil-forge/sprue@5954487(2026-08-28)9d85d24fil-forge/piri@804b1f0(2026-09-04)c450c9ffil-forge/smelt@6ae0dea(2026-09-04)d7adf43go 1.27.0, builder image →golang:1.27-bookworm(four of the five services now declare 1.27.0)892ba70TestExpiredEntryIsLiveCheckedBeforeJanitorminted a delegation expiring one second out, andAdddrops an entry whose remaining lifetime is not positive, so a second boundary between mint andAddfailed the test (it did, on run 33911230776). Two seconds out, 2.5 s sleep. Worth carrying upstream.aa535b0smelt/tests/s3,smelt/pkg/s3glue) re-derived from ingot'sitestat59fcb5e; the resurrectedingot/itestandingot/testingdropped again. See the ingot bullet below.Each sync is a three-way merge, not a copy: base = the import point after the module-path rewire (
18f05ac), ours = this repo, theirs = the upstream tree transformed the same way the original import was (module pathgithub.com/fil-forge/<svc>→github.com/fil-forge/forge/<svc>; per-repo.github/,Dockerfile,Dockerfile.releasedropped). Resolution policy, applied uniformly: upstream wins for code andgo.mod/go.sum; this repo keeps its infrastructure —.dockerignore,.goreleaser.yaml, the Makefileimage/image-devtargets, monorepo notes inAGENTS.md/CLAUDE.md, and, in smelt, everything that only exists here (below). Every sync commit message carries the exact command to reproduce its upstream side; the service directory differs from the transformed upstream tree only in the kept files it lists.How to review this
The five syncs: not line by line — the service diffs are five weeks of upstream history. Per commit: check the upstream SHA is the one you expect, run the reproduce command from the message, and read the residual diff (the kept files) — that residual is the whole review surface. The last two commits are hand edits and small; read those normally.
What this repo had that is deliberately not kept
PR #3 ("Refactor/drop replaces") moved hilt's S3 client,
sigv4,s3perm,zapucanand the/s3/*wire contract into libforge'sfeat/hilt-s3-clientbranch (libforge PR #52) and pointed hilt and ingot at that branch. Upstream never took that branch: live hilt still carries those packages in-tree, live ingot imports them fromfil-forge/hiltdirectly, and both build against libforgemain. Those edits are reverted here so both services return to the upstream shape; the same extraction is redone in #7 against this repo's own shared modules, which is where decision 9 (libforge PR #52 is obsolete as a libforge target) says it belongs. The three sprue/piri code hunks from that PR are also dropped — upstream carries the same/blob/releaseCausechange in a further-evolved form.Things worth knowing, per service
integration/→itest/, restructuredpkg/client/(admin.go,management/,upload/), and gained a revocation feature. Its Makefile now has both upstream'sitesttarget and this repo's image targets.itest/andtesting/, which this repo had moved tosmelt/tests/s3/andsmelt/pkg/s3glue/(PR Refactor/drop replaces #3/Ci/stack #5: the suite validates the whole stack, so it lives with the harness, and ingot's librarygo.modstops needing the smelt module). The sync brought both directories back, and the stack tier then ran the stale smelt copy against a five-weeks-newer ingot: 110 XFail rows "unexpectedly passed" (the suite's own signal that the surface they cover — checksums, conditional reads/writes,ListParts,ListMultipartUploads, the teardown-blocked rows released by hilt'sblob.Remove— has landed) andPutObject/racey_successfailed its teardown because upstream now runs it under the versioned teardown.aa535b0applies the move to the live content instead: each smelt file is a three-way merge of forge's copy with ingot'sitestat59fcb5eover the pre-move copy (96a672e), the five new upstream suites (object lock, tagging, versioning, deferred multipart, delete) come along, andingot/itest,ingot/testinggo again. ingot'sgo.modloses thefil-forge/smeltrequirement (itest was its only importer) and its Makefile'sitesttarget forwards tosmelt test-s3again, as it did before the sync.CLAUDE.mdwholesale; its text is taken and this repo's image-build note re-applied toAGENTS.mdwhere the anchor still exists.tests/compat/,tests/s3/+pkg/s3glue/, the workspace builder'sbuildTagssupport (merged with upstream's newalsoBinInfield and its new service entries), the stack SDK's image/binary plumbing, compose/env tweaks, docs. Two upstream changes are taken over this repo's: the e2e smoke matrix (upstream runsfilesystem,s3,postgres,s3_and_postgres; PR Refactor/drop replaces #3 had dropped the first two without saying why — if they fail in the stack tier that is signal, not noise), and everything inpkg/generate/pkg/manifest. Two pins in smelt'sgo.modare this repo's, not upstream's: libforge stays on928cf2a(the PR #52 head —tests/s3importslibforge/s3perm, which exists nowhere else; smelt is the only module still on that branch), andfil-forge/versitygwis pinned to7a65883, the revision whosego.modcarries the fork's module path (an unpinned resolve lands on an older revision still declaring itselfgithub.com/versity/versitygwand fails to load). Both pins disappear in Consolidate libforge and the satellite services into the repository #7. The merged suite also addsfilecoin-project/go-fee(at ingot's pin) and makesgo-multihasha direct dependency.New facts this surfaces (not changed here, but you should know)
fil-forge/hilt(its client,sigv4,s3perm, the auth/bucket wire types), live hilt requiresfil-forge/smelt(the stack harness, for its itest), and ingot requiresfil-forge/indexing-service. After this PR those resolve to the published external modules, not to the in-repo copies sitting next to them. Consolidate libforge and the satellite services into the repository #7 rewires them.fil-forge/swarf— a UCAN revocation service — is required by both hilt (8 files) and ingot (2). Forge has never seen it; smelt's workspace table already lists it as a stack service. It is treated as an external dependency in this and the next PR. Whether it belongs in the monorepo is a decision nobody has been asked yet.81372e7(hilt),c9252ac(ingot),3e6895b(sprue, piri),928cf2a(smelt); ucantone at25cf834everywhere except smelt (ccb7705). Workspace mode resolves each to the highest;GOWORK=off(CI) uses each module's own. Consolidate libforge and the satellite services into the repository #7 removes libforge entirely and unifies ucantone.Verification
Per module,
GOWORK=off:go build ./...,go vet ./...,go mod tidyleavesgo.mod/go.sumunchanged,go test ./...(piri with-tags skiff,CGO_ENABLED=0, as CI does). smelt's kept suites also vet under their tags (e2e,compat,itest). Workspace-modego build ./...passes for every module under the bumpedgo.work.Two test packages could not pass locally for lack of a Docker daemon and pass in CI:
piri/pkg/store/objectstore/minio(itsTestMainpanics rather than skipping without Docker — upstream behaviour) andsmelt/pkg/stack's three image-build tests.CI on this branch: the unit tier is green on every module. The stack tier's first run (33913565946) built all four images and passed the e2e smoke, then failed the S3 suite for the reason described under ingot; it re-runs on
aa535b0— see the checks.Not in this PR
libforge dissolution, the shared modules, the satellites (delegator, piri-signing-service, indexing-service), the client consolidation, CI changes — all in #7. This one is meant to be boring: same services, current upstream code, this repo's infrastructure preserved.
🤖 Generated with Claude Code
https://claude.ai/code/session_01CGAGAib517Ae1kg8SCdcEt