build(core): point the prod build at audius-mainnet-beta — *DO NOT MERGE* - #553
Open
rickyrombo wants to merge 1 commit into
Open
build(core): point the prod build at audius-mainnet-beta — *DO NOT MERGE*#553rickyrombo wants to merge 1 commit into
rickyrombo wants to merge 1 commit into
Conversation
DO NOT MERGE. Genesis is compiled in, so merging this and shipping it as stable IS the mass migration -- every node that pulls it leaves the current chain. The point is the image CI builds from the branch, which the bootstrap nodes run during Runbook steps 4-10. prod.json is the verified 2026-08-25 artifact: chain audius-mainnet-beta, one validator at power 100, migration end height 5839. ProdPersistentPeers is the bootstrap alone. The node key is the comet key (setup.go:105), so a node's P2P id is its validator address -- the bootstrap's id is its genesis validator address, lowercased. That also means the bootstrap must not inherit a node_key.json from the writer output, or its id will not match; the one in the artifact belongs to the verification node. v.audius.rickyrombo.com is a TODO: it does not exist yet, and until it does the second node reaches the network by dialling the first. ProdStateSyncRpcs names both bootstrap hosts. CometBFT refuses to state sync from fewer than two, so both must serve before any node joins. The two prod peer-list guards assert properties of a mature network -- twenty peers, and a per-node selection that avoids a gossip hub. Neither can hold for a chain with one node. They now step aside only when the embedded genesis has a single validator, and were confirmed to still fail against the nine-validator genesis.
rickyrombo
added a commit
that referenced
this pull request
Aug 28, 2026
Every step that cannot be executed without a code change now names it, so the dependency is visible where the work happens rather than only in a list at the top: #553 for the binary (step 3), api#1029 for play routing (5 and 13), #551 before the fleet state syncs (10), api#1018 before flushing (11), api#1028 for the indexer bounds (12). Also records two things found while deriving the bootstrap's identity for #553: the node key is the comet key, so a node's P2P id is its validator address and the bootstrap's is derivable before it runs; and the bootstrap must not inherit node_key.json from the writer output, or its id will not match the peer list.
rickyrombo
added a commit
that referenced
this pull request
Sep 8, 2026
…cutover plan (#550) * docs(genesis-writer): seal the artifact DB before starting any service A verify node started against the pristine chain DB with a binary that did not embed the artifact's genesis took its "generate new genesis" path and ran the core migrations down, destroying the writer output. OPENAUDIO_ENV=dev loads the embedded dev.json, so the binary looks normal right up until it drops the tables. Document sealing the artifact with ALLOW_CONNECTIONS false and serving from a TEMPLATE copy, so no connection string can reach the pristine write, and note that artifact verification belongs after services start, not before. * docs(genesis-writer): correct the height check and record the writer's real prerequisites Five fixes from a read-through: - The runbook told the operator to confirm block 5,819. Both verification artifacts end at 5,839, so the check inverted: a healthy chain reads as the dead-chain failure mode. Every prod run ends at a different height, so the check now points at the writer's reported height instead of any literal. - The writer's prerequisites were undocumented. rewards is its FIRST step and needs --core-dsn to the old core chain, the launchpad secrets from env, and the mints file; the destination database must exist and be empty in both Postgres and its CometBFT directory. Anything missing kills a ten-hour run in the first second. - Step 3 had no pass criterion, and parity legitimately reports divergences -- including several where the migration is more correct than the source. Lists the expected set so a reviewer does not flag the migration for being right. - The ~10h writer estimate is unsourced; verification runs took 3h09m and 3h36m. Says to re-derive rather than substituting another machine's number. - Step 13 fell outside the reversibility framing, but it retires the two old-chain state-sync servers and so ends the cheap rollback path. * docs(genesis-writer): replace the runbook with the sequenced cutover plan The previous runbook ordered the work but left the two hardest transitions underspecified: how plays avoid being split across chains while the fleet is migrating, and how the indexer moves chains without duplicating or dropping writes. Sixteen steps now, with the indexer switch pinned to a height L chosen in the future so it does not race the config rollout, and the flusher filtered below L so the boundary is exact rather than inferred. Plays are routed through two old-network nodes for the duration, which keeps them on the chain the indexer is actually reading. Records what each irreversible step costs: 14 is the point of no return, 15 retires the rollback anchors, and jailing stops entirely once the old network is down to ~30 active validators -- after which a halt cannot be recovered by waiting. * docs(genesis-writer): reconcile the reference with the new runbook and with the code Six places where the reference either contradicted the runbook or stated something the code does not do: - The facts table gave prod's validator count as 9. That is the genesis list; the live set is 67 of 72 registered nodes, and the runbook's quorum and jailing math is over the live set. - Section 7 said to leave BlockInterval at its default, which the runbook now overrides with 20,000. Records why the default is wrong here, that it is a producer-side setting, and that retention is Keep x BlockInterval so it cannot be lowered freely. - The state-sync truncation table listed blobs but not uploads, audio_previews or qm_audio_analyses, which are also wiped. Adds them with how each recovers, and notes #551 makes the row moot. - Section 3 asserted the bootstrap validator holds power 100. Its own registration rewrites that to ValidatorVotingPower, 10 on mainnet, because the writer never seeds core_registered_nodes. - The cursor discussion now points at api#1018, which is open and must land before flushing is first enabled. - Section 11 omitted the chain-aware fallback to core_indexed_blocks, which is only reached on ErrNoRows and so never runs on a database that has indexed the old chain. That is why the failure is a silent stall. * docs(genesis-writer): drop the genesis-vs-key startup refusal, it would block the rollout The reference suggested refusing to start when the validator key disagrees with genesis. New-chain genesis lists one validator and every other node joins by registration, so that check would stop the entire fleet from ever reaching the new chain -- it is the rollout mechanism, not a fault. It also named ensureGenesisFiles, which does not exist. The genesis-file handling is setup.go:114-126 and compares nothing, correctly. The check worth having is already there: ensurePrivValidator compares the key file against the key derived from the delegate key and refuses only when prior signing history makes a mismatch a double-sign risk. The T7 node that motivated the suggestion was behaving correctly -- a node whose key is not in genesis block-syncs and does not propose. It read as a dead chain only because it was the sole node. Replaces the recommendation with a diagnostic: log when the key is absent from the genesis set and say the node will not propose until registered. * docs(genesis-writer): reconcile the persistent-peers guidance with the new binary Reference SS5 said the bootstrap node would dial old-network hosts and should suppress them with OPENAUDIO_PERSISTENT_PEERS, no release needed. That assumed the node runs the existing binary with only genesis swapped. Runbook step 3 builds a binary whose ProdPersistentPeers is already the new bootstrap nodes, so there are no old-network hosts in the list and nothing to suppress. Outbound dialing of the old network does not happen on the step 3 binary. Inbound does, and no env var here changes it: old nodes hold this host in their address books and keep dialing until those entries age out. Those connections are rejected at the handshake on the network mismatch. The env override at step 4 is still worth setting, but for the narrower reason now stated there: the second bootstrap node does not exist until step 7, so the baked list would otherwise churn against a host that is not up. * docs(genesis-writer): clear the indexer bounds after the cutover Neither bound is consumed by the ETL; both are re-read on every startup. Left set, a start height makes each restart re-index from it and duplicate plays, and an end height stalls the indexer at L permanently. * docs(genesis-writer): add a TL;DR table of contents Sixteen steps across three phases is more than fits in someone's head, and the irreversible ones are buried at the end of a long document. One table, the three failure modes that actually bite, and the PRs that must merge first. * docs(genesis-writer): cite the PRs each step depends on Every step that cannot be executed without a code change now names it, so the dependency is visible where the work happens rather than only in a list at the top: #553 for the binary (step 3), api#1029 for play routing (5 and 13), #551 before the fleet state syncs (10), api#1018 before flushing (11), api#1028 for the indexer bounds (12). Also records two things found while deriving the bootstrap's identity for #553: the node key is the comet key, so a node's P2P id is its validator address and the bootstrap's is derivable before it runs; and the bootstrap must not inherit node_key.json from the writer output, or its id will not match the peer list. * docs(genesis-writer): record what the runbook alone did not carry Someone reading only this document would have hit each of these cold: - how to actually run the replay and parity, including that --db must be an ETL database and the replay must point at the serve copy, not the sealed original - calibration figures from the 2026-08-25 run, framed as sanity checks rather than expected values, since a fresh snapshot changes all of them - the row-level parity mismatches that are not in the divergence table: genre re-casing, musical_key dropped by the allowlist, and track_downloads disagreeing on parent_track_id for deleted tracks. All pre-existing ETL behaviour; every missing row should reconcile to a listed cause - why plays need routing at all: logTrackListen runs when a node serves audio, and plays never enter the relay queue, so a migrated node writes them to a chain nobody is indexing - the arithmetic behind ten-at-a-time: eligibility is ~4.5h of proposing nothing, not warden throughput, and attestation quorum never binds before the chain has already halted - that the routine DP backup is not restorable as taken, and that user_balance_history is ~85 GB the migration never reads * docs(genesis-writer): restructure into overview, steps, appendix The document had grown to where the instructions and the reasoning were the same text, and an operator working through it had to read an argument to find an action. Three parts now. The overview says what the migration is and why it is awkward, with a linked table of contents. The steps are imperative -- 22 to 110 words each, pointing at the appendix rather than explaining inline. The appendix holds the reasoning: nine sections keyed to the steps that need them, then the existing reference material. No content dropped. Every explanation that was inline is now an appendix section with an anchor, and all 26 cross-links resolve. * docs(genesis-writer): lead with why the migration exists The overview described the mechanism without saying what it is for. audius-mainnet-alpha-beta began part-way through the protocol's life, so most of the network's history -- the majority of users, tracks, playlists, follows, saves, reposts and plays -- is not on it and lives only in the application database. Reconstructing state from that chain alone yields a fraction of it. That is the reason the new chain is written rather than synced: syncing would carry the same gap forward. It is also why verification dominates step 1, since the artifact is the history and there is no second source once it ships. Also drops 'Discovery Provider' for 'production snapshot', per the terminology note in CLAUDE.md. * docs(genesis-writer): make the PR references clickable Thirteen references across the document were plain text, so following one meant knowing which repo it belonged to. api#N now resolves to AudiusProject/api and bare #N to OpenAudio/go-openaudio, which is also the disambiguation the plain text was silently relying on.
This was referenced Sep 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
⛔ DO NOT MERGE
Genesis is compiled into the binary (
//go:embed prod.json). Merging this and shipping it asstableis the mass migration — every node that pulls it leaves the current chain. This branch exists for the image CI builds from it, which the bootstrap nodes run during Runbook steps 4–10.*DO NOT MERGE*sits at the end of the title becausepr-title-lintrequires a conventional-commit prefix; it would reject the marker up front.What changed
pkg/core/config/genesis/prod.json— the verified 2026-08-25 artifact:ProdPersistentPeers— the bootstrap alone:The node key is the comet key (
p2p.NodeKey{PrivKey: envConfig.CometKey},setup.go:105), so a node's P2P id is its validator address. The bootstrap's id is therefore its genesis validator address, lowercased — derivable without the node running.Two consequences worth stating plainly:
node_key.jsonfrom the writer output. The one sitting in the artifact (56e4f272…) belongs to the verification node, which generated it on first start. Seeding it would give the bootstrap a P2P identity unrelated to its delegate key and unrelated to this entry. Delete it and let the node derive its own.v.audius.rickyrombo.comis a TODO. It does not exist yet, so its id cannot be known. Until it does, the second node reaches the network by dialling the first, and PEX propagates from there. This must be filled in before the fleet rolls (step 15).ProdStateSyncRpcs— both bootstrap hosts. CometBFT refuses to state sync from fewer than two servers, so both must be serving before any node tries to join.The test changes are the interesting part
Two existing guards failed, correctly:
Both assert properties of a mature network — many peers, and a per-node selection that keeps gossip off a single hub. Neither can hold for a chain with one node in it, which is what a chain being born is.
Rather than weaken them, they now step aside only when the embedded genesis has a single validator, and stay in force otherwise. Verified by swapping the nine-validator genesis back in: both fail again exactly as before. So the guard still protects the established network, and re-tightens on its own once validators register and the list is repopulated.
Not included
ProdPersistentPeersdeliberately does not carry the old-network hosts. That means a node on this build does not dial the old chain at all — see Reference §5 in the runbook, which used to say otherwise and was corrected in #550.