rfc: Forge Instances & Pilot Regions

[bajtos]

📖 Preview

Summary

We want to operate multiple Forge network instances to serve conflicting needs like continuous
deployment vs stability. This document collects different criteria we have for each instance and
proposes a set of instances to stand up and operate.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-21T09:19:01.789356Z	3d48805	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6949892ddf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[bajtos]

We may want to have at least 2 regions in dev, so that we can test multi-region support in central components. Thoughts?

[alanshaw]

Not initially. There is nothing to test(?).

[bajtos]

Ack 👍🏻

[bajtos]

How about verifying whether a single S3 access key can be scoped to buckets across multiple regions?

[alanshaw]

Do we have a stable network every Monday? My gut instinct is to release this whenever we believe we have a stable network, not on a weekly cadence.

[bajtos]

I see your point.

My concerns with "release whenever we believe we have a stable network":

1. If we don't release on a regular basis, then we need someone to remember to periodically ask whether we have a stable network. In my experience, this is a slippery slope leading to no releases for weeks.
2. IMO, we should treat the dev network the same way as CI. If it's broken, then it's the team's P0 priority to fix it ASAP. An unstable network on the day a scheduled upgrade was planned should be a rare exception, and we should iterate on improving our practices to make it less and less likely over time.

BTW, we can use feature flags to ship code for partially-implemented features before we make them available to users.

How about releasing biweekly on Mondays when a sprint ends?

• The network should be stable by the end of the sprint; otherwise, we cannot claim our work is "done done".
• This gives us a longer window for shipping & fixing destabilising changes, i.e. the entire sprint duration.
• This way, Monday demos can show things that are already available for non-engineers in the staging/preview environment. Which is the way it should be in Scrum.

Essentially, I am proposing to adopt "Release trains".

---

On the second thought, let's move the update frequency to open questions, and scope this RFC to describe the other aspects of the Forge instances we want to run.

[pwrepo]

Had a chat with @bajtos and @hannahhoward:

[pwrepo]

Chat with @bajtos and @hannahhoward concluded:

1. We are ready to add a POC region based on customer demand.

2. Ideally 48-72 hour deployment timeline from when a request is submitted

3. Use servers.com, vultr.com or akami.com to simulate what would eventually be a full buildout.

4. 90 minimum deployment of a POC region. Customers aware that the data storage is ephemeral and will not be migrated to full buildout.

5. HA deployments not needed to start. Single instance deployments with JBOD drives or large 10TB NFS or local filesystem mounts.

6. Regions will be deployed on Prod (fil.one). Customer's account or domain flagged for them to use it. It will not be available to all logins.

7. POC regions will be named: , example (london): eu-west-2-POC

[Copilot]

🟡 Changes recommended

The RFC contains several typos/grammar issues that should be corrected to avoid ambiguity in the proposal.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR introduces a new RFC proposing how to run and operate multiple Forge network instances to balance competing needs (e.g., continuous deployment vs. stability) and to define criteria per instance.

Changes:

• Added an RFC describing decision criteria (update cadence, stability, infra, money type, resets, regions).
• Proposed four instance tiers: Production, Dev, Staging, and Performance testing.

File summaries

File	Description
rfcs/2026-09-forge-instances.md	New RFC documenting criteria and a proposed set of Forge instances to operate.

Review details

Suppressed comments (1)

rfcs/2026-09-forge-instances.md:116

• Wording/grammar: "the person on the pager duty" is ungrammatical; "on pager duty" (or similar) reads correctly.

- **Stability & acceptable outages:** Reasonable stability and minimum unplanned downtime. Full monitoring with alerts routed to the person on the pager duty, with capped severity (no incident is critical).

• Files reviewed: 1/1 changed files
• Comments generated: 5
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

The RFC contains a few concrete typos plus a substantive inconsistency about S3 access-key scope that could mislead readers.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

rfcs/2026-09-forge-instances.md:63

• Typo: "gives use" should be "gives us".

On the FilOne side, we use Stripe sandbox in non-production environments, which gives use "test"

rfcs/2026-09-forge-instances.md:138

• This "Important" note seems inconsistent with the rest of the RFC: pilot regions are described as part of the production Forge instance, so S3 access keys would span production + pilot regions (within one instance), not "dev & staging" (which are separate instances per the later note). Consider correcting the wording to avoid confusing readers.

Important: by adding pilot regions to the production Forge instance, we allow customers to create S3
access keys with access to both dev & staging regions.

• Files reviewed: 1/1 changed files
• Comments generated: 2
• Review effort level: Lite

[parkan]

very good to think through this ahead of GA but a few concerns:

• I'm not sure how feature flagging here actually stops writes at the S3 boundary; users outside pilot can't access console but they can still direct writes here
• pilot data should either be very explicitly communicated as doomed to deletion upon conclusion or machinery from rfc: credible exit under encryption #24 can be used to exit from pilot into prod, though I have some bias here 😉
• "Same guarantees as in production" reads like durability is on the same order as production, which it is not -- need to be clear exactly what the guarantee scope is
• likewise for load isolation -- I don't see anything here that actually produces that

[bajtos]

very good to think through this ahead of GA but a few concerns:

• I'm not sure how feature flagging here actually stops writes at the S3 boundary; users outside pilot can't access console but they can still direct writes here

Great catch!

My initial thinking was that non-pilot users won't be able to create access keys with permissions to access non-pilot regions, and therefore we don't need the S3 boundary to be aware of pilots. However, that's not true if we keep the current option of creating an access key that can create new buckets everywhere.

• pilot data should either be very explicitly communicated as doomed to deletion upon conclusion or machinery from rfc: credible exit under encryption #24 can be used to exit from pilot into prod, though I have some bias here 😉

We will make it clear that pilot data will be deleted after the pilot ends.

I think we may eventually need to implement migration from pilot to production, but that's beyond our current horizon.

• "Same guarantees as in production" reads like durability is on the same order as production, which it is not -- need to be clear exactly what the guarantee scope is

I'll discuss internally and clarify.

• likewise for load isolation -- I don't see anything here that actually produces that

Can you please clarify what you mean?

[bajtos]

After discussing this with Alan, I realised I need to rework the proposal for Pilot regions. Adding & removing pilot regions from the main production network would require too much upfront work. We can get this faster by deploying a new Forge network for each pilot region.

[parkan]

After discussing this with Alan, I realised I need to rework the proposal for Pilot regions. Adding & removing pilot regions from the main production network would require too much upfront work. We can get this faster by deploying a new Forge network for each pilot region.

nice, cutting the knot is probably the better call than untangling it here 👍

[bajtos]

I reworked the text to propose creating a new Forge network instance for each customer pilot, based on the discussions we had.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3d488052ca

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[alanshaw]

Minor feedback worth addressing but LGTM otherwise.

[bajtos]

Thank you, @alanshaw, for the grammar fixes! ❤️ I applied all suggestions.