Demo Proposal: Chaos engineering - circuit breaker for a severed dependency link

[gruendlich]

Assignment Proposal

Title

Chaos engineering: circuit breaker for a severed dependency link

Names and KTH ID

• (agrund@kth.se)
• (derghazi@kth.se)

Deadline

Week 7

Category

Demo

Description

We containerise a Vite/React frontend, Python backend and Elasticsearch, with
Toxiproxy between backend and Elasticsearch. We sever that link live and show the
backend's circuit breaker open, degrade gracefully, then close again on its own once
the link returns, no manual intervention. Time permitting, we contrast this with a
naive watcher that kills and respawns the backend instead, showing live that this
just loops, since the new container hits the same severed link.

Relevance

Covers chaos and resilience engineering (week 7 preparatory material): steady state,
injected fault, observation, rollback, at the dependency level rather than the
infrastructure level of most prior demos. The respawn comparison shows that
remediation must act on the fault's actual location, tying into monitoring and
automated operational response.

[ericcornelissen]

Unfortunately, it looks like this idea is more or less identical to #2992, which was approved earlier. If you disagree please argue why. Otherwise, please close this and find another thing to demo.

[gruendlich]

@ericcornelissen We'd like to argue this is sufficiently distinct. #2992 tests network
resilience as a CI gate: Toxiproxy runs inside GitHub Actions, failing a
PR when integration tests catch missing retry/timeout handling. Our demo
injects the same class of fault against a live running system at
runtime, with no CI pipeline involved, and shows a circuit breaker
(closed/open/half-open) recovering automatically, a different
resilience mechanism to plain retry-with-timeout. We also show a naive
alternative, killing and respawning the app, failing live because
respawning doesn't fix a network-layer fault, a comparison #2992 doesn't
make. We see these as testing-before-merge versus self-healing-in-
production, different DevOps concerns sharing only the fault-injection
tool.

If this is still to similar, we shall find another topic...

[ericcornelissen]

You convinced me, thanks for the proposal and arguments. Best of luck with the preparations 🙂