Skip to content

Partition analysis: when each half's mesh reacted to the cut - #461

Open
shanurwan wants to merge 1 commit into
iftech:masterfrom
shanurwan:partition-mesh-reaction
Open

shanurwan wants to merge 1 commit into
iftech:masterfrom
shanurwan:partition-mesh-reaction

Conversation

@shanurwan

Copy link
Copy Markdown

The churn and degraded tables end with a mesh-peers row; the partition table has none. This
adds four rows to partition_result, from the mesh-peers gauge the run already scrapes:

item expectation result (example)
mesh reaction to the cut no bound yet side a: 6.0 to 3.0 between 60s and 120s after the cut; side b: held at 6.0 for the 120s observed
mesh peers under the split, side a between dLow (4) and dHigh (12), around D (6) median 4, range 3 to 5
mesh peers under the split, side b same median 6, range 5 to 8
mesh peers after the heal same median 6, range 4 to 8

Why. Delivery does not show how long a split goes unnoticed. A node whose mesh still
counts peers it cannot reach reports the same degree as before the cut. I ran into this
replaying a border shutdown against the regression node (150 nodes, K3s, cut with tc/netem):
the cut-off side kept its pre-cut mesh size for 535 to 574 s in four runs, then dropped from
7.8 to 6.4 peers, while delivery inside stayed at 100%. Report and raw runs:
https://github.com/shanurwan/cut-and-heal/tree/main/reports/iran-2019-11

What "reacted" means. The first snapshot under the split where a half's mean degree
differs by more than 10% from its last value before the cut. The move is reported as lying
between the two snapshots around it, because the gauge is scraped every 60 s. A half that
never moves reports "held at N for the Ms observed".

Not tested on the lab. I do not have access to it. The rows are covered by unit tests
on synthetic frames, and lint and the full suite pass locally (1014 passed, 3 skipped). Two
things I could not check and would like your view on:

  • whether the run's scrape window always starts before the cut. If it does not, the row
    says "no snapshot before the cut" instead of guessing.
  • whether a NetworkPolicy cut shows the same delay as a netem drop. With the default
    heal_at of 120 s there are only two snapshots under the split, so a delay like the one
    above would read as "held"; a longer split or a shorter scrape step would be needed to
    bound it.

Related to #321

The churn and degraded tables end with a mesh-peers row; the partition table
had none. A node whose mesh still counts peers it cannot reach reports the
same degree as before the cut, so delivery alone does not show how long a
split goes unnoticed.

Adds four rows from the mesh-peers gauge the run already scrapes: when each
half's mean mesh degree first moved by more than a tenth from its value at
the cut, bracketed by the two snapshots around the move, and the degree of
each half under the split and of the whole network after the heal.
@shanurwan

Copy link
Copy Markdown
Author

@radiken @AlbertoSoutullo this is ready for a look when you have time. The CI run needs a maintainer's approval to start, since it's my first PR here. Happy to change the row wording or the 10% threshold if you'd prefer something else.

@shanurwan

Copy link
Copy Markdown
Author

A follow-up on the second question in the description, in case it helps when you look at this.

I tested what sets the delay before a cut-off node drops its far-side mesh peers. Two scenarios that differ only in net.ipv4.tcp_retries2, three seeds each, 150 nodes, a 15-minute netem drop, predictions written before the runs:

seed 1 seed 2 seed 3
kernel defaults 577 s 572 s 537 s
tcp_retries2 = 5 20 s 22 s 19 s

So in these runs the node kept counting the peers until the kernel gave up on the TCP connection. Report and raw runs: https://github.com/shanurwan/cut-and-heal/tree/main/reports/long-cut

What that means for this PR: with the default heal_at of 120 s and default timers, the new "mesh reaction to the cut" row would read "held" for the whole split. It becomes informative with a longer split or a lower retransmission limit.

Two limits: this was a silent drop with netem, so a NetworkPolicy cut on your CNI may behave differently, and I did not record the effective default inside the pods (the delay matches 12 retransmissions, not 15).

@shanurwan

Copy link
Copy Markdown
Author

A correction to my comment above. I ran three more arms (tcp_retries2 at 8, 10 and 13) and recorded the value inside every pod.

The kernel is not the only timer. At 5, 8 and 10 the mesh reacts when the kernel closes the connection (20 s, 118 s, 351 s). At 13 and at the default of 15 it reacts at about nine minutes, well before the kernel would close anything (685 s and 925 s). So with default settings something above the kernel ends the connection, and I have not identified what.

Report with all sixteen runs: https://github.com/shanurwan/cut-and-heal/tree/main/reports/long-cut

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant