Skip to content

Take a context transfer into paths of its own before naming it - #93

Closed
MayCXC wants to merge 1 commit into
apple:mainfrom
MayCXC:fssync-concurrent-cache
Closed

MayCXC wants to merge 1 commit into
apple:mainfrom
MayCXC:fssync-concurrent-cache

Conversation

@MayCXC

@MayCXC MayCXC commented Aug 27, 2026

Copy link
Copy Markdown

Summary

The context cache is named for what it holds, so two builds carrying the same context meet on one name, and two builds carrying no context at all meet on the name of nothing, which is the common case on a machine running several builds at once. Both wrote the same tar, unpacked it to the same directory, and deleted that tar on the way out, so one build removed the file another was still writing to, and the build that lost failed with the tar it had just created not existing.

Stream into a file of this receive's own and unpack into a directory of its own, then move the finished tree under the shared name. A tree is already there when another receive got its first, and it holds the same content under the same name, so it stands and this copy is given up.

Fixes #92.

Motivation and Context

The cache key is content, which is the right key: two builds carrying identical context should share the unpacked tree rather than each unpacking their own. What did not follow from that key was the working state. The tar being streamed and the directory being unpacked into were both named by the key as well, so the shared name covered not just the finished artifact but every intermediate step, and concurrent receives collided on all of them.

Builds with no context at all are the case that makes this easy to hit, since they all derive the same name.

Testing

TestReceiver_Receive_ConcurrentSameContent drives eight concurrent receives of identical content and asserts each one gets a complete tree.

  • Against main without the fix, it fails with the error the builds reported:
    --- FAIL: TestReceiver_Receive_ConcurrentSameContent
        tarxfer_test.go:261: receiver 7: unexpected EOF
    
  • With the fix it passes.
  • go build ./..., go vet ./..., and go test ./pkg/... are clean.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update

The context cache is named for what it holds, so two builds carrying the
same context meet on one name, and two builds carrying no context at all
meet on the name of nothing, which is the common case on a machine
running several builds at once. Both wrote the same tar, unpacked it to
the same directory, and deleted that tar on the way out, so one build
removed the file another was still writing to and the build that lost
failed with the tar it had just created not existing.

Stream into a file of this receive's own and unpack into a directory of
its own, then move the finished tree under the shared name. A tree is
already there when another receive got its first, and it holds the same
content under the same name, so it stands and this copy is given up.

The test fails without this, with the error the builds reported.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Concurrent builds share one context cache name and delete each other's tar

2 participants