Skip to content

appsec: release the listeners when one of them fails - #4642

Open
hamodywe wants to merge 1 commit into
crowdsecurity:masterfrom
hamodywe:fix/appsec-listener-leak-on-error
Open

appsec: release the listeners when one of them fails#4642
hamodywe wants to merge 1 commit into
crowdsecurity:masterfrom
hamodywe:fix/appsec-listener-leak-on-error

Conversation

@hamodywe

Copy link
Copy Markdown

What and why

listenAndServe returned as soon as either listener sent to serverError, which skipped the shutdown block. The other listener stayed inside Serve, so its port was held for the lifetime of the process — and since nothing rebinds, every later reload failed on an address still in use while LAPI kept answering and the process looked healthy. One bad reload poisons all the following ones, which is the permanent death in #4545.

The server is now shut down on the way out whichever branch returns.

Second hole in the same function: a listener that never reaches Serve is not one the server tracks, so Shutdown cannot release it either. ServeTLS returns before Serve when tls.LoadX509KeyPair fails, and the missing key/cert checks in startServer return earlier still. The goroutine that opens a listener now closes it.

No retry loop — per @blotus in #4545, the bind is not meant to retry; the assumption that the previous server was already shut down is what was wrong.

Fixes #4545

How it was tested

go test ./pkg/acquisition/modules/appsec/... — the whole package, plus -race on the new tests:

ok  	github.com/crowdsecurity/crowdsec/pkg/acquisition/modules/appsec	4.254s

The four new tests are deterministic and need no race window: a socket path whose directory does not exist makes the UDS bind fail while TCP succeeds, and vice versa.

With run.go reverted and the tests kept, exactly the leak cases fail and the clean-shutdown case still passes:

--- FAIL: TestListenAndServeReleasesTCPWhenTheSocketFails (10.00s)
--- FAIL: TestListenAndServeReleasesSocketWhenTCPFails (10.00s)
--- FAIL: TestListenAndServeReleasesTCPWhenTLSIsMisconfigured (30.01s)
    --- FAIL: .../key_file_not_configured (10.00s)
    --- FAIL: .../cert_file_not_configured (10.00s)
    --- FAIL: .../cert_file_that_does_not_exist (10.00s)
FAIL

Before the fix, the leak is visible directly:

listenAndServe returned: listen unix .../no-such-dir/appsec.sock: bind: ...
LEAK: 127.0.0.1:59929 is still bound after listenAndServe returned

go vet ./pkg/acquisition/modules/appsec/... clean. make test and make lint were not run here (Windows, no containers); the pre-existing pkg/acquisition docker config failures and the cloudwatch vet finding reproduce on a pristine tree.

Checklist

  • One concern only.
  • No break to LAPI/CAPI payloads, database schema, config keys, or cscli -o json|raw output.
  • A human has reviewed this diff line by line.
  • A human has tested this change, not only an agent.

AI assistance used: mostly

/kind fix
/area appsec

listenAndServe returned as soon as either listener sent an error, skipping
the shutdown block entirely. The other listener kept serving, so its port
stayed held for the lifetime of the process. Nothing rebinds, so every later
reload failed on an address that was still in use, while LAPI carried on
answering and the process looked healthy.

Shut the server down on the way out instead of only when the tomb dies.

A listener that never reaches Serve is not one the server tracks, and so not
one Shutdown can release: ServeTLS returns before Serve when the cert fails
to load, and the missing key/cert checks return earlier still. The goroutine
that opened the listener now closes it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AppSec listeners can die permanently after SIGHUP reload while LAPI keeps running

1 participant