postoverflow: run them in a goroutine to not block writers, enforce max in-flight alerts, and warn if queue is becoming too full

[blotus]

Postoverflow parsing ran inline in the runOutput select loop. Postoverflow nodes do
blocking I/O (such as crowdsecurity/rdns that does a PTR lookup) and the
pipeline channels are unbuffered, so one slow lookup backpressured everything upstream:
buckets, parsers, acquisition, and the appsec runner, which emits into that same chain
before writing the in-band response. WAF requests then exceeded the remediation
component's APPSEC_PROCESS_TIMEOUT and failed open.

Postoverflow now runs in a worker behind a bounded queue, so the pipeline never waits on
it.
On a full queue the overflow is dropped and counted, rather than blocking: postoverflow
is where the whitelists run, so pushing an alert unparsed would turn a stall into false
positive bans. Dump mode (cscli explain, -dsn) still parses inline.

New config, with the flat *_routines keys kept as the fallback:

crowdsec_service:
  pipeline:
    parser:  {routines: 4}
    buckets: {routines: 2}
    output:  {routines: 1, queue_size: 256}

New metrics cs_postoverflow_queue_depth and cs_postoverflow_dropped_total, plus a
warning at 75% full so the queue is visible before anything is lost.

Fixes #4600

[github-actions]

@blotus: There are no 'kind' label on this PR. You need a 'kind' label to generate the release automatically.

• /kind feature
• /kind enhancement
• /kind refactoring
• /kind fix
• /kind chore
• /kind dependencies

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[github-actions]

@blotus: There are no area labels on this PR. You can add as many areas as you see fit.

• /area agent
• /area local-api
• /area cscli
• /area appsec
• /area security
• /area configuration

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[blotus]

/kind fix
/area agent

[codecov]

Codecov Report

❌ Patch coverage is 75.80645% with 30 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.73%. Comparing base (909b515) to head (9956a78).

Files with missing lines	Patch %	Lines
cmd/crowdsec/output.go	71.91%	20 Missing and 5 partials ⚠️
cmd/crowdsec/lpmetrics.go	71.42%	1 Missing and 1 partial ⚠️
pkg/csconfig/crowdsec_service.go	92.00%	1 Missing and 1 partial ⚠️
pkg/metrics/metrics.go	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4672      +/-   ##
==========================================
+ Coverage   64.71%   64.73%   +0.01%     
==========================================
  Files         522      522              
  Lines       39862    39958      +96     
==========================================
+ Hits        25798    25868      +70     
- Misses      11663    11683      +20     
- Partials     2401     2407       +6     

Flag	Coverage Δ
bats	41.18% <62.90%> (+0.09%)	⬆️
unit-linux	41.96% <54.03%> (-0.01%)	⬇️
unit-windows	30.64% <54.03%> (+0.13%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.