Skip to content

ncu-ci.md

Latest commit

 

History

History
554 lines (456 loc) · 29 KB

ncu-ci.md

File metadata and controls

554 lines (456 loc) · 29 KB
Raw

ncu-ci

Parse the results of a Jenkins CI run in https://ci.nodejs.org/ and display a summary for all the failures.

Supported jobs:

ncu-ci <command>

Commands:
  ncu-ci available          Check whether Jenkins is available for PR CI requests
  ncu-ci workload           Print the number of running and queued node-test-pull-request jobs
  ncu-ci rate <type>        Calculate the green rate of a CI job in the last 100
                            runs
  ncu-ci walk <type>        Walk the CI and display the failures
  ncu-ci run <prid>         Start a node-test-pull-request CI job for a PR
  ncu-ci resume <prid>      Resume the latest node-test-pull-request CI job for a PR
  ncu-ci url <url>          Automatically detect CI type and show results
  ncu-ci pr <jobid>         Show results of a node-test-pull-request CI job
  ncu-ci commit <jobid>     Show results of a node-test-commit CI job
  ncu-ci benchmark <jobid>  Show results of a benchmark-node-micro-benchmarks CI
                            job
  ncu-ci citgm <jobid>      Show results of a citgm-smoker CI job
  ncu-ci daily              Show recent results of node-daily-master

Options:
  --version          Show version number                               [boolean]
  --nobuild          If running cigtm, whether or not jobid is citgm-nobuild.
                                                      [boolean] [default: false]
  --json <path>      Write the results as json to <path>                [string]
  --markdown <path>  Write the results as markdown to <path>            [string]
  --help             Show help                                         [boolean]

ncu-ci available

ncu-ci available checks whether Jenkins is available for PR CI requests. It exits with status 0 and no output when the controller is not preparing for shutdown and node-test-pull-request is enabled and buildable. Otherwise it exits with status 1 and explains the reason on stderr. HTTP failures, timeouts, and unrecognized responses are treated as unavailable.

The command only reads Jenkins state. It does not start or resume a build, check individual PRs, or require idle executors. Availability can change after the check.

ncu-ci workload

ncu-ci workload prints the number of running and queued node-test-pull-request jobs, followed by a newline. This includes PR builds waiting for downstream tests. Each PR build counts once; downstream test jobs do not add to the count. If no PR builds are running or queued, it prints 0. A failed query exits with status 1, reports the reason on stderr, and does not print a count.

The command reads Jenkins executors and the waiting queue, so older active builds are counted without relying on a limited build history. A request that starts between the two reads is counted once. The count is a snapshot and can change before a new request starts.

Both commands use the configured username and jenkins_token; no GitHub token, repository configuration, or PR argument is required. Each command has a 20-second deadline for its Jenkins requests, including response bodies.

For example, skip processing requests unless Jenkins is available and fewer than five PR jobs are running or queued:

ncu-ci available || exit 0
workload=$(ncu-ci workload) || exit 0
[ "$workload" -lt 5 ] || exit 0

# Process requests here.

ncu-ci rate <type>

ncu-ci rate <type> calculate the success rate for CI jobs in the last 100 runs per CI Health History, where <type> can be either pr for node-test-pull-request or commit for node-test-commit. See ncu-ci rate --help for more.

Examples:

node on git:main ❯ ncu-ci rate pr
--------------------------------------------------------------------------------
[1/1] Running health
--------------------------------------------------------------------------------
✔  Done
| UTC Time         | RUNNING | SUCCESS | UNSTABLE | ABORTED | FAILURE | Green Rate |
| ---------------- | ------- | ------- | -------- | ------- | ------- | ---------- |
| 2020-07-03 16:28 | 1       | 7       | 34       | 10      | 48      | 7.87%      |
node on git:main ❯ ncu-ci rate commit
--------------------------------------------------------------------------------
[1/1] Running health
--------------------------------------------------------------------------------
✔  Done
| UTC Time         | RUNNING | SUCCESS | UNSTABLE | ABORTED | FAILURE | Green Rate |
| ---------------- | ------- | ------- | -------- | ------- | ------- | ---------- |
| 2020-07-03 16:28 | 2       | 8       | 27       | 5       | 58      | 8.60%      |

ncu-ci walk <type>

ncu-ci walk <type> walks CI and displays failures, where <type> can be either pr for node-test-pull-request or commit for node-test-commit. See ncu-ci walk --help for more.

Example:

node on git:main ❯ ncu-ci walk commit
✔  Done--------------------------------------------------------------------------------
[1/60] Running health
--------------------------------------------------------------------------------
| UTC Time         | RUNNING | SUCCESS | UNSTABLE | ABORTED | FAILURE | Green Rate |
| ---------------- | ------- | ------- | -------- | ------- | ------- | ---------- |
| 2020-07-03 16:50 | 1       | 8       | 27       | 5       | 59      | 8.51%      |

--------------------------------------------------------------------------------
[2/60] Running https://ci.nodejs.org/job/node-test-commit/39446/
--------------------------------------------------------------------------------
✔  Build data downloaded
✔  Data downloaded
----------------------------------- Summary ------------------------------------
Result     FAILURE
URL        https://ci.nodejs.org/job/node-test-commit/39446/
Source     https://api.github.com/repos/libuv/ci-tmp-libuv-node/git/refs/heads/jenkins-libuv-in-node-141
Commit     [b205f29c5b4d] Replace libuv version with refs/heads/v1.x from libuv/libuv
Date       2020-07-03 15:44:21 +0000
Author     ci <ci@iojs.org>
-------------------------------- freebsd11-x64 ---------------------------------
URL        https://ci.nodejs.org/job/node-test-commit-freebsd/nodes=freebsd11-x64/34327/console
Type       JENKINS_FAILURE
Built On   test-digitalocean-freebsd11-x64-2
Reason     
  Build timed out (after 6 minutes). Marking the build as failed.
--------------------------------- Other builds ---------------------------------
Unstable   https://ci.nodejs.org/job/node-test-commit-linux/35929/
Unstable   https://ci.nodejs.org/job/node-test-commit-arm-fanned/15212/
--------------------------------------------------------------------------------
[3/60] Running https://ci.nodejs.org/job/node-test-commit/39444/
...etc

Possible use cases:

  1. Walk CI for the latest 100 runs of node-test-pull-request, aggregate failures, write the results into a Markdown file, and then cache the responses from Jenkins so that the next time the command is run, it picks up cached data written on disk for jobs whose results are known.

    Note: results are cached in $tmpdir/ncu/cache, so you may want to clean it up from time to time.

    ncu-ci walk pr --stats --cache --markdown results.md

  2. Walk CI for the latest 100 runs of node-test-pull-request, and then write the failures into a JSON file named database.json.

    ncu-ci walk pr --json database.json

ncu-ci run <prid>

ncu-ci run <prid> starts a node-test-pull-request Jenkins job for a pull request.

<prid> can be one of the following:

  • a numeric pull request id, for example 34127
  • a pull request URL, for example https://github.com/nodejs/node/pull/34127
  • a commit URL for the head commit of the pull request, for example https://github.com/nodejs/node/pull/34127/commits/35ea6ded7315cf9d0585a5d1cd7d09f358ca5993

If you pass a GitHub URL, ncu-ci will infer the repository owner, repository name, and pull request id from the URL. When you pass a commit URL, it also uses the commit SHA as the expected tip commit.

By default, ncu-ci run only starts CI when it can verify that the tip of the PR head matches the latest approved commit. If that safety check cannot be satisfied, the command refuses to start CI.

Options:

  • --certify-safe <sha>: explicitly set the commit SHA that must be at the tip of the PR head
  • --check-for-duplicates: check recent Jenkins runs and refuse to start a duplicate job for the same commit
  • --owner <owner>: GitHub repository owner, used when <prid> is not a GitHub URL
  • --repo <repo>: GitHub repository name, used when <prid> is not a GitHub URL

With --check-for-duplicates, an existing CI run for the same approved commit prevents starting another run, including when a failed or aborted build has no eligible resume action. Errors checking resume availability also prevent starting another run.

Examples:

Run CI for a PR number using repository information from config or flags:

ncu-ci run 34127 --owner nodejs --repo node

Run CI by passing the pull request URL:

ncu-ci run https://github.com/nodejs/node/pull/34127

Run CI for a specific approved head commit and avoid starting a duplicate Jenkins job:

ncu-ci run https://github.com/nodejs/node/pull/34127/commits/35ea6ded7315cf9d0585a5d1cd7d09f358ca5993 --check-for-duplicates

If the PR has the v8 engine label, ncu-ci run also triggers the node-test-commit-v8-linux job after the main PR CI job is started successfully.

ncu-ci resume <prid>

ncu-ci resume <prid> resumes the latest node-test-pull-request CI run linked in the PR description, comments, or reviews. The job must have finished with FAILURE or ABORTED. Running jobs and jobs with other results are not resumed. If no PR CI run is found, or Jenkins rejects the resume request, the command reports the failure and exits unsuccessfully.

Before downloading failure logs or submitting a resume request, the command checks that Jenkins offers the Resume build action to your account. A failed or aborted result alone does not guarantee that this action is available. If it is missing, the command follows Jenkins' recorded resume ancestry to the nearest ancestor that offers the action, and reports which run it will resume. Each ancestor must have finished with FAILURE or ABORTED and match the PR, repository, and approved commit. It does not search unrelated older CI runs or skip past a mismatched ancestor. If no eligible ancestor is available, the command reports the build URL and exits unsuccessfully. Errors checking availability stop the command without attempting to resume.

The CI-approved commit (COMMIT_SHA_CHECK) must match the PR's current HEAD. The command refuses to resume if they differ or the approved commit cannot be determined.

Before resuming, the command streams failed-job console output and compares failure diagnostics with the PR's changed files. When recovering through resume ancestry, it checks the latest run and every ancestor visited. It refuses to resume if a failed test or a file referenced in a failure diagnostic is changed by the PR. Generic make recipe-failure summaries do not count as file references; direct Makefile diagnostics, such as syntax errors, still do. The refusal includes the first matching filename, failure excerpt, and console log URL. Large excerpts are truncated with an explicit marker. Logs are scanned one at a time with bounded memory. HTTP compression is decoded as the response arrives. A match cancels the download and skips remaining logs. Unknown or unavailable failure details do not prevent resuming; the check uses the available diagnostics. Failure to retrieve the PR's changed-file list prevents resuming.

Pass a PR number with repository information from config or flags, or a PR URL:

ncu-ci resume 34127 --owner nodejs --repo node
ncu-ci resume https://github.com/nodejs/node/pull/34127

This uses Jenkins' Resume build action on the existing job. It does not start a fresh CI run for the current PR head. Jenkins credentials with permission to resume the job are required.

ncu-ci pr <jobid>

ncu-ci pr <jobid> returns information about the results of a node-test-pull-request job.

For example, if you would like to see the results of node-test-pull-request for nodejs/node#34127, you would visit the PR and note that the node-test-pull-request job can be found at https://ci.nodejs.org/job/node-test-pull-request/32158, and therefore the <jobid> is 32158, not 34127. See ncu-ci pr --help for more.

Example:

node on git:main ❯ ncu-ci pr 32158
--------------------------------------------------------------------------------
[1/1] Running PR: 32158
--------------------------------------------------------------------------------
✔  Build data downloaded
----------------------------------- Summary ------------------------------------
Result     UNSTABLE
URL        https://ci.nodejs.org/job/node-test-pull-request/32158/
Source     https://github.com/nodejs/node/pull/34127/
Commit     [faa808326cb0] lib: handle missing callbackMap in esm logic
Date       2020-06-29 16:51:58 -0700
Author     Shelley Vohr <shelley.vohr@gmail.com>
--------------------------------- Other builds ---------------------------------
Unstable   https://ci.nodejs.org/job/node-test-commit-arm-fanned/15155/

ncu-ci commit <jobid>

ncu-ci commit <jobid> returns information about the results of a specified node-test-commit job.

For example, if you would like to see the results of node-test-commit for nodejs/node#34086, you would visit the PR and note that the node-test-commit job can be found at https://ci.nodejs.org/job/node-test-commit/39377, and therefore the <jobid> is 39377, not 34086. See ncu-ci commit --help for more.

Example:

node on git:main ❯ ncu-ci commit 39377
--------------------------------------------------------------------------------
[1/1] Running COMMIT: 39377
--------------------------------------------------------------------------------
✔  Build data downloaded
----------------------------------- Summary ------------------------------------
Result     UNSTABLE
URL        https://ci.nodejs.org/job/node-test-commit/39377/
Source     https://github.com/undefined/undefined/pull/34086/
Commit     [725a1df7b849] quic: remove redundant cast
Date       2020-06-27 22:14:09 +0800
Author     Jiawen Geng <technicalcute@gmail.com>
--------------------------------- Other builds ---------------------------------
Unstable   https://ci.nodejs.org/job/node-test-commit-linux/35868/
Unstable   https://ci.nodejs.org/job/node-test-commit-arm-fanned/15148/

ncu-ci url <url>

ncu-ci url <url> takes a url corresponding to a PR on nodejs/node, detects the CI type (either node-test-commit or node-test-pull-request) and corresponding job id for the latest run, and returns a summary of results about the job run. See ncu-ci url --help for more.

Example:

node on git:main ❯ ncu-ci url https://github.com/nodejs/node/pull/34127
--------------------------------------------------------------------------------
[1/1] Running PR: 32158
--------------------------------------------------------------------------------
✔  Build data downloaded
----------------------------------- Summary ------------------------------------
Result     UNSTABLE
URL        https://ci.nodejs.org/job/node-test-pull-request/32158/
Source     https://github.com/nodejs/node/pull/34127/
Commit     [faa808326cb0] lib: handle missing callbackMap in esm logic
Date       2020-06-29 16:51:58 -0700
Author     Shelley Vohr <shelley.vohr@gmail.com>
--------------------------------- Other builds ---------------------------------
Unstable   https://ci.nodejs.org/job/node-test-commit-arm-fanned/15155/

ncu-ci benchmark <jobid>

ncu-ci benchmark <jobid> displays the results of a specified benchmark-node-micro-benchmarks CI job. See ncu-ci benchmark --help for more.

Example:

node on git:main ❯ ncu-ci benchmark 636
--------------------------------------------------------------------------------
[1/1] Running BENCHMARK: 636
--------------------------------------------------------------------------------
✔  Data downloaded
                                                                                               confidence improvement accuracy (*) (**) (***)
 http2/compat.js duration=5 benchmarker='test-double-http2' clients=2 streams=100 requests=100         NA       NaN %           NA   NA    NA
 http2/compat.js duration=5 benchmarker='test-double-http2' clients=2 streams=10 requests=100          NA       NaN %           NA   NA    NA
 http2/compat.js duration=5 benchmarker='test-double-http2' clients=2 streams=10 requests=1000         NA       NaN %           NA   NA    NA
 http2/compat.js duration=5 benchmarker='test-double-http2' clients=2 streams=1 requests=100           NA       NaN %           NA   NA    NA
 http2/compat.js duration=5 benchmarker='test-double-http2' clients=2 streams=1 requests=1000          NA       NaN %           NA   NA    NA
 http2/compat.js duration=5 benchmarker='test-double-http2' clients=2 streams=200 requests=100         NA       NaN %           NA   NA    NA
 http2/compat.js duration=5 benchmarker='test-double-http2' clients=2 streams=20 requests=100          NA       NaN %           NA   NA    NA
 http2/compat.js duration=5 benchmarker='test-double-http2' clients=2 streams=20 requests=1000         NA       NaN %           NA   NA    NA
 http2/compat.js duration=5 benchmarker='test-double-http2' clients=2 streams=40 requests=100          NA       NaN %           NA   NA    NA

Be aware that when doing many comparisons the risk of a false-positive
result increases. In this case there are 9 comparisons, you can thus
expect the following amount of false-positive results:
  0.45 false positives, when considering a   5% risk acceptance (*, **, ***),
  0.09 false positives, when considering a   1% risk acceptance (**, ***),
  0.01 false positives, when considering a 0.1% risk acceptance (***)
++ mv output030720-170033.csv /w/bnch-comp
Collecting metadata...
Metadata collection done.
Notifying upstream projects of job completion
Finished: SUCCESS

ncu-ci citgm <jobid> [jobid2]

ncu-ci citgm <jobid> [jobid2] shows the results of a given citgm-smoker job, with the option to compare per-platform results of two jobs. You See ncu-ci citgm --help for more.

Example:

node on git:main ❯ ncu-ci citgm 2400
--------------------------------------------------------------------------------
[1/1] Running CITGM: 2400
--------------------------------------------------------------------------------
✔  Header data downloaded
✔  Report data downloaded
----------------------------------- Summary ------------------------------------
Result     FAILURE
URL        https://ci.nodejs.org/job/citgm-smoker/2400/testReport/
Source     https://github.com/nodejs/node/pull/34093/
Commit     [9ec07f42864c] 2020-06-30, Version 14.5.0 (Current)
Date       2020-06-29 21:17:56 -0700
Author     Shelley Vohr <shelley.vohr@gmail.com>
----------------------------------- Failures -----------------------------------
┌────────────────────────┬───────────────────────┬───────────────────────┬─────────────────────────┬─────────────────────┬─────────────────┬────────────────────┐
│        (index)         │           0           │           1           │            2            │          3          │        4        │         5          │
├────────────────────────┼───────────────────────┼───────────────────────┼─────────────────────────┼─────────────────────┼─────────────────┼────────────────────┤
│       debian9-64       │ 'coffeescript-v2.5.1' │   'through2-v4.0.2'   │                         │                     │                 │                    │
│      rhel7-s390x       │   'through2-v4.0.2'   │                       │                         │                     │                 │                    │
│   fedora-latest-x64    │ 'coffeescript-v2.5.1' │   'through2-v4.0.2'   │                         │                     │                 │                    │
│     ubuntu1604-64      │ 'coffeescript-v2.5.1' │   'through2-v4.0.2'   │                         │                     │                 │                    │
│        osx1014         │    'acorn-v7.3.1'     │ 'coffeescript-v2.5.1' │     'clinic-v6.0.2'     │ 'ember-cli-v3.19.0' │ 'semver-v7.3.2' │ 'watchify-v3.11.1' │
│     ubuntu1804-64      │ 'coffeescript-v2.5.1' │   'through2-v4.0.2'   │                         │                     │                 │                    │
│ fedora-last-latest-x64 │ 'coffeescript-v2.5.1' │   'through2-v4.0.2'   │                         │                     │                 │                    │
│     centos7-ppcle      │ 'coffeescript-v2.5.1' │    'clinic-v6.0.2'    │ 'torrent-stream-v1.2.0' │  'through2-v4.0.2'  │                 │                    │
└────────────────────────┴───────────────────────┴───────────────────────┴─────────────────────────┴─────────────────────┴─────────────────┴────────────────────┘

Comparison Example:

node-core-utils on git:allow-citgm-comparison ❯ ncu-ci citgm 2392 2390
--------------------------------------------------------------------------------
[1/1] Running CITGM: 2392
--------------------------------------------------------------------------------
✔  Summary data downloaded
✔  Results data downloaded
✔  Summary data downloaded
✔  Results data downloaded
----------------------------------- Summary ------------------------------------
Result     FAILURE
URL        https://ci.nodejs.org/job/citgm-smoker/2392/
Source     https://api.github.com/repos/nodejs/node/git/refs/heads/v12.x
Commit     [feed95cd4c2c] Working on v12.18.1
Date       2020-06-02 20:27:47 +0200
Author     Michaël Zasso <targos@protonmail.com>
----------------------------------- Summary ------------------------------------
Result     FAILURE
URL        https://ci.nodejs.org/job/citgm-smoker/2390/
Source     https://github.com/nodejs/node/pull/33811/
Commit     [9a60117875dd] 2020-06-16, Version 12.18.1 'Erbium' (LTS)
Date       2020-06-09 20:23:09 -0700
Author     Shelley Vohr <shelley.vohr@gmail.com>
----------------------------------- Results ------------------------------------



FAILURE: 5 failures in 2390 not present in 2392


┌────────────────────────┬───────────────────────────┬────────────────────────────┐
│        (index)         │             0             │             1              │
├────────────────────────┼───────────────────────────┼────────────────────────────┤
│     centos7-ppcle      │      'multer-v1.4.2'      │                            │
│   fedora-latest-x64    │    'spawn-wrap-v2.0.0'    │                            │
│ fedora-last-latest-x64 │                           │                            │
│       debian9-64       │ 'express-session-v1.17.1' │ 'yeoman-generator-v4.10.1' │
│        osx1014         │                           │                            │
│      rhel7-s390x       │  'torrent-stream-v1.2.0'  │                            │
│      aix71-ppc64       │                           │                            │
│     ubuntu1604-64      │                           │                            │
│     ubuntu1804-64      │                           │                            │
└────────────────────────┴───────────────────────────┴────────────────────────────┘

ncu-ci daily

ncu-ci daily show recent results of node-daily-master. You can also aggregate the results by passing --cache, or limit the maximum number of CIs jobs to get data from with --limit=N. See ncu-ci daily --help for more.

node on git:main ❯ ncu-ci daily
✔  Done--------------------------------------------------------------------------------
[1/16] Running health
--------------------------------------------------------------------------------
| UTC Time         | RUNNING | SUCCESS | UNSTABLE | ABORTED | FAILURE | Green Rate |
| ---------------- | ------- | ------- | -------- | ------- | ------- | ---------- |
| 2020-07-20 19:15 | 0       | 6       | 5        | 1       | 34      | 13.33%     |

--------------------------------------------------------------------------------
[2/16] Running https://ci.nodejs.org/job/node-daily-master/2004/
--------------------------------------------------------------------------------
✔  Build data downloaded
✔  Build data downloaded
✔  Data downloaded
----------------------------------- Summary ------------------------------------
Result     FAILURE
URL        https://ci.nodejs.org/job/node-test-commit/39692/
Source     https://api.github.com/repos/nodejs/node/git/refs/heads/main
Commit     [bf0d82c10247] test: remove common.localhostIPv6
Date       2020-07-16 16:57:30 -0700
Author     Rich Trott <rtrott@gmail.com>
-------------------------------- ubuntu1604-64 ---------------------------------
URL        https://ci.nodejs.org/job/node-test-commit-linux/nodes=ubuntu1604-64/36148/console
Type       JS_TEST_FAILURE
Built On   test-rackspace-ubuntu1604-x64-1
Reason     
  not ok 2783 benchmark/test-benchmark-streams
    ---
    duration_ms: 2.518
    severity: fail
    exitcode: 1
    stack: |-
      assert.js:385
          throw err;
          ^
      
      AssertionError [ERR_ASSERTION]: benchmark file not running exactly one configuration in test: 
      streams/creation.js
      streams/creation.js kind="duplex" n=1: 6,840.272790078869
      
      streams/pipe-object-mode.js
      streams/pipe-object-mode.js n=1: 134.18694119525676
      
      streams/pipe.js
      streams/pipe.js n=1: 127.79219557726546
      
      streams/readable-async-iterator.js
      2
      streams/readable-async-iterator.js sync="yes" n=1: 120.35939314793974
      
      streams/readable-bigread.js
      streams/readable-bigread.js n=1: 29.298658098020184
      
      streams/readable-bigunevenread.js
      streams/readable-bigunevenread.js n=1: 22.972824549899766
      
      streams/readable-boundaryread.js
      streams/readable-boundaryread.js type="string" n=1: 17.122757569423193
      
      streams/readable-readall.js
      streams/readable-readall.js n=1: 53.44909695078247
      
      streams/readable-unevenread.js
      streams/readable-unevenread.js n=1: 15.20828514524817
      
      streams/writable-manywrites.js
      streams/writable-manywrites.js len=1 callback="yes" writev="yes" sync="yes" n=1: 1,075.9039475991742
      
          at ChildProcess.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1604-64/test/common/benchmark.js:38:12)
          at ChildProcess.emit (events.js:314:20)
          at Process.ChildProcess._handle.onexit (internal/child_process.js:276:12) {
        generatedMessage: false,
        code: 'ERR_ASSERTION',
        actual: false,
        expected: true,
        operator: '=='
      }
    ...
  
--------------------------------- Other builds ---------------------------------
--------------------------------------------------------------------------------
[3/16] Running https://ci.nodejs.org/job/node-daily-master/2003/
--------------------------------------------------------------------------------
✔  Build data downloaded
✔  Build data downloaded
⠸ Querying console text for job/node-test-commit-osx/nodes=osx1015/35280/

Caveats

The CI failures are parsed using pattern matching and could be incorrect. Feel free to open a pull request whenever you find a case that ncu-ci does not handle well.