Repository navigation
Windows build times in the CI are highly variable #4537
Description
As part of the analysis I was doing for coming up with some CI improvement ideas for #4529 I noted that the Windows builds were extremely variable in terms of build times. While the Linux build tend to run in about 5-30 minutes (it's typically dwarfed by the test time which is now running at close to half an hour) the Windows builds are taking between 20 minutes (best case) but most days there are a few that seem to take over 2 hours for just the compile phase. win-vs2022_clang / win-vs2022_clang-arm64
Creating this issue so we can investigate whether there is any scope for improvement of the build times since (albeit subjectively) I've seen more occurrences of queued windows jobs in the CI recently than other platforms, and the spikiness of the compile jobs could potentially explain why.
I'll also note that here (because it's windows, although it's a different problem) there seems to be a relatively high proportion of failures occurring in the Windows js test suites that should perhaps also be looked at to see if any improvement to flakiness can be done. As an example https://ci.nodejs.org/job/node-test-binary-windows-js-suites/RUN_SUBSET=1,nodes=win10-COMPILED_BY-vs2022_clang/44048/console had one failure in test-sqlite-session. As noted in the CI improvement ideas issue we already have re-run testing but in this case (a 1/10 failure) the job is still marked as a fail so we possibly need to look at utilising this data as each failure in the js-suites one would likely have resulted in a PR test re-run.
08:46:45 failed 1 out of 10
08:46:45 not ok 1056 parallel/test-sqlite-session
That failure was a 08:46:45 # Fatal JavaScript out of memory: GC during deserialization condition - it would be useful to know how many of the other intermittent failures are similar problems.
Activity
sxa commentedon Oct 23, 2026
I wonder if this is just a side effect of having six machines each of which do one build for each of arm and x64. In the case where a V8 update or patch is done (causing the preprocessor directive for the V8 version to be bumped) that will potentially require 12 "slow" compile jobs to occur in the system (plus the debug job but that is typically not slow). While some other platforms have 3 executors (Same 3:1 ratio of machines to rebuilds) Windows has the disadvantage that the fanned test jobs could get delayed if we get a backlog of compile jobs in the queues waiting to get built that end up being run first.
Since we already have split build/test labels (e.g. win-vs2022-clang-arm64 vs win2022-COMPILED_BY=vs2022 I wonder if it's worth trying to disable the build labels on a couple of the machines and see if that prevents larger backlogs building up by having fewer builds on invalidated caches. Noting as a potential counterargument this will leave 4 build machines which, in the case of a V8 PR test or update, would tie up 3/4 of the executors for some time (x64, arm, debug) meaning that other node-test-commit jobs will have to share the one remaining node for a while, but overall my gut feel is that it would still be a positive.
sxa commentedon Oct 23, 2026
At the time of writing there are four windows compile jobs for each architecture in the queue waiting for a
win-vs2022_clang[-arm64]executor with an example job of https://ci.nodejs.org/job/node-test-commit/93504/ having completed all other platforms in 1h12 with windows not having started the compile yet. It is Job 81377 in the table below which took 7h23 in total including 2h34 compiling, 2h26 testing, and the rest presumably lost in queueing.Recent jobs were as follows. This only includes Node v27 builds:
(*) - ntprnw is the run time of the slowest non-windows platform from the parent note-test-pull-request job.
(†) - Job 81392 is notable because it had fast build times (once they got scheduled) but took 2h54 for the testing phase. From the twisty below which shows the log from the js-suites job you can see that the
Log snippets from fanned job 81392
So you can see from this that it was about 3½ hours for the compile job and nearly 3 hours for the completion of the test job. Looking in more details at the compile job you can see that while both x64 and arm completed in about 20 minutes they were held up before they started leading to the big gaps in the log:
You can see this from the initial timestamps in the logs. Here's x64:
and arm:
It's a similar store on the test suites (I'll show native hear rather than js because it's simpler without the fanned subsets):
And looking at the first line of each log to see the timestamps we can see that the all the others got going quickly (initial timestamps are within a second:
but the nodes=win2022-vs2022-COMPILED_BY-vs2022_clang one was queued up for over 2½ hours before it got scheduled
Current chart of build time trend of recent [node-test-pull-request](https://ci.nodejs.org/job/node-test-pull-request/buildTimeTrend) jobs
Chart of [node-test-commit-windows-fanned](https://ci.nodejs.org/job/node-test-commit-windows-fanned/buildTimeTrend) until things had quiesced
Chart of [node-test-binary-windows-js-suites](https://ci.nodejs.org/job/node-test-binary-windows-js-suites/buildTimeTrend) (slightly later than the other two charts)
Also noting that requiring all of the compile jobs to complete before the tests commence is probably suboptimal from a throughput perspective but it's down to the way this has been implemented in the multiconfiguration plugin which means that, for exampe, the arm cannot get queued start until the x64 build is complete - it's probably not a huge problem overall though. ON a counter point the current system does make it easier to see the full set of test results (arm/x64) from one instance of the test job (albeit without having the js and natives together)
sxa commentedon Oct 25, 2026
Due to the delays which are visible on some of the log snippets above I've disabled (Added a
-XXX4537suffix to the build labels) test-rackspace-win2022_vs2022-x64-6 from being able to run build jobs leaving it dedicated for testing and will keep an eye on the trend over the next week. I think I'd like to do the same on-5to leave two dedicated for testing (as we do for the other test labels, which would also mean that only four machines need ccaches to be repopulated) but we can leave that for a week or two before trialling - ideally including one cycle of a V8 patch being merged first. Fanned jobs from 81421 (test-commit windows 69946, js-suites 44129) are with this change applied.sxa commentedon Oct 26, 2026
There was a V8 PR test today at job https://ci.nodejs.org/job/node-test-commit-windows-fanned/81438/ which caused a bit of a backlog but not too severe (Up to 3 hours, with the test jobs occasionally going over an hour but not being as badly delayed as we've seen in the past).
While it has not fully got back to normal yet there are no windows compile jobs running at the moment so it's close to stable again. However the V8 PR (66820) has now been merged and so each machine will start repopulating its caches as of node-test-pull-request#78674 / windows fanned job 81454
sxa commentedon Oct 27, 2026
Looking at the windows jobs since the two at the end of the last comment:
rackspace-2have rebuilt both x64 and arm64 - that machine seems to have consistently been given the debug and test jobs overnight!) The slowest overall windows fanned job was just over 4h18m which I would say is acceptable for a cache invalidation scenario.Another point of note relating to something I said earlier in this issue:
For each build the x64 compile jobs are typically faster than the arm ones - around 2-2½ hours vs just over 3½ in the worst case for arm64. This shows that we are artificially delaying the start of x64 testing while the arm64 builds complete due to the way the fanning setup works.
For a sample job, node-compile-windows #69978 took 3h15. 1h39 for x64 and 3h15 for arm64. This will have meant that initiation of the x64 testing (including scheduling on the now "test only" machine) would have been delayed by about 1½ hours. It's not something that's necessarily critical to fix now, but worthy of consideration for any future redesign of the jobs. Similar for 69982 - 23 minutes on x64, over 4 hours on arm64.
Some charts which show the spikes from last week from V8 changes and the new ones from this week
Times mentioned in the text of the twisty are the worst case seen in the period while rebuilding with V8 changes.
node-test-commit windows fanned (Up to ~6 hrs last week, ~4¼ hrs now)
node-compile-windows (Hit 6-7 hrs last week, ~4hrs this week)
node-test-binary-windows-js-suites (Huge difference from freeing up one machine - ~3hrs peak last week, ~1hr today)
Overall I think we're in a much better position now. I may look at doing another table equivalent to the one I did for the initial analysis but I think the charts say all we need to know for now :-)
sxa commentedon Nov 4, 2026
Noting that over most of the last week it's been particularly bad. There has been v8 updates, some changes with V8 patches being merged and also a few security-release related jobs running. The V8 changes has there has been an unusually high number of cache invalidations plus there is additional load from V8 15.7 PR testing. On a positive the move to having one dedicated test machine which doesn't run any builds does mean there are fewer cases of the higher level fanned job being held up waiting for tests to be scheduled. The five build machines have been in constant operation for the last few days, and while node-tet-commit-windows-fanned has been running at over 10 hours to completion in many cases over the last two days it is coming down, albeit still about 5 hours at the time of writing.
This is the history of the arm64 axis of node-compile-windows at the time of writing:
Still incredibly spiky:
Linux/x64 (RHEL8) history in the last day for comparison
sxa commentedon Nov 5, 2026
Windows queues have cleared overnight after a fairly terrible few days of long delays
Equivalent chart for node-test-pull-request
Noting that the "flat" bit of this about 2/3 along the X axis was from a github outage that was causing jobs to fail (Slack thread) with
git@github.com: Permission denied (publickey).I do wonder if we could gain a further advantage by either:
dailyjobs.[*] - I'm suggesting arm64 first here because that platform takes longer to build in the invalidation scenario so have more to gain by building with fewer invalidations) It would also let us more directly compare how arm64 and x64 queue depths behaave if we leave x64 and x64/debug able to build across any of the five machines). Since the arm and x64 builds are almost always initiated together there should be little disadvantage from an overall utilisation perspetive to this as they'd generally both be initiated at the same time anyway.