Skip to content

Compatibility matrix failing on develop #4691

Closed

Description

@github-actions

The compatibility matrix failed on e890b4b (develop).

Run: https://github.com/wheels-dev/wheels/actions/runs/42355726719

Failed jobs

Wheels Test Matrix

Engine MySQL PostgreSQL SQL Server H2 CockroachDB Oracle SQLite
lucee6 ✅ ✅ ✅ ✅ ✅ ✅ ✅
lucee7 ✅ ✅ ✅ ✅ ✅ ✅ ✅
adobe2023 ✅ -- ✅ -- ✅ -- ✅
adobe2025 -- ✅ ✅ -- -- ✅ ✅
boxlang ✅ ✅ ✅ -- ✅ ✅ ✅
lucee7 (TZ America/New_York) ✅
adobe2025 (TZ America/New_York) ✅
boxlang (TZ America/New_York) ✅

A ':x: N tests' cell means the leg reported fewer than 4000 testcases (suite never ran; counted as failed). ':x: no result' means the leg returned no parseable result.

Results for commit e890b4b.

Activity

  1. github-actions commented on Nov 8, 2026

    @github-actions
    ContributorAuthor

    The compatibility matrix failed on e890b4b (develop).

    Run: https://github.com/wheels-dev/wheels/actions/runs/42464171033

    Failed jobs

    Wheels Test Matrix

    Engine MySQL PostgreSQL SQL Server H2 CockroachDB Oracle SQLite
    lucee6 -- -- -- -- -- -- --
    lucee7 -- -- -- -- -- -- --
    adobe2023 -- -- -- -- -- -- --
    adobe2025 -- -- -- -- -- -- --
    boxlang -- -- -- -- -- -- --

    A ':x: N tests' cell means the leg reported fewer than 4000 testcases (suite never ran; counted as failed). ':x: no result' means the leg returned no parseable result.

    Results for commit e890b4b.

  2. github-actions commented on Nov 8, 2026

    @github-actions
    ContributorAuthor

    The compatibility matrix failed on e890b4b (develop).

    Run: https://github.com/wheels-dev/wheels/actions/runs/42355726719

    Failed jobs

    Wheels Test Matrix

    Engine MySQL PostgreSQL SQL Server H2 CockroachDB Oracle SQLite
    lucee6 ✅ ✅ ✅ ✅ ✅ ✅ ✅
    lucee7 ✅ ✅ ✅ ✅ ✅ ✅ ✅
    adobe2023 ✅ -- ✅ -- ✅ -- ✅
    adobe2025 -- ✅ ✅ -- -- ✅ ✅
    boxlang ✅ ✅ ✅ -- ✅ ✅ ✅
    lucee7 (TZ America/New_York) ✅
    adobe2025 (TZ America/New_York) ✅
    boxlang (TZ America/New_York) ✅

    A ':x: N tests' cell means the leg reported fewer than 4000 testcases (suite never ran; counted as failed). ':x: no result' means the leg returned no parseable result.

    Results for commit e890b4b.

  3. github-actions commented on Nov 8, 2026

    @github-actions
    ContributorAuthor

    The compatibility matrix failed on e890b4b (develop).

    Run: https://github.com/wheels-dev/wheels/actions/runs/42464171033

    Failed jobs

    Wheels Test Matrix

    Engine MySQL PostgreSQL SQL Server H2 CockroachDB Oracle SQLite
    lucee6 -- -- -- -- -- -- --
    lucee7 -- -- -- -- -- -- --
    adobe2023 -- -- -- -- -- -- --
    adobe2025 -- -- -- -- -- -- --
    boxlang -- -- -- -- -- -- --

    A ':x: N tests' cell means the leg reported fewer than 4000 testcases (suite never ran; counted as failed). ':x: no result' means the leg returned no parseable result.

    Results for commit e890b4b.

  4. bpamiri commented on Nov 8, 2026

    @bpamiri
    Collaborator

    Triage: not a code regression — both runs fail before any test executes, on engine/asset downloads:

    • Run 42355726719: the Adobe legs died at Download ojdbc10 for Adobe engines with curl: (22) ... 502 from the Ortus downloads host.
    • Run 42464171033 (and its rerun/attempt 2): nearly every leg failed at Wait for CF engine to be ready / Start CF engine because downloads.ortussolutions.com returned 403 Forbidden for the engine archives (e.g. https://downloads.ortussolutions.com/lucee/lucee/6.2.5.48/cf-engine-6.2.5.48.zip, Adobe 2023.0.11, BoxLang bx-compat-cfml 1.27.0). No TestBox failures were reached.
    • Last green develop matrix: run 42191913487 at 8e08b919c (before feat(wheelstest): duration-weighted shard planner + runner shard params (prototype) #4689/docs(test): document full-suite shard-safety limits #4690); those commits only touch the test runner/planner and maintainer docs — no Docker, CI, datasource or framework runtime changes.

    The endpoints intermittently returned 200 during triage, so this looks like provider-side throttling/outage rather than a permissions change we made. I'm keeping this issue open until a matrix run is green, and a follow-up PR is in flight to cache the engine archives and Oracle JDBC (and add bounded retries) so the matrix stops depending on a live download at job start. If the 403 persists for GitHub-hosted runners, a mirror is the durable fix; that needs a decision on hosting.

  5. github-actions commented on Nov 8, 2026

    @github-actions
    ContributorAuthor

    The compatibility matrix failed on e890b4b (develop).

    Run: https://github.com/wheels-dev/wheels/actions/runs/42464171033

    Failed jobs

    Wheels Test Matrix

    Engine MySQL PostgreSQL SQL Server H2 CockroachDB Oracle SQLite
    lucee6 -- -- -- -- -- -- --
    lucee7 -- -- -- -- -- -- --
    adobe2023 -- -- -- -- -- -- --
    adobe2025 -- -- -- -- -- -- --
    boxlang -- -- -- -- -- -- --

    A ':x: N tests' cell means the leg reported fewer than 4000 testcases (suite never ran; counted as failed). ':x: no result' means the leg returned no parseable result.

    Results for commit e890b4b.

  6. bpamiri commented on Nov 8, 2026

    @bpamiri
    Collaborator

    Update after the retry and the cache work: this is IP-based blocking, not a transient outage.

    • From this maintainer host (non-GitHub IP), the exact failing URLs return 200 for HEAD and GET with curl/CommandBox/Mozilla user agents:
      • downloads.ortussolutions.com/lucee/lucee/7.0.5.41/cf-engine-7.0.5.41.zip
      • .../lucee/6.2.5.48/..., .../ortussolutions/boxlang-modules/bx-compat-cfml/1.27.0/...
    • From GitHub-hosted runners, the same URLs return 403 Forbidden (Lucee 6/7, Adobe 2023/2025, BoxLang module downloads), reproducible across reruns/attempts. Adobe engine paths also 404 from here because their exact path differs; the CI log's 403s are the authoritative evidence.

    So retries/backoff alone will not fix cold jobs, and a warm Actions cache can't be populated from a runner that is already blocked. The durable options (need a maintainer decision):

    1. Prebuilt engine images in GHCR — build/publish the test images (with engines installed) from a non-blocked network once, then have CI pull ghcr.io/wheels-dev/... instead of downloading at job start. Best fit for the cold-start/matrix jobs; images are public and version-pinned.
    2. Mirror the pinned engine archives somewhere GitHub runners can reach (GitHub release assets in a wheels-dev repo, S3/R2, or similar), with SHA-256 checks kept. Adobe's redistribution terms need checking.
    3. Ask Ortus to allowlist GitHub Actions egress / provide a token for downloads.ortussolutions.com. Fastest if they respond, least under our control.

    Until one lands, the matrix, cold-start and smoke jobs that need an engine download will stay red regardless of code. I'll keep the compatible cache/retry PR (#4692) ready, but it can only help once a runner can fetch something at least once.

  7. github-actions commented on Nov 9, 2026

    @github-actions
    ContributorAuthor

    The compatibility matrix failed on e890b4b (develop).

    Run: https://github.com/wheels-dev/wheels/actions/runs/42593003786

    Failed jobs

    Wheels Test Matrix

    Engine MySQL PostgreSQL SQL Server H2 CockroachDB Oracle SQLite
    lucee6 ✅ ✅ ✅ ✅ ✅ ✅ ✅
    lucee7 ✅ ✅ ✅ ✅ ✅ ✅ ✅
    adobe2023 -- ✅ ✅ -- ✅ ✅ ✅
    adobe2025 ✅ ✅ ✅ -- ✅ ✅ ✅
    boxlang ✅ ✅ ✅ -- ✅ ✅ ✅
    lucee7 (TZ America/New_York) ✅
    adobe2025 (TZ America/New_York) ✅
    boxlang (TZ America/New_York) ✅

    A ':x: N tests' cell means the leg reported fewer than 4000 testcases (suite never ran; counted as failed). ':x: no result' means the leg returned no parseable result.

    Results for commit e890b4b.

  8. github-actions commented on Nov 10, 2026

    @github-actions
    ContributorAuthor

    Compatibility matrix is green again on e890b4b (run). Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    compat-matrix-failureNightly compatibility matrix is failingphase:1-stabilizeBugs, cleanup, CI fixes — tackle first

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions