Skip to content

[CI] Optimize GPU CI and temporarily skip XPU jobs - #5219

Merged
scyyh11 merged 2 commits into
PaddlePaddle:developfrom
scyyh11:CI/speed-optimization
Oct 9, 2026
Merged

scyyh11 merged 2 commits into
PaddlePaddle:developfrom
scyyh11:CI/speed-optimization

Conversation

@scyyh11

@scyyh11 scyyh11 commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

GPU CI currently clones the full repository history and can install Paddle twice before running tests. The external test script also spends substantial time syncing host filesystem caches even when drop_caches is read-only. XPU hardware is temporarily unavailable, so XPU jobs are paused for every trigger while retaining the CI_XPU check.

This updates the GPU workflow to use the published optimized CI code and its configuration:

  • Fetch the event ref with depth 2, progress, bounded retries and timeouts. Verify the checked-out SHA and use the merge commit's base parent for PR change detection. Preserve the test=document_fix marker from the PR head.
  • Install Paddle once through PaddleTest, keeping paddlepaddle-gpu==3.2.2 explicit, and persist pip and pre-commit caches in Docker volumes.
  • Bypass the proxy for the Paddle package index and BOS CDN domains.
  • Pass CI_BASE_SHA for pre-commit change detection and disable unnecessary GPU host cache flushing with CI_CLEAR_HOST_CACHE=0.
  • Bound archive download retries and separate manual-run concurrency by ref.

The XPU workflow uses a job-level if: ${{ false }} for both pull requests and manual dispatches. The job is skipped before a runner is allocated, while its check remains visible and does not block required-check enforcement. The existing XPU test steps are retained; remove the condition when hardware is available again.

This PR changes .github/workflows/gpu_v100_ci.yml and .github/workflows/xpu_ci.yml. PaddleTest remains externally maintained.

Validation:

  • 17 offline regression checks passed against the GPU workflow and downloaded archive using Linux, Git 1.8.3.1 and Bash 4.2. These cover checkout, PR scope, retries, moved refs, proxy preservation, setup failures, and model command failure handling. Package installers and pre-commit were stubbed.
  • A real shallow fetch with Git 1.8.3.1 produced a 4.6 MiB .git directory, compared with approximately 968 MiB in the previous GPU run. The local smoke test took 3 seconds; this is not a GPU-runner timing.
  • The full GPU rerun on the GPU optimization commit succeeded.
  • XPU YAML parsing and the unconditional job-skip configuration were checked.
  • git diff --check passed.

The previous GPU run spent approximately 35 minutes across 32 sync calls. Its inference-only scope differs from the full-training scope triggered by this workflow change, so total runtimes are not directly comparable.

changdazhou
changdazhou previously approved these changes Oct 9, 2026
@scyyh11 scyyh11 changed the title [CI] Speed up GPU checkout and dependency setup [CI] Optimize GPU CI and temporarily skip XPU jobs Oct 9, 2026
@scyyh11
scyyh11 merged commit c852371 into PaddlePaddle:develop Oct 9, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants