Skip to content

[CI] Optimize GPU CI and temporarily skip XPU jobs #5219

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
scyyh11 merged 2 commits into from
Oct 9, 2026
Merged

[CI] Optimize GPU CI and temporarily skip XPU jobs #5219

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a103b6f
Speed up GPU CI checkout and dependency setup
scyyh11 Oct 8, 2026
2a4b350
Temporarily skip XPU CI while runners are unavailable
scyyh11 Oct 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
112 changes: 57 additions & 55 deletions .github/workflows/gpu_v100_ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,17 +10,18 @@ on:
workflow_dispatch:

concurrency:
group: ${{ github.event.pull_request.number }}-gpu-ci
group: ${{ github.event.pull_request.number || github.ref }}-gpu-ci
cancel-in-progress: true

env:
PADDLEX_IMAGE: ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlex/paddlex:paddlex_ce-paddlepaddle3.2.0-gpu-cuda11.8-cudnn8.6-trt8.6
PADDLETEST_TAR_URL: https://paddle-model-ecology.bj.bcebos.com/paddlex/tmp/PaddleTest.tar
PADDLETEST_TAR_URL: https://paddle-model-ecology.bj.bcebos.com/paddlex/tmp/CICE/PaddleTest.tar
PY_VERSION: "3.10"
CUDAID2: "0,1"
# Leave empty to use the paddle pre-installed in the image.
# Only used as a fallback install target when paddle is missing in the container.
PADDLE_INSTALL_VERSION: ""
# Keep the version used by PaddleTest explicit; ci_run.sh installs it once.
PADDLE_INSTALL_VERSION: "paddlepaddle-gpu==3.2.2"
# Functional tests do not need to flush the host's filesystem caches.
CI_CLEAR_HOST_CACHE: "0"

jobs:
CI_GPU:
Expand All @@ -37,65 +38,68 @@ jobs:
- name: Code Checkout
env:
docker_image: ${{ env.PADDLEX_IMAGE }}
CHECKOUT_SHA: ${{ github.sha }}
CHECKOUT_REF: ${{ github.ref }}
run: |
REPO="https://github.com/${{ github.repository }}.git"
FULL_REPO="${{ github.repository }}"
REPO_NAME="${FULL_REPO##*/}"
BASE_BRANCH="${{ github.base_ref }}"
# Pull proxy from the runner's global git config (set via
# `git config --global http.proxy ...` on the runner host).
HTTP_PROXY_VAL="$(git config --global --get http.proxy || true)"
HTTPS_PROXY_VAL="$(git config --global --get https.proxy || true)"
# Clean the repository directory before starting
docker run --rm --net=host -v $(pwd):/workspace -w /workspace \
docker run --rm --net=host -v "$(pwd):/workspace" -w /workspace \
-e "REPO_NAME=${REPO_NAME}" \
-e "BASE_BRANCH=${BASE_BRANCH}" \
${docker_image} /bin/bash -c '
if [ -d ${REPO_NAME} ]; then
"${docker_image}" /bin/bash -c '
if [ -d "${REPO_NAME}" ]; then
echo "Directory ${REPO_NAME} exists, removing it..."
rm -rf ${REPO_NAME}
rm -rf -- "${REPO_NAME}"
fi
'
git config --global user.name "PaddleCI"
git config --global user.email "paddle_ci@example.com"
export http_proxy="${HTTP_PROXY_VAL}"
export https_proxy="${HTTPS_PROXY_VAL}"
git clone ${REPO} ${REPO_NAME} -b ${BASE_BRANCH}
cd PaddleX
if [ "${{ github.event_name }}" = "pull_request" ]; then
git fetch origin pull/${{ github.event.pull_request.number }}/head:pr/${{ github.event.pull_request.number }}
git merge pr/${{ github.event.pull_request.number }}
git log -n 3 --oneline
git init "${REPO_NAME}"
cd "${REPO_NAME}"
git remote add origin "${REPO}"

# Old runner Git cannot fetch a raw SHA. Fetch the event ref, then
# verify its SHA so a moved ref cannot silently test another commit.
# PR refs point to GitHub's merge commit; depth 2 includes both parents.
for attempt in 1 2 3; do
if timeout 10m git -c http.lowSpeedLimit=1024 -c http.lowSpeedTime=60 \
fetch --progress --no-tags --depth=2 origin "${CHECKOUT_REF}"; then
break
fi
if [ "${attempt}" -eq 3 ]; then
echo "::error::Could not fetch ${CHECKOUT_REF} after 3 attempts"
exit 1
fi
sleep "$((attempt * 5))"
done
git checkout --detach FETCH_HEAD
if [ "$(git rev-parse HEAD)" != "${CHECKOUT_SHA}" ]; then
echo "::error::${CHECKOUT_REF} moved after this run was queued. Start a run for the latest revision."
exit 1
fi
git log -n 3 --oneline
if CI_BASE_SHA=$(git rev-parse --verify HEAD^1 2>/dev/null); then
echo "CI_BASE_SHA=${CI_BASE_SHA}" >> "$GITHUB_ENV"
else
git checkout ${{ github.sha }}
git log -n 3 --oneline
# A manually dispatched workflow may target the first commit.
echo "CI_BASE_SHA=$(git hash-object -w -t tree /dev/null)" >> "$GITHUB_ENV"
fi

- name: Detect test scope
id: scope
run: |
cd PaddleX
BASE_BRANCH="${{ github.base_ref }}"
# Collect files changed by the PR (vs. base branch)
if [ "${{ github.event_name }}" = "pull_request" ]; then
CHANGED=$(git diff --name-only "origin/${BASE_BRANCH}"...HEAD || true)
else
CHANGED=$(git diff --name-only HEAD~1 HEAD || true)
fi
# Compare with the merge commit's base parent (or the previous
# commit for workflow_dispatch). Do not turn diff errors into skips.
CHANGED=$(git diff --name-only "${CI_BASE_SHA}" HEAD)
echo "Changed files:"
echo "${CHANGED}"

filter_out() {
echo "${CHANGED}" | grep -v "^paddlex/pipeline" \
| grep -v "^paddlex/utils" \
| grep -v "^paddlex/paddlex_cli.py" \
| grep -v "^docs/" \
| grep -v "^api_examples/" \
| grep -v "^libs/" \
| grep -v "^paddlex/configs/pipelines" \
| grep -v "^paddlex/configs/pipeline" \
| wc -l
}
only_pipeline=$(echo "${CHANGED}" \
| grep -v "^paddlex/pipeline" \
| grep -v "^paddlex/utils" \
Expand Down Expand Up @@ -130,7 +134,12 @@ jobs:

# Skip if commit message contains test=document_fix
SKIP=0
if git log -1 | grep -qw "test=document_fix"; then
SKIP_COMMIT=HEAD
if [ "${GITHUB_EVENT_NAME}" = "pull_request" ]; then
# The generated merge message does not contain the PR head message.
SKIP_COMMIT=HEAD^2
fi
if git log -1 "${SKIP_COMMIT}" | grep -qw "test=document_fix"; then
echo "Detected test=document_fix, will skip tests"
SKIP=1
fi
Expand All @@ -149,7 +158,7 @@ jobs:
if: steps.scope.outputs.SKIP != '1'
run: |
rm -rf PaddleTest PaddleTest.tar
wget -q "${PADDLETEST_TAR_URL}" -O PaddleTest.tar
wget --timeout=30 --tries=3 "${PADDLETEST_TAR_URL}" -O PaddleTest.tar
tar -xf PaddleTest.tar && rm -f PaddleTest.tar
cp -r PaddleTest/models/PaddleX/ci PaddleX/

Expand Down Expand Up @@ -177,14 +186,16 @@ jobs:
--shm-size=128g \
--net=host \
-v "$(pwd)/PaddleX:/workspace" \
--mount "type=volume,source=paddlex_gpu_ci_pip_py${PY_VERSION},target=/root/.cache/pip" \
--mount "type=volume,source=paddlex_gpu_ci_precommit_py${PY_VERSION},target=/root/.cache/pre-commit" \
-w /workspace \
-e "PYTHONPATH=/workspace:" \
-e "http_proxy=${HTTP_PROXY_VAL}" \
-e "https_proxy=${HTTPS_PROXY_VAL}" \
-e "HTTP_PROXY=${HTTP_PROXY_VAL}" \
-e "HTTPS_PROXY=${HTTPS_PROXY_VAL}" \
-e "no_proxy=baidu.com,baidubce.com,bj.bcebos.com,mirrors.bfsu.edu.cn,paddlepaddle.org.cn,pypi.tuna.tsinghua.edu.cn,tuna.tsinghua.edu.cn,localhost,127.0.0.1" \
-e "NO_PROXY=baidu.com,baidubce.com,bj.bcebos.com,mirrors.bfsu.edu.cn,paddlepaddle.org.cn,pypi.tuna.tsinghua.edu.cn,tuna.tsinghua.edu.cn,localhost,127.0.0.1" \
-e "no_proxy=baidu.com,baidubce.com,bcebos.com,mirrors.bfsu.edu.cn,paddlepaddle.org.cn,pypi.tuna.tsinghua.edu.cn,tuna.tsinghua.edu.cn,localhost,127.0.0.1" \
-e "NO_PROXY=baidu.com,baidubce.com,bcebos.com,mirrors.bfsu.edu.cn,paddlepaddle.org.cn,pypi.tuna.tsinghua.edu.cn,tuna.tsinghua.edu.cn,localhost,127.0.0.1" \
-e "GIT_HTTP_LOW_SPEED_LIMIT=1000" \
-e "GIT_HTTP_LOW_SPEED_TIME=15" \
-e "AGILE_JOB_BUILD_ID=${{ github.run_id }}" \
Expand All @@ -196,6 +207,8 @@ jobs:
-e "MD_NUM=${MD_NUM}" \
-e "WITHOUT_MD_NUM=${WITHOUT_MD_NUM}" \
-e "TEST_RANGE=${TEST_RANGE}" \
-e "CI_BASE_SHA=${CI_BASE_SHA}" \
-e "CI_CLEAR_HOST_CACHE=${CI_CLEAR_HOST_CACHE}" \
-e "USE_NEW_INFERENCE=1" \
-e "PADDLE_INSTALL_VERSION=${PADDLE_INSTALL_VERSION}" \
"${docker_image}" \
Expand All @@ -215,18 +228,7 @@ jobs:

python${py_version} -m pip config set global.index-url https://mirrors.bfsu.edu.cn/pypi/web/simple

# Fallback install only when paddle is missing in the image.
if ! python${py_version} -c "import paddle" 2>/dev/null; then
echo "[INFO] paddle not installed in image, installing ${PADDLE_INSTALL_VERSION:-paddlepaddle-gpu==3.0.0}..."
env -u http_proxy -u https_proxy \
python${py_version} -m pip install ${PADDLE_INSTALL_VERSION:-paddlepaddle-gpu==3.0.0} \
-i https://www.paddlepaddle.org.cn/packages/stable/cu118/
else
echo "[INFO] paddle already installed in image, skip install"
fi

python${py_version} -c "import paddle; paddle.version.show()"

# PaddleTest installs and verifies PADDLE_INSTALL_VERSION.
git config --global --add safe.directory /workspace
export PADDLE_PDX_DISABLE_MODEL_SOURCE_CHECK=true
export DEVICE_ID=${cudaid2}
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/xpu_ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,9 @@ concurrency:

jobs:
CI_XPU:
# Temporarily pause all XPU jobs while no XPU runner is available.
# Remove this condition when XPU capacity is restored.
if: ${{ false }}
timeout-minutes: 60
runs-on: [self-hosted, XPU-P800-2Card]
steps:
Expand Down
Loading