Skip to content

[GLUTEN-12850][VL] Support multiple Dev Container configurations - #12853

Draft
felipepessoto wants to merge 1 commit into
apache:mainfrom
felipepessoto:gluten-12850-multiple-devcontainers
Draft

[GLUTEN-12850][VL] Support multiple Dev Container configurations#12853
felipepessoto wants to merge 1 commit into
apache:mainfrom
felipepessoto:gluten-12850-multiple-devcontainers

Conversation

@felipepessoto

Copy link
Copy Markdown
Contributor

What changes are proposed in this pull request?

Fixes #12850.

Follow-up to #12778, which made the dynamic-link image the default development environment.

  • Keep .devcontainer/devcontainer.json as the default dynamic-link configuration for daily Velox development and Spark unit tests.
  • Add .devcontainer/velox-static/devcontainer.json for portable jar packaging and vcpkg issue reproduction.
  • Share the post-create setup while printing build instructions for the selected configuration.
  • Keep dynamic and static ccache/Maven state separate, and persist the static image's /var/cache/vcpkg binary cache.
  • Keep cloud-filesystem dependencies and native tests disabled in the static packaging workflow unless explicitly requested.
  • Select the arm64 vcpkg triplet automatically when the static configuration runs on arm64.
  • Document configuration selection in VS Code and Codespaces, image limitations, resource requirements and safe switching.

The Spark 4.1 instructions now explicitly switch the dynamic environment from its default JDK 8 to JDK 17:

export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
export PATH="$JAVA_HOME/bin:$PATH"
java -version  # must report 17

buildbundle-veloxbe.sh adds the Java 17 Maven profile and release target for Spark 4.x, but a Maven profile cannot switch the JDK that is already running. Without this step Scala fails with '17' is not a valid choice for '-release'.

How was this patch tested?

  • Parsed both configurations with @devcontainers/cli 0.88.0 read-configuration.

  • Ran both post-create variants and verified the dynamic Spark 4.1 instructions, static packaging instructions, unknown-variant failure and idempotent arm64 environment setup.

  • Built Spark 4.1 under OpenJDK 17 with the same Maven command used by buildbundle-veloxbe.sh: BUILD SUCCESS in 4:23, producing gluten-package-1.8.0-SNAPSHOT-4.1.jar.

  • Validated the final static command end to end on CentOS 9/GCC 12/JDK 17 using the Maven repository and vcpkg binary cache extracted from apache/gluten:vcpkg-centos-9:

    ./dev/buildbundle-veloxbe.sh --enable_vcpkg=ON --build_arrow=OFF \
                                 --spark_version=3.5

    The build processed 124 vcpkg packages, completed in 24:55 and produced cpp/build/releases/libgluten.so plus the 88 MB gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.8.0-SNAPSHOT.jar.

  • Ran bash -n, the license-header check, git diff --check and VS Code diagnostics with no errors.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: GitHub Copilot CLI gpt-5.6-sol

- Keep the dynamic-link environment as the default for daily development.
- Add a static-link configuration for portable jars and vcpkg reproduction.
- Share post-create setup while printing variant-specific build instructions.
- Persist separate ccache, Maven and vcpkg binary-cache volumes.
- Select the arm64 vcpkg triplet automatically in the static environment.
- Document configuration selection, limitations and safe switching.
- Make the JDK 17 switch explicit for Spark 4.1 builds so Maven does not run
  the Java 17 release target under JDK 8.

Generated-by: GitHub Copilot CLI gpt-5.6-sol

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings August 22, 2026 02:54
@github-actions github-actions Bot added the DOCS label Aug 22, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends Gluten’s Dev Container support to a multi-configuration setup (dynamic-link as the default plus an additional static-link option), sharing common post-create initialization while updating developer documentation to describe configuration selection, intended use cases, and build/test workflows (including explicit JDK 17 switching for Spark 4.x in the dynamic image).

Changes:

  • Adds a new .devcontainer/velox-static/devcontainer.json configuration for static-link packaging and vcpkg reproduction while keeping .devcontainer/devcontainer.json as the default dynamic-link workflow.
  • Updates the shared .devcontainer/post-create.sh to tailor printed instructions and arm64 static vcpkg environment setup based on the selected configuration.
  • Refreshes developer docs to describe configuration choice in VS Code/Codespaces and to document build/test commands for each configuration.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file
File Description
docs/developers/NewToGluten.md Updates Dev Container references to the new multi-configuration layout under .devcontainer/.
docs/developers/dev-container.md Documents configuration selection and provides updated dynamic/static build and test instructions (including Spark 4.x JDK 17 guidance).
.devcontainer/devcontainer.json Marks the default dynamic-link config as the daily-dev option and sets GLUTEN_DEV_CONTAINER_VARIANT=velox-dynamic.
.devcontainer/velox-static/devcontainer.json Introduces a static-link configuration with separate persisted caches (including /var/cache/vcpkg) and higher host requirements.
.devcontainer/post-create.sh Shares setup across configs while printing variant-specific instructions and adding arm64 static vcpkg environment exports.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@felipepessoto
felipepessoto marked this pull request as draft August 22, 2026 03:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[VL] Support multiple Dev Container configurations

2 participants