feat(spark): support physical CPU requests and limits for Spark Connect by DUKartik · Pull Request #896 · kubeflow/sdk

DUKartik · 2026-10-23T04:06:23Z

Closes #578
Part of #837

Summary

Implements the SDK side of #578, Track SparkConnect CPU resource support after SparkConnect CRD enhancements.

For Spark Connect sessions, the existing cpu resource now represents physical Kubernetes CPU rather than Spark cores. The SDK maps it to equal coreRequest and coreLimit values on the SparkConnect CR:

SDK setting	SparkConnect field	Meaning
`Driver.resources["cpu"]`	`spec.server.coreRequest` and `spec.server.coreLimit`	Physical Kubernetes CPU request and limit for the server pod
`resources_per_executor["cpu"]` or `Executor.resources_per_executor["cpu"]`	`spec.executor.coreRequest` and `spec.executor.coreLimit`	Physical Kubernetes CPU request and limit for executor pods

The SDK no longer sets Spark cores for Spark Connect. When cpu is omitted, the physical CPU fields are left unset and Kubernetes/Spark defaults apply.

The operator-side support is kubeflow/spark-operator#3096 (the implementation for #2957). The SDK behavior requires an operator version containing #3096.

Motivation

The SparkConnect CRD previously represented CPU only through cores, so the SDK could only express Spark task parallelism. Users could not independently request a fractional or guaranteed Kubernetes CPU amount for the server and executor pods.

The SparkConnect CRD now has coreRequest and coreLimit. This PR exposes those fields through the existing Driver.resources and Executor.resources_per_executor dictionaries while keeping the SDK API unchanged:

Existing cpu is reused for physical Kubernetes CPU.
No new public resource keys are introduced.
Requests and limits are always equal, matching the simpler Trainer SDK behavior.
Spark cores are not configured by the SDK for Spark Connect.
Batch jobs remain unchanged.

What changed

Resource mapping

get_spark_connect_driver_spec() maps Driver.resources["cpu"] to:
- spec.server.coreRequest
- spec.server.coreLimit
get_spark_connect_executor_spec() maps executor cpu to:
- spec.executor.coreRequest
- spec.executor.coreLimit
Request and limit always receive the same value.
Spark cores is left unset for both server and executor specs.
If cpu is omitted, neither coreRequest nor coreLimit is set.
Existing driver/executor memory defaults and executor precedence rules are preserved.

The CPU value can be a Kubernetes quantity such as "500m", "0.5", or "2". Existing SDK CPU validation remains in place.

No public API changes

The following changes from the earlier version of this PR were removed:

New cpu_request and cpu_limit resource keys.
Public API/type/docstring changes.
Changes to SparkClient.connect() and submit_job() signatures.
Changes to backend interfaces or backend resource types.
New resource constants and separate request/limit resolution.
Effective CPU request/limit validation and fallback logic.
The backend serialization test.
Vendor CRD changes.
Papermill parameters for separate CPU request/limit values.
The batch-job note about unsupported new keys.

Batch jobs

Batch-job behavior is unchanged:

Batch jobs continue to use cpu as Spark cores.
Batch jobs keep their existing memory handling and 1-core default.
This PR does not modify batch-job resource mapping.

Docs, examples, and tests

Documented the distinction between Spark cores and physical Kubernetes CPU in:
- docs/source/spark/sessions.rst
- docs/source/spark/index.rst
Added a physical CPU table and example to the session documentation.
Updated the advanced-options example to keep existing CPU parameters and assert that the resulting coreRequest/coreLimit values match.
Updated unit tests to cover:
- Equal physical CPU request/limit mapping for server and executor.
- Fractional CPU quantities.
- Absence of cores in Spark Connect specs.
- No physical CPU fields when cpu is omitted.
- Driver/executor resource independence and existing precedence rules.

Usage

from kubeflow.spark import Driver, Executor

spark = client.connect(
    driver=Driver(
        resources={
            "cpu": "500m",
            "memory": "1Gi",
        },
    ),
    executor=Executor(
        num_instances=3,
        resources_per_executor={
            "cpu": "2",
            "memory": "4Gi",
        },
    ),
)

In this example:

The server pod receives a 500m CPU request and a 500m CPU limit.
Each executor pod receives a 2 CPU request and a 2 CPU limit.
Spark cores is not set by the SDK for Spark Connect.
Spark task slots remain at Spark's default configuration unless configured through spark_conf.

Existing usage with only cpu and memory remains valid, but cpu now controls physical Kubernetes CPU for Spark Connect rather than Spark cores. This is a behavioral change for existing Spark Connect users.

Compatibility and operational notes

Users need a Spark Operator release containing kubeflow/spark-operator#3096.
Server CPU changes take effect only after the server pod is recreated.
Spark Connect sessions no longer have spec.server.cores or spec.executor.cores set by the SDK.
Users who need Spark task-slot tuning can set Spark configuration properties through spark_conf.
Batch jobs retain their existing cpu-as-Spark-cores behavior.
The SDK does not infer a physical 1-CPU default for Spark Connect. Kubernetes and Spark defaults apply when cpu is omitted.

Validation

Local validation:

make verify
uv run pytest -q kubeflow/spark

Results:

make verify: passed
Spark unit tests: 286 passed
Sphinx documentation build: succeeded, with existing unrelated warnings
Notebook JSON validation: passed

The end-to-end validation from the earlier version of this PR used separate cpu_request/cpu_limit keys and is not applicable to the simplified mapping. E2E should be rerun after the released Spark Operator and kubeflow-spark-api versions are available.

Dependency and release readiness

The CR builders require kubeflow-spark-api fields core_request and core_limit, which are not available in the current stable PyPI release. Development currently resolves the API from the Spark Operator master revision containing #3096 through the existing [tool.uv] override-dependencies entry and uv.lock.

Before merge/release:

Land kubeflow/spark-operator#3226 and publish kubeflow-spark-api v2.6.0rc0.
Bump pyproject.toml to the released kubeflow-spark-api version and align uv.lock.
Pin the e2e setup to a Spark Operator release containing #3096.
Rerun the advanced-options notebook e2e against the released operator/API versions.
Trigger CI against the released dependencies.

Key files

Area	Files
Resource mapping	`kubeflow/spark/backends/kubernetes/utils.py`
Unit tests	`kubeflow/spark/backends/kubernetes/utils_test.py`
Documentation	`docs/source/spark/sessions.rst`, `docs/source/spark/index.rst`
Example	`examples/spark/spark_advanced_options.ipynb`
Development lock	`uv.lock`

Signed-off-by: Kartik Nath Khatri <masterbro770@gmail.com>

review-notebook-app · 2026-10-23T04:06:29Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

google-oss-prow · 2026-10-23T04:06:31Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign electronic-waste for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

github-actions · 2026-10-23T04:06:34Z

🎉 Welcome to the Kubeflow SDK! 🎉

Thanks for opening your first PR! We're happy to have you as part of our community 🚀

Here's what happens next:

If you haven't already, please check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards
Our team will review your PR soon! cc @kubeflow/kubeflow-sdk-team

Join the community:

Slack: Join our #kubeflow-ml-experience and #kubeflow-trainer Slack channels
Meetings: Attend the Kubeflow SDK and ML Experience bi-weekly meetings

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

tariq-hasan

Thanks for raising the PR @DUKartik!

Let's keep the implementation fairly simple and make CPU requests equal CPU limits - same as what is done in the Trainer SDK. A data scientist should not have to worry about requests and limits. The overall goal of this PR is primarily to fix what resource specification means and how it's mapped to the control plane. The complexity of requests vs limits falls outside the scope of use cases currently considered for Spark SDK.

On another note I will ask you to bump kubeflow-spark-api to v2.6.0rc0 in pyproject.toml once kubeflow/spark-operator#3226 is merged and v2.6.0-rc0 is cut in preparation for v2.6.0 release. We will then trigger the CI and test the flow in this PR. For now my inline comments are primarily focused on the general shape of the PR.

DUKartik · 2026-10-27T04:13:23Z

hi @tariq-hasan,
Thanks for the review! I just had one doubt.
In PR#3096 of sparkconnect : cores and coreRequest/coreLimit are independent. Should the SDK’s existing cpu key now control only physical CPU request/limit and leave Spark cores at Spark defaults, or should cpu also continue setting Spark task slots via cores?

tariq-hasan · 2026-10-27T04:17:21Z

There is no need to configure Spark cores from the SDK. This was a workaround for a missing feature. Now that we have requests and limits we do not need to use this workaround any longer. So Spark cores should not be configured in the SDK.

Update resource types in SparkClient, RuntimeBackend, and KubernetesBackend to accept only string CPU values. Simplify driver and executor resource documentation, remove deprecated CPU request and limit fields, and update tests to reflect the revised validation. Allow any valid Kubernetes resource name in Driver and Executor resource configurations. Signed-off-by: Kartik Nath Khatri <masterbro770@gmail.com>

DUKartik · 2026-10-31T04:41:05Z

Hi @tariq-hasan,
I've updated PR #896 based on your feedback and simplified the implementation.

I noticed the e2e setup still uses a vendored SparkConnect CRD that lacks coreRequest and coreLimit. Since this CRD is planned for removal in #777, would you recommend using the CRD from the Spark Operator checkout instead?

Could you please review the latest changes and share your thoughts?
Thanks!

tariq-hasan · 2026-11-04T15:13:35Z

/ok-to-test
/retest

tariq-hasan

Thanks @DUKartik for addressing the prior comments!

Correcting my prior comment pinning to the latest commit on the spark operator master branch is good enough to pull and test with the latest Python models.

I have added some additional review comments.

Co-authored-by: Tariq Hasan <mmtariquehsn@gmail.com> Signed-off-by: Kartik Nath Khatri <kartikkhatri8210@gmail.com>

…and remove unused CPU validation code Signed-off-by: Kartik Nath Khatri <kartikkhatri8210@gmail.com>

…-resources Signed-off-by: Kartik Nath Khatri <kartikkhatri8210@gmail.com> # Conflicts: # kubeflow/spark/backends/kubernetes/utils.py

DUKartik · 2026-11-06T06:13:12Z

Hi @tariq-hasan,

I've addressed all the requested changes and updated the PR accordingly.

I'm currently waiting for Spark Operator release PR #3287 to be merged and kubeflow-spark-api v2.6.0rc0 to be published on PyPI. Once it's available, I'll update the dependency, run the final validations, and push the changes.

Thanks for your feedback and guidance!

Use the published 2.6.0rc0 models for the physical CPU request/limit fields and remove the Spark API master override so development and CI resolve the release artifact from PyPI. Signed-off-by: Kartik Nath Khatri <kartikkhatri8210@gmail.com>

tariq-hasan · 2026-11-11T09:08:06Z

If you include the latest commits from master once #930 merges that should help pass the tests - #930 pins the latest commit from spark operator, updates the Python models and also updates the E2E tests to more accurately reflect upstream operator behavior.

tariq-hasan

I have a few additional comments. Otherwise lgtm from me.
Thanks for working on the PR.

tariq-hasan · 2026-11-11T08:51:01Z

+        core_request=quantity,
+        core_limit=quantity,


Thinking again we should include cores as well since they determine the number of tasks that can run.

Suggested change

core_request=quantity,

core_limit=quantity,

cores=_validate_cpu_value(cpu),

core_request=quantity,

core_limit=quantity,

tariq-hasan · 2026-11-11T08:51:34Z

+        core_request=cpu,
+        core_limit=cpu,


Same here.

Suggested change

core_request=cpu,

core_limit=cpu,

cores=_validate_cpu_value(cpu),

core_request=cpu,

core_limit=cpu,

tariq-hasan · 2026-11-11T08:52:31Z

        assert spark_connect.spec.spark_version == constants.DEFAULT_SPARK_VERSION
        assert spark_connect.spec.executor.instances == constants.DEFAULT_NUM_EXECUTORS
-        assert spark_connect.spec.executor.cores == constants.DEFAULT_EXECUTOR_CPU
+        assert spark_connect.spec.executor.cores is None


Suggested change

assert spark_connect.spec.executor.cores is None

assert spark_connect.spec.executor.cores == constants.DEFAULT_EXECUTOR_CPU

tariq-hasan · 2026-11-11T08:53:09Z


    elif test_case.name == "spark connect cr with executor resources":
-        assert spark_connect.spec.executor.cores == 2
+        assert spark_connect.spec.executor.cores is None


Suggested change

assert spark_connect.spec.executor.cores is None

assert spark_connect.spec.executor.cores == 2

tariq-hasan · 2026-11-11T08:53:42Z

+        assert spark_connect.spec.server.core_limit.actual_instance == "500m"
+
+    elif test_case.name == "spark connect cr with fractional executor cpu":
+        assert spark_connect.spec.executor.cores is None


Suggested change

assert spark_connect.spec.executor.cores is None

assert spark_connect.spec.executor.cores == 1

tariq-hasan · 2026-11-11T08:54:57Z

    elif test_case.name == "kep107 level2 simple mode":
        assert spark_connect.spec.executor.instances == 5
-        assert spark_connect.spec.executor.cores == 5
+        assert spark_connect.spec.executor.cores is None


Suggested change

assert spark_connect.spec.executor.cores is None

assert spark_connect.spec.executor.cores == 5

tariq-hasan · 2026-11-11T08:55:15Z

        assert spark_connect.spec.server.template.spec.service_account_name == "spark-driver-prod"
        assert spark_connect.spec.executor.instances == 20
-        assert spark_connect.spec.executor.cores == 8
+        assert spark_connect.spec.executor.cores is None


Suggested change

assert spark_connect.spec.executor.cores is None

assert spark_connect.spec.executor.cores == 8

tariq-hasan · 2026-11-11T08:55:43Z


    assert spec.instances == constants.DEFAULT_NUM_EXECUTORS
-    assert spec.cores == constants.DEFAULT_EXECUTOR_CPU
+    assert spec.cores is None


Suggested change

assert spec.cores is None

assert spec.cores == constants.DEFAULT_EXECUTOR_CPU

tariq-hasan · 2026-11-11T08:56:05Z

    elif test_case.name == "file job with executor configuration":
        assert app.spec.executor.instances == 3
-        assert app.spec.executor.cores == 2
+        assert app.spec.executor.cores is None


Suggested change

assert app.spec.executor.cores is None

assert app.spec.executor.cores == 2

tariq-hasan · 2026-11-11T08:56:24Z

    elif test_case.name == "function job with executor configuration":
        assert app.spec.executor.instances == 3
-        assert app.spec.executor.cores == 2
+        assert app.spec.executor.cores is None


Suggested change

assert app.spec.executor.cores is None

assert app.spec.executor.cores == 2

DUKartik added 6 commits October 22, 2026 14:20

chore(spark): sync Spark operator artifacts

2ea3659

Signed-off-by: Kartik Nath Khatri <masterbro770@gmail.com>

feat(spark): map CPU requests and limits to SparkConnect CR

d1148be

Signed-off-by: Kartik Nath Khatri <masterbro770@gmail.com>

test(spark): cover SparkConnect CPU request and limit mapping

f4bd489

Signed-off-by: Kartik Nath Khatri <masterbro770@gmail.com>

docs(spark): document SparkConnect CPU request and limit support

6759531

Signed-off-by: Kartik Nath Khatri <masterbro770@gmail.com>

test(spark): exercise CPU requests and limits in advanced-options e2e

a8b605e

Signed-off-by: Kartik Nath Khatri <masterbro770@gmail.com>

feat(spark): support CPU requests and limits in resource configuration

75f03eb

Signed-off-by: Kartik Nath Khatri <masterbro770@gmail.com>

github-project-automation Bot added this to Kubeflow SDK Oct 23, 2026

github-project-automation Bot moved this to Ready to Merge in Kubeflow SDK Oct 23, 2026

google-oss-prow Bot requested review from Shekharrajak and tariq-hasan October 23, 2026 04:06

google-oss-prow Bot added the size/XL label Oct 23, 2026

tariq-hasan suggested changes Oct 25, 2026

View reviewed changes

github-project-automation Bot moved this from Ready to Merge to Changes Requested in Kubeflow SDK Oct 25, 2026

google-oss-prow Bot assigned tariq-hasan Oct 25, 2026

google-oss-prow Bot added size/L and removed size/XL labels Oct 27, 2026

DUKartik requested a review from tariq-hasan October 27, 2026 11:34

google-oss-prow Bot added the ok-to-test label Nov 4, 2026

tariq-hasan suggested changes Nov 4, 2026

View reviewed changes

DUKartik and others added 3 commits November 5, 2026 22:39

Update kubeflow/spark/backends/kubernetes/utils.py

71e3cfb

Co-authored-by: Tariq Hasan <mmtariquehsn@gmail.com> Signed-off-by: Kartik Nath Khatri <kartikkhatri8210@gmail.com>

refactor(spark): update resource handling for Spark Connect sessions …

cae4b0f

…and remove unused CPU validation code Signed-off-by: Kartik Nath Khatri <kartikkhatri8210@gmail.com>

Merge remote-tracking branch 'origin/main' into feat/sparkconnect-cpu…

f43d773

…-resources Signed-off-by: Kartik Nath Khatri <kartikkhatri8210@gmail.com> # Conflicts: # kubeflow/spark/backends/kubernetes/utils.py

tariq-hasan suggested changes Nov 11, 2026

View reviewed changes

	assert spark_connect.spec.executor.cores is None
	assert spark_connect.spec.executor.cores == constants.DEFAULT_EXECUTOR_CPU

	assert spec.cores is None
	assert spec.cores == constants.DEFAULT_EXECUTOR_CPU

	assert app.spec.executor.cores is None
	assert app.spec.executor.cores == 2

Conversation

DUKartik commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

What changed

Resource mapping

No public API changes

Batch jobs

Docs, examples, and tests

Usage

Compatibility and operational notes

Validation

Dependency and release readiness

Key files

Uh oh!

review-notebook-app Bot commented Oct 23, 2026

Uh oh!

google-oss-prow Bot commented Oct 23, 2026

Uh oh!

github-actions Bot commented Oct 23, 2026

Uh oh!

tariq-hasan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DUKartik commented Oct 27, 2026

Uh oh!

tariq-hasan commented Oct 27, 2026

Uh oh!

DUKartik commented Oct 31, 2026

Uh oh!

tariq-hasan commented Nov 4, 2026

Uh oh!

tariq-hasan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DUKartik commented Nov 6, 2026

Uh oh!

tariq-hasan commented Nov 11, 2026

Uh oh!

tariq-hasan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DUKartik commented Oct 23, 2026 •

edited

Loading