[Shardul369-coder]

Closes #909

Hi Kubeflow maintainers! 👋

This PR aligns the Kubeflow SDK's Spark defaults with Spark Operator 4.2.1.

Changes made:

• Updated DEFAULT_SPARK_VERSION to 4.2.1.
• Updated DEFAULT_SPARK_IMAGE to apache/spark:4.2.1.
• Upgraded pyspark-client to 4.2.1.
• Regenerated uv.lock to reflect the dependency changes.

Validation:

• 225 targeted Spark tests passed.
• All configured pre-commit checks passed.

Thank you for reviewing my contribution! I welcome any feedback or suggestions.

[github-actions]

🎉 Welcome to the Kubeflow SDK! 🎉

Thanks for opening your first PR! We're happy to have you as part of our community 🚀

Here's what happens next:

• If you haven't already, please check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards
• Our team will review your PR soon! cc @kubeflow/kubeflow-sdk-team

Join the community:

• Slack: Join our #kubeflow-ml-experience and #kubeflow-trainer Slack channels
• Meetings: Attend the Kubeflow SDK and ML Experience bi-weekly meetings

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign electronic-waste for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[andreyvelich]

/ok-to-test

[andreyvelich]

/hold for this PR to release: kubeflow/spark-operator#3245

[thisis-Shitanshu]

Thanks @Shardul369-coder for picking this up!

kubeflow/spark-operator#3245 has now merged, and the final Spark runtime version there is 4.2.1 rather than 4.2.0.

Could you please update this PR to align with 4.2.1 across:

• DEFAULT_SPARK_VERSION
• DEFAULT_SPARK_IMAGE
• pyspark-client
• the regenerated uv.lock

Please also update the PR title from 4.2.0 to 4.2.1.

pyspark-client==4.2.1 is available, so the SDK client and Spark runtime can remain aligned as intended.

[thisis-Shitanshu]

Hi @andreyvelich, small update: kubeflow/spark-operator#3245 has now been merged, with the final Spark runtime updated to 4.2.1.

PR #910 is currently aligned to 4.2.0, so I've requested a small update to 4.2.1.

Once that update is in and the checks pass, this should be aligned with the Spark Operator change, and the hold can be removed. Thanks!

[Shardul369-coder]

Hi @thisis-Shitanshu,

Thanks for the review and for pointing this out!

I'll update the Spark SDK defaults to 4.2.1, including DEFAULT_SPARK_VERSION, DEFAULT_SPARK_IMAGE, and pyspark-client. I'll also regenerate uv.lock, update the PR title, and run the relevant tests and checks.

I'll push the updates to the existing PR #910.

Thanks for your guidance!

[thisis-Shitanshu]

Thanks @Shardul369-coder, the 4.2.1 update looks good.

I validated the updated PR locally:

• uv lock --check ✅
• uv sync --extra spark ✅
• confirmed pyspark-client==4.2.1
• confirmed DEFAULT_SPARK_VERSION == "4.2.1"
• confirmed DEFAULT_SPARK_IMAGE == "apache/spark:4.2.1"
• targeted Spark Kubernetes tests: 226 passed
• Ruff lint and format checks passed

This is now aligned with the final Spark 4.2.1 runtime merged in KSO

@andreyvelich, the upstream PR has merged and the requested 4.2.1 alignment here is now complete and validated, so this should be ready for the hold to be removed.

/lgtm

[Shardul369-coder]

Thank you, @thisis-Shitanshu, for reviewing and validating my changes!

I really appreciate your guidance throughout this PR. Glad to see the Spark 4.2.1 alignment is now complete and validated.

Thanks again for your support!

[thisis-Shitanshu]

/lgtm cancel

[Shardul369-coder]

@thisis-Shitanshu I investigated the Spark Examples E2E failure and wanted to get your guidance before making any changes.

Here's what I found:

1. CI checks out the current kubeflow/spark-operator master branch and builds the controller image from it.
2. The SDK E2E setup still installs Helm chart 2.1.0 by default.
3. The published chart 2.1.0 does not include the required networkpolicies RBAC permissions.
4. The upstream controller commit used by CI (1a7133c) includes NetworkPolicy permissions (create and patch) in its RBAC and Helm chart templates.
5. In my local Kind reproduction, the Spark 4.2.1 notebook failed before I manually added NetworkPolicy RBAC in the spark-test namespace. After adding the permissions, the notebook completed successfully.

This suggests a possible mismatch between the newer controller image and the older Helm chart RBAC configuration. I haven't captured the exact authorization error from the failed CI driver's logs, so I don't want to assume this is definitively the root cause.

Would you recommend updating the E2E setup to use a compatible Helm chart, adding the required NetworkPolicy RBAC there, or taking another approach?

I haven't made any additional changes to the SDK code or pushed another commit. I'll proceed based on your guidance.

[thisis-Shitanshu]

Thanks again for investigating this.

I opened #933 to track the E2E setup issue separately.

I was able to reproduce and isolate the failure locally. The same notebook fails with the SDK 4.0.4 baseline when the current Spark Operator controller is used with Helm chart 2.1.0, due to the missing networkpolicies create/patch RBAC. After adding only those permissions, the notebook passes with both Spark 4.0.4 and 4.2.1.

So this looks separate from the Spark 4.2.1 changes in this PR.

@Shardul369-coder since you've already spent time debugging this, feel free to pick up #933 if you'd like. I added the reproduction and a possible direction there.

[Shardul369-coder]

Thanks, @thisis-Shitanshu, for investigating and isolating the E2E failure. This confirms that the issue is independent of the Spark 4.2.1 changes in this PR.

I'll keep #910 focused on the Spark version alignment and take up the E2E setup fix separately in #933.

Thanks for the guidance!

[thisis-Shitanshu]

I validated this PR together with #930 locally.

Using Spark 4.2.1 / pyspark-client 4.2.1 on a fresh Kubernetes 1.32.11 cluster, with the Spark Operator controller, Helm chart, CRDs, RBAC, and workload service account all coming from the same current Spark Operator checkout, all 8 Spark example notebooks passed.

So the previous E2E failure was caused by the old E2E harness drift rather than the Spark 4.2.1 changes in this PR.

I think we should wait for #930 to merge, then update/rebase this PR onto the new harness and let the Spark Examples E2E checks run again there. Once those are green, this should be ready to proceed for #909 🚀

[Shardul369-coder]

Thanks @thisis-Shitanshu for validating #910 together with #930. Glad to see all 8 Spark example notebooks passing with Spark 4.2.1 on Kubernetes 1.32.11. I'll keep #910 focused on the Spark 4.2.1 alignment and consolidate the E2E harness changes under #930 as suggested. Appreciate your guidance!