[Shardul369-coder]

Summary

Fixes the Spark Examples E2E failure caused by a mismatch between the Spark Operator controller and its Helm chart RBAC permissions.

Changes

• Add optional SPARK_OPERATOR_CHART_PATH support to hack/e2e-setup-cluster.sh.
• Preserve the existing published chart and version as defaults for backward compatibility.
• Pass the local Spark Operator chart path through the Makefile.
• Update the Spark Examples E2E GitHub Actions workflow to use the locally checked-out Spark Operator chart.

Validation

• Bash syntax check passed.
• Local Spark Operator controller and Helm chart installation succeeded.
• Verified controller service account has create and patch permissions for NetworkPolicies.
• Spark Connect notebook E2E test completed successfully.

Closes #933

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign astefanutti for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[github-actions]

🎉 Welcome to the Kubeflow SDK! 🎉

Thanks for opening your first PR! We're happy to have you as part of our community 🚀

Here's what happens next:

• If you haven't already, please check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards
• Our team will review your PR soon! cc @kubeflow/kubeflow-sdk-team

Join the community:

• Slack: Join our #kubeflow-ml-experience and #kubeflow-trainer Slack channels
• Meetings: Attend the Kubeflow SDK and ML Experience bi-weekly meetings

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

[thisis-Shitanshu]

Thanks @Shardul369-coder. I validated this locally on a fresh Kind 1.32.0 cluster using the controller and Helm chart from the same current spark-operator master commit.

The fix works as intended: the required networkpolicies create/patch permissions are present without any manual RBAC changes, and I ran the full Spark Examples E2E notebook set successfully.

I have two small changes I'd suggest before LGTM:

1. When SPARK_OPERATOR_CHART_PATH is set, skip helm repo add/update since the local chart is used directly.
2. In local-chart mode, avoid reporting Spark Operator version: 2.1.0, since that value only applies to the published-chart fallback. Reporting the local chart path instead would be more accurate.

With those adjustments, the implementation looks good to me.

[thisis-Shitanshu]

/ok-to-test

[Shardul369-coder]

Thanks @thisis-Shitanshu for the validation and feedback! Both suggestions make sense. I'll implement these changes and push an update shortly.

[thisis-Shitanshu]

Thanks @Shardul369-coder for picking this up and for the quick turnaround on the fix.

Since #930 already covers this issue as part of a broader cleanup of the Spark E2E harness, I think we should stop the remaining updates here and consolidate the implementation and review there as Tariq suggested.

I also validated 930 together with #910 locally, including the complete Spark example notebook suite, and the combined setup passes.

@tariq-hasan, could you close this PR in favor of 930 so we keep a single implementation path?

[Shardul369-coder]

Thanks @thisis-Shitanshu for the clarification! I’ll pause further updates to this PR and consolidate any further implementation and review under #930, as suggested. I appreciate the validation and guidance.

[tariq-hasan]

Thanks again for the contribution @Shardul369-coder!
Let me close this PR since we consolidate as part of #930.
Appreciate your review on that PR as well.
/close

[google-oss-prow]

@tariq-hasan: Closed this PR.

Details

In response to this:

Thanks again for the contribution @Shardul369-coder!
Let me close this PR since we consolidate as part of #930.
Appreciate your review on that PR as well.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.