[tariq-hasan]

What this PR does / why we need it:

Makes the Spark E2E tests run against an unmodified, default Helm install of the Spark Operator, so the SDK catches upstream drift instead of masking it.

The scope of changes covered in this PR is as follows:

• Remove vendored CRD and pull the latest upstream artifacts from the master head for controller image and helm chart
• Remove the bespoke RBAC and use the chart-provisioned service account so drivers can create executors
• Align Spark E2E workflow with Trainer workflow
• Move tool installation as Makefile targets
• Test over the full suite of K8s versions that are also tested in spark operator project

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):

Fixes #933

Checklist:

• Docs included if any changes are user facing

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign andreyvelich for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tariq-hasan]

/cc @Neha-Nimsatkar @RobuRishabh @vikas-saxena02 @Shekharrajak

[google-oss-prow]

@tariq-hasan: GitHub didn't allow me to request PR reviews from the following users: Neha-Nimsatkar.

Note that only kubeflow members and repo collaborators can review this PR, and authors cannot review their own PRs.

Details

In response to this:

/cc @Neha-Nimsatkar @RobuRishabh @vikas-saxena02 @Shekharrajak

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[thisis-Shitanshu]

Validated this locally as well.

I combined this PR with #910, which updates the SDK defaults to Spark 4.2.1, and tested against a fresh Kind Kubernetes 1.32.11 cluster.

The setup used Spark Operator master at 511100c, with the controller image, CRD hook image, Helm chart, CRDs, RBAC, and Spark workload service account all coming from that same checkout.

I also verified:

• controller networkpolicies create/patch: yes
• Spark workload SA pod permissions: yes
• ConfigMap deletecollection: yes

Then ran the complete Spark example notebook suite with Spark 4.2.1 / pyspark-client 4.2.1:

• batch_job_lifecycle ✅
• batch_func_job_lifecycle ✅
• batch_job_options ✅
• batch_failed_job ✅
• connect_existing_session ✅
• demo_existing_sparkconnect ✅
• spark_advanced_options ✅
• spark_connect_simple ✅

This also confirms that the failure previously seen on #910 was caused by the old E2E controller/chart/RBAC drift rather than the Spark 4.2.1 update itself.

[tariq-hasan]

Thanks for the verification @thisis-Shitanshu!

[Shardul369-coder]

@tariq-hasan, I noticed a potential edge case in the cluster reuse and cleanup logic.

For example, if the current kubectl context points to kind-dev-cluster while SPARK_TEST_CLUSTER=spark-test, ensure_cluster() may reuse dev-cluster. However, delete_cluster() always attempts to delete spark-test.

Since the setup and cleanup commands run in separate processes, the name of the reused cluster isn't retained between them.

Could we ensure that cleanup targets the same cluster selected during setup, without accidentally deleting an unrelated cluster?

Happy to help implement a solution if needed.

[tariq-hasan]

@tariq-hasan, I noticed a potential edge case in the cluster reuse and cleanup logic.

For example, if the current kubectl context points to kind-dev-cluster while SPARK_TEST_CLUSTER=spark-test, ensure_cluster() may reuse dev-cluster. However, delete_cluster() always attempts to delete spark-test.

Since the setup and cleanup commands run in separate processes, the name of the reused cluster isn't retained between them.

Could we ensure that cleanup targets the same cluster selected during setup, without accidentally deleting an unrelated cluster?

Happy to help implement a solution if needed.

Thanks for the feedback. I have addressed this in one of the latest commits.

[tariq-hasan]

The Python models have also been bumped to 2.6.0 in the PR given that they have been released now: https://pypi.org/project/kubeflow-spark-api/.

[tariq-hasan]

/cc @andreyvelich

This PR unblocks other Spark SDK PRs on their failing CI.