Support users to set more fields in suggetion deployment

/kind feature

Describe the solution you'd like
[A clear and concise description of what you want to happen.]

Is it possible to allow users to set more fields in suggetion deployment like we set the trailSpec
such as imagePullSecrets 、args、command and so on

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

I think it is a useful feature. WDYT @kubeflow/wg-automl-leads

@tingweiwu

Thank you for creating this @tingweiwu!
We can allow to specify the Deployment spec or spec.template.spec in Config Suggestion Settings, but then setting just Suggestion image will be more complicated for our users since they have to follow the correct API structure.
For example:

"random": {
  "containers": [
    {
      "image": "docker.io/andreyvelichkevich/suggestion-chocolate:latest"
    }
  ]
 },

Does it look good to us @gaocegege @johnugeorge @tingweiwu ?

It seems the Suggestion settings with configmap don't support imagePullSecrets and other similar spec settings in Deployment

It will probably be more flexible for user to set deployment attributes in algorithmSpec template in different scenarios.
two simple examples as the following
case 1: set imagePullSecrets and resource limits

apiVersion: kubeflow.org/v1beta1
kind: Experiment
metadata:
  namespace: kubeflow-test
  name: job-test
spec:
    algorithm:
      algorithmName: random
      algorithmSpec:
        template:
          spec:
            containers:
            - image: docker.io/kubeflowkatib/suggestion-hyperopt:latest
              resources:
                limits:
                  cpu: 500m
                  ephemeral-storage: 5Gi
                  memory: 100Mi
                requests:
                  cpu: 50m
                  ephemeral-storage: 500Mi
                  memory: 10Mi 
            imagePullSecrets:
            - name: default-secret

case 2: set start commond and persistent volume, as the new suggetion service may not build in image.

apiVersion: kubeflow.org/v1beta1
kind: Experiment
metadata:
  namespace: kubeflow-test
  name: job-test
spec:
    algorithm:
      algorithmName: random
      algorithmSpec:
        template:
          spec:
            containers:
            - image: docker.io/kubeflowkatib/suggestion-new:latest
              args:
              - python /opt/apps/main.py 
              command:
              - /bin/bash
              - -c
              volumeMounts:
                - mountPath: /opt/apps
                  name: kubeflow-test-pvc
                  readOnly: true
            imagePullSecrets:
            - name: default-secret  
            volumes:
            - name: kubeflow-test-pvc
              persistentVolumeClaim:
                claimName: kubeflow-test-pvc

@tingweiwu

I think we should consider if we can support SuggestionSpec as @tingweiwu said. WDYT @andreyvelich

@johnugeorge

Basically, Katib config was designed to simplify Experiment specification and give additional settings to more advanced users who are Kubernetes experts.
Therefore, I am not sure that algorithmSpec should live under Experiment, but we can extend Katib config settings as I mentioned above: #1737 (comment).

What do you think @johnugeorge ?

/priority p2
/help

@andreyvelich

@andreyvelich:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

Details

In response to this:

/priority p2
/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

I think, this is a valid ask for many deployments. We can provide this as an option. If AlgorithmSpec is provided in Experiment, it is picked up. Else, defaults will be used(like the current)

@andreyvelich

@andreyvelich @johnugeorge would need this feature for our deployments. which katib release will have this feature implemented?

This hasn't been implemented fully yet.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

/lifecycle frozen

@andreyvelich

Hi @andreyvelich, I'd like to work on the remaining part of this issue.

Since #2000, SuggestionConfig inlines corev1.Container, so container-level fields (command, args, env, resources, ...) can already be set in the Katib config. What's still missing are the pod-level fields: DesiredDeployment() in pkg/controller.v1beta1/suggestion/composer/composer.go only sets ServiceAccountName and the resume volume on the pod spec, so things like imagePullSecrets (the original ask here) can't be configured.

Proposal, keeping the simple image: setup unchanged as you suggested above:

Add optional pod-level fields to SuggestionConfig in pkg/apis/config/v1beta1/types.go:

ImagePullSecrets []corev1.LocalObjectReference `json:"imagePullSecrets,omitempty"`
NodeSelector     map[string]string             `json:"nodeSelector,omitempty"`
Tolerations      []corev1.Toleration           `json:"tolerations,omitempty"`
Affinity         *corev1.Affinity              `json:"affinity,omitempty"`

Copy them into d.Spec.Template.Spec in DesiredDeployment() when they're set. Existing configs don't change.
Add unit tests in composer_test.go and update the generated deepcopy.
Follow up with a docs update for the Katib config page on the website.

Example config:

suggestions:
  - algorithmName: random
    image: ghcr.io/kubeflow/katib/suggestion-hyperopt:latest
    imagePullSecrets:
      - name: my-registry-secret
    nodeSelector:
      kubernetes.io/os: linux

Does this approach look good to you? I'm also happy to keep the first PR to just imagePullSecrets if you'd prefer something smaller.

Opened #2744 with the approach above (all four fields, with a unit test). Happy to trim it down to just imagePullSecrets if you'd prefer a smaller change.

Navigation Menu

Repository navigation

Support users to set more fields in suggetion deployment #1737

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Support users to set more fields in suggetion deployment #1737

Description

Activity

gaocegege commented on Nov 19, 2021

andreyvelich commented on Nov 19, 2021

tingweiwu commented on Nov 27, 2021

gaocegege commented on Nov 27, 2021

andreyvelich commented on Nov 29, 2021

andreyvelich commented on Feb 11, 2022

google-oss-prow commented on Feb 11, 2022

johnugeorge commented on Feb 15, 2022

ramdootp commented on Apr 21, 2022

andreyvelich commented on Jan 23, 2024

github-actions commented on Apr 22, 2024

andreyvelich commented on Apr 22, 2024

dipesh-svg commented on Oct 27, 2026

dipesh-svg commented on Oct 31, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions