Repository navigation
Sanitize ensemble IDs for Kubernetes label compatibility - #158
Merged
Merged
Conversation
Contributor
Author
|
@saintstack have a look please |
ploxiln
reviewed
Jul 6, 2026
| v1.patch_namespaced_pod(pod_name, namespace, {"metadata":{"labels":{"ensemble":ensemble}}}) | ||
| sanitized_ensemble = joshua_model.sanitize_for_k8s_label(ensemble) | ||
| if pod_desc.metadata.labels.get('ensemble', '') != sanitized_ensemble: | ||
| v1.patch_namespaced_pod(pod_name, namespace, {"metadata":{"labels":{"ensemble":sanitized_ensemble}}}) |
Contributor
There was a problem hiding this comment.
FWIW I think we do not really need this "belt-and-suspenders" change, because this problem has existed for so long ... in the rare case that someone already specified an ensemble username with unacceptable characters, it has already been killing all their joshua agents, so they presumably already figured out that they have to manually delete such pre-existing ensembles.
Contributor
Author
|
@ploxiln Yeah, it's mostly there as defense-in-depth to catch older client versions or manual DB edits. I'm happy to remove it though if you think it's unnecessary noise |
Contributor
|
Yeah, I'd either delete that bit, or add a short comment like: # Just in case of ensembles created before joshua_model sanitize id fix |
Contributor
Author
|
@ploxiln done please check |
ploxiln
approved these changes
Jul 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Ensemble IDs that contain characters invalid for Kubernetes labels (e.g.,
/in usernames like
praza-dev/tclinkenbeard) causejoshua-agentpods to crashwith an unhelpful
ApiException(422)when trying to patch their pod label.The agent then enters an infinite crash-loop as the scaler recreates it.
Fixes #129
Root Cause
joshua_model.create_ensemble()constructs the ensemble ID directly from theusername without sanitization:
timestamp + "-" + userid:27 + "-" + hash:16
Kubernetes label values only allow
[a-zA-Z0-9_.-]and must start/end withalphanumeric. When this unsanitized ID is later used as a label value in
joshua_agent._run_ensemble(), the K8s API rejects it.Changes
joshua/joshua_model.pysanitize_for_k8s_label()function — Replaces characters invalid forK8s labels with
-, strips leading/trailing non-alphanumeric characters,and falls back to
'x'if the result is empty.create_ensemble()— The ensemble ID is now sanitized atcreation time, preventing invalid IDs from entering the system.
joshua/joshua_agent.pybefore the K8s label patch call. This catches any pre-existing ensembles
(created before this fix) that may have invalid characters.
Example
Username
praza-dev/tclinkenbeardpreviously produced ensemble ID:20260320-194628-praza-dev/tclinkenbeard...(contains/→ K8s rejects)After fix:
20260320-194628-praza-dev-tclinkenbeard...(/→-→ valid)