Fix variable collisions in orchestrator provisioning - #5134
Closed
SAYUK09 wants to merge 3 commits into
Closed
Conversation
This commit resolves multiple variable precedence and collision issues
that were preventing the orchestrator.yml playbook from completing
successfully.
1. cluster_name collision: k8s_config set_fact overrode openchami cluster_name
- Pin openchami cluster_name at set_fact level in register_nodes.yml
- Ensures metadata-service health checks use correct cluster identity
2. nodes variable collision: configure_boot_svc set_fact persisted across plays
- Renamed bare 'nodes' references to avoid collisions:
* boot_svc_nodes - for boot-service templates
* smd_group_nodes - for SMD group templates
* mapping_nodes - for hostname/node templates
- Updated all affected templates and task files
3. Additional fixes from previous session:
- Fixed kube_vip undefined in metadata-service templates
- Fixed service_k8s_version extraction from catalog
- Added fallback for missing service_k8s_v<version>.json
- Double-escaped cloud-init Jinja vars in metadata-service templates
- Fixed packages_base_dir variable precedence collision
- Fixed offline_tarball_path propagation
- Fixed image validation _first suffix mismatch
Signed-off-by: sayuri <sayuri.kamble@dell.com>
- Remove debug tasks from register_nodes.yml (Jinja templates in name) - Fix line length issues by using YAML folding - Fix key order (when before block) in configure_metadata_svc.yml Signed-off-by: sayuri <sayuri.kamble@dell.com>
- Split long Jinja line for calico_cidr calculation - Remove duplicate 'when' clause (outer block already has the condition) Signed-off-by: sayuri <sayuri.kamble@dell.com>
abhishek-sa1
reviewed
Sep 3, 2026
| show_task_path_on_failure = false | ||
| stdout_callback = omnia_default | ||
| show_task_path_on_failure = true | ||
| stdout_callback = default |
Collaborator
There was a problem hiding this comment.
omnia_default should be used. any reason for change?
Collaborator
Author
There was a problem hiding this comment.
was added for debugging purposes
Collaborator
Author
|
Closing this PR. As this change has been addressed in PR |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix Variable Collisions in Orchestrator Provisioning
Problem
The
orchestrator.ymlplaybook was failing due to multiple variable precedence and collision issues that prevented successful completion of the provisioning workflow.Root Causes
1.
cluster_namecollisionThe
k8s_configrole was settingcluster_nameviaset_fact, which unintentionally overrode the OpenCHAMIcluster_nameduring the Slurm provisioning pass. As a result, metadata-service health checks were executed against an incorrect cluster identity, causing failures in the provisioning flow.2.
nodesvariable collisionA generic
nodesvariable was defined as a host-scoped dictionary inconfigure_boot_svc.yml. Becauseset_factvariables persist across plays, this host-scoped value conflicted with later usages ofvars: nodes:in templates and tasks. This resulted in errors such as:object of type 'dict' has no attribute 'XNAME'
Solution
1. Pin OpenCHAMI
cluster_nameusingset_factAdded an explicit
set_factat the beginning ofregister_nodes.ymlto ensure the correct OpenCHAMI cluster identity is always used.Changes include:
cluster_nameandcluster_domaindirectly from the configuration file usingjson_queryset_factvalues from shadowing the correct configuration during the Slurm pass2. Rename
nodesvariables to avoid host-scoped collisionsReplaced generic
nodesreferences with context-specific variable names to eliminate conflicts across plays, tasks, and templates.Renamed variables:
boot_svc_nodesconfigure_boot_svc.ymlboot-svc.yaml.j2configure_boot_svc_metadata_svc.ymlsmd_group_nodescreate_smd_group.ymlgroups.yaml.j2create_groups.ymlcreate_groups_additional_fg.ymlgroups_additional_fg.yaml.j2mapping_nodeshostname.yaml.j2register_nodes.ymlAdditional Fixes
kube_vipreferences in metadata-service templatesservice_k8s_versionextraction from the catalogservice_k8s_v.jsonis missingpackages_base_dirvariable precedence collisionoffline_tarball_pathpropagation_firstsuffix mismatchResult
These changes resolve variable precedence and collision issues within the provisioning workflow, ensure consistent cluster identity throughout the orchestration process, and eliminate cross-play variable contamination that was causing template rendering and node registration failures.