Fix variable collisions in orchestrator provisioning - #5143
Conversation
This commit resolves multiple variable precedence and collision issues
that were preventing the orchestrator.yml playbook from completing
successfully.
1. cluster_name collision: k8s_config set_fact overrode openchami cluster_name
- Pin openchami cluster_name at set_fact level in register_nodes.yml
- Ensures metadata-service health checks use correct cluster identity
2. nodes variable collision: configure_boot_svc set_fact persisted across plays
- Renamed bare 'nodes' references to avoid collisions:
* boot_svc_nodes - for boot-service templates
* smd_group_nodes - for SMD group templates
* mapping_nodes - for hostname/node templates
- Updated all affected templates and task files
3. Additional fixes from previous session:
- Fixed kube_vip undefined in metadata-service templates
- Fixed service_k8s_version extraction from catalog
- Added fallback for missing service_k8s_v<version>.json
- Double-escaped cloud-init Jinja vars in metadata-service templates
- Fixed packages_base_dir variable precedence collision
- Fixed offline_tarball_path propagation
- Fixed image validation _first suffix mismatch
Signed-off-by: sayuri <sayuri.kamble@dell.com>
- Remove debug tasks from register_nodes.yml (Jinja templates in name) - Fix line length issues by using YAML folding - Fix key order (when before block) in configure_metadata_svc.yml Signed-off-by: sayuri <sayuri.kamble@dell.com>
- Split long Jinja line for calico_cidr calculation - Remove duplicate 'when' clause (outer block already has the condition) Signed-off-by: sayuri <sayuri.kamble@dell.com>
…ariables to prevent undefined variable errors The slurm_support and service_k8s_support variables were only conditionally defined, which caused "undefined variable" errors when referenced in subsequent tasks. This fix ensures these variables are always initialized with their default values before use. Signed-off-by: sayuri <sayuri.kamble@dell.com>
…th_on_failure settings This reverts the debugging changes to ansible.cfg that were made during variable collision investigation. The omnia_default callback is the custom Omnia-specific callback designed for better output formatting, and show_task_path_on_failure=false provides cleaner error messages. Signed-off-by: sayuri <sayuri.kamble@dell.com>
Signed-off-by: Abhishek S A <abhishek.sa3@dell.com> Signed-off-by: Abhishek S A <abhishek.sa3@dell.com>
The metadata-service v0.2.1 uses Pongo2 for template validation, which
rejects unknown Jinja2 variables. This caused template validation failures
when Ansible-rendered templates contained cloud-init Jinja2 variables that
Pongo2 couldn't validate.
The issue manifested in two ways:
1. Mount logic blocks with {% raw %}{{ ip}}{% endraw %} patterns caused
YAML parsing errors in cloud-init, rejecting entire FG templates
2. Kube templates with {% raw %}{{ ip}}{% endraw %} were rejected by Pongo2
with "missing required variables: [ip]"
Fixes:
- Removed mount logic {% raw %} blocks from all 10 FG templates (dead code,
host_mount_map is empty in current environment)
- Replaced {% raw %}{{ ip}}{% endraw %} with {{ "{{" }} ds.meta_data.instance_data.v1.local_ipv4 {{ "}}" }}
in all kube templates and configure-ib-network.sh.j2
- Fixed phone_home URL to use ds.instance_id instead of v1.instance_id
- Removed ## template: jinja from phone_home (metadata-service uses Pongo2,
not cloud-init Jinja2)
This ensures templates pass Pongo2 validation and cloud-init Jinja2 can
resolve variables correctly at node boot time.
Signed-off-by: sayuri <sayuri.kamble@dell.com>
c84dec8 to
f17843e
Compare
| - echo "{{"{{"}} mount | join(' ') {{"}}"}}" >> /etc/fstab | ||
| {% endfor %} | ||
| - mount -av | ||
| {% for decoded_cmd in mymounts.get("runcmd", []) %} |
There was a problem hiding this comment.
Need fo recheck, aarch64 was working code. Have we removed same task in x86_64 also?
There was a problem hiding this comment.
Yes, we removed the same mount logic block from ALL 10 FG templates, including both aarch64 and x86_64 variants.
The mount logic block was breaking cloud-init YAML parsing due to the Pongo2/cloud-init Jinja2 incompatibility.
The template fails because Ansible leaves literal {% tokens in the rendered output, which cloud-init Jinja2 interprets and causes a YAML parsing error (found character '%' that cannot start any token).
As a result, cloud-init rejects the configuration as an empty cloud config, and the users section (including ssh_authorized_keys) is never applied, breaking passwordless SSH.
However, the actual NFS mounts are still happening - just in a different place.
- ms-group-service_kube_control_plane_first_x86_64.yaml.j2 (lines 381-386)
- ms-group-service_kube_control_plane_x86_64.yaml.j2 (lines 288-293)
- ms-group-service_kube_node_x86_64.yaml.j2 (lines 184-187)
| {% for mount in mymounts.get("mounts", []) %} | ||
| - mkdir -pv {{ mount[1] }} | ||
| - echo "{{ mount | join(' ') }}" >> /etc/fstab | ||
| {% endfor %} |
There was a problem hiding this comment.
@snarthan cab you check if this block requires or not
| {% raw %} | ||
| {% if ds.meta_data.instance_data.v1.vendor_data.groups.ssh.host_mount_map is mapping and ds.meta_data.instance_data.v1.vendor_data.groups.ssh.host_mount_map %} | ||
| {% set mymounts = ds.meta_data.instance_data.v1.vendor_data.groups.ssh.host_mount_map.get(local_hostname, {}) %} | ||
| {% if mymounts %} |
There was a problem hiding this comment.
please re-visit this block it should not break the existing functionality
There was a problem hiding this comment.
Added a comment explaining this below
|
host_mount_map block removed from FG templates -- incompatible with Fabrica metadata-service Pre-Fabrica, templates were served via OpenCHAMI's cloud-init server which had no template validation. Cloud-init Jinja2 loop variables like {{ mount[1] }} inside {% for %} worked natively. Post-Fabrica, templates go through metadata-service which validates all {{ }} expressions via Pongo2. Pongo2 rejects any variable not in the ds.* namespace (loop variables, {% set %} variables, cloud-init builtins like {{ local_hostname }}). The {{"{{"}} escaping workaround passes Pongo2 but produces literal text instead of resolved values -- so the block was fundamentally broken in either case. Verified on live node: the block was never executing (empty host_mount_map = "{}" string, always skipped by the {% if %} guard). Existing slurm/K8s/login NFS mounts are unaffected -- they use Ansible-time resolved runcmd entries, not cloud-init Jinja2 loops. Going forward without groups feature in storage_config.yml for this release. |
5b5af5e
into
dell:issue-4849-omnia-modernization
3 participants
Fix Variable Collisions in Orchestrator Provisioning
Problem
The
orchestrator.ymlplaybook was failing due to multiple variable precedence and collision issues that prevented successful completion of the provisioning workflow.Root Causes
1.
cluster_namecollisionThe
k8s_configrole was settingcluster_nameviaset_fact, which unintentionally overrode the OpenCHAMIcluster_nameduring the Slurm provisioning pass. As a result, metadata-service health checks were executed against an incorrect cluster identity, causing failures in the provisioning flow.2.
nodesvariable collisionA generic
nodesvariable was defined as a host-scoped dictionary inconfigure_boot_svc.yml. Becauseset_factvariables persist across plays, this host-scoped value conflicted with later usages ofvars: nodes:in templates and tasks. This resulted in errors such as:object of type 'dict' has no attribute 'XNAME'
Solution
1. Pin OpenCHAMI
cluster_nameusingset_factAdded an explicit
set_factat the beginning ofregister_nodes.ymlto ensure the correct OpenCHAMI cluster identity is always used.Changes include:
cluster_nameandcluster_domaindirectly from the configuration file usingjson_queryset_factvalues from shadowing the correct configuration during the Slurm pass2. Rename
nodesvariables to avoid host-scoped collisionsReplaced generic
nodesreferences with context-specific variable names to eliminate conflicts across plays, tasks, and templates.Renamed variables:
boot_svc_nodesconfigure_boot_svc.ymlboot-svc.yaml.j2configure_boot_svc_metadata_svc.ymlsmd_group_nodescreate_smd_group.ymlgroups.yaml.j2create_groups.ymlcreate_groups_additional_fg.ymlgroups_additional_fg.yaml.j2mapping_nodeshostname.yaml.j2register_nodes.ymlAdditional Fixes
kube_vipreferences in metadata-service templatesservice_k8s_versionextraction from the catalogservice_k8s_v.jsonis missingpackages_base_dirvariable precedence collisionoffline_tarball_pathpropagation_firstsuffix mismatchResult
These changes resolve variable precedence and collision issues within the provisioning workflow, ensure consistent cluster identity throughout the orchestration process, and eliminate cross-play variable contamination that was causing template rendering and node registration failures.