feat(orchestrator): implement tag-based cleanup framework with NFS se…

[Nagachandan-P]

Description of the Solution

Summary: Implements a comprehensive tag-based cleanup framework for the Omnia orchestrator with selective component cleanup, opt-in credential cleanup,
and NFS server data deletion capabilities. The framework ensures cleanup only runs with explicit tags, preventing accidental execution during normal orches
trator operations.

Changes

Cleanup Framework Core

• Created new tag-based cleanup system in cleanup_orchestrator.yml with component-level granularity
• Added when condition in orchestrator.yml to gate cleanup execution, preventing runs without explicit cleanup tags
• Removed reserved all tag to avoid Ansible conflicts
• Added never tags to legacy cleanup playbooks (cleanup_openchami.yml, cleanup_openldap.yml) to prevent execution without tags

Credentials Cleanup Component

• Created new credentials component with opt-in cleanup via cleanup_credentials tag
• Added credentials/tasks/cleanup.yml and credentials/vars/component_spec.yml
• Updated orchestrator_credentials.yml to skip credential collection when cleanup tags are present
• Added credentials to cleanup configuration with enabled: false by default

NFS Server Cleanup

• Implemented NFS server data deletion for Slurm and K8s components
• Added cleanup_nfs_server: true to Slurm and K8s component specs (default behavior)
• NFS server cleanup executes before unmounting to ensure data accessibility
• Correct cleanup order: delete NFS server data → unmount → clean client-side directories

Tag Validation and Configuration

• Updated orchestrator_setup/vars/main.yml to remove conflicting tag combinations for cleanup
• Added cleanup_credentials to supported tags list
• Updated default_cleanup.yml with component configurations and removed redundant settings
• Simplified configuration architecture by removing cleanup_config.yml override file

Documentation

• Created comprehensive cleanup/README.md with usage examples, available tags, and troubleshooting
• Documented NFS server cleanup behavior and configuration options
• Added dry run mode documentation

Files Changed

File	Change Type	Description
src/orchestrator/playbooks/cleanup/README.md	Added	Comprehensive cleanup framework documentation
src/orchestrator/playbooks/cleanup/cleanup_all.yml	Deleted	Replaced by new tag-based framework
src/orchestrator/playbooks/cleanup/cleanup_openchami.yml	Modified	Added never tags to prevent accidental execution
src/orchestrator/playbooks/cleanup/cleanup_openldap.yml	Modified	Added never tags to prevent accidental execution
src/orchestrator/playbooks/cleanup/cleanup_orchestrator.yml	Modified	Implemented tag-based component selection and removed cleanup_config merge log
ic
src/orchestrator/playbooks/credentials/orchestrator_credentials.yml	Modified	Added when condition to skip credential collection during cleanup
src/orchestrator/playbooks/orchestrator.yml	Modified	Added when condition to gate cleanup execution
src/orchestrator/roles/cleanup/components/credentials/tasks/cleanup.yml	Added	Credentials cleanup tasks
src/orchestrator/roles/cleanup/components/credentials/vars/component_spec.yml	Added	Credentials component specification
src/orchestrator/roles/cleanup/components/k8s/tasks/cleanup.yml	Modified	Added NFS server cleanup with delegate_to
src/orchestrator/roles/cleanup/components/k8s/vars/component_spec.yml	Modified	Set cleanup_nfs_server: true
src/orchestrator/roles/cleanup/components/slurm/tasks/cleanup.yml	Modified	Added NFS server cleanup with delegate_to
src/orchestrator/roles/cleanup/components/slurm/vars/component_spec.yml	Modified	Set cleanup_nfs_server: true
src/orchestrator/roles/cleanup/config/default_cleanup.yml	Modified	Updated comments and removed redundant cleanup_nfs_server settings
src/orchestrator/roles/orchestrator_setup/vars/main.yml	Modified	Updated tag validation for cleanup tags

Testing

• Verified cleanup does not run when orchestrator.yml is called without tags
• Tested full cleanup with --tags cleanup (6 components, preserves credentials)
• Tested credentials-only cleanup with --tags cleanup_credentials
• Tested combined cleanup with --tags cleanup,cleanup_credentials (all 7 components)
• Tested NFS server cleanup for Slurm (deleted data from 172.16.0.254:/nfs-slurm before unmounting)
• Verified correct cleanup order: NFS server data deletion → unmount → client-side directory cleanup
• Tested various tag combinations for both orchestrator.yml and cleanup_orchestrator.yml

Backward Compatibility

• Breaking change: Legacy cleanup_all.yml playbook removed (replaced by new framework)
• Cleanup now requires explicit tags - will not run automatically
• Users must use new tag-based syntax (--tags cleanup, --tags cleanup_credentials, or component-specific tags)
• NFS server cleanup is enabled by default (users can disable by editing component_spec.yml)