FabricChaincodeInstall keeps reinstalling chaincode in a loop — thousands of duplicate .tar.gz generated. Has anyone seen this?

[joaquinBlocknitive]

Environment

• hlf-operator: 1.13.0
• Hyperledger Fabric: 3.1.0
• Kubernetes: EKS (AWS) with Istio ingress gateway
• Storage: EFS (NFS-backed PVCs, efs-sc StorageClass)
• Chaincode type: ccaas

What is happening

After applying FabricChaincodeInstall resources for a ccaas chaincode, the operator continuously reinstalls the chaincode in a non-stop loop. The peer logs show the full ccaas builder cycle (detect → build → release → InstallChaincode) repeating every few hundred milliseconds, and the peer filesystem accumulates thousands of duplicate .tar.gz files in /var/hyperledger/production/lifecycle/chaincodes/. We ended up with over 25,000 files for just 5 chaincodes.

The FabricChaincodeInstall CRs stay stuck in RUNNING status with an empty message field and empty installedPeers, never reaching a stable completed state.

Each reinstallation generates a different package hash (packageID), so the ccaas packageID is not deterministic across runs even though the chaincodePackage spec has not changed. This creates a mismatch with the already-committed chaincode definition on the channel, making the chaincode unusable without a new approve + commit cycle with an incremented sequence.

Additional context: nightly scale-down

Our infrastructure scales peer nodes down to 0 replicas every night and back up to 1 every morning (cost-saving measure on EKS). We suspect this may be a contributing trigger: every time the peer pods restart after the scale-up, the operator detects the chaincodes are no longer installed (since the peer starts fresh) and begins reinstalling them. However, it never marks the installation as complete, so it keeps retrying in a loop until the pods are manually restarted or the operator is stopped.

Peer log excerpt (repeating continuously)

INFO [chaincode.externalbuilder.ccaas_builder] ::Detect command=detect
INFO [chaincode.externalbuilder.ccaas_builder] ::Type detected as ccaas command=detect
INFO [chaincode.externalbuilder.ccaas_builder] ::Build command=build
INFO [chaincode.externalbuilder.ccaas_builder] ::Build phase completed command=build
INFO [chaincode.externalbuilder.ccaas_builder] ::Release command=release
INFO [chaincode.externalbuilder.ccaas_builder] ::Release phase completed command=release
INFO [lifecycle] InstallChaincode -> Successfully installed chaincode with package ID 'tracechaincodeqa:a1b2c3...'
INFO [lifecycle] InstallChaincode -> Successfully installed chaincode with package ID 'tracechaincodeqa:d4e5f6...'
INFO [lifecycle] InstallChaincode -> Successfully installed chaincode with package ID 'tracechaincodeqa:7g8h9i...'
... (repeating indefinitely)

What we tried

• Deleting and recreating the FabricChaincodeInstall CRs — the loop resumes immediately after recreation.
• Stopping the peer pods — the loop stops but resumes as soon as the peers come back up.
• Stopping the operator (kubectl scale deployment hlf-operator-controller-manager --replicas=0) — this stops the reinstallation loop entirely, confirming the operator is the source.

Questions

1. Has anyone else experienced this issue with FabricChaincodeInstall and ccaas chaincodes going into an install loop, especially after a peer scale-down/scale-up cycle?
2. What condition causes the FabricChaincodeInstall controller to keep reinstalling indefinitely instead of detecting that the chaincode is already installed and marking the CR as completed?
3. Is the ccaas packageID expected to be non-deterministic across reinstallations (i.e. the same chaincodePackage spec generates a different hash each time)? If so, how is the operator supposed to detect "already installed" after a peer restart?
4. Is there a known fix or workaround in hlf-operator 1.13.0, or should we upgrade to a newer version?

[pfi79]

As I understand you, the problem is with the hlf-operator. They don't develop hlf-operator here. Rather, all the best to you here https://github.com/hyperledger-bevel/bevel-operator-fabric

[joaquinBlocknitive]

That's right, please excuse my mistake.