CI: find a solution for including custom PlatformIO build envs from PRs (usermods) in CI builds

[coderabbitai]

Problem

When a PR introduces a new usermod (or other feature) that requires a custom PlatformIO build environment, there is currently no clean way to include that environment in CI builds for the PR.

• platformio_override.ini is listed in .gitignore and must not be committed to the repository, so it cannot be used to ship a custom build env in a PR.
• The established workaround is to place an example as usermods/<modname>/platformio_override.ini.sample, but .sample files are not picked up by PlatformIO or the CI pipeline.
• Even the VSCode UI workflow does not solve the CI-build side of the problem — a second file would still be needed.

Impact

• New usermods/features that need non-default build environments cannot be fully validated by CI during code review.
• PR authors resort to committing platformio_override.ini directly (as seen in PR Add DALI support - As a light #5645), which clobbers a developer's local override when they pull the branch.

Possible directions

• A dedicated, version-controlled file (e.g., usermods/<modname>/platformio_build_envs.ini) that CI explicitly includes/merges.
• A CI mechanism that discovers and appends *.ini.sample files from changed usermods.
• Documentation of an agreed-upon interim workaround.

References

• Raised during review of PR Add DALI support - As a light #5645: Add DALI support - As a light #5645 (comment)
• Related docs issue: (Doc request) compiling in platformio - selecting build envs via dropdown WLED-Docs#161

/cc @softhack007

[willmmiles]

The issue description here is describing a solution, not a problem. So I understand clearly: is the actual issue that there exist usermods that put constraints on the environment that are illegible to the per-usermod CI, so that we will have to find some way to override the default configuration to test-build them?

I'm very leery of encouraging this broadly -- it's rife for abuse. I especially want to avoid requiring every usermod to have a sample file that merely replicates the documentation on how to add a single usermod. Replicating the documentation like that is a maintenance nightmare.

A bad example (which we should not support!) is a module requiring pin specifications as -D entries in build_flags.

A weak example would be inter-usermod dependencies -- the library resolver won't pick them up. This is arguably more of an internal tooling bug and is best corrected in load_usermods.py.

A mid example would be a module that requires a specific platform -- documented in library.json, but harder to discover in CI.

I'm probably missing something else though?

[netmindz]

So the most minimal requirement is that our CI for usermods is simultaneously not enough and also a sledgehammer!

While pushing changes you your own repo (pre-pr) there is no way to build your code for the PR without either editing platformio.ini which we tell people not to, or adding a platformio.overrude.ini to git - which we tell people not to

Once you then open the PR our full matrix build kicks in and builds ever usermod for every chip, with no filtering for the usermod you actually changed. This fixes that to filter to just what you are editing

If you have concerns about the per usermod platformio.override.ini duplicating what is in the readme, then we can just remove the text from the readme. Alternatively I remove and discovery of the ini file and we keep to just doing the "default" build of that usermod but this feels too restrictive when people working on a usermod that does for some reason want to use build flags to create certain variants (e.g if hub75 was a usermod driver)

[willmmiles]

Thanks for following up!

While pushing changes you your own repo (pre-pr) there is no way to build your code for the PR without either editing platformio.ini which we tell people not to, or adding a platformio.overrude.ini to git - which we tell people not to

Once you then open the PR our full matrix build kicks in and builds ever usermod for every chip, with no filtering for the usermod you actually changed. This fixes that to filter to just what you are editing

This is a good goal, and a feature worth having - no arguments from me on this objective. This has no strong coupling with per-usermod platformio fragments, though? The CI for a given usermod should be triggered if any file in that usermod's folder is changed.

---

If you have concerns about the per usermod platformio.override.ini duplicating what is in the readme, then we can just remove the text from the readme. Alternatively I remove and discovery of the ini file and we keep to just doing the "default" build of that usermod

It's not about the readme, it's the copy-and-paste boilerplate in every usermod. That's what's not OK by me. I would much prefer a "if no custom file is present, do the default thing" approach -- it keeps the maintenance centralized, and encourages people to design "self-contained" usermods.

but this feels too restrictive when people working on a usermod that does for some reason want to use build flags to create certain variants (e.g if hub75 was a usermod driver)

That's a bit of what I was trying to poke at -- I think there's some value in deliberately discouraging the use of compile-time configuration. I don't feel bad making life hard for bad designs!

I'm digging deeper here because I can sense there's a stronger argument lurking but I haven't been able to put a finger on it yet. "Making it easier to do something we recommend against" isn't the best selling point.

What's the use case I'm missing?

---

Some other thoughts in nearby concept-space, so we're all in the same context:

• I think we should be generally discouraging usermod PRs in favour of better out-of-tree support. (Cf. Why bother with PRs for usermods #5318)
• That said, I agree that we do still need better tooling to support the things we do want to keep in-tree. (Add FSEQ + FPP usermods for local playback #5641, I'm looking at you; and my WIP design for Log buffer #5583 has some of that too.)
• Cross-usermod dependencies in tree are a pain point. (Out of tree, it's easy.)
• I experimented with using PlatformIO scripting and custom_variables for managing build-time variants in TTGO-T-Display usermod fixup #5479 - see ‎usermods/TTGO-T-Display/set_build_flags.py. This kind of approach works for cases where there can be a representative 'default' configuration, but won't handle a strong case for multiple variants.
• Sorry to scoop you on out-of-tree usermod CI -- Claude and I sketched it yesterday: Add nightly CI workflow building against WLED main wled-usermod-example#6
• ... but we might also want to add the example out-of-tree usermod to mainline CI, so we can get a warning if we've broken the API before merging. I had a WIP branch with updates to the in-tree documentation on how-to-write-a-usermod (ie. do it out of tree); I'd intended to add the out-tree smoke test CI there, but I wasn't (yet) satisfied with the implementation.

[netmindz]

While pushing changes you your own repo (pre-pr) there is no way to build your code for the PR without either editing platformio.ini which we tell people not to, or adding a platformio.overrude.ini to git - which we tell people not to
Once you then open the PR our full matrix build kicks in and builds ever usermod for every chip, with no filtering for the usermod you actually changed. This fixes that to filter to just what you are editing

This is a good goal, and a feature worth having - no arguments from me on this objective. This has no strong coupling with per-usermod platformio fragments, though? The CI for a given usermod should be triggered if any file in that usermod's folder is changed.

Better I open a fresh PR that only fixes performance that issue in our current CI then?

It's not about the readme, it's the copy-and-paste boilerplate in every usermod. That's what's not OK by me. I would much prefer a "if no custom file is present, do the default thing" approach -- it keeps the maintenance centralized, and encourages people to design "self-contained" usermods.

I'm unclear what copy and paste you are meaning? just a new env that extends the right base?

but this feels too restrictive when people working on a usermod that does for some reason want to use build flags to create certain variants (e.g if hub75 was a usermod driver)

That's a bit of what I was trying to poke at -- I think there's some value in deliberately discouraging the use of compile-time configuration. I don't feel bad making life hard for bad designs!

Yeah I agree about discouraging the use of build flags, I'll remove any of the changes were I have added any new platformio_override.ini

[willmmiles]

While pushing changes you your own repo (pre-pr) there is no way to build your code for the PR without either editing platformio.ini which we tell people not to, or adding a platformio.overrude.ini to git - which we tell people not to
Once you then open the PR our full matrix build kicks in and builds ever usermod for every chip, with no filtering for the usermod you actually changed. This fixes that to filter to just what you are editing

This is a good goal, and a feature worth having - no arguments from me on this objective. This has no strong coupling with per-usermod platformio fragments, though? The CI for a given usermod should be triggered if any file in that usermod's folder is changed.

Better I open a fresh PR that only fixes performance that issue in our current CI then?

Sure. Small PRs are the easiest to merge. :)

It's not about the readme, it's the copy-and-paste boilerplate in every usermod. That's what's not OK by me. I would much prefer a "if no custom file is present, do the default thing" approach -- it keeps the maintenance centralized, and encourages people to design "self-contained" usermods.

I'm unclear what copy and paste you are meaning? just a new env that extends the right base?

Yes, exactly that. If platformio ini fragment files are strictly required for every usermod, it becomes another 10 line boilerplate file that every usermod has to drag around: and for most modules differs only in the module name in the custom_usermods line, or ends up duplicating information from library.json. IMO library.json is enough of a stretch -- I really don't want to make it any worse. And if we ever want to change the "standard" environment, we have to push an update to all those files. Standard code should be in one place, DRY, etc. etc.

I'm sure there's a case for something that really, truly has a need for some expanded target CI list of options that can't be met with custom scripts or unique CI -- but I really don't want to add to the maintenance burden of every module to pay for that feature. The little costs add up over time.