Read store repository app folders in parallel#6963
Conversation
StoreData.update reads each repository's app folder in sequence: the core repository, the local repository, then every custom git repository one after another. Each read does a recursive config glob and parses all add-on configs for that repository, so the total time grows with the number of repositories. The repositories are independent and each produces apps under its own repository-prefixed slug, so read all app folders concurrently with asyncio.gather instead. Results are merged in the original order, so the behavior is unchanged.
frenck
added
the
refactor
There was a problem hiding this comment.
Pull request overview
This PR optimizes the Supervisor’s store reload path by changing StoreData.update() to read add-on (app) folders from the core, local, and custom git repositories concurrently, reducing store reload wall-clock time as the number of repositories grows.
Changes:
- Discover custom git repositories, then read all repository app folders concurrently via
asyncio.gather. - Merge per-repository app dictionaries in the same order as before (core → local → git repos) to preserve precedence behavior.
| # Read all repository app folders in parallel; they are independent and | ||
| # each produces apps under its own repository-prefixed slug. | ||
| folders = [ | ||
| (self.sys_config.path_apps_core, REPOSITORY_CORE), | ||
| (self.sys_config.path_apps_local, REPOSITORY_LOCAL), | ||
| *((repo.path, repo.slug) for repo in git_repositories), | ||
| ] | ||
| folder_apps = await asyncio.gather( | ||
| *(self._read_apps_folder(path, slug) for path, slug in folders) | ||
| ) | ||
|
|
||
| # Merge in folder order to preserve the previous precedence | ||
| apps: dict[str, dict[str, Any]] = {} | ||
| for result in folder_apps: | ||
| apps.update(result) |
There was a problem hiding this comment.
so behavior is unchanged; only the wall-clock time drops toward the slowest single repository instead of the sum of all of them.
Is that proven? 🤔
Typically such recursive search in directory trees leads to lots of random I/O. I am quite sure that at least on spinning disks doing such operation in parallel actually kills performance. Probably modern SSD with multiple I/O queues can actually parallelize such operations, but I'd guess most cheaper SSD and eMMC/SD cards run with a single I/O queue, where running these in parallel most likely leads to no or negligible performance differences.
Unless measurements really show that this improves reload time on typical hardware in real world, I'd rather not parallelize workloads with lots of random I/O.
Proposed change
StoreData.update()reads each repository's app folder in sequence: the core repository, the local repository, then every custom git repository one after another. Each read does a recursive**/config.*glob and parses every add-on config for that repository, so the total time grows with the number of repositories. This runs on every store reload (startup, adding/removing a repository, and the periodic reload), and is more expensive with large or network-backed repositories.The repositories are independent and each produces apps under its own repository-prefixed slug (
{repository}_{slug}), so this reads all app folders concurrently withasyncio.gatherinstead of awaiting them one by one. The results are merged in the original order (core, local, then git repositories), so behavior is unchanged; only the wall-clock time drops toward the slowest single repository instead of the sum of all of them.Type of change
Additional information
Checklist
ruff format supervisor tests)If API endpoints or add-on configuration are added/changed: