[Bug]: High CPU/disk usage when indexing large generated data folders

[juleshall4]

Which fff frontend?

MCP server (fff-mcp)

has logs

No response

Description

fff-mcp caused sustained high CPU and disk usage on Windows after it encountered a large generated data folder with many files.

After quitting/disabling fff-mcp, system performance returned to normal.

Expected behavior:

• Large/generated folders should not cause prolonged system slowdown.
• Indexing should be throttled, skipped, or easy to exclude.

Suggested fixes:

• Add folder exclusions.
• Auto-skip very large/high-file-count folders.
• Limit concurrent indexing work.
• Show which path is currently being indexed.

[dmtrKovalenko]

can you give any numbers? What was the usage? RAM/CPU? how large was a directory?

[juleshall4]

can you give any numbers? What was the usage? RAM/CPU? how large was a directory?

Yep. Rough numbers from my machine:

• OS: Windows
• Directory size: ~107 GB total
• Largest subdirectory: ~98 GB
• File count in largest subdirectory: ~95k files
• CPU: system CPU was around 75-92% while the slowdown was happening
• Disk: system disk activity was around 59-90%
• RAM: not the bottleneck, about 18-20 GB free
• GPU: not the bottleneck, ~37C / low power / no compute-heavy process
• fff-mcp: multiple fff-mcp.exe processes were running and actively consuming CPU
• After disabling/quitting fff-mcp, disk usage dropped to low single digits and the system became responsive again

My guess: the large/high-file-count directory made the problem much worse, but it probably isn't the whole bug.

It looks like fff-mcp can get into sustained indexing/scanning work that is too aggressive on Windows. A ~107 GB generated-data directory with ~95k files is likely a strong trigger/repro case, because it creates lots of filesystem work, but another user also saw performance issues without necessarily having the same dataset shape.

So I would treat this as an indexing/throttling issue, not just "one weird huge folder".

[gustav-fff]

[triage-bot] NEEDS INFO:

Symptom overlaps three known problem classes and the fix path depends on which one this is. @juleshall4 please clarify:

1. Where was fff-mcp launched from? What was its cwd when spawned? Is the ~107 GB generated folder under that cwd, or is cwd itself deep inside a repo that contains this folder as a subdir?
2. Is that folder under a git repo? If yes, is it listed in a .gitignore (root or nested)? fff-mcp walks with gitignore filtering — if the folder is ignored it should be skipped.
3. fff-mcp log. Attach the log from ~/.cache/fff_mcp*.log (or %LOCALAPPDATA%\fff_mcp\*.log on Windows) covering startup + a few minutes. Lines starting with SCAN: and Initializing FilePicker with base_path are the ones we need.
4. Which MCP client? Claude Code, Codex Desktop, Cursor, other? Version?
5. Multiple fff-mcp.exe processes — how many? Same PPID or different? This may be Every Time i copy files i get 16 fff-mcp processes spawning killing my cpu #497 / [Bug]: MCP server with Claude code keeps leaving orphan processes that take up of CPU usage #633 (client leaks orphans on session end), which is a different bug.
6. fff-mcp version: fff-mcp --version.

Two possibilities we can already rule in/out with the above:

• If cwd=/ or cwd=%USERPROFILE% and no git repo → known fff-mcp indexes entire filesystem when launched with cwd=/ (Codex Desktop) #320 pattern, walking whole disk.
• If the generated folder is not gitignored and lives inside the repo → walker is doing its job; API to control what is ignored and --no-ignore ability to index everything #530 (exclude API) or .gitignoreing the folder is the answer, not an indexing bug.

Honk-Honk 🪿

[juleshall4]

• MCP client: Codex Desktop.
• OS: Windows.
• fff-mcp --version: fff-mcp 0.9.6 (98af56c393a596292a24fb00d5b672306706b07c).
• There were multiple fff-mcp.exe processes. I observed at least 3 active at once; after reinstall/re-enable I also saw 3 spawned. I did not capture PPIDs at the time.
• The large generated folder was outside my Codex workspace/repo, not intentionally part of a code repo I was working in.
• It was not a folder I expected fff to index.
• I do not know the exact cwd fff-mcp was launched with during the bad run. Given Codex Desktop + Windows + the folder being outside my repo, the cwd=%USERPROFILE% / broad scan pattern sounds plausible.
• The generated folder is not knowingly under a git repo or controlled by .gitignore.
• During the bad state, system CPU was around 75-92%, disk activity around 59-90%, RAM was fine with ~18-20 GB free, and GPU was idle.
• Disabling/quitting fff-mcp made disk usage drop to low single digits and the machine became responsive again.

I’ll try to reproduce and attach logs from %LOCALAPPDATA%\fff_mcp\*.log if it happens again. I do not want to intentionally let it hammer the machine for long, but I can grab SCAN: / Initializing FilePicker with base_path lines next time.

Based on your classification, this sounds more like the Codex Desktop broad-cwd / broad-scan case than “folder inside repo but not gitignored”.