Repository navigation
Conversation
github-actions
Bot
added
a:infra+ops
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved routing, checkout isolation, and lifecycle defects prevent safe concurrent test runs.
8 open issues
What changed in this PR
Extends checkout-scoped test infrastructure with ephemeral host ports so separate worktrees can share one Docker daemon.
Changes:
- Rewrites test ports, stages deployment, and updates embedded endpoints.
- Adds checkout claims, swarm coordination, scoped lookups, and regression tests.
- Updates director-v2 fixtures and maintenance-skill metadata.
Validation: Two isolated Python checks ran. Project tests and pre-commit were not run in the dependency-free review environment.
| File | Description |
|---|---|
| services/docker-compose.local.yml | Documents test/debug port behavior; comment-only change. |
| services/director-v2/tests/integration/02/test_mixed_dynamic_sidecar_and_legacy_project.py | Removes fixed stack override. |
| services/director-v2/tests/integration/02/test_dynamic_sidecar_nodeports_integration.py | Removes fixed stack override. |
| services/director-v2/tests/integration/02/conftest.py | Derives checkout-specific network name. |
| services/director-v2/tests/conftest.py | Uses checkout-specific stack configuration. |
| packages/pytest-simcore/tests/test_helpers_swarm_coordination.py | Tests swarm membership and cleanup. |
| packages/pytest-simcore/tests/test_helpers_stack_names.py | Tests stack names and run directories. |
| packages/pytest-simcore/tests/test_helpers_host_ports.py | Tests port rewriting and collection. |
| packages/pytest-simcore/tests/test_helpers_docker.py | Checks lookup names against compose services. |
| packages/pytest-simcore/tests/test_helpers_checkout_claim.py | Tests checkout claim lifecycle. |
| packages/pytest-simcore/src/pytest_simcore/traefik_service.py | Scopes Traefik port lookups. |
| packages/pytest-simcore/src/pytest_simcore/simcore_webserver_service.py | Scopes webserver port lookup. |
| packages/pytest-simcore/src/pytest_simcore/simcore_storage_service.py | Scopes storage port lookup. |
| packages/pytest-simcore/src/pytest_simcore/simcore_services.py | Scopes generic service lookups. |
| packages/pytest-simcore/src/pytest_simcore/simcore_dask_service.py | Scopes scheduler port lookup. |
| packages/pytest-simcore/src/pytest_simcore/s3_storage_service.py | Uses checkout-specific ops stack. |
| packages/pytest-simcore/src/pytest_simcore/redis_service.py | Corrects and scopes Redis lookup. |
| packages/pytest-simcore/src/pytest_simcore/rabbit_service.py | Scopes broker and management lookups. |
| packages/pytest-simcore/src/pytest_simcore/postgres_service.py | Scopes database lookup and assertion. |
| packages/pytest-simcore/src/pytest_simcore/httpbin_service.py | Scopes container names by checkout. |
| packages/pytest-simcore/src/pytest_simcore/helpers/xdist.py | Identifies the run-owner process. |
| packages/pytest-simcore/src/pytest_simcore/helpers/swarm_coordination.py | Coordinates daemon-wide swarm ownership. |
| packages/pytest-simcore/src/pytest_simcore/helpers/stack_names.py | Generates stack and run identifiers. |
| packages/pytest-simcore/src/pytest_simcore/helpers/host_ports.py | Implements port policy, rebaking, and reporting. |
| packages/pytest-simcore/src/pytest_simcore/helpers/docker.py | Adds stack-filtered port discovery. |
| packages/pytest-simcore/src/pytest_simcore/helpers/checkout_claim.py | Guards concurrent checkout use. |
| packages/pytest-simcore/src/pytest_simcore/docker_swarm.py | Stages deployment and coordinates cleanup. |
| packages/pytest-simcore/src/pytest_simcore/docker_compose.py | Applies scoped names and port policy. |
| packages/pytest-simcore/src/pytest_simcore/docker_api_proxy.py | Scopes proxy port lookup. |
| .github/skills/update-user-messages/SKILL.md | Updates metadata and example formatting. |
| .github/skills/refactor-pydantic-annotated-fields/SKILL.md | Updates metadata and example formatting. |
| .github/skills/refactor-aiohttp-appkey/SKILL.md | Updates metadata and example formatting. |
| .github/skills/fix-pre-commit-hooks/SKILL.md | Updates skill metadata and lint comments. |
Review effort: Balanced
Comment on lines
+143
to
+144
| for old, new in replacements.items(): | ||
| replaced, n = patterns[old].subn(f":{new}", new_value) |
Comment on lines
+228
to
+232
| claim_checkout( | ||
| get_checkout_token(osparc_simcore_root_dir), | ||
| run_id=token, | ||
| watched_pid=watched_pid, | ||
| ) |
| log.info("published host ports of this run:\n%s", lines) | ||
| if run_dir is not None: | ||
| report_path = run_dir / "published_ports.json" | ||
| report_path.write_text(yaml.dump(table, default_flow_style=False, sort_keys=True)) |
pcrespov
approved these changes
Oct 19, 2026
sanderegg
marked this pull request as draft
October 19, 2026 15:21
Stack/network/volume/image names and host ports are daemon-global, so two
checkouts (e.g. git worktrees) collided on `make up-*`, `make build` and the
integration tests. Every checkout now leases a dev slot N (0-9):
- stacks master-simcore/ops/vendors (slot 0, unchanged) or simcore-sN/ops-sN/
vendors-sN; pytest stacks pytest-sN-{core,ops}; traefik zone per slot
- host ports = slot-0 port + N*100 via ${*_HOST_PORT:-<slot-0 port>} in the
compose files and in the .env-devel URLs (S3_ENDPOINT, ...)
- local images tagged production-sN/development-sN; frontend compiler
container/compose project and qx serve port scoped per slot
- the registry is the daemon: a config-only network osparc-dev-slot-N labelled
with the checkout path (atomic lease, stale markers reclaimed on lease)
- `make down` only removes this checkout's stacks; `make leave` refuses while
other checkouts' stacks run unless force=1; VS Code launch.json rendered
with the slot's debug ports
- pytest leases the checkout's slot, fails fast when the slot's ports are
published by anything else (e.g. this checkout's dev stack), and replaces
the branch+uid stack names
See docs/parallel-worktrees.md and scripts/dev_slot.py.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- docker_compose.py and docker_swarm.py are stored with LF line endings - shorter comments, docstrings and messages in the dev-slot code - docs/parallel-worktrees.md rewritten concisely Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
sanderegg
force-pushed
the
maintenance/pytest-per-run-host-ports
branch
from
October 22, 2026 16:15
a423fda to
7919702
Compare
github-actions
Bot
added
a:frontend
sanderegg
changed the title
- the slot keeps checkouts apart, the branch (cut to 18 chars) tells humans what a stack is - before deploying, test stacks of the same slot left from another branch are removed - dev stacks keep their slot-only names (simcore-sN, ...) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…worktrees Git worktrees share .git/hooks, and `pre-commit install` writes the interpreter that ran it into the hook. With `make devenv` that was the .venv of whichever worktree ran it last, so commits in every worktree broke once that .venv was removed or recreated. `make devenv` now installs pre-commit as a uv tool and installs the hook from there. `make clean-hooks` removes the hook from the common git dir, which also works inside a worktree. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
With pre-commit in .venv, `pre-commit install` from an activated venv pointed the shared hook back to that .venv. pre-commit is now only installed as a uv tool by `make devenv`, which also removes it from existing venvs. - requirements/devenv.txt: drop pre-commit - */requirements/_tools.txt: regenerated without --upgrade; only pre-commit and the packages it alone pulled in are removed Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved lease and swarm-ownership races, bootstrap compatibility, and stale configuration or image selection can break checkout isolation.
10 open issues
4 resolved since last review
Review effort: Balanced
| # Creating config for stack with 'local/{service}:production' (except of static-webserver -> static-webserver:development) to $@ | ||
| @export DOCKER_REGISTRY=local && \ | ||
| export DOCKER_IMAGE_TAG=production && \ | ||
| export DOCKER_IMAGE_TAG=production$(LOCAL_IMAGE_TAG_SUFFIX) && \ |
| .init-swarm: | ||
| # Ensures swarm is initialized (careful we use a default pool of 172.20.0.0/14. Ensure you do not use private IPs in that range!) | ||
| $(if $(SWARM_HOSTS),,docker swarm init --advertise-addr=$(get_my_ip) --default-addr-pool 172.20.0.0/14) | ||
| @$(_lease_dev_slot) |
| if checkout and Path(checkout).is_dir(): | ||
| alive[slot] = checkout | ||
| else: | ||
| _remove_marker(slot, run) |
|
|
||
| all: $(VSCODE_CONFIGS) ## creates all vscode config files from templates | ||
|
|
||
| launch.json: launch.template.json ## creates launch.json from its template, with the dev slot's debug ports |
Comment on lines
+694
to
+696
| leased=$$($(DEV_SLOT_SCRIPT) lease --checkout $(CURDIR) --env-file $(CURDIR)/.env $(if $(slot),--slot $(slot))); \ | ||
| if [ "$$leased" != "$(OSPARC_DEV_SLOT)" ]; then \ | ||
| echo "ERROR: this checkout now holds dev slot $$leased (this make run used $(OSPARC_DEV_SLOT)): re-run make"; \ |
|
|
||
| # slot markers ---------------------------------------------------------------------------- | ||
|
|
||
| type RunDocker = Callable[[list[str]], subprocess.CompletedProcess[str]] |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved ownership checks can overwrite or remove another checkout’s resources, and frontend-only deployment still selects an unsuffixed image.
13 open issues
1 resolved since last review
Review effort: Balanced
Comment on lines
+532
to
533
| $(filter $(shell docker stack ls --format={{.Name}} 2>/dev/null),$(SWARM_STACK_NAME) $(SWARM_VENDORS_STACK_NAME) $(SWARM_OPS_STACK_NAME)),\ | ||
| docker stack rm $(stack);) |
| $(if $(wildcard $@), \ | ||
| @echo "WARNING ##### $< is newer than $@ ####"; diff -uN $@ $<; false;,\ | ||
| @echo "WARNING ##### $@ does not exist, cloning $< as $@ ############"; cp $< $@) | ||
| @$(_lease_dev_slot) |
Comment on lines
+297
to
+299
| for extra in owned: | ||
| if extra != slot: | ||
| _remove_marker(extra, run) |
|
|
||
| static-webserver: | ||
| image: local/static-webserver:${BUILD_TARGET:?build_target_required} | ||
| image: local/static-webserver:${BUILD_TARGET:?build_target_required}${LOCAL_IMAGE_TAG_SUFFIX:-} |
sanderegg
force-pushed
the
maintenance/pytest-per-run-host-ports
branch
from
October 23, 2026 15:18
7919702 to
cdb7483
Compare
sanderegg
force-pushed
the
maintenance/pytest-per-run-host-ports
branch
2 times, most recently
from
October 24, 2026 13:37
8c639d4 to
dc0750c
Compare
github-actions
Bot
added
a:storage
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important
Developers, after pulling this:
make devenvonce, in any worktree. It installs pre-commit as a uv tool in~/.local/bin, removes it from.venvand installs the git hook from the uv tool. To check, runsed -n 6p .git/hooks/pre-commit. It should print~/.local/share/uv/tools/pre-commit/bin/python..envfrom.env-devel. Until you do,makeprints the diff and stops. Your main checkout then gets slot 0 and keeps its current stack names, ports and docker volumes, including the postgres and S3 data.docker stack ls | grep '^pytest-'and remove each withdocker stack rm <name>.To work in a second checkout, run
git worktree add ../osparc-simcore-<topic> <branch>, thenmake devenvandmake build(ormake build-devel) in it. Its firstmaketakes the next free slot and asks you to run it again once.make infolists the slot's stack names, andmake up-*prints its ports.Details are in
docs/parallel-worktrees.md.What do these changes do?
Several checkouts of the repo, for example git worktrees, can now build, deploy and run integration tests on one docker daemon at the same time.
Each checkout leases a dev slot
Nfrom 0 to 9. The slot decides the stack names, host ports and local image tags of the checkout.master-simcore,ops,vendorssimcore-sN,ops-sN,vendors-sNmaster-simcore_postgres_data,_redis-data,_rabbit_data,_computational_shared_datasimcore-sN_postgres_data,_redis-data,_rabbit_data,_computational_shared_dataops_s3_storage_data,ops_portainer_dataops-sN_s3_storage_data,ops-sN_portainer_datalocal/<svc>:production,:developmentlocal/<svc>:production-sN,:development-sNN*100, e.g. 9381 in slot 3pytest-s0-<branch>-core,-opspytest-<branch>-core-<uid>pytest-sN-<branch>-core,-opspytest-s0-<branch>-core_postgres_data, ...pytest-sN-<branch>-core_postgres_data, ...Slot 0 keeps the current dev stack names, volume names, ports and image tags, so a single checkout keeps its database and S3 data, and CI behaves as before. The dev volumes depend only on the slot, not on the branch, so the data stays when a worktree switches branch. Test stacks name their volumes after the stack (
pytest-sN-<branch>-core_postgres_data, ...) and remove them on teardown.Changes:
scripts/dev_slot.pyleases the slots. It uses only the standard library because make runs it before any venv exists, and pytest loads the same file. A config-only docker networkosparc-dev-slot-N, labelled with the checkout path, holds slotN. Docker refuses a second network with the same name, so two checkouts cannot lease the same slot. Each lease deletes the networks of checkouts that no longer exist.${X_HOST_PORT:-<slot-0 port>}. The ops volumes take their names from the ops stack..env-devel, the URLs with a host port (S3_ENDPOINT,AGENT_VOLUMES_CLEANUP_S3_ENDPOINT,DIRECTOR_PUBLISHED_HOST_NAME,INVITATIONS_OSPARC_URL) use the slot's port variables.make .envandmake up-*lease the slot, andslot=Nmoves the checkout to slotN.make downremoves only this checkout's stacks.make leavestops with an error while other checkouts have stacks, unless you passforce=1. The build and tag targets add the slot's suffix to the image tags.qx serveport depend on the slot..vscode/Makefilewriteslaunch.jsonwith the slot's debug ports. The clusters-keeper attach config now uses port 3015 instead of 3006, which belongs to api-server.pytest-sN-<branch>-{core,ops}instead ofpytest-<branch>-{core,ops}-<uid>(🔨 Scope test docker stacks to each checkout for concurrent runs #9832). Long branch names are cut to 18 characters.tests/public-apireads the traefik and api-server ports from the slot.docker_compose.pyanddocker_swarm.pynow use LF line endings instead of CRLF.make devenvinstalls pre-commit as a uv tool and installs the git hook from it. All worktrees share.git/hooks, andpre-commit installwrites the path of its own interpreter into the hook. Before, the hook pointed to the.venvof the worktree that ranmake devenvlast, and commits failed in every worktree once that.venvwas gone. This PR removes pre-commit fromrequirements/devenv.txtand from all 33_tools.txtfiles. I regenerated them without--upgrade, so the diff only deletes pre-commit and the packages that only it needed.Related issue/s
Follow-up of #9832 and #9834.
How to test
pytest packages/pytest-simcore/tests/. The tests cover the slot names and ports, port collisions between all slots, the compose defaults, leasing (reuse, moving, concurrent leases, deleted checkouts) and the port check.make up-develin two worktrees.docker stack lsshowsmaster-simcoreandsimcore-s1, and the endpoint tables list different ports.make up-develin a checkout, then run its integration tests. They stop and ask you to runmake down.Dev-ops
.env-devel,docker-compose.local.yml,docker-compose-ops*.yml,docker-compose-build.ymland the Makefiles. The deployed compose files are unchanged.OSPARC_DEV_SLOT,*_HOST_PORT,SWARM_OPS_STACK_NAME,SWARM_VENDORS_STACK_NAME,LOCAL_IMAGE_TAG_SUFFIX.