pathocore-api

THIS REPO IS IN ACTIVE DEVELOPMENT.

Table of contents

• Installation
• Installation outside Docker
• Documentation

Installation

Docker test installation

This is the recommended entry point for developers who want to run PathoCore API locally for installation testing, smoke tests, demos, or frontend integration work.

The local test stack starts two services:

• app: Django API running inside a container
• db: MySQL database running inside a container

The database is stored in a Docker volume, so the stack can be stopped and started without losing data unless the volumes are explicitly removed.

Prerequisites

Before starting, make sure the machine has:

• git
• Docker Engine
• Docker Compose plugin (docker compose)

Check the tooling in a terminal:

git --version
docker --version
docker compose version

1. Clone the repository

git clone https://github.com/BIPLAT-CIBERINFEC/pathocore-api.git
cd pathocore-api
git checkout develop

2. Build and install the local test stack

Run the container installer from the repository root:

bash container_install.sh --test --git_revision current

This command will:

• build the application image
• start app and db
• install the Django project inside the container
• run database migrations

To install the isolated API stack and import a provided PathoCore API SQL dump in the same flow:

bash container_install.sh --test --git_revision current \
  --pathocore_api_sql ../pathocore_api_testing_seed.sql.gz

The SQL dump can be .sql or .sql.gz. This option is intended for the Compose-managed test database. Production deployments with external databases should import dumps through the database administration flow agreed for that environment.

3. Check that the containers are running

docker compose -f docker-compose.test.yml ps

4. Follow the application logs

In a separate terminal:

docker compose -f docker-compose.test.yml logs -f app

To inspect the database container logs:

docker compose -f docker-compose.test.yml logs -f db

5. Open the API documentation

Once the stack is up, Swagger UI should be available at:

http://localhost:8000/v1/swagger/

The OpenAPI schema is available at:

http://localhost:8000/v1/openapi/

6. Use the test administrative user

The test stack creates an idempotent Django superuser from conf/docker_test_settings.txt so Django admin access and protected API calls can be tested after a fresh Docker install:

admin / admin_pass

Override DJANGO_SUPERUSER_USERNAME, DJANGO_SUPERUSER_EMAIL and DJANGO_SUPERUSER_PASSWORD before running container_install.sh to use different local credentials. Production compose keeps PATHOCORE_CREATE_DEFAULT_SUPERUSER disabled by default.

7. Optional: load a small non-sensitive test dataset

This public repository does not include production data or large internal datasets. If maintainers provide a small test dump separately, pass it to the installer:

bash container_install.sh --test --git_revision current \
  --pathocore_api_sql /path/to/pathocore_api_test_dump.sql.gz

The installer reads database settings from conf/docker_test_settings.txt and imports the dump into the local Compose db service.

Useful commands

Open a shell in the API container:

docker compose -f docker-compose.test.yml exec app bash

Open a MySQL shell in the database container:

docker compose -f docker-compose.test.yml exec db mysql -u<db_user> -p<db_password> pathocore_api

Stop the containers but keep the database volume:

docker compose -f docker-compose.test.yml down

Stop the containers and remove the database volume:

docker compose -f docker-compose.test.yml down -v

Use down -v only when you want to discard the local test database completely.

Docker Production Installation

docker-compose.prod.yml is intended for controlled server deployments behind a reverse proxy. By default, the API binds only to 127.0.0.1 on the host:

PATHOCORE_API_BIND_HOST=127.0.0.1
PATHOCORE_API_PORT=8000
PATHOCORE_HOST_LOG_DIR=/var/log/local/pathocore-api/apps

Prepare a non-committed production install file on the server, based on conf/docker_production_settings.txt. It can live outside the repository, for example under /srv/containers/bind/pathocore-api/production_settings.txt. Do not commit credentials.

The production install file is consumed by install.sh inside the running container to generate the installed Django settings and /opt/pathocore-api/.env. container_install.sh also writes .env.prod.file in the repository root for Docker Compose interpolation. That generated file contains runtime metadata such as ports, paths and Gunicorn tuning, not database/SMTP/Keycloak secrets.

Start or upgrade the production container with container_install.sh so the Django installation, migrations and runtime .env are applied consistently:

bash container_install.sh \
  --install_conf /srv/containers/bind/pathocore-api/production_settings.txt \
  --git_revision current

For upgrades:

bash container_install.sh \
  --install_conf /srv/containers/bind/pathocore-api/production_settings.txt \
  --action upgrade \
  --git_revision current

To repair production bind-mount permissions without rebuilding or bootstrapping:

bash container_install.sh \
  --install_conf /srv/containers/bind/pathocore-api/production_settings.txt \
  --action fix-permissions

Validate from the server itself:

curl -I http://127.0.0.1:8000/v1/openapi/
curl -I http://127.0.0.1:8000/v1/swagger/
curl -I http://127.0.0.1:8000/v1/databrowser/overview-summary

Important Environment Variables

For isolated API deployments, start from conf/docker_test_settings.txt for local testing or conf/docker_production_settings.txt for server deployments. Do not commit real production settings.

Database Settings

Variable	Example	Purpose
DB_USER	pathocore	Database user used by Django.
DB_PASS	change_me	Database password used by Django.
DB_NAME	pathocore_api	Database name.
DB_SERVER_IP	pathocore_db or database host/IP	Database host.
DB_PORT	3306	Database port.

In the isolated test stack, DB_SERVER_IP is the Compose service name. In production it should point to the agreed database host.

Public API Host Settings

Variable	Example	Purpose
PATHOCORE_API_LOCAL_SERVER_IP	127.0.0.1	Host/IP inserted into Django ALLOWED_HOSTS.
PATHOCORE_API_DNS_URL	mepram-api-pathocore.<domain>	Public API hostname inserted into Django ALLOWED_HOSTS.
PATHOCORE_API_ALLOWED_HOSTS	localhost,127.0.0.1,api.<domain>	Optional comma-separated override for Django ALLOWED_HOSTS.
PATHOCORE_API_CSRF_TRUSTED_ORIGINS	https://api.<domain>	Public HTTPS origins trusted by Django forms such as admin login.

These values configure what hostnames the API considers valid. When the API is deployed behind the pathocore-web orchestrator, the orchestrator passes these values to the API installer.

When the API is deployed behind Apache or another reverse proxy, set PATHOCORE_API_CSRF_TRUSTED_ORIGINS to the public API origin. The API trusts the proxy-provided X-Forwarded-Proto and X-Forwarded-Host headers so Django can validate HTTPS form submissions correctly.

Keycloak Settings

Variable	Example	Purpose
KEYCLOAK_ISSUER	https://keycloak.example.org/realms/ciberisciii_datahub	Issuer expected in Bearer tokens.
KEYCLOAK_JWKS_URL	http://keycloak:8080/realms/.../certs	URL used by the API to fetch Keycloak public keys.
KEYCLOAK_REALM	ciberisciii_datahub	Keycloak realm name.
KEYCLOAK_AUDIENCE	pathocore-api	Expected token audience.
KEYCLOAK_CLIENT_ID	pathocore-web	Frontend client ID.
KEYCLOAK_ADMIN_BASE_URL	http://keycloak:8080	URL used for Keycloak admin API operations.
KEYCLOAK_ADMIN_USERNAME	admin	Keycloak admin user used for provisioning access requests.
KEYCLOAK_ADMIN_PASSWORD	change_me	Keycloak admin password.
KEYCLOAK_ADMIN_SEND_ACTION_EMAILS	true or false	Whether Keycloak sends password setup / verification action emails.
KEYCLOAK_ADMIN_ACTION_EMAIL_REDIRECT_URI	https://mepram-datahub.<domain>/signin	Optional redirect after Keycloak action emails.

KEYCLOAK_JWKS_CACHE_TTL_SECONDS and KEYCLOAK_JWKS_TIMEOUT_SECONDS have safe defaults and normally do not need to be changed.

Access Request Email Settings

Variable	Example	Purpose
EMAIL_HOST	mailpit or SMTP host	SMTP server used by Django.
EMAIL_PORT	1025 or 587	SMTP port.
EMAIL_HOST_USER	SMTP username	Optional SMTP auth username.
EMAIL_HOST_PASSWORD	SMTP password	Optional SMTP auth password.
EMAIL_BACKEND	django.core.mail.backends.smtp.EmailBackend	Django email backend. Use django.core.mail.backends.console.EmailBackend for CLI-only development checks.
EMAIL_USE_TLS	false or true	Whether SMTP uses TLS.
DEFAULT_FROM_EMAIL	no-reply@pathocore.local	Sender shown in PathoCore API emails.
ALLOWED_EMAIL_DOMAINS	ciberisciii.es,externos.isciii.es	Optional comma-separated recipient domain allow-list for the test email command.
PATHOCORE_ACCESS_REQUEST_ADMIN_EMAILS	admin@example.org	Optional fallback/copy recipients if Keycloak use-case admins are not found.

New access requests notify admins from the Keycloak group /use-cases/<use_case>/admin. PATHOCORE_ACCESS_REQUEST_ADMIN_EMAILS should be used only as a fallback or general copy list.

Local Admin and Public Endpoint Settings

Variable	Purpose
PATHOCORE_ENABLE_LEGACY_BASIC_AUTH	Enables Basic Auth compatibility for local/testing workflows.
PATHOCORE_ENABLE_PUBLIC_READ_ENDPOINTS	Allows selected read-only endpoints without Bearer auth.
PATHOCORE_CREATE_DEFAULT_SUPERUSER	Creates/updates the configured Django superuser during container startup.
DJANGO_SUPERUSER_USERNAME	Optional local Django superuser username.
DJANGO_SUPERUSER_EMAIL	Optional local Django superuser email.
DJANGO_SUPERUSER_PASSWORD	Optional local Django superuser password.

Databrowser summary cache

The databrowser summary endpoints use precomputed global summaries for the default unfiltered view:

• /v1/databrowser/overview-summary
• /v1/databrowser/metadata-summary
• /v1/databrowser/schema-summary

Refresh the cache manually after large sample or metadata ingests:

docker compose -f docker-compose.test.yml exec app bash
cd /opt/pathocore-api
source virtualenv/bin/activate
python manage.py refresh_databrowser_cache

In Docker, a lightweight scheduler runs inside the app container and refreshes the cache every Friday at 12:00 by default. The schedule can be overridden with:

DATABROWSER_CACHE_REFRESH_WEEKDAY=4
DATABROWSER_CACHE_REFRESH_TIME=12:00
DATABROWSER_CACHE_REFRESH_ON_START=false

Keycloak authentication

PathoCore validates Bearer JWTs issued by Keycloak. The API needs these settings:

KEYCLOAK_ISSUER=https://<keycloak-host>/realms/<realm>
KEYCLOAK_JWKS_URL=https://<keycloak-host>/realms/<realm>/protocol/openid-connect/certs
KEYCLOAK_AUDIENCE=pathocore-api
KEYCLOAK_CLIENT_ID=pathocore-web

Optional settings:

KEYCLOAK_JWKS_CACHE_TTL_SECONDS=300
KEYCLOAK_JWKS_TIMEOUT_SECONDS=5
PATHOCORE_ENABLE_LEGACY_BASIC_AUTH=true
PATHOCORE_ENABLE_PUBLIC_READ_ENDPOINTS=true
PATHOCORE_CREATE_DEFAULT_SUPERUSER=false
DJANGO_SUPERUSER_USERNAME=
DJANGO_SUPERUSER_EMAIL=
DJANGO_SUPERUSER_PASSWORD=

Authorization is derived from the JWT groups claim. Supported group paths are /use-cases/<project>/<view|admin> and /superusers. The older viewer role is accepted as view during migration.

Configuration flow:

• Bash install: copy conf/template_install_settings.txt, fill the KEYCLOAK_* values, then run install.sh. The installer writes those values to <INSTALL_PATH>/.env; template_settings.py loads that file at runtime.
• Docker install: fill conf/docker_test_settings.txt or conf/docker_production_settings.txt, then run container_install.sh. The selected install settings are exported for Compose and also written to the installed app .env.
• Runtime environment variables override the installed .env values.

Missing Keycloak values warn while legacy auth is enabled and fail when legacy auth is disabled.

Swagger UI, Redoc and the OpenAPI schema are public documentation views. Protected API endpoints still require authentication. Use-case endpoints require a valid Keycloak Bearer token or an allowed administrative API user.

When using Keycloak:

1. Open the configured Keycloak realm URL.
2. Log in with your use-case credentials.
3. Copy the access token.
4. Send API requests with: Authorization: Bearer <token>

For local Docker testing, Django admin access and protected API requests can use the default superuser credentials:

admin / admin_pass

API rate limiting

Public databrowser/read endpoints remain unauthenticated for the web public area, while the API uses Django REST Framework AnonRateThrottle and UserRateThrottle as basic overuse protection. Public endpoints use the anonymous throttle path. Configure the shared rate with:

PUBLIC_API_THROTTLE_RATE=500/hour

Requests over the public limit return the standard DRF 429 Too Many Requests response. This is operational protection for repeated public API calls, not a replacement for production reverse-proxy or DoS controls.

The same configured DRF rate is used for anonymous and authenticated requests.

Access Request Workflow

PathoCore keeps pending registration and approval state in its own database. Keycloak remains the final identity and group store only after approval.

Public users can create requests:

curl -sS -X POST http://localhost:8000/v1/access-requests \
  -H "Content-Type: application/json" \
  -d '{
    "username": "new_user",
    "email": "new.user@example.org",
    "first_name": "New",
    "last_name": "User",
    "message": "I collaborate with the MEPRAM and RELECOV networks.",
    "requests": [
      {"use_case": "mepram", "role": "view"},
      {"use_case": "relecov", "role": "view"}
    ]
  }' | jq

The canonical API prefix is /v1. The legacy /api/v1 prefix remains available temporarily while clients migrate back to /v1.

The user receives an email confirming that the request was received and remains pending review.

Admins can review pending requests:

curl -sS -u admin:admin_pass \
  "http://localhost:8000/v1/access-requests?status=pending" | jq

Approve a request:

curl -sS -X POST -u admin:admin_pass \
  http://localhost:8000/v1/access-requests/<request_id>/approve \
  -H "Content-Type: application/json" \
  -d '{"review_note": "Approved for MEPRAM view access."}' | jq

Reject a request:

curl -sS -X POST -u admin:admin_pass \
  http://localhost:8000/v1/access-requests/<request_id>/reject \
  -H "Content-Type: application/json" \
  -d '{"review_note": "Missing project justification."}' | jq

Revoke a previously approved request:

curl -sS -X POST -u admin:admin_pass \
  http://localhost:8000/v1/access-requests/<request_id>/revoke \
  -H "Content-Type: application/json" \
  -d '{"review_note": "Access no longer required."}' | jq

Approval calls the Keycloak Admin REST API to create or enable the user, trigger execute-actions-email with UPDATE_PASSWORD and VERIFY_EMAIL, and assign the approved group path, for example:

/use-cases/mepram/view

Configure these values for approval:

KEYCLOAK_ADMIN_BASE_URL=http://keycloak:8080
KEYCLOAK_REALM=ciberisciii_datahub
KEYCLOAK_ADMIN_TOKEN_REALM=master
KEYCLOAK_ADMIN_CLIENT_ID=admin-cli
KEYCLOAK_ADMIN_USERNAME=admin
KEYCLOAK_ADMIN_PASSWORD=admin

Do not send temporary passwords by email. Keycloak must have SMTP configured for execute-actions-email; otherwise approval can fail before permissions are granted.

Request received, rejection, and revocation notifications are sent by pathocore-api through Django email settings. Approval notifications include a link to the corresponding use-case section in PathoCore Web:

EMAIL_HOST=mailpit
EMAIL_PORT=1025
EMAIL_USE_TLS=false
DEFAULT_FROM_EMAIL=no-reply@pathocore.local

New pending requests are notified to enabled Keycloak users with email in the matching use-case admin group, for example /use-cases/mepram/admin. Configure PATHOCORE_ACCESS_REQUEST_ADMIN_EMAILS only as a fallback or general copy. Rejected and revoked notifications include the review note as the reason and the available use-case admin contact emails. Access request workflow emails are sent as multipart plain text and HTML. The HTML version presents the requested use-case, role and web section in a structured summary instead of exposing raw group paths in the main sentence.

In the local Docker test stack these messages are captured by Mailpit:

http://127.0.0.1:8025

Send a test email with the active Django settings:

python manage.py send_test_email user@example.org

For a console-only check, override the backend:

EMAIL_BACKEND=django.core.mail.backends.console.EmailBackend \
  python manage.py send_test_email user@example.org

Revocation removes the approved Keycloak group but does not disable the whole account.

For local Docker testing without SMTP or Mailpit, keep:

KEYCLOAK_ADMIN_SEND_ACTION_EMAILS=false

The generic databrowser and variant read endpoints are intentionally public for the no-login web experience. They are read-only, query-limited where applicable, rate-limited through PUBLIC_API_THROTTLE_RATE, and can be disabled globally with:

PATHOCORE_ENABLE_PUBLIC_READ_ENDPOINTS=false

Disabling public read endpoints also disables anonymous access from the web app.

For a Dockerized API validating local host-issued tokens, keep the issuer as the token sees it and point JWKS through the Docker host gateway:

KEYCLOAK_ISSUER=http://127.0.0.1:8090/realms/ciberisciii_datahub
KEYCLOAK_JWKS_URL=http://host.docker.internal:8090/realms/ciberisciii_datahub/protocol/openid-connect/certs

Installation outside Docker

PathoCore API can also be installed directly on a Linux server. This mode is intended for controlled server deployments where the host already provides the required system services such as MySQL/MariaDB and, if needed, a reverse proxy.

The installer expects:

• a reachable MySQL/MariaDB database
• a writable installation path, typically /opt/pathocore-api
• a configuration file based on conf/template_install_settings.txt

Basic flow:

cp conf/template_install_settings.txt install_settings.txt
nano install_settings.txt
sudo bash install.sh --install dep --conf install_settings.txt
bash install.sh --install app --git_revision develop --conf install_settings.txt

If the application is already installed and you need to deploy code changes, use upgrade mode:

bash install.sh --upgrade app --git_revision develop --conf install_settings.txt

Documentation

This is an API-only project. Public documentation is served via Swagger UI at /v1/swagger/ and the OpenAPI schema at /v1/openapi/.