cmgr Artifact Server

A simple daemon to automatically handle the distribution of cmgr build artifacts.

cmgr-artifact-server is intended to run alongside cmgrd and supports multiple file hosting backends. Like cmgrd, it is a single binary and requires minimal configuration.

The CMGR_ARTIFACT_DIR environment variable (also used by cmgrd) determines which artifacts to distribute, while the backend and any additional settings are specified via command-line options.

Behind the scenes, cmgr-artifact-server maintains a cache (.artifact_server_cache) within the specified CMGR_ARTIFACT_DIR. A full synchronization of all existing local artifacts to the backend is performed upon startup. Any further changes to local artifacts (due to build creation, updates, or deletion) are automatically handled as they occur.

What that cache holds depends on the backend. The selfhosted backend serves files off local disk, so each tarball is unpacked into it. The S3 backend copies bytes to a bucket and reads them out of the tarballs directly, so it keeps only a checksum per build — unpacking for it would mean a second copy of every artifact on the machine that builds them, which for a corpus of any size is most of a disk.

Artifact namespaces

A cork build plane builds for several orchestrators at once, and sorts each schema's tarballs into a subdirectory of CMGR_ARTIFACT_DIR named for the destination it was built for. Those subdirectories are picked up automatically, and a build found in one is published under a matching path: a tarball at library/7.tar.gz becomes library/7/file.c rather than 7/file.c, so one server can publish several events' artifacts without their build IDs colliding, and an event can be retired by deleting a single prefix.

cork marks each of these directories with an empty .cork-artifact-namespace file, and only marked directories are treated as namespaces. Any other subdirectory is ignored — which matters because CMGR_ARTIFACT_DIR is often the challenge directory itself, whose subdirectories are challenges.

A tarball directly in CMGR_ARTIFACT_DIR keeps a bare build ID for its path, exactly as before, so a plain cmgr/cmgrd deployment is unaffected.

Installation

Download the latest release for your platform, extract the tarball, and copy the binary to an appropriate location:

$ tar xzf cmgr-artifact-server_linux_amd64.tar.gz
$ cp cmgr-artifact-server /usr/local/bin

Alternatively, build and install from source:

$ cargo install --locked --path .

Backends

selfhosted backend

The selfhosted backend provides a simple way to serve build artifacts over HTTP without exposing the full cmgrd API to end users. cmgr-artifact-server itself will act as a web server for generated artifact files.

# Artifact files can be served by the cmgrd API, but this also exposes other endpoints:
$ cmgrd &
$ curl http://localhost:4200/builds/1/file.c            # 200 OK
$ curl http://localhost:4200/builds/1                   # 200 OK
$ curl http://localhost:4200/builds/1/artifacts.tar.gz  # 200 OK
$ curl http://localhost:4200/challenges                 # 200 OK

# With the selfhosted backend, cmgr-artifact-server serves individual artifact files only:
$ cmgr-artifact-server -b selfhosted &
$ curl http://localhost:4201/1/file.c                   # 200 OK
$ curl http://localhost:4201/1                          # 404 Not Found
$ curl http://localhost:4201/1/artifacts.tar.gz         # 404 Not Found

When using the this backend with the picoCTF platform (note: not yet publicly available), specify http://hostname:4201 as the challenge server's artifact base URL.

S3 backend

This backend syncs artifact files to a specified S3 bucket. It can also automatically generate invalidations for an associated CloudFront distribution.

$ cmgr-artifact-server -b S3 \
> --backend-option bucket=sample-bucket-name \
> --backend-option path-prefix=ctf-artifacts \
> --backend-option cloudfront-distribution=EDFDVBD6EXAMPLE &

# Creates a new build:
$ cmgr build cmgr/examples/custom-socat 1
  Build IDs:
    4

# Potentially updates existing builds:
$ cmgr update

# In either case, any modified artifact files are synced to S3:
$ curl https://your-cloudfront-distribution.com/ctf-artifacts/4/file.c  # 200 OK

Note that there will necessarily be some delay between cmgr(d) reporting a build as successful and the completed upload of its associated artifacts.

IAM user credentials are read from the same sources used by the AWS CLI, e.g. the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, the ~/.aws/config and ~/.aws/credentials files, etc. The provided IAM user requires the following permissions for the associated resources:

• s3:ListBucket
• s3:GetObject
• s3:PutObject
• s3:DeleteObject
• cloudfront:CreateInvalidation (if a CloudFront distribution is specified)

The backend will check that all necessary IAM actions can be performed before starting.

When using this backend with the picoCTF platform (note: not yet publicly available), specify your bucket or CloudFront distribution URL (including path prefix, if applicable) as the challenge server's artifact base URL.

Using Build Digests

By default, cmgr-artifact-server exposes artifacts at paths which include their associated build IDs, as shown in the above examples.

However, this can be undesirable as it allows players to easily enumerate artifact URLs. Specifically, since multiple builds of a challenge will have adjacent build IDs, players can decrement and increment the build ID portion of their download URLs in order to discover artifacts for other builds of the same challenge.

This can lead to players discovering flags for other builds, or gaining unintended information by diffing multiple builds' versions of an artifact.

To mitigate this, the build ID component of artifact URL paths can be replaced with the lowercase SHA-256 hexadecimal digest of {build_id}:{salt} by specifying a salt value with the -s / --salt command-line flag.

Accordingly, any client applications that generate URLs served by cmgr-artifact-server must also apply this same transformation.

Options

short	long	description
-b	--backend	File hosting backend. Options: selfhosted, S3.
-o	--backend-option	Backend-specific option in key=value format. May be specified multiple times. Some options may be required - see backend-specific documentation.
-s	--salt	Optional salt for build digests. See using build digests.
-l	--log-level	Specify log level. Options: error, warn, info, debug, trace. Defaults to info.
-h	--help	Prints help information.
-V	--version	Prints version information.

selfhosted backend options

key	required?	description
address	no	Socket address to bind to. Defaults to 0.0.0.0:4201.

S3 backend options

key	required?	description
bucket	yes	S3 bucket name
path-prefix	no	Slash-delimited path prefix to use when uploading artifacts.
cloudfront-distribution	no	CloudFront distribution ID. If specified, will automatically create invalidations when artifacts are updated. Uses path-prefix if set (assumes distribution's origin path is the bucket root).
prune-orphans	no	Whether the startup synchronization removes bucket directories that have no local artifact. Defaults to true. See orphan removal.

Orphan removal

On startup, artifacts in the bucket with no corresponding local artifact are removed, so that builds deleted while this was not running do not stay published. Deletions that happen while it is running are propagated as they occur and do not depend on this pass.

Two safeguards, because this pass is destructive:

• If the local artifact directory holds no builds at all while the bucket holds some, nothing is removed and a warning is logged. An empty artifact directory means the host has not built yet — a fresh disk, a restored machine, a build server brought up on demand — and is not the statement that every build was deleted.
• -o prune-orphans=false disables the pass entirely. Use it where the artifact directory is not the durable record of what exists, such as a build server whose disk does not outlive it.

The pass treats every directory under the bucket's path-prefix (the whole bucket, if none is set) as this server's. To share a bucket with something this server did not publish, mark each such directory instead of turning the pass off: create a directory with the same name in CMGR_ARTIFACT_DIR and put an empty .dont-purge file in it. The bucket directory of that name, and everything under it, is then never removed by this pass.

# Keep s3://my-bucket/other-stuff/ (no path-prefix), while library/ and fooEvent/ are still pruned:
$ mkdir -p "$CMGR_ARTIFACT_DIR/other-stuff" && touch "$CMGR_ARTIFACT_DIR/other-stuff/.dont-purge"

Names are relative to path-prefix, and only a top-level subdirectory is read. The marks are read at startup, which is also the only time the pass runs. Removing the directory lets the next startup prune the bucket directory like any other.