Skip to content

[STACKED on #880] feat(simulator): add deterministic seeded attestations - #964

Merged
kvinwang merged 13 commits into
masterfrom
codex/feat-simulator-seeded-attestation
Aug 5, 2026
Merged

[STACKED on #880] feat(simulator): add deterministic seeded attestations#964
kvinwang merged 13 commits into
masterfrom
codex/feat-simulator-seeded-attestation

Conversation

@kvinwang

@kvinwang kvinwang commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Builds on #880 (merged).

Problem

A development guest cannot verify the KMS and the gateway it talks to when those
run under the simulated PKI, and two independent defects stood in the way.

1. The seeded mock TDX PKI was not reproducible across processes. The seed
only fed key derivation; everything around it was nondeterministic:

  • cert_params and the root CRL used OffsetDateTime::now_utc(), so validity
    windows differed per process and per run.
  • Certificates were signed through rcgen's own KeyPair, whose ECDSA
    backend draws a random nonce, so the same key over the same TBS bytes
    produced a different signature each time.

The host collateral service and the in-guest simulator therefore reconstructed
different root certificates from the same seed. A quote produced in one
process could not be verified against collateral produced in the other, which is
exactly the arrangement a simulated deployment runs in.

2. The guest verifier was hard-wired to production roots. Both
dstack-util (stage0 and the gateway path) and guest-agent called
AttestationVerifier::new_prod(...) directly, so a simulated guest had no way to
authenticate a simulated key provider. The obvious fix — let the host hand the
guest a root through sys-config — is not available: the host sits outside the
trust boundary, and a host-chosen trust anchor lets it stand up a fake KMS and
hand the guest keys it never earned.

Fix

Make the seeded PKI byte-identical across processes. Fixed
MOCK_PKI_NOT_BEFORE / MOCK_PKI_NOT_AFTER constants replace wall-clock
validity, and every certificate is now signed through
rcgen::RemoteKeyPair backed by a p256::ecdsa::SigningKey, whose RFC 6979
nonce makes the signature a pure function of key and message. Same seed ⇒ same
roots, same CRL, same collateral, in any process.

Move the trust anchor handoff entirely inside the guest.
dstack-tee-simulator derives the roots from the same seed it already uses to
sign evidence, and writes them to /run/dstack/attestation — guest tmpfs the
host cannot reach — before dstack-prepare runs. The host supplies a seed for a
fake TEE device through .tee-simulator.json and never names a trust anchor.

One decision point on the reader side. dstack_attest::default_verifier is
now the only place a guest component picks a trust anchor, and both
dstack-util and guest-agent go through it. It reads
trust_anchors::load_anchors, which accepts only that one directory and
validates what it finds:

  • owner must be the current euid, checked on symlink_metadata so a planted
    symlink cannot pass while resolving elsewhere;
  • nothing group- or world-writable;
  • every root path must live directly inside the published directory, so a stale
    or tampered roots.json cannot redirect the verifier at a host-shared file.

When the directory is absent — the only outcome on a production image, which
does not ship the simulator and whose contents are measured — new_prod roots
apply, unchanged. The systemd unit removes the directory in ExecStopPost, so
external roots never outlive the simulator run that published them.

Services still configured with a mock root through hand-written TOML (KMS,
gateway, dstack-verifier) keep needing an explicit
insecure_allow_external_trust_anchors = true. The guest path sets it
internally because there is no operator to warn: one program in this guest wrote
the roots and the next authenticated the directory before reading them.

Seeded guest-agent-simulator. An optional mock_attestation_seed makes the
simulator re-sign fixture attestations with the seeded PKI, preserving the
fixture's mr_td and RTMRs while binding the requested report_data, so the
quote it returns actually verifies.

Changed paths

Path Change
dstack/crates/mock-attestation/src/tdx.rs deterministic signing and fixed validity windows
dstack/crates/mock-attestation/README.md document the guest handoff and where the insecure flag does/doesn't apply
dstack/dstack-attest/src/lib.rs new default_verifier — the single guest-side trust anchor decision
dstack/dstack-attest/src/trust_anchors.rs new — ANCHOR_DIR, roots_path, load_anchors plus the ownership/mode/containment checks
dstack/dstack-util/src/system_setup.rs stage0 and gateway paths use default_verifier
dstack/guest-agent/src/rpc_service.rs same
dstack/tee-simulator/src/main.rs publish seeded roots to the runtime dir before mounting anything
dstack/tee-simulator/tests/process_e2e.rs assert the published root signed the quote; report an early-exiting child instead of a timeout
dstack/guest-agent-simulator/src/{main,simulator}.rs optional seed; re-sign fixture evidence with measurements preserved
dstack/vmm/src/app.rs test asserting sys-config never carries a trust anchor
os/.../dstack-tee-simulator.service drop /run/dstack/attestation on stop

Verification

cargo clippy --all-targets is clean workspace-wide; tests pass for every
touched crate, including the process-level e2e. New coverage:

  • mock-attestation: seeded_hierarchies_are_cross_process_compatible — two
    TdxGenerator::from_seed instances produce identical roots and CRL, and a
    quote from one verifies under dcap_qvl against collateral from the other.
    This is the test that would have failed before the determinism fix.
  • dstack-attest::trust_anchors: absent directory selects production roots;
    published roots round-trip and load into a real AttestationVerifier; a
    malformed root fails verifier construction; world-writable roots are rejected;
    a root outside the published directory is rejected.
  • dstack-tee-simulator: the default runtime dir resolves to ANCHOR_DIR; the
    published TDX root equals the seeded PKI's root; a seedless config publishes
    nothing and fails loudly.
  • dstack-tee-simulator e2e: a separately spawned simulator process publishes a
    root that verifies the quote read back through the simulated TSM device.
  • guest-agent-simulator: seeded_simulator_resigns_certificate_attestation
    the re-signed certificate attestation verifies under the seeded root and
    carries the RA-TLS report_data for the requested public key.
  • dstack-vmm: make_sys_config output contains no root_ca / trust_anchor
    key in any deployment mode.

Copilot AI review requested due to automatic review settings July 31, 2026 03:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

…r-seeded-attestation

# Conflicts:
#	dstack/dstack-attest/src/attestation.rs
#	dstack/dstack-util/src/system_setup.rs
@kvinwang
kvinwang force-pushed the codex/feat-simulator-seeded-attestation branch 6 times, most recently from 4ae5f1e to 8c50307 Compare August 5, 2026 10:24
@kvinwang
kvinwang force-pushed the codex/feat-simulator-seeded-attestation branch from 8c50307 to 11a334c Compare August 5, 2026 10:33
@kvinwang
kvinwang merged commit d9796c0 into master Aug 5, 2026
18 checks passed
kvinwang added a commit that referenced this pull request Aug 5, 2026
…file

[STACKED on #964] fix(verifier): validate one-shot certificate profile
kvinwang added a commit that referenced this pull request Aug 5, 2026
…ap-collateral

[STACKED on #964] fix(mock-attestation): model complete DCAP collateral
kvinwang added a commit that referenced this pull request Aug 5, 2026
…testation

[STACKED on #964] fix(simulator): preserve legacy attestation responses
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants