Skip to content

CONFIGURATION.md

Latest commit

 

History

History
458 lines (354 loc) · 15.2 KB

CONFIGURATION.md

File metadata and controls

458 lines (354 loc) · 15.2 KB
Raw

Configuration Reference

This document describes the configuration options for PTD deployments.

Configuration Files

PTD uses YAML configuration files organized by deployment type:

targets/
├── __ctrl__/                    # Control room configurations
│   └── <control-room-name>/
│       └── ptd.yaml
│
└── __work__/                    # Workload configurations
    └── <workload-name>/
        ├── ptd.yaml             # Workload infrastructure
        └── site_<name>/
            └── site.yaml        # Site (products) configuration

Control Room Configuration

File: __ctrl__/<name>/ptd.yaml

apiVersion: posit.team/v1
kind: AWSControlRoomConfig
spec:
  # Required: AWS account ID
  account_id: "123456789012"

  # Required: AWS region
  region: us-east-2

  # Required: Primary domain
  domain: control-room.example.com

  # Required: External access domain
  front_door: cr.example.com

  # Amazon Elastic Kubernetes Service (EKS) cluster version
  eks_k8s_version: "1.33"

  # Enable Tailscale for remote access
  tailscale_enabled: true

  # IAM principals allowed to assume admin role
  trusted_principals:
    - admin@example.com
    - arn:aws:iam::123456789012:role/github-actions

  # Resource tags for all AWS resources
  resource_tags:
    rs:owner: team@example.com
    rs:project: posit-team
    rs:environment: production

  # AWS tag keys PTD must never add or remove on managed resources (AWS-only).
  # Wired into the Pulumi AWS provider's ignoreTags.keys so customer-applied tags
  # (e.g. from an org-wide tagging policy) are left untouched by our IaC. Exact key match.
  # Honored on every `ptd ensure`, including `--refresh` (state is not polluted with ignored tags).
  ignore_tags:
    - customer:cost-center
    - customer:owner

Workload Configuration

File: __work__/<name>/ptd.yaml

apiVersion: posit.team/v1
kind: AWSWorkloadConfig
spec:
  # Required: AWS account ID
  account_id: "123456789012"

  # Required: Control room reference
  control_room_account_id: "123456789012"
  control_room_cluster_name: control-room-production
  control_room_domain: cr.example.com

  # Required: AWS region
  region: us-east-2

  # VPC configuration
  vpc_az_count: 2  # Number of availability zones (default: 3, max: 3)

  # RDS PostgreSQL configuration
  db_engine_version: "15.18"
  db_instance_class: db.m5d.large
  db_performance_insights_enabled: true
  db_deletion_protection: true
  db_max_allocated_storage: 1024

  # Additional databases
  extra_postgres_dbs:
    - analytics

  # FSx for OpenZFS (shared storage)
  fsx_openzfs_storage_capacity: 900      # GB
  fsx_openzfs_throughput_capacity: 320   # MB/s

  # Domain configuration
  domain_source: ANNOTATION_JSON  # or ROUTE53_PRIVATE_ZONE

  # Authentication
  keycloak_enabled: false

  # Cluster autoscaling
  autoscaling_enabled: true

  # EKS clusters (supports blue/green)
  clusters:
    "20250115":  # Date-based naming recommended
      spec:
        cluster_version: "1.33"
        mp_min_size: 3
        mp_max_size: 10
        mp_instance_type: r6a.2xlarge
        root_disk_size: 200
        routing_weight: "100"  # For blue/green: 0-255
        force_maintenance: false  # Enable to bypass PDBs during upgrades
        components:
          traefik_forward_auth_version: "0.0.14"

  # AWS tag keys PTD must never add or remove on managed resources (AWS-only).
  # Wired into the Pulumi AWS provider's ignoreTags.keys so customer-applied tags
  # (e.g. from an org-wide tagging policy) are left untouched by our IaC. Exact key match.
  # Honored on every `ptd ensure`, including `--refresh` (state is not polluted with ignored tags).
  ignore_tags:
    - customer:cost-center
    - customer:owner

  # Sites in this workload
  sites:
    main:
      spec:
        domain: analytics.example.com
        use_traefik_forward_auth: false
    dev:
      spec:
        domain: analytics-dev.example.com

Azure Postgres version (postgres_version)

For AzureWorkloadConfig, postgres_version sets the Postgres Flexible Server major version for both the main and grafana servers. It takes a major version only (e.g. "17", not "17.2") and defaults to "14" when unset.

spec:
  postgres_version: "17"

Raising it performs an in-place, irreversible major version upgrade with downtime on both servers. The value must be greater than or equal to the servers' live major version (Azure rejects downgrades).

Preview first with ptd ensure <target> --only-steps persistent --dry-run. If the step has unrelated pending changes, scope the apply to the two servers with --target-resources <main Server URN>,<grafana Server URN>.

Adopting ignore_tags on an existing target

ignore_tags is enforced by the AWS provider's ignoreTags, which only takes effect after a state-persisting ptd ensure (an apply) has registered it on the target's AWS provider. On the run that first registers it, Pulumi still plans against the previously-registered provider — so if a matching tag is already tracked in Pulumi state, that apply will strip it once. Follow this procedure when enabling ignore_tags on an existing target:

  1. Add the keys to ignore_tags, then run ptd ensure <target> --dry-run.
  2. If the preview shows an ignored key being removed (- <key> on a resource), that key is already tracked in state (a prior --refresh pulled it in). Do not apply — the apply would strip the customer tag. Remove it from state first (a state-only ptd ensure <target> --refresh filters ignored keys out of state without modifying AWS), then repeat step 1.
  3. If the preview is clean, run ptd ensure <target> to apply. This registers ignoreTags on the provider with no tag changes.

After this one-time registration, tags matching ignore_tags are ignored on every subsequent ptd ensure (including --refresh) — never added, removed, or pulled into state. A key added to ignore_tags later requires another registering ptd ensure (on then-clean state) before it is protected.

Fleet rollout: running ptd ensure across all targets while their state is clean of the configured keys registers ignoreTags everywhere, so customer tags applied afterward are ignored from then on. Use the --dry-run check above to catch any target that already has a matching tag in state.

Site Configuration

File: __work__/<workload>/site_<name>/site.yaml

apiVersion: core.posit.team/v1beta1
kind: Site
spec:
  # Disable pre-pulling images
  disablePrePullImages: true

  # Posit Workbench (IDE)
  workbench:
    replicas: 2
    image: ghcr.io/rstudio/rstudio-workbench:ubuntu2204-2026.01.0
    defaultSessionImage: ghcr.io/rstudio/workbench-session:ubuntu2204-r4.4.3_4.3.3-py3.12.11_3.11.13
    imagePullPolicy: Always
    domainPrefix: dev  # -> dev.analytics.example.com

    # Authentication
    auth:
      type: "oidc"  # oidc, saml, ldap, pam
      clientId: "your-client-id"
      issuer: "https://your-idp.com"
      scopes:
        - "offline_access"
        - "groups"

    createUsersAutomatically: true

    # API access
    apiSettings:
      workbenchApiEnabled: 1
      workbenchApiAdminEnabled: 1

    # Session timeouts (hours)
    vsCodeConfig:
      sessionTimeoutKillHours: 4
    positronConfig:
      sessionTimeoutKillHours: 4

    # Session label injection — presence of this block enables the feature.
    # The controller reads a field from each session pod, matches entries against
    # searchRegex, and writes one numbered label per match (user-group-1, user-group-2, …).
    # See docs/team-operator/README.md for full details.
    sessionLabels:
      # Dot-path into the pod spec identifying the field with the comma-separated group list.
      # Supports array index notation. Default: "spec.containers[0].args"
      sourceField: "spec.containers[0].args"

      # Sub-key used to extract the value from the resolved field:
      #   args slice   → flag name whose next element is the value
      #   map field    → map key to look up (e.g. annotation or label key)
      #   string field → ignored; field value used directly
      # Default: "--container-user-groups"
      sourceKey: "--container-user-groups"

      # Regex applied to each comma-separated entry. Only matching entries become labels.
      # Default: "_entra_[^ ,]+"
      searchRegex: "_entra_[^ ,]+"

      # Prefix stripped from the start of each matched entry before it becomes the label value.
      # Default: "_"
      trimPrefix: "_"

      # Prefix for generated label keys. Default "user-group-" produces user-group-1, user-group-2, …
      labelKeyPrefix: "user-group-"

      # When true, forces re-labeling of all existing session pods for this site.
      # Use after changing searchRegex or trimPrefix to update already-processed pods.
      # Set back to false (or omit) once done. Default: false
      reprocess: false

  # Posit Connect (Publishing)
  connect:
    replicas: 2
    image: ghcr.io/rstudio/rstudio-connect:ubuntu2204-2025.12.1
    sessionImage: ghcr.io/rstudio/rstudio-connect-content-init:ubuntu2204-2025.12.1
    imagePullPolicy: Always
    domainPrefix: pub  # -> pub.analytics.example.com

    auth:
      type: "oidc"
      clientId: "your-client-id"
      issuer: "https://your-idp.com"

  # Posit Package Manager
  packageManager:
    image: ghcr.io/rstudio/rstudio-package-manager:ubuntu2204-2025.09.2
    imagePullPolicy: Always
    domainPrefix: pkg  # -> pkg.analytics.example.com

  # Posit Chronicle (Audit Logging)
  chronicle:
    image: ghcr.io/rstudio/chronicle:2025.08.0
    agentImage: ghcr.io/rstudio/chronicle-agent:2025.08.0

  # Custom ingress annotations
  ingressAnnotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "100m"

  # Extra service accounts for custom integrations
  extraSiteServiceAccounts:
    - nameSuffix: custom-app
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/custom-role

CLI Configuration

File: ~/.config/ptd/ptdconfig.yaml

# Path to targets directory
targets_config_dir: /path/to/targets

# Default AWS profile (optional)
aws_profile: my-profile

# Default region (optional)
region: us-east-2

Environment Variables

Variable Description
PTD_TARGETS_CONFIG_DIR Path to targets directory
PTD_AWS_ACCOUNT_STAGING AWS account ID for staging
PTD_AWS_ACCOUNT_PRODUCTION AWS account ID for production
AWS_PROFILE AWS credential profile to use
AWS_REGION Default AWS region

Authentication Options

OIDC (OpenID Connect)

auth:
  type: "oidc"
  clientId: "your-client-id"
  issuer: "https://your-idp.com"
  scopes:
    - "offline_access"
    - "groups"

Security Assertion Markup Language (SAML)

auth:
  type: "saml"
  idpMetadataUrl: "https://your-idp.com/metadata"

Lightweight Directory Access Protocol (LDAP)

auth:
  type: "ldap"
  ldapUrl: "ldap://ldap.example.com"
  ldapBindDn: "cn=admin,dc=example,dc=com"

Cloud Provider Integrations

Databricks

workbench:
  databricks:
    my-workspace:
      name: "Databricks Workspace"
      url: "https://workspace.cloud.databricks.com"
      clientId: "databricks-oauth-client-id"

Snowflake

workbench:
  snowflake:
    clientId: "snowflake-client-id"
    accountId: account-name

VPC Availability Zone Configuration

The vpc_az_count setting controls how many availability zones the VPC subnets span. This affects where EKS nodes and EBS volumes can be placed.

Value Use Case
3 (default) Maximum redundancy across AZs
2 Recommended for workloads with StatefulSets using EBS volumes

Why Use 2 AZs?

EBS volumes are bound to a single availability zone. When using StatefulSets (like Loki or Mimir for monitoring), the PersistentVolumes are created in a specific AZ. If a node group rolls and new nodes are placed in different AZs than where the PVs exist, pods cannot schedule.

With 3 AZs and 2 nodes, there's no guarantee that nodes will cover all AZs where PVs may exist. Restricting to 2 AZs ensures that after any node operation, there will always be a node available in each AZ where PVs are located.

Note: This setting only affects new VPCs. Changing this value on an existing workload will cause Pulumi to attempt to delete subnets in the removed AZ.

Cluster Maintenance Options

force_maintenance

The force_maintenance option enables cluster version upgrades to proceed even when they would normally be blocked by safety checks.

clusters:
  "20250115":
    spec:
      cluster_version: "1.33"
      force_maintenance: true  # Bypass upgrade-blocking checks
Cloud Provider Behavior
AWS EKS Sets ForceUpdateVersion on the cluster, which overrides upgrade-blocking readiness checks including EKS Insights validations (deprecated APIs, compatibility issues, cluster health checks)
Azure AKS Sets UpgradeSettings.OverrideSettings.ForceUpgrade with a 24-hour expiration window, which bypasses PodDisruptionBudget (PDB) constraints and takes precedence over all other drain configurations

When to use:

  • During planned maintenance windows when you accept workload disruption
  • When PDBs are blocking necessary security or version upgrades (Azure)
  • When EKS upgrade insights are blocking an upgrade you've assessed as safe (AWS)
  • When you need to force through an upgrade that has stalled

Caution:

  • Azure: Bypasses PodDisruptionBudget protections, which may cause service disruption. Pods protected by PDBs may be evicted without respecting minimum availability guarantees.
  • AWS: Bypasses pre-upgrade validation checks. Review EKS Insights warnings before forcing an upgrade to understand what issues are being overridden.
  • Only enable temporarily during maintenance windows, then set back to false

Default: false (safety checks are respected during upgrades)

Secret Management Options (Azure)

external_secrets_enabled

Deploys the External Secrets Operator (ESO) to an AKS cluster and generates a per-site ExternalSecret, so Azure Key Vault secrets sync into native Kubernetes Secrets instead of being applied by hand. AKS only — AWS uses the Secrets Store CSI driver.

clusters:
  "20250115":
    external_secrets_enabled: true
    components:
      external_secrets_version: "2.8.0"  # optional; chart version

Key Vault secrets follow the <compound>-<site>-<field> convention; the site's ExternalSecret selects ^<compound>-<site>- and strips that prefix so the resulting Secret keys match what team-operator reads.

Before enabling on an existing cluster, the Key Vault entries must already exist under the new names and match the live cluster values — otherwise the ExternalSecret (which owns the target Secret) will reproduce it incompletely. Some Key Vault secrets are created by PTD code and some must be created by hand.

Default: false

See External Secrets on AKS for the secret-ownership tables, the migration procedure, and the post-migration cleanup checklist.

See Also