Deploy¶

Deployment moves a released artifact into a running environment. It is the last gate before software is exposed to real traffic, and it is where supply-chain verification, environment configuration, and runtime access converge. Secure deployment ensures that only verified artifacts reach production, that they run with least privilege, and that the blast radius of any problem is contained.

Verify before you run: admission control and deployment gates¶

The deploy stage is where signing and provenance pay off. Do not allow an artifact to run unless you can cryptographically verify where it came from and that it passed required checks. Enforce verification across both deployment pipelines and admission control so that:

Only signed artifacts with valid signatures are admitted (see Artifact Signing and Provenance).
Only artifacts with verifiable, policy-compliant provenance (SLSA level, trusted builder identity, and approved source repository revision) run in production.
Verification targets the exact immutable artifact identified by its cryptographic digest (image@sha256:...), preventing time-of-check to time-of-use (TOCTOU) tampering associated with mutable tags.
Verification fails closed: if provenance is missing, invalid, or violates policy, deployment is aborted in the pipeline and blocked at admission, never allowed to proceed silently or fail open.

Pre-deployment verification in deployment pipelines¶

Before applying deployment manifests or triggering workload rollouts, the deployment pipeline must verify the immutable artifact digest against trusted builder and source identities. The verification step must fail closed on any policy violation:

# Verify provenance for the exact immutable digest before deploying.
# Fails closed (non-zero exit code) if provenance is missing, invalid, or violates policy.
slsa-verifier verify-image "myregistry.io/myapp@sha256:${IMAGE_DIGEST}" \
  --source-uri "github.com/myorg/myrepo" \
  --builder-id "https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/v2.1.0" || {
    echo "ERROR: Provenance verification failed against trusted builder and source. Aborting deployment." >&2
    exit 1
}

# Alternatively, verify SLSA provenance attestations using cosign:
cosign verify-attestation \
  --type slsaprovenance1 \
  --certificate-identity-regexp "^https://github.com/myorg/myrepo/\.github/workflows/release\.yml@" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "myregistry.io/myapp@sha256:${IMAGE_DIGEST}" || {
    echo "ERROR: Provenance attestation missing or invalid. Aborting deployment." >&2
    exit 1
}

Enforcing provenance at admission control¶

Kubernetes admission controllers (Kyverno, OPA/Gatekeeper with Ratify, Sigstore policy-controller) enforce these rules as code at runtime, providing an automated backstop if pipeline checks are bypassed. Kubernetes' built-in ValidatingAdmissionPolicy (CEL, GA since v1.30) covers simple non-image rules without an extra controller. Signatures from both cosign and Notation can be verified at admission.

Configure admission policies to inspect both the cryptographic signature and the SLSA provenance attestation, enforcing validationFailureAction: Enforce so any unverified or non-compliant artifact fails closed at cluster entry:

# Kyverno policy: require verified SLSA provenance attestation and fail closed
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-image-provenance
spec:
  validationFailureAction: Enforce   # newer Kyverno releases move this to a per-rule failureAction
  rules:
    - name: check-image-provenance
      match:
        resources:
          kinds: [Pod]
      verifyImages:
        - imageReferences: ["myregistry.io/*"]
          attestors:
            - entries:
                - keyless:
                    issuer: "https://token.actions.githubusercontent.com"
                    subject: "https://github.com/myorg/myrepo/.github/workflows/release.yml@refs/tags/*"
          attestations:
            - predicateType: https://slsa.dev/provenance/v1
              conditions:
                - all:
                    - key: "{{ builder.id }}"
                      operator: Equals
                      value: "https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/v2.1.0"

GitOps and declarative deployment¶

GitOps makes the desired state of an environment a version-controlled, auditable artifact:

The git repository is the single source of truth for deployment state. Changes go through reviewed pull requests — not SSH sessions, not manual Helm installs.
A reconciler (Argo CD, Flux) continuously applies the desired state and corrects drift. If someone manually modifies a running workload, the reconciler reverts it.
Every change has a full audit trail: who opened the PR, who approved it, when it merged, what the diff was — all in git history.
Rollback is a git revert — fast, auditable, and not dependent on tribal knowledge.

Use image digest pinning in deployment manifests rather than mutable tags:

# Prefer this (immutable):
image: myregistry.io/myapp@sha256:a1b2c3d4...
# Not this (mutable):
image: myregistry.io/myapp:latest

Limit blast radius with progressive delivery¶

Roll out changes gradually so a bad release — whether a bug, a security regression, or a compromised artifact — affects as few users and systems as possible:

Canary deployment — route a small percentage of traffic (e.g., 5%) to the new version. Monitor error rates, latency, and security signals. Promote only if metrics are healthy.
Blue-green deployment — run the new version alongside the old. Instantly switch traffic back if issues appear; the old version is not terminated until confidence is established.
Feature flags — deploy code without activating new behavior; enable features for specific users or percentages via a feature flag service. Decouples deploy from release.
Automated rollback — define health criteria and automatically revert if they breach thresholds within a deployment window.

Least privilege and secrets at deploy¶

Deploy identity¶

The service account or role used by the deployment pipeline should have:

Write access only to the target environment (not all environments).
Permission to update the specific workloads it manages, not cluster-admin.
Time-limited credentials: use OIDC federation (GitHub Actions or GitLab CI OIDC → AWS IAM role, GCP Workload Identity Federation, Azure federated credentials) rather than long-lived access keys.

Secrets injection¶

Never bake secrets into images or deployment manifests. Inject them at runtime from a trusted source:

Vault agent sidecar — a secrets management sidecar injects secrets into the pod filesystem or environment at startup and rotates them without restart.
Kubernetes External Secrets Operator — syncs secrets from AWS Secrets Manager, HashiCorp Vault, or GCP Secret Manager into Kubernetes Secrets automatically.
Secrets Store CSI Driver — mounts secrets from Vault, AWS Secrets Manager, Azure Key Vault, or GCP Secret Manager directly as files in the pod, optionally without creating Kubernetes Secret objects.
Cloud-native workload identity — services authenticate to cloud APIs using platform identity (EKS Pod Identity or IRSA, GKE Workload Identity Federation, Azure Workload Identity) with no stored credential at all.

See Secrets Management.

Environment parity and configuration security¶

Use the same base image and the same configuration parameters across environments — the only differences should be externally injected (vault paths, endpoint URLs, TLS certificates).
Never disable security controls in lower environments to make testing easier. A control disabled in staging often gets forgotten and reaches production.
Configuration drift between environments is a leading cause of "works in staging, fails in production" security incidents. IaC and GitOps eliminate drift by making configuration declarative.

Common pitfalls and anti-patterns¶

Manual deployments with shared credentials — no audit trail, broad permissions, and credentials that are never rotated. Replace with automated pipelines and workload identity.
latest tags in production — a mutable tag means any registry push can change what runs in production. Use digest-pinned references.
Admission control in audit-only mode forever — many teams set admission controllers to warn but never enforce, defeating the purpose. Set a deadline to switch to enforcement.
Failing open on missing or unverified provenance — allowing deployment when provenance is absent, timed out, or unverified. Verification must fail closed: if an artifact cannot cryptographically prove its source and builder identity against policy, deployment must be blocked.
Secrets in environment variables or build arguments — environment variables leak into logs, crash dumps, and /proc on Linux. Use mounted secret files or vault agent injection instead.
No rollback plan — deployment without a tested rollback plan is high-risk. Verify rollback in staging before every major production deploy.

Maturity progression¶

Starter — Deploy via reviewed PRs to a private registry. Use a scoped deploy identity. Inject secrets from a vault rather than baking them in. Document a manual rollback procedure.

Intermediate — GitOps with Argo CD or Flux. Admission control enforcing signed images (Kyverno/policy-controller). Canary or blue-green delivery for tier-1 services. Automated rollback on health signals. Image digest pinning in all manifests.

Advanced — Full SLSA provenance verification at admission. Zero standing credentials — all deploy identities are short-lived OIDC tokens. Progressive delivery with automated promotion gates (error rate, latency, security signals). Configuration drift detection and auto-remediation. Deployment decisions logged to an immutable audit store.

Metrics and KPIs¶

Deployment frequency — higher frequency with stable failure rates indicates a healthy deployment process.
Change failure rate — percentage of deployments causing incidents or rollbacks; security-related failures tracked separately.
Mean time to restore (MTTR) — time from a bad deployment detection to rollback completion.
Policy violation blocks at admission — number of deployments rejected by admission control; should reach zero as teams fix upstream processes.
Percentage of workloads with digest-pinned images — target 100% for production.

Tools¹¶

Open-source¶

Argo CD — Declarative GitOps continuous delivery for Kubernetes; provides a UI dashboard for deployment status, drift detection, and rollback.
Argo Rollouts — Progressive delivery controller for Kubernetes: canary, blue-green, and automated analysis-based promotion.
External Secrets Operator — Syncs secrets from external vaults (AWS, GCP, Azure, Vault) into Kubernetes Secrets; avoids storing secrets in git.
Flagger — Progressive delivery operator (Flux project) automating canary, A/B, and blue-green releases with Istio, Linkerd, Cilium, or ingress controllers and metric-based rollback.
Flux — GitOps toolkit for keeping clusters in sync; lightweight and composable; strong multi-tenancy and multi-cluster support.
Kyverno — Kubernetes admission policy engine including image signature verification, resource validation, and mutation; no Rego required.
OPA Gatekeeper — Rego-based Kubernetes admission controller built on Open Policy Agent; constraint templates enforce organization-wide policy.
Ratify — Verification engine that plugs into Gatekeeper to check cosign and Notation signatures, SBOMs, and other artifact attestations before admission.
Secrets Store CSI Driver — Kubernetes SIG driver that mounts secrets from external stores into pods as volumes using provider plugins.
Sigstore policy-controller — Enforces cosign image signature and attestation policies at Kubernetes admission; integrates with Fulcio and Rekor.

Commercial¶

Harness — CI/CD and progressive delivery with built-in governance policies, approval workflows, and audit trails.
Octopus Deploy — Release management and deployment automation with environment promotion gates and change management integration; strong for non-Kubernetes workloads.

Links¶

Listed in alphabetical order. ↩