📖Docs · Architecture & integration

mDL/mdoc incident runbook — DSC/IACA expiry, runtime config corruption, key rotation

Operational procedures for the mdoc PKI and runtime configs described across
docs/deploy.md's "mDL / mdoc issuance" sections and
docs/mdl-issuance-manual-checklist.md. Those documents cover setup and
verification; this one covers what to do when something breaks in an
already-running deployment.

Scope: the walt.id/issuer-api2 path and the Inji Certify Pre-Auth path (see
docs/dpg/inji-certify-preauth.md) — both emit real mso_mdoc credentials and
both depend on the DSC/IACA generated by cmd/mdl-pki-gen for walt.id, or on
Inji Certify's own mock-HSM PKI for Inji. Not in scope: rotating the
general-purpose SD-JWT issuer key (internal/signer/) — see
docs/superpowers/specs/2026-07-23-issuer-key-rotation-design.md, a design
document for future work, not implemented.

1. The DSC is approaching or has passed its 457-day expiry

Symptom: wallets start rejecting newly-issued mdocs with a certificate
validity error, or a conformant reader flags the DSC as expired.
cmd/mdl-pki-gen prints the expiry date at generation time
(mdl-pki-gen: DSC expires <date> (Annex B caps this at 457 days)) — check
that log line from the original deploy, or read it directly:

openssl x509 -in deploy/k8s/config/issuer2/certs/dsc.pem -noout -enddate

Why 457 days: ISO/IEC 18013-5 Annex B caps DSC validity at 457 days, and
requires it not to outlive its IACA. internal/mdl/serversigner.go and
cmd/mdl-pki-gen/main.go both encode this cap; it is not configurable
without also changing the standard's conformance requirement.

What re-issuing a DSC does NOT do: it does not invalidate credentials
already issued under the old DSC — those remain verifiable as long as the
IACA that signed the old DSC is still published and trusted by wallets. It
only affects new issuance going forward.

Procedure — walt.id path:

Back up the current material (you may need it later to explain a support
ticket about an old credential):
```
cp -r deploy/k8s/config/issuer2/certs deploy/k8s/config/issuer2/certs.bak-$(date +%Y%m%d)
```
Remove dsc.pem (and dsc-key.pem) specifically — not iaca.pem or
iaca-key.pem. cmd/mdl-pki-gen's idempotence gate is keyed on
dsc.pem's existence; removing only the DSC forces a fresh DSC to be
issued under the SAME IACA, which is what you want — the IACA is the
trust anchor already distributed to wallets (see §3 below), and rotating
it unnecessarily forces every wallet to re-import.
```
rm deploy/k8s/config/issuer2/certs/dsc.pem deploy/k8s/config/issuer2/certs/dsc-key.pem
```
Re-run the provisioning step (or just ./deploy.sh up waltid, which calls
it as part of the normal up flow):
```
./deploy.sh up waltid
```
provision_issuer2_certificates (scripts/gen-caddy.sh) regenerates
dsc.pem signed by the existing iaca.pem, writes the new
VERIFIABLY_ISSUER2_KEY_X/_Y/_D coordinates to .env, and re-renders
issuer2-profiles.conf's defaultIssuerX5chain — but only if that
file still carries the OLD DSC's x5chain, which it will, since nothing
else rewrites it between deploys.

Verify the new DSC is live:

openssl x509 -in deploy/k8s/config/issuer2/certs/dsc.pem -noout -enddate
curl -s http://localhost:<issuer2-port>/trust/mdoc-anchors | python3 -m json.tool

Issue one test credential and verify it end-to-end (same smoke test as
docs/mdl-issuance-manual-checklist.md's Step 5/6, or the independent
Node harness at internal/mdl/testdata/verify/).

Procedure — Inji Certify Pre-Auth path: Inji Certify's mock-HSM PKI is
managed by Inji Certify itself, not by cmd/mdl-pki-gen — this repo only
extracts its ROOT for distribution (provision_inji_root_anchors in
deploy.sh), it does not generate or rotate Inji's signing key. Consult
Inji Certify's own documentation for its key lifecycle; a full compose
volume reset (docker volume rm on Inji Certify's key/HSM volume, then
./deploy.sh up inji) forces regeneration but invalidates every credential
Inji has issued — the same trade-off as §2's "wipe" path below, and just as
disruptive.

2. `issuer2-profiles.conf` (runtime) is corrupted or in an inconsistent state

Symptom: issuer-api2 fails to boot with a HOCON parse error, or boots
but every mdoc issuance 500s, or GET /issuer2/profiles returns something
that doesn't match what you expect.

First, determine whether this is a migration case. If this deployment
predates the baseline/runtime config split (commit history before
2026-08-21), see docs/mdl-issuance-manual-checklist.md's "One-time
migration" section first — that is a different procedure (adopting the
split) from what follows here (recovering a runtime file that already
exists under the split).

Recovery, in order of preference:

If you have a recent backup of issuer2-profiles.conf (you should —
see §4 below for why you need one), restore it directly:
```
cp /path/to/backup/issuer2-profiles.conf deploy/k8s/config/issuer2/issuer2-profiles.conf
./deploy.sh up waltid
```

If you do NOT have a backup but dsc.pem/iaca.pem still exist
(deploy/k8s/config/issuer2/certs/), the x5chain can be re-rendered from
them without regenerating any key material — this is the recoverable
path:

# Reset the runtime file back to the tracked baseline (loses any custom
# mdoc schema display names an operator saved — those live in
# credential-issuer-metadata.conf, a SEPARATE file, unaffected by this).
rm deploy/k8s/config/issuer2/issuer2-profiles.conf
./deploy.sh up waltid   # seed_issuer2_configs re-seeds from baseline,
                        # provision_issuer2_certificates re-renders the
                        # x5chain from the EXISTING dsc.pem/iaca.pem —
                        # confirm this by checking the x5chain in the
                        # regenerated file matches:
openssl x509 -in deploy/k8s/config/issuer2/certs/dsc.pem -noout -fingerprint
grep -A2 defaultIssuerX5chain deploy/k8s/config/issuer2/issuer2-profiles.conf
# (visually confirm the PEM block matches; there is no automated check for this)

If dsc.pem/iaca.pem are ALSO gone or corrupted — this is not
recoverable. A fresh dsc.pem/iaca.pem pair means a fresh trust
anchor, and every credential issued under the old IACA becomes
unverifiable the moment wallets stop trusting the old anchor (which they
will, once GET /trust/mdoc-anchors stops serving it — see §3). This is
why .gitignore's own comment on deploy/k8s/config/issuer2/certs/
says "regenerating is not a recovery path". Treat this as a full reset:
generate fresh material (rm -rf deploy/k8s/config/issuer2/certs && ./deploy.sh up waltid),
accept that every previously-issued mdoc is now unverifiable, and notify
whoever depends on those credentials that re-issuance is required.

3. What happens to wallets on IACA rotation (planned or forced)

The trust anchor mechanism (GET /trust/mdoc-anchors,
internal/handlers/mdoc_anchors.go) is explicitly a POC mechanism — see
docs/superpowers/adr/2026-08-23-mdl-trust-anchor-distribution.md. On any
redeploy that generates a NEW IACA (any of the "not recoverable" paths
above, or a genuinely fresh deployment):

A wallet built against cdpi-wallet's mdocTrustAnchors.ts fetches the
current anchor from the exact origin it resolved the credential offer's
credential_issuer from, unions it with a compiled-in fallback (so
credentials issued before the rotation keep verifying if the wallet
cached the old anchor), and falls back to stale-cache-or-static on any
fetch error. In practice: a wallet that has already claimed a credential
under the old IACA keeps working; a wallet claiming a NEW credential
after the rotation picks up the new anchor automatically, no manual
import needed, PROVIDED the wallet is online at claim time.
A wallet that manually imported the old IACA as a static trust anchor
(e.g. for testing with multipaz-identity-reader, per
docs/mdl-issuance-manual-checklist.md Step 7) does NOT automatically
pick up the new one — re-import is required. announce_issuer2_trust_anchor
(scripts/gen-caddy.sh) prints the current IACA's path at the end of
every deploy.sh up specifically so this step is not missed.
This is a POC gap, not a design flaw to route around silently: the
ADR's documented production replacement (a Hub-signed VICAL-style list)
would make this section largely unnecessary — a rotation would update one
Hub-signed list instead of requiring wallet-side re-fetch/re-import logic
per issuer. See TODO.md's "VICAL-style trust anchor distribution" entry
for that work's status (not started as of 2026-08-31).

4. Preventive: back up what recovery in §2 depends on

There is no automated backup of deploy/k8s/config/issuer2/certs/ or
issuer2-profiles.conf — both are gitignored runtime state by design (see
.gitignore's own comments on both paths), which means "not in git" is not
the same as "backed up". Before any planned maintenance that touches this
host (OS upgrade, disk migration, etc.), copy these off-host:

tar czf mdoc-pki-backup-$(date +%Y%m%d).tar.gz \
  deploy/k8s/config/issuer2/certs/ \
  deploy/k8s/config/issuer2/issuer2-profiles.conf \
  deploy/k8s/config/issuer2/credential-issuer-metadata.conf \
  deploy/compose/stack/inji/certify/certify-postgres-dataprovider.properties \
  deploy/compose/stack/inji/esignet/credential-scopes.properties

Store this somewhere with the same access-control posture as any other
private-key material — certs/ contains the DSC and IACA private keys in
cleartext PEM.

Source: docs/mdl-incident-runbook.md

mDL/mdoc incident runbook — DSC/IACA expiry, runtime config corruption, key rotation

1. The DSC is approaching or has passed its 457-day expiry

2. issuer2-profiles.conf (runtime) is corrupted or in an inconsistent state

3. What happens to wallets on IACA rotation (planned or forced)

4. Preventive: back up what recovery in §2 depends on

2. `issuer2-profiles.conf` (runtime) is corrupted or in an inconsistent state