# Production activation incident - 28 August 2026

## Customer-visible symptom

SmartCut64 1.0.2 displayed:

> Licence activation is temporarily unavailable.

Lemon Squeezy nevertheless showed the installation as activated, while the
desktop application remained in Evaluation mode.

## Diagnosis

The production service was reachable over trusted HTTPS and its Lemon licence
API path was available. A production KV activation record was created at the
reported activation time. In the Worker flow, that means the request passed
rate limiting, Lemon activation and record storage before failing while issuing
the signed `.sc64` entitlement.

The most probable cause was an unusable `SMARTCUT_PRIVATE_KEY_PEM` Worker
secret. The independently retained production PFX was loaded ephemerally,
matched the 3072-bit public key compiled into SmartCut64 1.0.2, and passed an
in-memory PKCS#1 SHA-256 sign/verify test.

No customer email address, order reference, licence key, Installation ID or
private-key material is retained in this incident note.

## Remediation

At 2026-08-28 16:05 UTC, the production Worker secret
`SMARTCUT_PRIVATE_KEY_PEM` was replaced from the verified DPAPI-protected local
PFX. The PKCS#8 PEM existed only in process memory and standard input; it was
not written to a plaintext file, placed on a command line or printed.

Cloudflare created production version:

`c14dfdbb-7c8e-49b5-8184-e524d171c00a`

Existing Lemon and KV activation records were preserved. Retrying the same
installation follows the existing-record validation path and should not consume
another activation seat.

## Verification

- The new Worker version lists both required production secrets.
- `GET /health` returned HTTP 200 with the expected service/schema response.
- A syntactically valid, deliberately nonexistent diagnostic licence reached
  the production Lemon path and returned the expected HTTP 403
  `licence_not_valid`, rather than an availability error.
- Full closure requires the affected customer, or a separately controlled live
  licence, to receive and install a signed entitlement successfully.

## Follow-up hardening

- Implemented locally, pending controlled production deployment: `/ready` and
  `/health` now return success only after a fixed-challenge sign/verify test.
  The expected public-key fingerprints are pinned in source to the test and
  production Lemon product IDs, so a private key and configuration variable
  cannot be changed together to bless the wrong key.
- Implemented locally, pending controlled production deployment: activation
  performs the same key-pair proof and prepares the signed entitlement before
  Lemon consumes a seat or KV publishes an activation record. Deterministic
  partial storage failures roll the seat back; failed rollbacks retain a
  `rollback_pending` recovery record when storage remains writable and are
  reconciled before another attempt. A simultaneous total KV outage can prevent
  that recovery handle and requires manual reconciliation.
- Implemented locally, pending controlled production deployment: readiness and
  activation read a product-and-physical-namespace-pinned, non-customer KV
  marker. An unseeded namespace or accidental test/live binding swap therefore
  fails before Lemon can consume an activation. A production namespace must
  never be replaced merely by copying this marker; state continuity requires a
  frozen, verified migration and a newly pinned marker. Existing instances
  reported invalid are also key-validated and retired by their exact UUID
  before replacement activation.
- Implemented locally, pending controlled production deployment: a durable
  privacy-safe `activation_pending` intent is written before Lemon is asked to
  consume a seat. A lost or malformed activation response and a compound
  rollback/storage failure therefore fail closed on same-isolate and sequential
  retries once that intent is visible, instead of deliberately attempting
  another activation without a known instance UUID. Concurrent identical
  requests handled by one Worker isolate are serialized; cross-location KV
  visibility remains subject to the limitation below.
- Implemented locally: pending recovery phases use `status=revoked` plus an
  explicit `recovery_status`. The current production v1 Worker understands the
  legacy status and blocks without activating, while the hardened Worker can
  still reconcile the recovery phase. The current v1 version therefore remains
  a fail-closed rollback target after candidate traffic.
- Implemented locally: the bridge runbook now requires authoritative Lemon
  inspection, deactivation of every exact `SmartCut64 <installation_id>` match,
  production-marker verification, an audited transition from
  `activation_pending` to `deactivated`, and remote readback before retry.
- Implemented locally: the deployment runbook puts a candidate at 0% traffic,
  targets that exact version through Cloudflare's override header, checks the
  returned version ID and readiness result, and only then permits promotion.
- Add a correlation identifier and privacy-safe structured error category for
  activation failures without logging purchase emails, raw licence keys or
  Installation IDs.
- Retain a controlled live activation check after future secret rotations.
- A future concurrency hardening phase should move each licence/installation
  state machine to strongly consistent coordination. Workers KV is retained as
  the privacy-safe lookup/index store and cannot alone guarantee at-most-once
  activation under simultaneous cross-region requests.
- Lemon's License API exposes no idempotency key. If Lemon commits an
  activation but the network loses the response before the Worker receives the
  instance ID, automatic compensation is impossible from that response alone.
  The Worker retains the pre-activation intent, returns no entitlement and
  blocks identical retries; operational reconciliation is still required to
  identify and release any stranded seat. Future strongly consistent instance
  coordination remains required for simultaneous cross-region requests.
