Let's Encrypt: OCSP "unauthorized" responses

The case reports an incident involving Let’s Encrypt’s OCSP responder service. On 2018-08-23, a configuration change deployed to the OCSP responder resulted in about 90% of traffic to the origin receiving OCSP "unauthorized" statuses for valid OCSP requests. The change was reverted the same day, but cached OCSP responses at the CDN could have caused affected statuses to be served for a limited period after resolution. The incident was attributed to a bug in the OCSP request validation implementation that improperly rejected OCSP requests unless they matched the last configured serial prefix rather than any configured serial prefix; the underlying implementation issue was later fixed. The CA reported the incident and described remediation steps, including reviewing monitoring procedures to ensure monitoring parity between production and staging, extending OCSP monitoring to include OCSP statuses (including "unauthorized"), and adding alerts for unusually high fractions of unauthorized or revoked OCSP responses. The CA stated that these remediation items were completed and considered the remediation for the incident complete. The bug is marked RESOLVED with resolution FIXED.

1. 2018-08-23 Let’s Encrypt deployed an OCSP responder configuration change that caused many valid OCSP requests to receive OCSP "unauthorized" statuses.
2. 2018-08-23 Let’s Encrypt reverted the OCSP responder configuration change to resolve the issue.
3. 2018-08-27 Let’s Encrypt disclosed the OCSP incident and remediation details in the Mozilla CA Program bug.
4. 2018-09-23 Let’s Encrypt reported completion of the listed remediation items and stated remediation was complete.

1. 2018-08-27 Fastly representative — Wayne Thayer posted an incident report describing the OCSP "unauthorized" status problem, the cause, the revert, and three remediation items.
2. 2018-08-27 Kflag representative — Josh Aas asked to update the bug as remediation items were completed.
3. 2018-09-23 Fastly representative — Wayne Thayer reported that monitoring parity was applied to staging, OCSP status monitoring was extended (with a referenced Boulder change), and alerts for unauthorized/revoked fractions were added, and stated remediation was complete.