Let's Encrypt: 2019.08.20 Incident: Incorrect OCSP responses under certain conditions

On 2019-08-20 08:48 UTC, Let’s Encrypt received a report from community member and Apache httpd developer Stefan Eissing that, under certain conditions, its OCSP caching layer could return a valid OCSP response but not the one that was requested. Let’s Encrypt stated this caused its OCSP service to act in violation of RFC 6960, and it believed the behavior was triggered by making the OCSP request via POST with the “Expect: 100-continue” header described in RFC 7231 section 5.1.1. Let’s Encrypt determined the issue was with its CDN provider (Akamai), because the origin servers were not seeing the requests in question. Let’s Encrypt reported the problem to Akamai, and Akamai fixed the issue; Let’s Encrypt also applied a temporary production workaround by stripping the problematic header. Let’s Encrypt stated it did not stop certificate issuance, and it reported that no certificates were problematic, though a very small number of OCSP responses were incorrect. In response to Mozilla’s request for an incident report, Let’s Encrypt provided a detailed incident report and later indicated remediation was complete; the bug is resolved as FIXED.

1. 2019-08-20 A report was received that Let’s Encrypt’s OCSP caching layer could return an incorrect OCSP response under certain request conditions.
2. 2019-08-21 A temporary production workaround was applied to strip the problematic header.
3. 2019-08-26 Akamai confirmed a global permanent fix and public disclosure was made.
4. 2020-01-23 Mozilla requested a full incident report; Let’s Encrypt provided it and later indicated remediation was complete.

1. 2019-08-27 Kflag representative — Reported that the OCSP caching layer could return the wrong response under specific POST requests with the “Expect: 100-continue” header, identified Akamai as the cause, and said Akamai fixed the issue.
2. 2019-08-27 Kflag representative — Provided a more complete timeline including the initial report, Akamai ticket, temporary workaround, private disclosures to root programs, and Akamai’s confirmation of a permanent fix.
3. 2020-01-23 Fastly representative — Requested a full incident report per Mozilla’s incident-report instructions and noted consistent reporting helps ensure valuable information is not missed.
4. 2020-01-23 Kflag representative — Submitted answers for the incident report questions, including that issuance was not stopped, no certificates were problematic, and the issue was attributed to a CDN bug.
5. 2020-01-23 Fastly representative — Indicated remediation was complete after confirming the questions were answered.