Let's Encrypt: TLS-ALPN-01 challenge certificates allowed additional SAN identifiers (RFC 8737 non-compliance)

This case is a preliminary incident report from Let’s Encrypt / ISRG about a non-compliance in its TLS-ALPN-01 challenge implementation (RFC 8737). The issue was triggered when a bug report was received at 19:41 UTC on 2022-01-28 stating that Let’s Encrypt issued a TLS-ALPN-01 validated certificate that was non-compliant with RFC 8737 because the challenge certificate’s subjectAltName contained identifiers other than the dNSName being validated (including an IP Address). Let’s Encrypt confirmed the report and began incident response, including revoking the known misissued certificate at 21:22 UTC. The CA developed and merged a fix into Boulder and deployed it to production at 01:56 UTC on 2022-01-29. Let’s Encrypt also audited issuance/validation logs to identify additional affected authorizations and certificates, and it stated that affected certificates were revoked within five days of becoming aware of the issue. The bug was resolved as FIXED, and later updates directed reviewers to Bug 1751984 for remediation items; Mozilla indicated it would close the ticket if no further questions were raised.

1. 2022-01-28 Let’s Encrypt received a report of TLS-ALPN-01 non-compliance and began incident response.
2. 2022-01-28 Let’s Encrypt revoked the known misissued certificate.
3. 2022-01-29 Let’s Encrypt merged and deployed a Boulder fix to production for TLS-ALPN-01 validation.
4. 2022-02-02 Let’s Encrypt revoked all identified affected authorizations and issued certificates based on the affected authorizations.
5. 2022-03-03 Let’s Encrypt reported remediation items were completed on Bug 1751984 and requested closure.

1. 2022-01-29 Internet Security Research Group — Reported the incident details: RFC 8737 non-compliance due to extra SAN entries in the TLS-ALPN-01 challenge certificate, confirmed the issue, revoked the misissued certificate, merged a Boulder fix, deployed it to production, and planned log review and revocations by 2022-02-02.
2. 2022-01-31 Internet Security Research Group — Stated ongoing auditing of validation logs to find additional affected authorizations/certificates and reiterated revocation and a full incident report by 2022-02-03.
3. 2022-02-02 Internet Security Research Group — Provided a full incident report summary, stating the fix was merged/deployed, affected authorizations and issuances were identified, and affected certificates were revoked within five days; included a detailed timeline and stated only three certificates were affected.
4. 2022-02-10 Internet Security Research Group — Gave a weekly update pointing to Bug 1751984 for remediation items and asked to close this ticket if no further questions, otherwise set Next Update to 2022-03-11.
5. 2022-03-03 Internet Security Research Group — Reported remediation items were completed on Bug 1751984 and requested closure if no further questions.
6. 2022-03-03 Mozilla representative — Indicated Mozilla would close the ticket on 2022-03-04 unless there were questions or concerns.