GoDaddy: Failure to revoke 210 subscriber certificates within 24 hours
This case concerns GoDaddy’s failure to process 210 customer subscriber certificate revocation requests within 24 hours, which GoDaddy stated was a violation of the Baseline Requirements (BRs) for Publicly Trusted SSL certificates (Section 4.9.1.1). GoDaddy reported that the RA team noticed delayed revocation requests and escalated the issue to PKI Engineering on Sep 23, 2022. GoDaddy stated that PKI Engineering confirmed the issue and identified a root cause related to a bug introduced with the integration of a new event queue system (RabbitMQ) in April 2020, where the event queue could become unresponsive and revocation requests could remain persisted but unprocessed. GoDaddy reported that PKI Engineering revoked all 210 certificates on Sep 23, 2022 (noting that 123 were still active) and added automated alerts to notify PKI Engineering if revocations were not processed as scheduled. GoDaddy also stated that it deployed a failsafe on 10/24/2022 to automatically process delayed revocation requests and that it formally completed the last step in its remediation plan. The bug was resolved as FIXED, and Mozilla indicated it planned to close the case on or about Wed. 23-Nov-2022.
- GoDaddy integrated a new event queue system (RabbitMQ), introducing a write-failure logic oversight that could leave revocation requests unprocessed if the queue became unresponsive.
- GoDaddy’s RA team escalated a delayed revocation issue to PKI Engineering, which confirmed 210 revocation requests had not been processed as expected.
- PKI Engineering revoked all 210 certificates and added monitoring/automated alerts to mitigate further delayed revocation processing.
- GoDaddy deployed a failsafe to automatically process delayed revocation requests.
- GoDaddy — Jessica Coover created the incident report describing the delayed processing of 210 revocation requests and GoDaddy’s remediation timeline.
- GoDaddy — Brittany Randall stated there were no formal updates and that GoDaddy would continue to monitor for questions/comments.
- Google representative — Chris Clements asked follow-up questions about how the April 2020 introduction was determined, what caused the queue to become unresponsive, and how the automated alerts were tested.
- GoDaddy — Brittany Randall acknowledged the questions and said GoDaddy expected to post answers early the next week.
- GoDaddy — Jessica Coover answered that the bug related to RabbitMQ write-failure logic introduced in April 2020, that networking changes/failures caused the queue to become unresponsive, and that alerting was tested by reproducing unprocessed requests and observing successful alerting.
- GoDaddy — Brittany Randall reported that on 10/24/2022 a failsafe was deployed to process delayed revocation requests automatically and that the last remediation step was formally completed.
- Mozilla representative — Mozilla asked whether there were any additional questions/issues and stated an intent to close the bug on or about Wed. 23-Nov-2022.