Google Trust Services: Incorrect revocation data temporarily served for GTS Y3 & Y4

Google Trust Services (GTS) reported that during an internal review on 2020-04-17 it identified that revocation data rolled out on 2020-04-08 erroneously overwrote previously published revocation data for its old subordinate CAs GTS Y3 and GTS Y4. The issue occurred when “Batch 2” revocation data was installed on 2020-04-08 14:55 UTC, overwriting revocation data produced on 2020-01-31 for the old GTS Y3 and old GTS Y4, effectively un-revoking them. After discovery, GTS began a partial rollback of “Batch 2” at 2020-04-17 17:02 UTC and completed the rollout of correct CRLs by 2020-04-17 18:31 UTC, with correct OCSP responses served by 2020-04-18 01:25 UTC. The thread states that the outdated revocation data was rolled back and replaced with correct CRL and OCSP responses. The affected certificates were identified by two crt.sh IDs in the report. In follow-up comments, Mozilla asked for updates on implementing incident-reporting procedure changes; GTS responded that presubmits and procedure updates were in progress and later stated that mitigations were complete. Mozilla indicated it would schedule the bug to be closed on or after 7-August 2020 unless additional questions or issues arose. The bug is marked RESOLVED with resolution FIXED.

1. 2020-04-08 GTS installed revocation data “Batch 2,” overwriting previously published revocation data for old GTS Y3 and GTS Y4.
2. 2020-04-17 GTS identified the overwrite during an internal review and began a partial rollback of “Batch 2” to restore intended revocation data.
3. 2020-04-18 GTS completed rollout of correct OCSP responses for old GTS Y3 and GTS Y4.

1. 2020-05-01 Google representative — Opened the incident report describing how Batch 2 revocation data overwrote old GTS Y3/Y4 revocation data and how CRLs/OCSP were rolled back and corrected.
2. 2020-05-17 Community commenter — Asked for an update including a timeline/ETA for implementing changes to ensure incident reports include required information and timely weekly updates.
3. 2020-05-20 Google representative — Responded that procedure updates happened in early May, most presubmits were in place, and an ETA for the final presubmit would be provided.
4. 2020-06-01 Google representative — Reported the final presubmit did not make the last production push but was likely to make the next one during June.
5. 2020-06-08 Google representative — Stated it was still on track for the next production push but had not happened yet.
6. 2020-06-19 Google representative — Said the status was still the same.
7. 2020-07-09 Google representative — Reported the final presubmit was in place, exercised enough to be confident it worked, and that all mitigations were complete.
8. 2020-07-29 Mozilla representative — Indicated Mozilla would schedule the bug to be closed on or after 7-August 2020 unless additional questions or issues arose.