Google Trust Services: OCSP serving issue 2020-04-09
Google Trust Services reported an OCSP serving issue affecting its EJBCA-based CAs (GIAG4, GIAG4ECC, GTSY1-4) during 2020-04-08 16:25 UTC to 2020-04-09 05:40 UTC, when empty OCSP data was served and OCSP responders returned unauthorized. Mozilla’s CA Program case describes how monitoring detected the issue on 2020-04-08 at 16:35 UTC and how the problem self-corrected after correct OCSP archives and responses were generated and pushed to the CDN. Google stated that the affected CAs are used for infrequent, manual custom certificate issuance for test sites for inactive roots, and that no certificate issuance aside from a manually issued post-update test certificate occurred during the period. Google attributed the root cause to a scripting/error-handling problem: a non-zero exit code from a tool fetching OCSP responses was suppressed in an AND/OR context, leading the script to use empty tar.gz files when EJBCA was not running. Google said it added safeguards to prevent recurrence, including additional pre-push checks for readable tar.gz, at least one OCSP response in the archive, and at least one OCSP response per CA. The bug was resolved as FIXED, and Google later confirmed the described changes went into production at the end of April and that remediation was complete.
- Incorrect OCSP archives were generated and incorrect OCSP responses were pushed to the CDN for affected EJBCA-based CAs.
- Production monitoring detected the OCSP serving issue via the first monitoring alert.
- Correct OCSP archives and responses were generated and pushed to the CDN; monitoring confirmed probes were passing.
- Google representative — Opened the bug describing the OCSP serving issue, its timeline, root cause, and the safeguards added to prevent recurrence; stated the issue self-corrected and remediation was implemented.
- Community commenter — Asked for clarification on why prior OCSP testing did not catch the issue and requested more detail on GTS OCSP processes and monitoring.
- Google representative — Provided additional context on GTS OCSP/revocation infrastructure stacks and described improvements, including adding a pre-push check for the empty-bundle condition.
- Community commenter — Acknowledged the context and asked for more detail on the existing pre-push checks and what is being added.
- Google representative — Explained additional pre-push (presubmit) checks planned for the next production update, including tar.gz readability and presence of OCSP responses.
- Community commenter — Noted another production revocation incident in a separate bug and suggested a pattern of revocation-related issues.
- Fastly representative — Asked whether the update described in the safeguards comment had been deployed and when it was expected.
- Google representative — Confirmed the changes went into production at the end of April shortly after the referenced update.
- Fastly representative — Confirmed remediation appeared complete after questions were answered.