Google Trust Services: Invalid OCSP responses
This case documents an incident involving Google Trust Services (GTS) OCSP generation. On 2018-01-19, GTS initiated a code push to improve OCSP generation for a subset of Google-operated Certificate Authorities, related to packaging generated OCSP responses. The change triggered unexpected behavior in GNU tar that manifested as empty tarballs, which were distributed to the global CDN and effectively dropped some OCSP responses while continuing to serve updates. The issue was not noticed by Google employees until after a post-mortem investigation began, and remediation procedures were triggered after discovery. The incident report states the issue was resolved about 2 days and 6 hours from when it was introduced, with additional time for the fix to be fully deployed. In this Bugzilla thread, Ryan Sleevi opened the bug for bookkeeping after noticing a prior incident bug was missing, and Andy Warner stated GTS had nothing additional to add and that the affected CA infrastructure was turned down and new pipelines include checks for the January 2018 issue. The bug was resolved as FIXED.
- GTS deployed a code push intended to improve OCSP generation packaging for a subset of Google-operated CAs.
- GTS discovered the issue via a mailing list post and began remediation procedures.
- GTS resolved the OCSP issue and completed deployment after additional propagation time.
- A Bugzilla entry was filed for bookkeeping regarding the January 2018 OCSP incident.
- Community commenter — Filed the bug for bookkeeping, describing the January 2018 OCSP incident and providing the incident report details.
- Community commenter — Noted that in responding to Bug 1630040 they found no prior bug for the previous incident and suggested resolving/fixing for accounting purposes.
- Google representative — Confirmed GTS had nothing additional to add and stated the affected infrastructure was turned down and new pipelines include checks for the January 2018 issue.
- Fastly representative — Resolved the bug per comment #1.