Certainly: Missing audit log entries during capacity testing; incident report closed
Certainly reported a self-discovered compliance incident involving missing audit log entries for 5,868 certificates issued during planned capacity characterization testing on 2026-04-01 and 2026-04-02. The incident was found during annual WebTrust audit work, and Certainly said the affected certificates all expired by 2026-05-02, so none remained valid. The root cause was described as silent audit-log loss in the host logging pipeline under sustained high-volume issuance, with an overloaded forwarding step as the dominant loss path and a logging-service queue overflow as a secondary contributor. Certainly reported that it replaced the lossy transport, added DB-vs-log reconciliation and related monitoring, removed a duplicate output action, and added audit-log throughput validation to the load-test procedure. In the latest thread activity, Certainly said all six action items were complete and alerting had been fully elevated to paging. CCADB then issued a final call for comments on 2026-08-29 and said the incident report would be closed around 2026-09-05; the bug is now resolved.
- Capacity characterization testing began and audit log loss started under sustained high-volume issuance.
- A second testing window occurred and additional audit log loss was observed.
- All affected certificates expired.
- Investigation identified the silent log-loss pathway.
- The transport replacement remediation was deployed to production.
- All disclosed action items were reported complete and alerting was fully elevated to paging.
- The bug was resolved.
- Fastly representative — Posted a preliminary incident report saying Certainly found 5,868 certificates missing expected audit log entries during annual WebTrust audit work.
- Fastly representative — Added the full incident report with the impact, timeline, analysis, and related incidents.
- Fastly representative — Said remediation was still on track for 2026-07-30.
- Fastly representative — Reported several action items as complete or in progress and said the next update would follow by 2026-08-01 or upon completion.
- Fastly representative — Said five of six action items were complete, the transport replacement was in production, and the reconciliation service was estimated for 2026-08-06.
- Fastly representative — Reported the reconciliation service deployed to production and the alerts were in a testing period.
- Fastly representative — Said the reconciliation service remained deployed, thresholds needed minor tuning, and alerts were active but not yet paging.
- Fastly representative — Said the reconciliation service continued operating in production and further tuning was in progress.
- Fastly representative — Reported that all action items were complete and alerting had been fully elevated to paging.
- CCADB representative — Issued a final call for comments and said the incident report would be closed around 2026-09-05.