Microsec: Expired Certificates on test Pages for Revocation
Microsec reported an incident involving its revocation test web pages: certificates protecting those pages had expired, making the pages not applicable for testing the CA revocation service. The issue was triggered when Microsec received an email from Chris Clements (Google) reporting a potential problem with two revoked test pages; Microsec then investigated and found three affected test web pages. Microsec corrected the problem by generating new keys and issuing new certificates with longer lifetimes, then replacing the certificates on the test web pages and notifying the affected party. Microsec attributed the root cause to the focus on automatic renewal for VALID test pages, which led to the expired certificates not being replaced on time, and to a separate issue related to the REVOKED certificate on the new dedicated hierarchy. In follow-up comments, Microsec described plans to improve monitoring and reconfigure its monitoring system to properly handle revoked test certificates, and it provided multiple status reports on process reviews and automation/testing changes. The bug was resolved as FIXED, with a closing status report dated 2025-01-10 and a note that Mozilla would close it on or about 13-Jan-2025.
- Microsec received an external report that revoked test pages were potentially problematic due to expired certificates.
- Microsec opened a JIRA ticket to manage the incident.
- Microsec provided a closing status report indicating related monitoring and manual checks had been completed or terminated.
- Microsec representative — Opened an incident report in Bugzilla describing expired certificates on revoked test pages and the corrective actions taken (new keys, new longer-lifetime certificates, and replacement on the test pages).
- Community commenter — Asked whether Microsec was considering automated monitoring (e.g., daily) to check expected certificate status on all deployed test web pages.
- Microsec representative — Responded that Microsec uses Nagios and daily certificate monitoring, but that warnings were turned off for revoked test certificates in this special case; stated it would reconfigure monitoring to handle these revoked test certificates properly.
- Microsec representative — Provided a status report listing review of external requirements and internal processes, clarifying requirements for test websites, and action items including issuing new long-lifetime REVOKED test certificates and improving supervision/automation.
- Microsec representative — Provided a status report describing completed external requirement review and planned changes to test website organization and automation for certificate exchange and daily testing.
- Microsec representative — Provided another status report with updates to test website setup, improved automatic testing, and action item statuses.
- Microsec representative — Reported decisions for REVOKED/EXPIRED test sites (manual renewal triggers and manual revocation) and continued work on audit log processing and weekly manual checks.
- Microsec representative — Submitted a closing status report stating log entries had been processed since end of November 2024 and that weekly manual checks were terminated after automated tests succeeded on 2025-01-01.
- Mozilla representative — Indicated the case would be closed on or about Monday, 13-Jan-2025.