Let's Encrypt: Deployed Unreviewed Boulder Code

This case is an incident report from Let’s Encrypt about a deployment of Boulder CA software that was tagged using an unreviewed commit. The incident was triggered when a core developer accidentally tagged a Boulder release (release-2025-06-09) with a commit that had not been reviewed and merged into the main branch, and that release was deployed. After detection, the Boulder team reviewed the commit and stated it only removed dead code and did not cause incorrect operation of the CA, and they did not stop issuance during the incident period. The report explains that the gap was between automated systems enforcing reviewed commits on main and the release process assumptions for signed release tags, including that release tag verification was not automated. Let’s Encrypt reported remediation by improving automation for release tagging and release artifact builds to verify that tagged commits are reviewed and integration tests pass, and by reviewing historical release tags to ensure no other unreviewed code had been deployed to production. The bug is resolved as FIXED, with a report closure summary stating that all disclosed action items were completed and requesting closure.

1. 2025-06-09 A Boulder release tag was created and deployed based on an unreviewed commit.
2. 2025-06-16 The incident was detected and Let’s Encrypt began incident response and remediation.
3. 2025-07-21 Let’s Encrypt posted a report closure summary stating remediation and action items were completed.

1. 2025-06-18 Internet Security Research Group — Posted a full incident report describing the unreviewed commit tagged into a deployed Boulder release and the impact (0 certificates), plus the remediation and timeline.
2. 2025-06-19 Community commenter — Asked why issuance was not stopped and why the event was not treated as a revocation scenario.
3. 2025-06-23 Internet Security Research Group — Explained that the significance was determined quickly, that mandatory revocation circumstances did not apply, and described additional protections added for hotfix release branches and tag protection rules.
4. 2025-06-30 Internet Security Research Group — Provided an update on action items, including completion of a GitHub Actions step verifying tag commit ancestry and progress on other items.
5. 2025-07-07 Internet Security Research Group — Updated that automated release tagging support was merged and that remaining analysis and script checks were in progress.
6. 2025-07-14 Internet Security Research Group — Reported completion of all action items, including analysis of historical release tags and confirmation that no other unreviewed code was deployed.
7. 2025-07-21 Internet Security Research Group — Posted the report closure summary, stating remediation improvements and historical tag review were completed and requesting closure.
8. 2025-07-22 CCADB representative — Issued a final call for comments and noted the bug would be closed on approximately 2025-07-29.