Revoked DigiCert Digital Certificates: 27% Not Yet Replaced

Many Customers Apparently Still Struggling to Deploy Certificates, Researchers Say Mathew J. Schwartz (euroinfosec) • August 15, 2024

Thousands of organizations are struggling to comply with a forced, mass revocation of thousands of digital certificates issued by DigiCert.

Utah-based DigiCert, the world's largest certificate authority, on July 29 announced the forced revocation of 83,267 certificates it issued to 6,807 customers, warning they had just 24 hours to replace the certificates with new ones (see: Bad Certificate Revocation: DigiCert Offers Temporary Pause).

The company eventually revoked them all on Aug. 3, at which point the certificates became unusable for establishing secure connections with websites and services. DigiCert blacklisted the expired certificates by adding them to its revocation list, meaning any processes relying on the certificate for "authentication, encryption, or other secure connections will fail if certificate validation is enabled," Himaja Motheram, a security researcher at cybersecurity firm Censys, told Information Security Media Group.

Of the 83,267 revoked certificates, Censys said that 33,201 of them were being used to secure 455,999 distinct IPv4 IP addresses, including both physical and virtual hosts, as well as 30,528 domains and 33,201 leaf certificates, aka end-entity certificates, which are issued for servers. The majority of the revoked certificates were issued to telecommunications companies - predominantly Vodafone - and technology firms.

In a root cause analysis, DigiCert said it was forced to revoke the certificates after it discovered a bug in its internal processes, which in some cases led the company to incorrectly validate ownership for domains owned or controlled by some customers, before it issued certificates to those customers.

Censys counted 22,659 of the revoked digital certificates still in place on public-facing hosts as of Tuesday, 15 days after DigiCert first announced the pending revocation.

Customers' failure to replace 27% of the revoked certificates suggests that affected DigiCert customers "may be facing challenges" with managing their certificates, it said. Motheram said the percentage actually isn't surprising, since the bug didn't exist in DigiCert's high-volume system for issuing certificates, which is used by content delivery networks and cloud providers, but rather in its low-volume system, designed for use by customers with more configuration management requirements.

"The act of reissuing a new certificate within a short amount of time on a large number of servers can take a high level of effort. This work doesn’t even include the act of requesting a new certificate from the CA," Motheram said.

It's also possible that many of the expired certificates belong to systems no longer in production, she added.

Brian Trzupek, senior vice president of product at DigiCert, told ISMG that the company has been working closely with customers to get the revoked certificates reissued, after which customers then need to get them deployed. According to DigiCert's analysis, by last week about 98% of the revoked certificates had been reissued, which involves the customer working with the CA to generate a new one.*

Of the revoked certificates, "what we see via our tools is that fewer than 3,000 are still publicly accessible," Trzupek said, although cautioned that such analysis, due to the vagaries of mass scanning the internet, internal networks and analyzing certificate transparency logs, can be "more art more than science." Customers who have yet to deploy certificates that had been revoked and reissued may be using other workarounds in the interim, or decided to retire their systems altogether, he said. Also, he said of the revoked certificates, about 40% are used only for internal systems.*

What accounts for the discrepancy between what Censys and DigiCert are seeing? Censys told ISMG one possibility is that its numbers "include counts of virtual hosts, but without more information on DigiCert’s scanning methods, it’s hard to confirm." (This story will be updated with additional information when available.)*

DigiCert faced an immediate deadline for revoking the certificates. CA/Browser Forum rules require that erroneously issued certificates "must be revoked within 24 hours, without exception," and that "failure to comply can result in a distrust of the certificate authority." Distrust would mean that all DigiCert certificates, which the companies says secure 28 billion web connections and underpin 40 billion DNS queries, would be rendered invalid.

The need to comply with CA/Browser Forum rules and to demonstrate trustworthy behavior is no joke for certificate authorities. In June, Google announced that the Chrome browser will soon no longer trust any digital certificates issued by certificate authority Entrust, owing to what it characterized as six years of untrustworthy behavior by the CA.

"Over the past several years, publicly disclosed incident reports highlighted a pattern of concerning behaviors by Entrust that fall short of the above expectations, and has eroded confidence in their competence, reliability, and integrity as a publicly trusted CA owner," Google said in its explanation for the move.

After announcing the 24-hour deadline, DigiCert backtracked slightly, saying that after customer feedback and consulting with browser makers, it would offer a five-day extension to customers, upon request.

"We originally planned to revoke all of these certificates within 24 hours, but a few legal and critical infrastructure related concerns, as well as the scale of the number of organizations we were dealing with, caused us to take a few more days to get everything resolved and revoked safely," Timothy Hollebeek, an industry technology strategist at DigiCert, posted to the Bugzilla bug-tracking site. "Those concerns have now all been dealt with."

DigiCert said it ultimately revoked all of the certificates by Aug. 3 at 20:47 UTC. The company said it also identified 1,308 S/MIME certificates that it issued using the buggy validation method and said it would be revoking those too. Such certificates can be used to verify email senders and ensure privacy.

At least one of the legal measures Hollebeek referenced involved financial technology firm Alegeus Technologies, which on July 30 filed a complaint in Utah federal court seeking a temporary restraining order prohibiting DigiCert from invalidating its certificates.

Alegeus, a benefit funding and payment solutions SaaS provider to the healthcare sector, told the court it needed to coordinate with each one of its 300 customers to recertify affected websites and said it requested at least three days to do so, which DigiCert "refused" to allow.

"Revoking the certificates as proposed will result in significant access issues to our clients and their participants, which means that millions of plan participants and consumers will not be able to access funds reserved for and designed to pay for healthcare benefits until the access issues are resolved," Alegeus told DigiCert in emails reproduced in a court filing.

U.S. District Judge Howard C. Nielson granted the company's request on Aug. 1 "for a period of seven days, or until the court is able to schedule a hearing on the motion." On Aug. 2, attorneys for both Alegeus and DigiCert collectively petitioned the court to vacate the temporary restraining order. "The parties are pleased to report that the affected websites have had their certificates recertified," they said. The judge granted their motion on Aug. 3, revoking his temporary restraining order.

In theory, DigiCert's failure to correctly validate domain ownership before issuing digital certificates for those domains could have been exploited by attackers to falsely obtain legitimate certificates for subdomains of domains they didn't own. Use cases for such certificates might include supporting phishing attacks or espionage operations.

DigiCert said that after a review, it has found no signs of bad behavior resulting from its validation failures, which it has now fixed.

"We've had our DNS experts looking closely at whether any certificates were improperly issued due to this bug and have found no evidence that any of these certificates were issued to anyone other than the intended recipients," Hollebeek said.

"We've examined the list of affected domains and compared them to the theoretical attacks that have been suggested and found in most cases the suggested actions cannot be carried out successfully, and there's no evidence anyone even attempted to do so," he said. "If people have additional scenarios or evidence that needs to be investigated, we can do so."

*Update Aug. 17, 2024 10:52 UTC: This story has been updated with comment from DigiCert, and to note that the company ultimately provided a five-day (not three-day) revocation deadline extension to customers.