When a botched software update from security firm CrowdStrike inadvertently caused digital chaos around the world last month, the first signs were Windows computers displaying the Blue Screen of Death. As websites and services went down and people scrambled to understand what was going on, conflicting and inaccurate information was everywhere. Patrick Wardle, a longtime Mac security researcher, was quick to understand the crisis and knew there was one place he could turn to get the facts: crash reports from computers affected by the bug.
“Even though I’m not a Windows researcher, I was intrigued by what was happening and there was a dearth of information,” Wardle tells WIRED. “People were saying it was a Microsoft problem, because Windows systems were bluescreening, and there were a lot of wild theories. But it actually had nothing to do with Microsoft. So I turned to the bug reports, which to me contain the absolute truth. And if you looked there, you could identify the underlying cause long before CrowdStrike came out and said it.”
At the Black Hat security conference in Las Vegas on Thursday, Wardle argued that crash reports are an underused tool. These system snapshots give software developers and maintainers insight into potential problems with their code. And Wardle notes that they can be, in particular, a source of information about potentially exploitable vulnerabilities in software, for both defenders and attackers.
In his talk, Wardle presented several examples of vulnerabilities he found in software when the app crashed and he reviewed the report for the possible cause. Users can easily view their own crash reports on Windows, macOS, and Linux, and they are also available on Android and iOS, though they can be harder to access on mobile operating systems. Wardle notes that to gain insight from crash reports, a basic understanding of instructions written in the low-level machine code known as assembly is needed, but he emphasizes that the payoff is worth it.
At his Black Hat talk, Wardle presented multiple vulnerabilities he discovered simply by examining crash reports on his own devices, including bugs in the YARA analysis tool and in the current version of Apple’s macOS operating system. In fact, when Wardle discovered in 2018 that an iOS bug was causing apps to crash whenever they displayed the Taiwanese flag emoji, he got to the bottom of what was going on using — you guessed it — crash reports.
“We conclusively revealed that Apple had agreed to China’s demands to censor the Taiwanese flag, but its censorship code was flawed, which was ridiculous,” he says. “My friend, who was the first to notice this, said to me, ‘The Chinese are hacking my phone. Every time you text me, it crashes. Or are you hacking me? ’ And I said, ‘How rude, I wouldn’t hack you. And also, how rude, if I hacked you, I wouldn’t crash your phone. ’ So I pulled up the crash reports to see what was going on.”
Wardle stresses that if you can find so many vulnerabilities just by looking at crash reports from your own devices and those of your friends, software developers should be looking there too. Both sophisticated criminal actors and well-funded state-backed hackers are probably already getting insights from their own crash reports. Over the years, press reports have indicated that intelligence agencies like the US National Security Agency Mining crash logs. Wardle notes that crash reports are also a valuable source of information for detecting malware, as they can reveal anomalous and potentially suspicious activity. Well-known spyware broker NSO Group, for example, used to build mechanisms into its malware specifically to delete crash reports immediately after infecting a device. And the fact that malware is often buggy makes crashes more likely, and crash reports valuable to attackers as well to understand what went wrong with their code.
“In accident reports, the truth is out there,” Wardle says. “Or, I suppose, in there.”