Online blackmailers repeatedly succeed in paralyzing the digital infrastructure of entire districts and sending authorities into the analogue Stone Age, most recently recognizable by the lack of corona reporting numbers for the district of Ludwigslust-Parchim, for example in the Corona warning app. We were interested in whether the authorities have learned from this and what the general situation is with the IT security of servers in the public sector. At the beginning of November, we searched 59,000 web servers in counties, federal authorities, and cities across Germany for Exchange web apps and tested them for a known, critical vulnerability.
From this, our test script filtered 460 publicly accessible Exchange server web apps, which can be recognized by the path /owa/, and analyzed them, according to which 20 servers were vulnerable. Among the Exchange servers found are a theatre, an adult education center and several city administrations and districts. Four server operators had not yet run a patch against this at the time of going to press and therefore do not appear in our list. In each individual case, attackers could easily have taken control of the vulnerable mail server and, in the worst case, even the entire network.
The Microsoft Exchange servers were vulnerable via the so-called “proxyshell vulnerability”. . Microsoft has been providing security updates for this problem since April 2021. The situation came to a head at the beginning of August, as the updates had not yet been installed in many places: Heise Security and the Federal Office for Information Security (BSI) warned in August of an ongoing wave of attacks, in the context of which criminals were gaining access to the unpatched servers and blackmailed the operators with ransomware. So the danger is very real. But even three months after the urgent warning, there were still Exchange servers in public administration without the latest security updates, as the current c’t investigation shows.
Strictly speaking, this is not a single vulnerability, but rather three separate vulnerabilities that combine to allow an attacker to take over the Exchange server. To find out if the servers are vulnerable, we tested the first vulnerability CVE-2021-34473, i.e. “Access to backend URLs”, which showed us the version of the Exchange server. We first generated a list with the domains of all federal states, over 200 administrative districts, several federal authorities and over 2,000 cities. From this list, we filtered the active Outlook Web Apps (OWA) to review only the relevant servers. We then called up a test URL on the OWA servers to find out whether the important security updates were already installed and saved relevant metadata such as URL, version number, user and mailbox. To our surprise, 20 servers were still vulnerable to Proxyshell.
A status page with service account (user) and mailbox is displayed for vulnerable Exchange servers.
Test your own server
If you run an Exchange server yourself, you can easily check whether it is vulnerable. It is sufficient to send a GET request to the mail server’s Outlook web app, for example by calling up a URL in the browser:
Replace yourdomain.de with your domain name, but leave @foo.com – this must be a third-party domain for the test to be meaningful. In a vulnerable system, the Exchange server spits out a page with HTTP status code 200 and the title “Exchange MAPI/HTTP Connectivity Endpoint”. This shows the running Exchange version, the service account (NT AuthoritySystem, depending on the system language) and the mailbox name. This page should not be reachable at all because it belongs to the Exchange backend and is only accessible via the vulnerability. We even recommend calling the URL multiple times over a longer period of time, because we noticed that even with current Exchange servers, the backend page was only accessible at certain times.
An attacker would use the same URL, but send a POST request with attack code (exploit) to the server. The appropriate exploit code has been public since at least August and is so easy to use that a burglar only has to choose a backdoor to upload to the Exchange server. There are now also scripts that carry out the attack automatically. This allows an intruder to execute any commands on the server at any later time. c’t informed the operators themselves or reported the cases to the BSI. Most responded quickly when contacted and updated their servers.
Response from the uninvited visitor
A few weeks after we emailed the operator of an Exchange server with an open vulnerability, we received an unusual response: “Hello! Please ensure that all documentation can be found via the next link.” – the link led to a website that had nothing to do with the municipality concerned or us. The e-mail itself did not come from the city administration, but was sent via a hacked WordPress website, with the city only in the sender name. As a preceding conversation, the email quoted our warning about the vulnerable Exchange server. As the city administration informed us, our e-mail was copied when the mail server was infected. We can imagine the vulnerability that caused this to happen. We are working to ensure that this does not happen again in the future.
Already hacked, right?
If you haven’t secured your server quickly, you now face a problem: Has the server already been hacked? Just looking at the IIS logs and looking for suspicious HTTP requests is not enough. Admins should at least take a look at the directories “C:inetpubwwwrootaspnet_client” and “C:Program FilesMicrosoftExchange ServerV15FrontEndHttpProxyowaauth”, because these are known ones Places where attackers place their backdoor.
But even if there are no unusual files to be found here, that does not necessarily mean that the server was spared: if an attacker with system rights was on the server, he could and would have covered his tracks. In the worst case, he has even taken over other computers in the network.
With a three-month-old critical vulnerability that wasn’t patched in time, expect the worst. Searching for a potential intruder is expensive, so prompt maintenance of servers would be a better strategy in the future. Install security updates quickly, without waiting for the BSI to warn of a large wave of attacks. This applies in particular to publicly accessible servers.
At the end of November, the BSI’s computer emergency team again warned on Twitter of mail servers with critical vulnerabilities.
There are still a number of vulnerable Exchange servers outside of public administration. Overall, the situation is highly critical: the BSI warned via Twitter on November 30 that around 12,000 servers in Germany were affected by at least one critical gap – that corresponds to 30 percent of the servers known to the BSI. The BSI gets information about the patch status from the computer search engine Shodan. The fact that Exchange maintenance is sometimes forgotten is apparently not an exclusive problem for public administration.
No happy ending
Our research shows that the public sector sometimes cannot keep up with closing critical gaps quickly. Even if it has been patched, the situation in government IT often does not look rosy. The ransomware cases in public institutions are piling up: The district of Anhalt-Bitterfeld was particularly badly hit in the summer, and even had to declare a disaster in order to get support from the German armed forces. The IT infrastructure had to be rebuilt for weeks. The door to the district administration remained locked. The citizens could no longer deal with the authorities there and the administration could not even assign social assistance. Such attacks are increasingly taking place via Trojans in an email. With the unpatched Exchange servers, not even a thoughtless click on an email attachment is necessary: the barn door is wide open there.
Wilhelm Drehling and Dennis Schirrmacher, Microsoft’s late awakening, The Exchange vulnerabilities in detail, c’t 8/2021, p. 12