We’re approaching the one-year anniversary of the Kaseya VSA attacks, and here at Automation Theory, we were curious about the state of security for Connectwise Automate. Many MSPs we work with are concerned about similar threats and attacks against the Automate platform. To this end, we wanted to track and provide meaningful reporting about the general state of Automate security (particularly around a similar zero-day style threat).
Methodology
In constructing this report we gathered data from the Shodan search engine. This tool is used by attackers and defenders alike and is likely to be used by any bad actors targeting the Automate platform. We used the introductory paid tier to obtain API access, which we used to export the data. We then imported the raw data into a database for additional analytics. We did not perform any direct scanning to collect any of the information for this blog.
After importing the data we performed some basic cleanup to remove duplicates and eliminate any devices that were not Automate servers. From there, we queried the data to produce the data presented below. The sections below cover various topics of interest from the defensive security perspective.
Common Connectwise Automate security concerns
Enumerability
In a typical attack, reconnaissance is the first phase, and enumeration of the target is the first step. Using Shodan falls under technique 1596 in the MTIRE ATT&CK framework , and other direct scanning methods (such as those performed by DIVD before the Kaseya attacks) fall into this category as well.
We were able to enumerate ~7500 Automate servers by searching for patterns in the default HTML of the homepage of the application. This is significant since an attacker with a vulnerability would likely use similar methods to find would-be targets. There are ways to defend against this, which we’ll discuss later.
Predictiable subdomains
The Shodan data enumerates based on public IP address. However, other enumeration tactics also leverage DNS and/or certificate creation. It’s a general industry best practice to use descriptive naming patterns, so as a result there are some very common names for Automate servers — which attackers can use for targeting. From our dataset, we filtered for on-premise Automate servers, and then pulled the top subdomains for Automate servers. Below we have the top names by percentage that are likely to be traced back to the Automate application:
In total, 29.3% of on-premise Automate servers are named after the application itself. Other common subdomains included “support”, “remote”, “monitor”, and “connect” — and more generic names such as these are ideal for preventing enumeration of the specific application.
Plain-text connections
One very striking pattern is that a majority of Automate instances are still listening on plain-text HTTP. In our initial searches, we discovered that ~89% of Automate servers are still listening on port 80. This is a concern for single-agent attacks, as a bad actor could attempt to force plain-text connections to intercept communications for the purpose of discovering credentials.
The other concern would be the ability to perform downgrade attacks as a platform for further lateral movement. As soon as the communications are in plain text from the perspective of the attacker, they can be tampered with. For example, normal agent commands could be manipulated to deploy malicious payloads or to gain footholds on the devices (create accounts, disable security layers, etc.). Unfortunately, with HTTP it’s difficult to protect against a MITM attack, and the best course of action would be to disable plain-text connections and force the use of HTTPS.
Self-signed certificates
While any encrypted channel is better than plain text; Automate by default will allow for self-signed certificates (and a variety of other certificate issues) to be accepted and used in production. This is an unfortunate security architecture choice (likely meant to avoid communication loss when certificates expire) — but as a result, 115 Automate servers are running with self-signed certificates in production.
The risks of this configuration are almost identical to those of plain-text connections; if the identity of the certificate issuer can’t be proven anyone could generate a certificate with the same name and then perform a MITM attack and decrypt the traffic. While certain methods could be used to reduce this risk, it would be far better to configure a properly signed certificate given the simplicity and low cost of certificates.
Weak SSL Ciphers
The dangers of weak SSL Ciphers are well understood; over the years cryptographic flaws have been discovered that make certain protocols almost trivial to decrypt in the correct circumstances. In our survey, we found production cipher suites as old as SSLv2 (deprecated in 2011 for cryptographic weaknesses) out in the wild. Some of these servers were obviously honeypots created by security researchers, but the majority appear to be MSPs running insecure ciphers.
This trend continues into the subsequent cipher suites, with production Automate stacks running on SSLv3, TLSv1, and TLSv1.1, all of which contain known cryptographic flaws. The risk posed by the weak encryption can vary depending on the particulars of a given environment — but disabling weak ciphers is definitely a best practice (and something increasingly scrutinized by insurance providers). The cipher hardening can be done inside of IIS, but even more granular controls can be applied using a reverse proxy.
Improperly configured reverse proxies
After SQL injection vulnerabilities were discovered in June 2020, using a reverse proxy as a security layer for Automate became increasingly popular. Various guides emerged for various technology stacks, some more popular than others. In reviewing our dataset, we were able to enumerate 10 Automate instances running behind improperly configured proxies.
A properly configured reverse proxy should return an error message when scanned by IP address (thus requiring the requestor to know what they are asking for and thus keeping the server data out of IP-based IoT scanners). However, these proxies passed the request to Automate when scanned by IP, and that resulted in them being enumerated by Shodan. Reverse proxies can be an extremely valuable security layer, but they must be properly configured.
Unpatched vulnerabilities
In the dataset gathered from Shodan, details on certain vulnerabilities are detected by the scanner. We found 10 Automate servers running with vulnerabilities detected by Shodan. We won’t speak to the particulars — but we were able to contact a majority of the MSPs and alert them to the danger, and we also passed along the information to Connectwise as well.
This highlights an additional danger with running Automate servers directly on the Internet — the web server details can be fingerprinted. When securing Automate, the whole stack must be considered — including the OS and the database. We’ve covered this topic in-depth, including a webinar recording showing attack vector examples, in our post on LAN security for Automate servers.
Fixing the holes
It’s clear that the overall state of Automate security isn’t what it should be. Here at Automation Theory, we want to help MSPs implement the security layers needed for Connectwise Automate security and to keep their clients safe.
Awareness
As with most problems, the first step is awareness. Here at Automation Theory we regularly hear that MSPs were unaware of just how enumerable the Automate application stack can be — not to mention other best practices not implemented by default. We encourage MSPs to review their external infrastructure in Shodan to determine how much surface area is visible and to also review secondary items such as TLS cipher strength and HTTP header configuration. Once any issues (or variances from security preferences) are found, a solution can be sought.
Technical solutions
The easiest solution to a majority of the common best-practice issues is to deploy a reverse proxy. A properly configured reverse proxy will:
- Prevent enumeration by responding with an error when scanned by IP
- Allow for TLS cipher suite hardening
- Redirect plain text connections (as needed)
Here at Automation Theory, we offer a reverse proxy service. Along with the above benefits, it will also:
- Leverage a randomly generated FQDN to prevent subdomain enumeration
- Remove headers used to fingerprint the underlying operating system
- Use certificate best practices to avoid certificate enumeration
- Provide custom ACLs to restrict application access
- Apply GeoIP rules and perform IPS scanning
Beyond proxying the web traffic, other technical controls such as server hardening and regular patching of components (MySQL, database connectors, etc.) are also important steps to take (especially since most Automate servers don’t receive any MySQL patches post-implementation).
We hope you’ve found this post helpful and informative. If you are looking for assistance with Connectwise Automate security, details of our services can be found below:
