We had the pleasure of being a vendor and speaking at MSPGeekCon23 — and it was a fantastic conference! We presented our talk, “Defending the MSP tool stack in a zero-day world,” and converted it into a blog format below.
Setting the stage: the zero-day world
Zero-day attacks are increasing, and it’s all fun and games (and chaos) until it’s an MSP tool vulnerability. Since MSPs are the gatekeepers to a massive number of downstream networks, the stakes increase exponentially for both attackers and defenders in this sector. MSP tools have problem security behaviors, plus the guidance on securing them hasn’t changed with the attack trends — leading to a desperate need for change in the industry.
In 2005, security researcher Marcus Ranum published a paper titled “The Six Dumbest Ideas in Computer Security.” Specific points have been debated, but the first item on his list was “Default Permit”:
This dumb idea crops up in a lot of different forms; it’s incredibly persistent and difficult to eradicate. Why? Because it’s so attractive. Systems based on “Default Permit” are the computer security equivalent of empty calories: tasty, yet fattening….This put the security practitioner in an endless arms-race with the hackers. Suppose a new vulnerability is found in a service that is not blocked – now the administrators need to decide whether to deny it or not, hopefully, before they got hacked.
Marcus Ranum, 2005 “The Six Dumbest Ideas in Computer Security”
MSP tools typically are default permit applications, as we’ve written about in blog posts like Is your Automate server open to the world? and State of Connectwise Automate Security 2022. This default permit behavior makes it easy to index MSP tools in IoT scanners like Shodan — which would make it trivially easy for an attacker to launch mass attacks.
The outdated guidance for MSPs is also problematic. Even in the last year, other RMM consulting firms have led the security conversation with patching, authentication, OS hardening, and application configuration. While we are proponents of all of that (please patch your systems and enable MFA!), let’s take a look at the CVEs from the Kaseya attack:
| CVE | Description |
| CVE-2021-30118 | Remote Code Execution Vulnerability |
| CVE-2021-30117 | SQL Injection Vulnerability |
| CVE-2021-30121 | Local File Inclusion Vulnerability |
| CVE-2021-30201 | XML External Entity Vulnerability |
| CVE-2021-30116 | Credentials Leak and Business Logic Vulnerability |
| CVE-2021-30119 | Cross-Site Scripting Vulnerability |
| CVE-2021-30120 | Two-Factor Authentication Bypass Vulnerability |
Taking a look at the list, note a couple of items:
- Credentials were stolen
- MFA was bypassed
- The OS was not part of the equation
Those non-application items are essential to the overall security picture. However, MSPs could have followed all of those best practices to a T and still be exploited because of the vulnerability in the application itself. The days of brute-force credential attacks will soon be behind us, and the trend of software attacks will continue to increase (if the other zero-day trends hold true).
Defending the MSP tool stack
In defending the MSP tool stack, we want to create an architecture that is resistant to these attacks (not just spot-fixing an issue). In doing this, we have two needs emerging:
- Armor: we need defenses against application layer attacks like injections, inclusions, XSS, etc.
- Camouflage: MSP tools are easy to find and, therefore, easy to attack
The easiest way to demonstrate the idea behind this architecture is an army tank. It has offensive and defensive security measures, which might be considered secure. However, militaries worldwide take one more step: they paint them in colors that don’t stand out. Consider the two images below — which one is more attack resistant?
While one brave soul at MSPGeekCon was up for the adventure of a pink army tank, it becomes evident in this context that the different layers in concert provide the total net security protection — and any one layer out of place can reduce the full effectiveness of the other controls.
Here’s how we’d construct something like this:
We start with DNS and certificates. While wildcard configurations are often used for convenience and scalability, they offer another advantage: they are resistant to enumeration. Attackers can find DNS and certificate records — but if they only point to the reverse proxy, the applications behind the proxy are still unknown to the attacker (and it’s impossible to weaponize a payload or deliver an attack without a known target).
The reverse proxy acts as our camouflage (and it has some armor components to it). We can thwart IoT scanners at the proxy layer, keeping prying eyes away from the MSP tool stack. We can also implement a variety of other controls at this layer, such as GeoIP, IP whitelisting, HTTP header and TLS hardening, and a variety of application layer ACLs (like restricting access by IP except for a specific URL parameter so clients can jump over the IP restrictions to get to a join-session page).
The WAF is the armor component in the diagram. The proxy sends requests to it, which are scanned to determine if they are malicious (being checked for injection/inclusion/XSS/etc. attacks). The WAF replies to the proxy — and from there, the proxy will either block or allow the request. Even if the camouflage of the proxy is circumvented, and an attacker can maneuver around the other security controls, the deep inspection of inbound requests will still prove to be a robust last line of defense.
We hope this information has been useful! If you’re interested in seeing our reverse proxy service and WAF in action, you can request a trial with the form below:
