Defending Against DoS Attacks: Defense Part 1, the Network:
In
Attacks, we discussed both network-based and application-targeting Denial of Service (DoS) attacks. Given the radically different techniques between the types, it’s only logical that we use different defense strategies for each type. But be aware that aspects of both network-based and application-targeting DoS attacks are typically combined for maximum effect. So your DoS defenses need to be comprehensive, protecting against (aspects of) both types. Anti-DoS products and services you will consider defend against both. This post will focus on defending against network-based volumetric attacks.
First the obvious: you cannot just throw bandwidth at the problem. Your adversaries likely have an unbounded number of bots at their disposal and are getting smarter at using shared virtual servers and cloud instances to magnify the amount of evil bandwidth at their disposal. So you can’t just hunker down and ride it out. They likely have a bigger cannon than you can handle. You need to figure out how to deal with a massive amount of traffic and separate good traffic from bad, while maintaining availability. Find a way to dump bad traffic before it hoses you somehow without throwing the baby (legitimate application traffic) out with the bathwater.
We need to be clear about the volume we are talking about. Recent attacks have blasted upwards of 80-100gbps of network traffic at targets. Unless you run a peering point or some other network-based service, you probably don’t have that kind of inbound bandwidth. Keep in mind that even if you have big enough pipes, the weak link may be the network security devices connected to them. Successful DoS attacks frequently target network security devices and overwhelm their session management capabilities. Your huge expensive IPS might be able to handle 80gbps of traffic in ideal circumstances, but fall over due to session table overflow. Even if you could get a huge check to deploy another network security device in front of your ingress firewall to handle that much traffic, it’s probably not the right device for the job.
Before you just call up your favorite anti-DoS service provider, ISP, or content delivery network (CDN) and ask them to scrub your traffic, that approach is no silver bullet either. It’s not like you can just flip a switch and have all your traffic instantly go through a scrubbing center. Redirecting traffic incurs latency, assuming you can even communicate with the scrubbing center (remember, your pipes are overwhelmed with attack traffic). Attackers choose a mix of network and application attacks based on what’s most effective in light of your mitigations.
No, we aren’t only going to talk about more problems, but it’s important to keep everything in context. Security is not a problem you can ever
solve – it’s about figuring out how much loss you can accept. If a few hours of downtime is fine, then you can do certain things to ensure you are back up within that timeframe. If
no downtime is acceptable you will need a different approach. There are no right answers – just a series of trade-offs to manage to the availability requirements of your business, within the constraints of your funding and available expertise.
Handling network-based attacks involves mixing and matching a number of different architectural constructs, involving both customer premise devices and network-based service offerings. Many vendors and service providers can mix and match between several offerings, so we don’t have a set of vendors to consider here. But the discussion illustrates how the different defenses play together to blunt an attack.
Customer Premise-based Devices
The first category of defenses is based around a device on the customer premises. These appliances are purpose-built to deal with DoS attacks. Before you turn your nose up at the idea of installing another box to solve such a specific problem, take another look at your perimeter. There is a reason you have all sorts of different devices. The existing devices already in your perimeter aren’t particularly well-suited to dealing with DoS attacks. As we mentioned, your IPS, firewall, and load balancers aren’t designed to manage an extreme number of sessions, nor are they particularly adept at dealing with obfuscated attack traffic which looks legitimate. Nor can other devices integrate with network providers (to automatically change network routes, which we will discuss later) – or include out-of-the-box DoS mitigation rules, dashboards, or forensics, built specifically to provide the information you need to ensure availability under duress.
So a new category of DoS mitigation devices has emerged to deal with these attacks. They tend to include both optimized IPS-like rules to prevent floods and other network anomalies, and simple web application firewall capabilities which we will discuss in the next post. Additionally, we see a number of anti-DoS features such as session scalability, combined with embedded IP reputation capabilities, to discard traffic from known bots without full inspection. To understand the role of IP reputation, let’s recall how email connection management devices enabled anti-spam gateways to scale up to handle spam floods. It’s computationally expensive to fully inspect every inbound email, so dumping messages from known bad senders first enables inspection to focus on email that might be legitimate, and keeps mail flowing. The same methodology applies here.
These devices should be as close to the perimeter as possible, to get rid of the maximum amount of traffic before the attack impacts anything else. Some devices can be deployed out-of-band as well, to monitor network traffic and pinpoint attacks. Obviously monitor-and-alert mode is less useful than blocking, which helps maintain availability in real time. And of course you will want a high-availability deployment – an outage due to a failed security device is likely to be even more embarrassing than simply succumbing to a DoS.
But anti-DoS devices include their own limitations. First and foremost is the simple fact that if your pipes are overwhelmed, a device on your premises is irrelevant. Additionally, SSL attacks are increasing in frequency. It’s cheap for an army of bots to use SSL to encrypt all their attack traffic, but expensive for a network security device to terminate all SSL sessions and check all their payloads for attacks. That kind of computational cost arbitrage puts defenders in a world of hurt. Even load balancers, which are designed to terminate high SSL volumes, can face challenges dealing with SSL DoS attacks, due to session management limitations.
So an anti-DoS device needs to integrate a number of existing capabilities such as IPS, network behavioral analysis, WAF, and SSL termination, combining them with highly scalable session management to cope with DoS attacks. And all that is still not enough – you will always be limited by the amount of bandwidth coming into your site. That brings us to network services, as a compliment to premise-based devices.
Proxies & CDN
The first service option most organizations consider is a Content Delivery Network (CDN). These services enhance web site performance by strategically caching content. Depending on the nature of your site, a CDN might be able to dramatically reduce your ingress network traffic – if they can cache much of your static content. They also offer some security capabilities, especially for dealing with DoS attacks. The CDN acts as a proxy for your web site, so the provider can protect your site by using its own massive bandwidth to cope with DoS attacks for you. They have significant global networks, so even a fairly large volumetric attack shouldn’t look much different than a busy customer day – say a software company patching an operating system for a hundred million customers. Their scale enables them to cope with much larger traffic onslaughts than your much smaller pipes. Another advantage of a CDN is its ability to obscure the real IP addresses of your site, making it more difficult for attackers to target your servers. CDNs can also handle SSL termination if you allow them to store your private keys.
What’s the downside? Protecting each site individually. If one site is
not running through the CDN, attackers can find it through some simple reconnaisance and blast the vulnerable site. Even for sites running through the CDN, if attackers can find your controlling IPs they can you directly, bypassing the CDN. Then you need to mitigate the attack directly. Attackers also can randomize web page and image requests, forcing the CDN to request what it thinks is dynamic content directly from your servers over and over again. These cache misses can effectively cause the CDN to attack your servers. Obviously you want the CDN to be smart enough to detect these attacks before they melt your pipes and servers.
Also be wary of excessive bandwidth costs. At the low end of the market, CDNs charge a flat fee and just eat the bandwidth costs if a small site is attacked. But enterprise deals are a bit more involved, charging for both bandwidth and protection. A DoS attack can explode bandwidth costs, causing an “economic DoS”, and perhaps shutting down the site when the maximum threshold (by contract or credit card limit) is reached. When setting up contracts, make sure you get some kind of protection from excessive bandwidth charges in case of attack.
Anti-DoS Service Providers
CDN limitations require some organizations to consider more focused network-based anti-DoS service providers. These folks run
scrubbing centers – big data centers with lots of anti-DoS mitigation gear to process inbound floods and keep sites available. You basically flip a switch to send your traffic through the scrubbing center once you detect an attack. This switch usually controls BGP routing, so as soon as DNS updates and the network converge the scrubbing center handles all inbound traffic. On the backend you receive legitimate traffic through a direct connection – GRE tunnels to leverage the Internet, or a dedicated network link from the scrubbing center. Obviously there is latency during redirection, so keep that in mind.
But what does a scrubbing center actually do? The same type of analysis as a premise-based device. The scrubbing center manages sessions, drops traffic based on network telemetry and IP reputation, blocks application-oriented attacks, and otherwise keeps your site up and available. Most scrubbing centers have substantial anti-DoS equipment footprints, amortized across all their customers. You pay for what you need, when you need it, rather than overprovisioning your network and buying a bunch of anti-DoS equipment for whenever you are actually attacked.
Getting back to our email security analogy, think of an anti-DoS service provider like an cloud email security service. Back in the early days of spam, most organizations implemented their own email security gateways to deal with spam. When the inbound volume of email overwhelmed the gateways, organizations had to deploy more gateways and email filter hardware. This made anti-spam gateways a good business, until a few service providers started selling cloud services to deal with the issue. Just route your mail through their networks, and only good stuff would actually get delivered to your email servers. Spam flood? No problem – it’s the provider’s problem. Obviously there are differences – particularly that email filtering is full-time, while DoS filtering is on-demand during attacks.
There are, of course, issues with this type of service, aside from the inevitable latency, which causes disruption while you reroute traffic to the scrubbing center. Scrubbing centers have the same SSL requirement as CDNs: termination requires access to your private key. Depending on your security tolerance, this could be a serious problem. Many large sites have tons of certificates and can-cross sign keys for the scrubbing center, but it does complicate management of the service provider.
You will also need to spell out a process for determining when to redirect traffic. We will talk about this more when we go through the DoS defense process, but it generally involves an internal workflow to detect emerging attacks, evaluation of the situation, and then a determination to move the traffic – typically rerouting via BGP. But if your anti-DoS provider uses the same equipment as you have on-site, that might offer proprietary signaling protocols to automatically shift traffic based on thresholds. Though some network operations folks don’t enjoy letting Skynet redirect their traffic through different networks. What could possibly go wrong with that?
Selection of an anti-DoS service provider is a serious decision. We recommend a fairly formal procurement process, which enables you to understand the provider’s technical underpinnings, network architecture, available bandwidth, geographic distribution, ability to handle SSL attacks, underlying anti-DoS equipment, and support for various signaling protocols. Make sure you are comfortable with the robustness of their DNS infrastructure, because DNS is a popular target and critical to several defenses. Also pay close attention to process hand-offs, responsiveness of their support group, and their research capabilities (to track attackers and mitigate specific attacks).
The Answer: All of the Above
Ultimately your choice of network-based DoS mitigations will involves trade-offs. It is never good to over-generalize, but most organizations will be best suited by a hybrid approach, involving both a customer premise-based appliance and a contracting with a CDN or anti-DoS service provider to handle severe volumetric attacks. It is simply not cost-effective to run all your traffic through a scrubbing center constantly, and many DoS attacks target the application layer – demanding use of a customer premise device anyway.
In terms of service provider defense, many organizations can (and should) get started with a CDN. The CDN may be more attractive initially for its performance benefits, with anti-DoS and WAF capabilities as nice extras. Until you are attacked – at which point, depending on the nature of the attack, the CDN may save your proverbial bacon. If you are battling sophisticated attackers, or have a complicated and/or enterprise class infrastructure, you are likely looking at contracting with a dedicated anti-DoS service provider. Again, this will usually be a retainer-based relationship which gives you the ability to route your traffic through the scrubbing center when necessary – paying when you are under attack and sending them traffic.
All this assumes your sites reside within your data center. Cloud computing fundamentally alters the calculations, requiring different capabilities and architectures. If your apps reside in the cloud you don’t
have a customer premise where you can install devices, so you would instead consider either virtual instance, routing traffic through your site before it hits the cloud, or using a CDN for all inbound traffic. You could also architect your cloud infrastructure to provision more instances as necessary to handle traffic, but it is easy to convert a DoS attack into an economic attack as you pay to scale up in order to handle bogus traffic. There are no clear answers yet – it is still very early in the evolution of cloud computing – but it is something to factor in as your application architects keep talking about this
cloud thingy.
Next we will address the
application side of the DoS equation, before we wrap up with the DoS Defense process.
- Mike Rothman
(0)
Comments
