Incident Report – Outage Sunday 7th October 2007

Sorry for the delay in providing this report regarding the last network issue

Date: October 7th 2007
Timeline: 02:15 – 03:35am (Irish time)
Affected Customers: Any customer on the shared firewall that has a dedicated server or has colocation with us was affected during this incident.
This also included our shared hosting clients.
What happened?
At around 2:15am on Sunday 7th of October a segment of our main network was sluggish and people would have experienced latency and packet loss.
Why?
As you may know our main network is firewalled. We have a pair of firewalls setup in HA (high availability) to protect the bulk of our clients, which includes all our shared hosting clients on both windows and linux, as well as a large number of clients on dedicated servers or with colocated machines.
Similar to the events of September 11 this year this was mostly because the firewalls we’re using have 100meg ports and as such are easily flooded by this simple attacks. We’ve already put the wheels in motion to upgrade these and we hope to announce the upgrade at the end of this week.
A brief timeline of events is shown below.
02:15: Alerted that sites on the network are not reachable.
02:20: A check reveals that any site behind the shared firewall is
not accessible.
02:30: A reboot of the firewalls is not successful in getting a
response. so an engineer is dispatched to Dublin
03:30: A check by the Engineer on site indicate that one customer
box is sending out masses of UDP traffic. The firewall is attempting to stop all the traffic at the cost of bring down everything else includingthe local console.
03:35: The customer port is disabled and the firewall becomes responsive
again.