Incident Report – Outage Sunday 7th October 2007

By on October 10, 2007 in Blacknight, Hosting, irishblogs, network

Sorry for the delay in providing this report regarding the last network issue

Date: October 7th 2007
Timeline: 02:15 – 03:35am (Irish time)
Affected Customers: Any customer on the shared firewall that has a dedicated server or has colocation with us was affected during this incident.
This also included our shared hosting clients.
What happened?
At around 2:15am on Sunday 7th of October a segment of our main network was sluggish and people would have experienced latency and packet loss.
Why?
As you may know our main network is firewalled. We have a pair of firewalls setup in HA (high availability) to protect the bulk of our clients, which includes all our shared hosting clients on both windows and linux, as well as a large number of clients on dedicated servers or with colocated machines.
Similar to the events of September 11 this year this was mostly because the firewalls we’re using have 100meg ports and as such are easily flooded by this simple attacks. We’ve already put the wheels in motion to upgrade these and we hope to announce the upgrade at the end of this week.
A brief timeline of events is shown below.
02:15: Alerted that sites on the network are not reachable.
02:20: A check reveals that any site behind the shared firewall is
not accessible.
02:30: A reboot of the firewalls is not successful in getting a
response. so an engineer is dispatched to Dublin
03:30: A check by the Engineer on site indicate that one customer
box is sending out masses of UDP traffic. The firewall is attempting to stop all the traffic at the cost of bring down everything else includingthe local console.
03:35: The customer port is disabled and the firewall becomes responsive
again.

About Michele Neylon

Known for his outspoken opinions on technology and the Internet, Michele Neylon is the award winning author of several blogs and co-host of the Technology.ie podcast. A thought leader in the Internet community, Neylon is active with ICANN and an expert on policy, security, ICANN, Nominet and Internet Governance.
View all posts by Michele Neylon →

, , , , , , ,

4 Responses to Incident Report – Outage Sunday 7th October 2007

  1. Bock the Robber October 11, 2007 at 21:29 #

    Would a customer normally be notified if this happened?

  2. Michele Neylon October 11, 2007 at 21:33 #

    If what happened?

  3. Brian Greene October 16, 2007 at 14:31 #

    have to say it was spotted and responded to very quickly.
    I was awake and witnessed it, but the team were on hand to fix. I fell asleep and woke up at 6.55 to the start of the Chinese F1 Grand Prix and all was well. now I must subscribe to this here feed.

  4. Michele Neylon October 16, 2007 at 15:59 #

    Brian – we try to be as responsive as possible, though our focus will always be primarily on the problem at hand. With regard to incident reports – I’d prefer to wait 48 hours and provide a proper one, rather than throwing something together
    Michele