The expression "it never rains but it pours" springs to mind today ....

Earlier today we had a serious network issue and as soon as that was resolved we discovered a glitch in our online credit card processing.

I'm not a programmer, so I won't even pretend to understand exactly what happened, but I do know what the consequences were - no online CC processing for a few hours!

Fortunately our technical team were able to find the problem and fix it, so credit card processing is back in operation.

Sorry about the hiccups today !!

Posted by Michele Neylon at 4:53 PM | Permalink | Comments (0) | TrackBacks (0)

As some of you know there were connectivity issues earlier this morning.

At this juncture all service should be restored, however if anyone is experiencing issues still please let us know

We'll provide more details on what happened as soon as one of the technical team provides a report

Posted by Michele Neylon at 12:32 PM | Permalink | Comments (3) | TrackBacks (0)

Summary:

@ 10:00am on Monday 31st December we experienced a DOS in the form of a
compromised host on our network in InterXion. This issue caused serious packet loss
for us on our core network in DEG. It also caused dns issues because the dns
servers couldn't communicate with the mysql cluster that hosts this data.

There are 3 main causes for this issue.
1) Compromised host in InterXion flooded the core network with 100Megabits of traffic
2) Legacy dns scripts caused the include file for bind from our dns management
system to become empty and for dns to stop working.
3) The router upgrade last week left 1 link (which happens to carry all outbound traffic) to be auto negotiated at 100M instead of the usual 1000M.

The above 3 issues compounded each other and caused the 2 hour outage we
experienced this morning.

* To remedy the issue we've checked all core router -> core switch ports to ensure
they are all running at the correct speed.

* We're examining the scripts that write out dns changes to ensure they don't break
the bind includes if they are unable to contact the mysql cluster that stores this
information.

* We're going to get an OOB(out of band) network access solution for all core
equipment including all switches in each location where we have core network. This
will allow us to access equipment from another non Blacknight network in the
event we loose connectivity or we're having an issue such as we had today.

* We'll also setup blacknightstatus.com which we'll host elsewhere and ensure
that this is kept up2date with info as it becomes available to us regarding outages
and issues. We've had this in the pipe line for sometime but we didn't feel it was
necessary until now.

Further updates with fault diagnosis will follow once we return back to full staffing
levels on Wednesday the 2nd of Jan 2008.

We apologise for this issue and the problems it has caused our customers.

Posted by Paul Kelly at 12:21 PM | Permalink | Comments (0) | TrackBacks (0)

Hi Folks,

Looks like the hardware for Morgana has become faulty in some way. We've tried some remote diagnostics with the guys on site. However they are unable to determine the issue. Our on-call Engineer is currently en route to the Data Centre to perform fault diagnosis.

We've spare hardware on site that we can swap the drives into if that is required. Time to fix will be approx 2-4 hours from the time of this blog post.

The on-call engineer will post updates when he has them.

Posted by Paul Kelly at 9:11 PM | Permalink | Comments (1) | TrackBacks (0)

We're going to maintain the price of .eu registrations and transfers @ 5 euro for the foreseeable future ie. we've no intention of upping the price.

We are hoping that Eurid (the entity that is in charge of .eu namespace) will change some of its procedures and policies with regard to renewals, as the present regime does not benefit either registrars or registrants.

Posted by Michele Neylon at 3:14 PM | Permalink | Comments (2) | TrackBacks (0)

When: Thursday 27th Of December 2007 at 22:00

Duration: 2 Hours

What: Upgrade of two core routers within the Blacknight network. Each
core router will be swapped out one at a time. We do not anticipate any
downtime of network connectivity for customers during this maintenance
window, although customers will see routing path changes and/or brief
latency hits as traffic is rerouted.

Update: Friday Dec 28 @ 11:00am

This work was carried out successfully and without interruption to customers traffic. We'll be scheduling one more maintenance window for the core network for sometime in Jan to swap out another one of the older routers for one of our new Juniper routers.

Posted by Niall Donegan at 4:59 PM | Permalink | Comments (0) | TrackBacks (0)

When: Tuesday-Wednesday 18th/19th December 2007 @ 23:45 hours

What:

Firewall issues. We've been experiencing problems with the HA pair of firewalls that
look after the shared hosting network. Config changes are particularly problematic
and we've had to stop all changes.

Tonight we're going to move shared windows and linux hosting off of the current
firewall onto another set of firewalls. This is completely temporary as we wish to
investigate the issues we're having with the Fortigates without affecting service.

Who will be affected:

All shared hosting customers. We estimate 5-10 minutes per shared hosting
customergroup of down time.

So at approx 23:50 all shared hosting machines will dissappear from the network.
Once they have been all verified as "UP" from inside and outside we'll switch to
the windows machines.

At approx 00:10 or shortly after we'll started the shared windows hosting machines.

The above all requires moving the machines to seperate vlans also, this will take
the bulk of the time.

Summary:

Starting from 23:45 tonight the shared hosting network will disappear and should
be back at the latest 00:50.

As usual, please contact us via e-mail to support@blacknight.com if there are any
issues once the proposed time period has passed.

We do apologise for the late notification on this work, but due to staffing constraints on the run up to Christmas this is the only evening we can perform this task.

Happy Christmas to all!

Posted by Paul Kelly at 11:13 PM | Permalink | Comments (0) | TrackBacks (0)

Rather than leave it to the last minute I thought I'd take a moment today to give a quick outline of our Christmas 2007 opening hours.

Office Opening Hours:

Friday 21st December - open
Monday 24th December - closed
Tuesday 25th December - closed
Wednesday 26th December - closed
Thursday 27th December - open
Friday 28th December - open
Monday 31st December - closed
Tuesday 1st January 2008 - closed
Wednesday 2nd January 2008 - open. Normal operations resume

Accounts

As above

Technical Support

Our support desk is accessible 24/7/365 and we will have oncall staff throughout the holiday period to deal with any emergency issues.
If you have a dedicated server or colo you should have the "out of hours" number to call. If you don't have the number or have misplaced it please let us know as soon as possible.

IE Domain Updates

The IE Domain Registry will be closed from Friday 21st until January 2nd, so no dns updates will be processed during the Christmas period. If you need any changes made then you need to get them done by close of business on Thursday 20th.

Other Domains

Most other domain registries are automated, so changes and requests maybe processed.

If you have any queries please let us know

Finally, on behalf of all the staff at Blacknight, I would like to take this opportunity to wish you all a peaceful and pleasant Christmas break.

Posted by Michele Neylon at 2:17 PM | Permalink | Comments (0) | TrackBacks (0)

A couple of months ago (time flies!) I was asking people both here and on LinkedIn for product name suggestions.

I was expecting some feedback, but I was totally unprepared for the sheer volume of replies I got.

Wow!

Thanks for taking the time to exercise your grey matter and help us out!

Some of the suggestions were good, some were amusing and some were, well, em, a little, mad!

In the end we'll be opting for something that is hopefully simple enough for people to "get" without being too restrictive:

minimus, medius, maximus

Thanks for the input!

Posted by Michele Neylon at 3:25 PM | Permalink

Following up from my earlier post....

Here are some of our staff in their Christmas attire, which goes "nicely" with the one with myself and Paul.

Posted by Michele Neylon at 4:03 PM | Permalink | Comments (2)

It seems like every ecommerce site I've ever bought from is doing something special in the runup to Christmas.

Here at Blacknight we've spruced up the offices a bit - no singing Santas, but a few decorations here and there. I think there might even be a miniature tree floating around somewhere!

And of course tonight's the annual party!

Over the past few years the numbers have gone up and up as we've expanded. Hopefully the number of embarrassing photos that end up on Flickr won't!

Posted by Michele Neylon at 1:51 PM | Permalink | Comments (4)

Hi All,

ns2.blacknightsolutions.com and all its various alias name servers is currently down. We're working with our Data Centre partner in Germany to find out what is wrong. We'll have further information to post in relation to this shortly.

Update: This issue has been resolved

Posted by Paul Kelly at 1:23 PM | Permalink

A lot of our clients share a common trait.

They are confident that their idea will succeed.

It's great to see so much optimism, but people shouldn't take themselves too seriously!

Have a look at this (pump up the volume if you can!):

With thanks to Jimmy for sharing the link!

Edit: Pointed to update YouTube link / version.

Full credits available here

Posted by Michele Neylon at 1:38 PM | Permalink | Comments (1)

Next Monday, 10th December, Mobile Monday will be taking place in downtown Dublin.

James Pearce, dotMobi's CTO posted the following to the Mobile Monday group:

When: Monday 10th December, 6:30 for 7:00
What: Bob Last and Chris Moisan, Taptu
Where: The Porterhouse, 16-18 Parliament Street
http://www.porterhousebrewco.com/templebar.html

We're thrilled to be able to hear about mobile search from Taptu
(http://taptu.com/ and http://taptu.com/blog/).

Taptu has been hot news in mobile this year, and the service has had
rave reviews in such illustrious places as the Scoble Show, WAP Review,
and Mobile2.0. Now they're coming to a pub near you!

So... if you thought that the mobile version of the search game was
wrapped up already, come and hear Bob and Chris dispel a few
assumptions. Mobile's a little different, right?

If you're interested in attending let them know so they can gauge numbers

Posted by Michele Neylon at 12:48 PM | Permalink

There have been some issues with the connection into our second location this lunchtime.

We will post a more detailed explanation as soon as we have one.

Update: (13:57)

After investigating the issue with our Data Centre and Connectivity partners we've narrowed down the issue to being a faulty patch panel and/or patch leads.

To minimise further downtime in the next 24-48 hours we're going to hold off debugging this issue further. We'll schedule a network maintenance window in the coming days so we can test both links to DEG from InterXion fully.

The lunch time fault today occurred because both metro connections dropped at the same time as InterXion were carrying out cabling work in their patch room. We're working with InterXion to get a resolution on this, but as I said above we want to minimise downtime for our customers by scheduling a window in 24-48 hours to solve this issue fully.

If you have any queries about this please contact us via phone/email/forum or leave a comment on this post.

Posted by Michele Neylon at 1:10 PM | Permalink

Hi All,

@ 1:50am we had all servers back online. There were some complications that were not foreseen by us which were over looked by InterXion. This delayed us somewhat getting the last rack backup and on the network.

There are some other customer machines that did not come back up correctly. We'll liase with you regarding these tomorrow morning.

Posted by Paul Kelly at 3:29 AM | Permalink