We informed you some time ago of an upcoming upgrade to our billing / control panel system.

You will probably receive an email from us later today regarding our billing system.

The text of the email is below:

Dear XXX

As you may know we are currently deploying a new billing and control panel system.

In order to facilitate the migration we would appreciate your help.

Please follow the link below to create your new control panel login.

NB: We will not ask you for any personal information at any time.

To create your new control panel login, please visit

Should you encounter difficulties please contact support@blacknight.com

--
Regards
Team Blacknight
http://www.blacknight.com

Once you get this email you will be able to setup your new login for our new billing system which will be going live very soon.
If you have any queries or issues please let us know.

NB: This change affects ALL Blacknight clients

Posted by Michele Neylon at 10:40 AM | Permalink | Comments (0) | TrackBacks (0)

If you want to know about any service affecting maintenance, technical updates or anything else of a technical nature, we recommend that you check out our new Technical Blog.

The site is hosted outside our core network (we don't even use our own nameservers just to be 100% safe!) and is part of our backup / contingency plans for emergency situations.

While our network uptime has been and hopefully will continue to be exemplary there's no reason to be lazy. We need to make sure that we have a system in place in case there is an issue NOT after the issue arises.

You can subscribe to the site's RSS feed OR to the email alerts.

Your choice :)

You can sign up for the email alerts by filling out the form below:

Posted by Michele Neylon at 12:14 PM | Permalink | Comments (0) | TrackBacks (0)

When: Tonight Tue March 18th @ 21:00 hours

What: Disk replacement in shared server bors.blacknight.ie, the spare in the Raid 1 mirror failed last night so we're going to swap it out tonight. This machine does not have hot swap drives so we need to take it down for the change over.

This will take between 15 minutes and 1 hour to complete, please also note that performance will be degraded today and tonight post the swap out until the raid array returns to normal.

Summary: @ 21:00 tonight Tue March 18th the server hosting server bors will be taken offline for between 15 minutes and 1 hour while we replace a dead drive.

Update @ 21:48:

The disk has been replaced, the raid array is now rebuilding and the server is backup. It'll be a little slow for a couple of horus until the array is completely rebuilt.

Posted by Paul Kelly at 11:03 AM | Permalink | Comments (0) | TrackBacks (0)

If you're not an existing Blacknight client this post isn't going to be that interesting or relevant to you, so you might want to move along...

The short version of this post:

We'll be emailing you to get you to choose a new username for the new control panel and billing system. The email will be clearly identified as coming from us and we won't be asking you for any sensitive information (ie. it won't be a phish!)

And for those with more patience ...

As some of you know we are currently working on a new shiny hosting system which is going to make your lives better and cure the world's ills (marketing speak gone mad! I know!).

If you can get past the marketing blurb one important fact remains. We will be moving away from Modernbill before too long. While it has served us well over the last few years our relationship has had its ups and downs, so now it only seems fair that we part our ways.

Now we know that migrating people to new systems can be disruptive and we also know how annoying it is when you're forced to use an insanely complicated username and password pair.

So our little elves (forgive the terrible metaphors!) have devised simple solution.

If you are an existing client of ours you will be getting an email at some point in the next few weeks. Within that email will be the basic details you need to choose your new username for the new billing system. It will all be quite painless we hope (and pray!).

In any case if you have any questions do let us know.

When is this going to happen?

I can't give a fixed date just yet, as we're still ironing out some issues and I hate making promises that I can't deliver on.

Posted by Michele Neylon at 4:03 PM | Permalink | Comments (0) | TrackBacks (0)

When: INEX LAN#1 connection being upgraded @ 23:00 on Monday 10th of March.

What: We currently have 2 x 100M connections to INEX. Our LAN#1 connection carriers a lot of our INEX traffic and as such we're upgrading it to 1000M to prevent it being a bottle neck for traffic originated in Ireland. This is a simple software configuration change for the port speed on one of our routers and requires INEX operations staff to do the same on their end.

There'll be a brief hit as all our INEX peerings on LAN#1 go down and traffic re-routes over LAN#2 and transit. This should only be temporary and peerings should re-establish automatically after a few minutes.

Summary: @ 23:00 on March 10th we're upgrading our primary Connection to INEX to 1000M. Traffic reaching us via INEX peerings on LAN#1 should be re-routed via LAN#2 and transit with minimal downtime being incurred, just the time it takes for BGP to reconverge.

Update: 23:25 March 10th 2008

This upgrade went ahead without a hitch. We're now running at GE @ INEX on Lan#1. We'll upgrade Lan#2 later in the year as necessary.

Posted by Paul Kelly at 5:04 PM | Permalink | Comments (2) | TrackBacks (0)

When: Monday 10th of March @ 22:00 hours

What: Firstly we've recently lit our own protected wavelength between DEG and InterXion. It has been in place and in testing for a few weeks now. We need to test the failover on both the long and short legs of this new connectivity and also check the failover to the backup layer 2 paths in the event of both the short and long legs getting damaged at the same time.

We don't expect any downtime during this testing as our layer 2 network normally fails over within a few milliseconds.

Secondly we're moving the InterXion firewalls to the new Distribution routers in this location. This change should take 30 seconds or so to propagate within our network as it's a logical Layer 3 change.

This will mean the firewalled network in DEG and InterXion will be seperated from each other and traffic originating and destined for each data centre doesn't need to traverse our metro network.

For complete testing we'll allocate 2 hours to perform these tests. We don't envisage anything more than a few 10-30 seconds hits on metro traffic (so won't affect everyone) and it will only cause slow loading times for some websites and not others.

Summary: Works begin @ 22:00 hours on Monday 10th and end at midnight on Monday 10th. There should only be a few short hits on our metro links as they failover while we simulate fibre cuts, switch failures, port failures etc.

Update: 23:58 March 10th:

These tests have been completed. The inter DC links have been tested in several scenarios are we're happy it's quite resilient now. We also moved the InterXion firewalls to a new distribution router pair in InterXion from the DEG routers. This took a little longer than expected, around 3 minutes and 40 seconds or so, slight OSPF glitch in the config which took a minute or two to find. All went to plan except that firewall move in the initial stages.

Posted by Paul Kelly at 12:07 PM | Permalink | Comments (0) | TrackBacks (0)

Summary: Kay.blacknight.ie (one of the windows shared hosting servers) is going to be moved to new hardware. The highlights are:

a) More memory
b) 64bit Windows 2003 server
c) Faster disks
d) Newer version of Smartermail

Time lines:

@22:00 on the 27th of Feb 2008 we'll be shutting down the smartermail service.

Once we're sure the service is stopped and all end users disconnected we'll be syncing the config, mailboxes and spools to the new server. Once we've moved everything we'll do some extensive testing to make sure all is well.

We estimate around 3 hours or so for the change over to take place, if it's looking like it'll take longer, we'll abort the move but keep syncing the data.

I'll post more info tomorrow when I know more. Mark one of our Windows engineers will be on-site for the duration of the change over.

Please address any queries regarding this to the support desk http://support.blacknight.ie, support@blacknight.ie

Update @ 7:30am Thursday 28th:

Mark aborted the transfer during the night as it was taking far too long. We're re-scheduling this for this weekend coming and we'll try an alternative means of syncing the data. Can I urge people to download all their e-mail and not leave it on the server for the next few days as this will help us migrate the data faster.

Update @ 14:30 Friday Feb 29th:

We've been investigating this further over the last few days and we've found a faster method to complete the transfer and replicate the data. The tests we've run indicate between 1 and 2 hours to sync the changes since the major file copy on Wednesday last. So starting at 22:00 tonight the mail server kay.blacknight.ie will be unavailable for 2-3 hours to complete this migration. I'll update the blog on Saturday morning with more information.

Update @ 23:49 Friday Feb 29th:

We have completed this move. There is a noticeable performance increase which can be attributed to new software, 64 bit OS and Mail services and better hardware.

Changes:

* New Webmail interface with a built in RSS reader
* New Antispam measures in the back end
* ClamAV now installed by default
* Better attack mitigation features
* Improved accessibility built into the webmail interface itself (Daragh please comment on how good/bad Smartermail got it this time)

Posted by Paul Kelly at 8:52 PM | Permalink | Comments (2) | TrackBacks (0)

We are currently experiencing issues with the server known as "Balin".

Our technical team are working on a fix and we will update you as soon as possible

Update: the server has been rebooted. Mail and most other services have been restored. Web traffic is currently disabled

Update 2 - Web traffic is now enabled.

Posted by Michele Neylon at 1:33 PM | Permalink | Comments (2) | TrackBacks (0)

When: Thursday 14:30 in the afternoon.

Summary: There is a software upgrade due for a while for the vendor we use. So we're going to do it during the day when the system is at its quietest.

We estimate between 30 minutes and 2 hours for the complete upgrade and testing phase. The backup system won't be available during this upgrade.

If you have any questions please address them to our support team. There may be client upgrades and new features available after this upgrade so we'll post about these once we've tested them all fully.

Posted by Paul Kelly at 9:49 AM | Permalink | Comments (1) | TrackBacks (0)

There's a "lovely" aroma of burning in our offices in Carlow this lunchtime.

I say "lovely" as I'm trying really hard to find a silver lining to this morning's electrical "mishap".

I am not an electrician and I would never pretend to be one. Neither it seems was the person or persons responsible for this morning's problems!

They may advertise themselves as electricians, but I'd love to know how someone competent could have unbalanced a 3 phase so easily.

It seems that while doing some work on the other half of the building ie. outside Blacknight territory, they managed to disconnect one of the phases which caused an imbalance and a rather nasty power spike.

The surge took out several of the lights in our offices, three desktop PCs (the power supplies are literally burntout - you can actually see black smoke marks), the phone system blew and more than half of the headset base stations melted.

On a positive note the UPS worked fine and kept the office servers up and running and protected them from the nasty spike.

The guys down in PC Pro in Cork will be happy, as we've had to replace a load of office equipment..
I'm still compiling a list of stuff that got damaged, but the bill will run to several thousand euro.

There isn't a whole lot you can do in a normal office building when crazy people with screwdrivers get involved, so I'm thankful that we don't have any customer equipment onsite at present. The data centres we use in Dublin are incredibly redundant!

Posted by Michele Neylon at 12:44 PM | Permalink | Comments (2) | TrackBacks (0)

Hi All,

Up until recently we've left register_globals on, for the sake of cutting down on support issues with particularly iffy php code writing styles.

Today we're officially announcing that we are disabling register_globals across the board now on all our shared hosting servers.

The simple fix for this if it affects you is to enter the following string into as a .htaccess file

php_value "register_globals" "1"

We've turned it off on a few machines already, but we'll be doing the rest as the day goes on. So please wait and see if your site is affected before effecting the fix.

Posted by Paul Kelly at 12:07 PM | Permalink | Comments (0) | TrackBacks (0)

Issue: We're seeing errors regarding a bad PSU in a switch in DEG. We're not going to take any chances leaving it as is, so we're going to RMA the switch.

Outage Window: For customers on the affected switch, an e-mail will be sent notifying you of this downtime window.

Duration: 5-15 minutes total for the window. Each customer should only notice a small blip of around 10 seconds as they are re-patched to the new switch.

If you notice any problems please e-mail support or call the on-call number.

Update:

This will affect the following blacknight services for the duration of the move. Our primary mail relay host for dedicated customers is in this rack and will be affected by the outage. There are some other non critical services that will be affected also.

Posted by Paul Kelly at 10:59 AM | Permalink | Comments (0) | TrackBacks (0)

When: Starting Wednesday 9th @ 22:00 and ending Thursday 10th @ 01:00

What: Migration of Dedicated, Colocation and IP transit customers
to new Juniper network layer.

In December we bought a bunch new of Juniper routers to upgrade
our core network with. The ones that were there, were almost 2 years
old and were due an upgrade.

We'll have the new Juniper router pair pre-configured with all prefixes
and BGP sessions. We'll slot it into place and clear the arp cache
on all affected layer 2 devices and shut down the old device. There will
be approx 10-30 minutes where routes to certain parts of our network
are unavailable.

This will also remove the need for our old IPv6 configuration. We'll now
have end to end native IPv6 core running on the Juniper platform. We're
the first hosting company in Ireland to build a native IPv4, IPv6 network
core on the Juniper platform and we're very proud of this fact.

Who will be affected:

Customers on our unfirewalled network (who have their own routers or
firewalls) or IP Transit customers.

This affects both customer groups in InterXion and DEG locations. If you
are unsure if this affects you or not, give us a call or drop an e-mail
into support@blacknight.com

Summary:

On Wednesday 9th starting @ 22:00 hours we'll be performing maintenance
on the routers that run our un-firewalled and IP Transit networks.

Posted by Paul Kelly at 10:27 AM | Permalink | Comments (3) | TrackBacks (0)

Summary:

@ 10:00am on Monday 31st December we experienced a DOS in the form of a
compromised host on our network in InterXion. This issue caused serious packet loss
for us on our core network in DEG. It also caused dns issues because the dns
servers couldn't communicate with the mysql cluster that hosts this data.

There are 3 main causes for this issue.
1) Compromised host in InterXion flooded the core network with 100Megabits of traffic
2) Legacy dns scripts caused the include file for bind from our dns management
system to become empty and for dns to stop working.
3) The router upgrade last week left 1 link (which happens to carry all outbound traffic) to be auto negotiated at 100M instead of the usual 1000M.

The above 3 issues compounded each other and caused the 2 hour outage we
experienced this morning.

* To remedy the issue we've checked all core router -> core switch ports to ensure
they are all running at the correct speed.

* We're examining the scripts that write out dns changes to ensure they don't break
the bind includes if they are unable to contact the mysql cluster that stores this
information.

* We're going to get an OOB(out of band) network access solution for all core
equipment including all switches in each location where we have core network. This
will allow us to access equipment from another non Blacknight network in the
event we loose connectivity or we're having an issue such as we had today.

* We'll also setup blacknightstatus.com which we'll host elsewhere and ensure
that this is kept up2date with info as it becomes available to us regarding outages
and issues. We've had this in the pipe line for sometime but we didn't feel it was
necessary until now.

Further updates with fault diagnosis will follow once we return back to full staffing
levels on Wednesday the 2nd of Jan 2008.

We apologise for this issue and the problems it has caused our customers.

Posted by Paul Kelly at 12:21 PM | Permalink | Comments (0) | TrackBacks (0)

Hi Folks,

Looks like the hardware for Morgana has become faulty in some way. We've tried some remote diagnostics with the guys on site. However they are unable to determine the issue. Our on-call Engineer is currently en route to the Data Centre to perform fault diagnosis.

We've spare hardware on site that we can swap the drives into if that is required. Time to fix will be approx 2-4 hours from the time of this blog post.

The on-call engineer will post updates when he has them.

Posted by Paul Kelly at 9:11 PM | Permalink | Comments (1) | TrackBacks (0)

When: Tuesday-Wednesday 18th/19th December 2007 @ 23:45 hours

What:

Firewall issues. We've been experiencing problems with the HA pair of firewalls that
look after the shared hosting network. Config changes are particularly problematic
and we've had to stop all changes.

Tonight we're going to move shared windows and linux hosting off of the current
firewall onto another set of firewalls. This is completely temporary as we wish to
investigate the issues we're having with the Fortigates without affecting service.

Who will be affected:

All shared hosting customers. We estimate 5-10 minutes per shared hosting
customergroup of down time.

So at approx 23:50 all shared hosting machines will dissappear from the network.
Once they have been all verified as "UP" from inside and outside we'll switch to
the windows machines.

At approx 00:10 or shortly after we'll started the shared windows hosting machines.

The above all requires moving the machines to seperate vlans also, this will take
the bulk of the time.

Summary:

Starting from 23:45 tonight the shared hosting network will disappear and should
be back at the latest 00:50.

As usual, please contact us via e-mail to support@blacknight.com if there are any
issues once the proposed time period has passed.

We do apologise for the late notification on this work, but due to staffing constraints on the run up to Christmas this is the only evening we can perform this task.

Happy Christmas to all!

Posted by Paul Kelly at 11:13 PM | Permalink | Comments (0) | TrackBacks (0)

Hi All,

@ 1:50am we had all servers back online. There were some complications that were not foreseen by us which were over looked by InterXion. This delayed us somewhat getting the last rack backup and on the network.

There are some other customer machines that did not come back up correctly. We'll liase with you regarding these tomorrow morning.

Posted by Paul Kelly at 3:29 AM | Permalink

As part of upgrading our network we are changing access to our primary
name servers (217.114.173.6 and 82.96.97.64) so that they are authoritative only. If you have servers on our network that are set to use these, then you will need to update the DNS settings.

We have attempted to notify all customers whose servers are currently using our authoritative name servers. However, if you have not been contacted and believe your server(s) may be using them, then you can contact us directly for more information. Any customers who put in a server within the last six months should be already using the new servers.

On Friday the 7th of December, the servers will be made authoritative only. If your name servers are not updated by then this may cause issues with connectivity.

Posted by Niall Donegan at 4:19 PM | Permalink

Our support area has been upgraded to the latest version of Kayako, which is the software we use to run it.

We'll be tweaking the settings, and customising the look and feel over the coming days, so don't be put off if it looks a bit different!

A few things may not have been transferred over during the upgrade, but they will be restored to all their glory over the coming days.. It will also give you a chance to get a first look at the new style design that we will be unveiling shortly. We hope you like it!

Posted by Michele Neylon at 1:46 AM | Permalink

The maintenance window for Cold Fusion has been cancelled

Our technical team found a resolution that did not require any upgrades

Posted by Michele Neylon at 3:13 PM | Permalink

Customer Migrations:

Due to restraints on colo space in the Dublin area at the moment we've had to make provisions to allow us to grow for the next 12 months at a minimum.

With this in mind, we've contracted with InterXion Ireland to get our own cage in their Dublin faciltiy in Parkwest Business Park.

On Sunday 2nd of December 2007 we wish to move our existing racks from the shared floor in InterXion to our caged area. Customers affected will be a small subset of our old and new shared hosting on the following two servers and their associated IP addresses:

balin
81.17.252.15
81.17.252.16
81.17.252.17
81.17.252.18
81.17.252.19
81.17.252.20
ragnell
81.17.252.110
81.17.252.111
81.17.252.112
81.17.252.113
81.17.252.114
81.17.252.115
81.17.252.116
81.17.252.117
81.17.252.118
81.17.252.119
81.17.252.120
81.17.252.121
81.17.252.122
81.17.252.123
81.17.252.124
81.17.252.125
81.17.252.126
81.17.252.127
81.17.252.128

Starting at 21:00 hours on Sunday 2nd of December, we'll be powering down the racks that contain these servers and transporting them down the corridor to our new Caged area. We estimate around 2-4 hours of downtime for the complete migration. This estimate is worst case scenario.

Colo and dedicated customers located in InterXion will be contacted separately with more information regarding their hardware.

As with any move of this magnitude we wish for anyone with any thoughts or questions regarding this please contact us ASAP so we can consider their ideas or issues prior to the move.

Contact details:

+353599183072
support [at] blacknight.ie

Blacknight Operations

Posted by Paul Kelly at 11:09 AM | Permalink

What: Coldfusion JDBC Drivers
When: Friday, 21st November, between 10:00AM and 10:15AM

What Will Be Affected:

Coldfusion websites on the Bediver server will be shutdown down while the JDBC drivers are upgraded.

Posted by Niall Donegan at 10:09 AM | Permalink

What: Blacknight Support

When: Thursday, 22nd November, between 18:00 and 21:00

What Will Be Affected:

Online support via the support website and email will be down for the duration of the upgrade. All email to Blacknight staff and other Blacknight addresses will be delayed.

Kayako, the system we use for support, has had extra features added which should allow for greater integration into our other systems. It is a major upgrade, which requires a lot of the database to be rebuilt and our email to be queued.

Phone support will still be available to CoLo and Dedicated customers at the out of hours number.

Summary:

All customers will be affected.

Posted by Niall Donegan at 4:19 PM | Permalink