Inter DataCentre connectivity testing

When: Monday 10th of March @ 22:00 hours
What: Firstly we’ve recently lit our own protected wavelength between DEG and InterXion. It has been in place and in testing for a few weeks now. We need to test the failover on both the long and short legs of this new connectivity and also check the failover to the backup layer 2 paths in the event of both the short and long legs getting damaged at the same time.
We don’t expect any downtime during this testing as our layer 2 network normally fails over within a few milliseconds.
Secondly we’re moving the InterXion firewalls to the new Distribution routers in this location. This change should take 30 seconds or so to propagate within our network as it’s a logical Layer 3 change.
This will mean the firewalled network in DEG and InterXion will be seperated from each other and traffic originating and destined for each data centre doesn’t need to traverse our metro network.
For complete testing we’ll allocate 2 hours to perform these tests. We don’t envisage anything more than a few 10-30 seconds hits on metro traffic (so won’t affect everyone) and it will only cause slow loading times for some websites and not others.
Summary: Works begin @ 22:00 hours on Monday 10th and end at midnight on Monday 10th. There should only be a few short hits on our metro links as they failover while we simulate fibre cuts, switch failures, port failures etc.
Update: 23:58 March 10th:
These tests have been completed. The inter DC links have been tested in several scenarios are we’re happy it’s quite resilient now. We also moved the InterXion firewalls to a new distribution router pair in InterXion from the DEG routers. This took a little longer than expected, around 3 minutes and 40 seconds or so, slight OSPF glitch in the config which took a minute or two to find. All went to plan except that firewall move in the initial stages.