Just to let you know that next Sunday, 22nd November during the afternoon we will be rebooting the machines that handle email.

This will allow us to boot with an updated kernel which has a number of security fixes we regard as needed.

The outage should last less than 10 minutes for each machine, and we will only reboot one machine at a time to ensure that incoming email is not blocked. Doing this means no inbound email will be lost, only that for the duration of the reboot your will not be able to collect email.

We are going to be taking an outage on our primary machines this Sunday to replace some disks that are being problematic.

We will take each machine down in turn to ensure that there is always a machine for incoming email, but mail will not be available to fetch/read whilst your primary machine is down.

This is a precautionary measure as we have properly configured RAID systems to ensure availability so no data will be lost and these drives could normally be replaced without downtime, but we have some configuration updates to install that will require at least a reboot so we are combining the events to minimise the downtime.

The outages will commence sometime after 14:00 on Sunday and we hope to complete by 18:00 though some overrun on individual services may occur.

We took a one-hour hit starting just after midnight today.

Apparently the psu blew in one box in one of our supplier’s racks, and took the whole rack out.  I guess a breaker tripped.  Our 2MX went down, and that system is also the outbound used by most of our customers (we’d prefer they used the 1MX, actually).  Worse, the routers upstream from us went out – so although the 1MX stayed up, the outside world couldn’t get to it!

Sorry, everyone.  All well now.