A nationwide authentication problem with our DSL supplier has caused problems for a large number of customers.
Our last update at 09:26 confirmed that engineers were on site, had located the problem and were in the process of bringing systems back on line. Intermittent connectivity is expected during this period but normal service should resume shortly.
A failed disk in one of the storage systems for our virtual hosting platform has resulted in file system errors, taking down the virtual machines currently stored on that system.
At the moment engineers are attempting to recover the file system, hopefully bringing all machines back on line.
Update 1: 15/01 00:10
The storage is now on line and all virtual machines have been restarted. It’s advised for customers to run a file system check inside their guest operating system just in case errors have made in into the guest images.
Update 2: 15/01 08:35
It appears errors have accumulated over night and the storage system went offline around 02:45. Engineers are currently looking into a solution.
Update 3: 15/01 09:00
Machines are now back on line and all errors appear to have been cleared. We will keep a close eye on the system but all servers should now be running as normal.
A switch failure inside the network of one of our core network providers has caused disruption to one of our primary network feeds. This happened at around 9:30 this morning. Some users may have noticed a small blip in their connection although all traffic is now flowing via our other links.
We are currently waiting for the equipment to be replaced by our provider but expect full service to be resumed shortly.
The switch was finally brought back online around 16:30, restoring full service and resilience.
We are experiencing some issues that may effect some users. Engineers are currently resolving the issue we are sorry for any inconvenience caused by this.
A faulty switch was replaced and all service resumed at around 16:30.
An issue with a switch at our Wrekin point of presence may have caused connection problems for some uConnect wireless broadband users, starting at around 20:55 this evening.
The problematic equipment has been restarted, and as of 21:25, all connections appear to be up and running. Our network staff will continue to monitor the relevant equipment closely as replacement may be necessary. If any work is required that will impact user connections, updates will be posted here on the NOC website.
We apologise for any inconvenience this outage may of caused to users.
Please contact email@example.com for more information.
Between 09:30 and 16:00 on Friday, 21st of November, we will be performing maintenance on the north facing equipment at The Wrekin.
We are expecting some downtime for wireless broadband customers while the work is completed and apologise for any disruption in service.
We are currently experiencing outages on all our connections provided by Virgin Media. This issue started around 11:20.
The majority of customers may have noticed a small outage while our routers re-routed traffic via alternate links.
Customers who connect to us via a dedicated Virgin link will still be experiencing problems.
We are currently waiting for an update from the supplier on the exact cause of the problem and the time for resolution. Please keep checking this network operations site for updates.
UPDATE: As of 12:57, these links have all come back online
We have now brought additional servers online and our primary email queues are clear. We expect email services to have now returned to normal.
We are monitoring the system closely and will provide further updates if any additional problems surface.
We are continuing to see delays with our email scanning servers which is causing email to be queued by our backup system. This can cause emails to arrive anywhere between a few minutes or a few hours late. All email is queued until delivery so no messages will be lost.
We are now in the process of provisioning additional servers to increase the load that can be handled by the system before email is diverted to our backup servers.
We have been experiencing issues with email delays over the weekend due to high volume of email. We are currently investigating and looking into measures to prevent build up of email queues.