We are currently experiencing an issue at the EECS network border which is causing intermittent outages. The connection affected is between EECS and the outside world.
This means that communication within the EECS network should be unaffected, and connectivity to campus or the Internet is intermittently unavailable.
Impact to the research computing cluster in Soda Hall is currently undetermined.
Network staff are coming onsite to better diagnose the issue, and an update will be posted here when we have more detail or an ETA.
Update 11:54 AM:
We are troubleshooting an issue where OSPF control traffic is intermittently dropped between our core and edge router, causing all data traffic to stop.
Traffic is affected between:
- EECS network and campus/Internet
- EECS network and Soda research compute network
Traffic is not affected:
- within the EECS network
- between Soda research compute and campus/Internet
We are temporarily disabling network uplinks to the outside world in order to troubleshoot without impacting network security.
We need to determine whether the network firewall is implicated in this issue before engaging vendor support. In order to do this without making the network vulnerable from a security perspective, we must disable the links to campus and the Internet while this work is performed.
During this time no traffic will be able to enter or leave the department. We expect this will last for up to 30 minutes.
The Soda Hall machine room has lost power at approximately 11:50am on Dec 20, 2017, during what was supposed to be routine work on the room’s UPS. This has caused an interruption in many IRIS services. Staff are bringing services back as soon as they can.
The Soda Hall machine room has lost power at approximately 3:10pm on Dec 1, 2017, during what was supposed to be routine work on the room’s UPS. This has caused an interruption in many IRIS services. Staff are bringing services back as soon as they can.
Services Affected Wired Networking and Wireless Networking Begins 2:00 a.m. Nov 29, 2017
There will be a power shutdown for Soda Hall on the early morning of Wednesday, November 29th, scheduled for 2 to 6 AM. The shutdown and short notice are due to an urgent need for the campus high voltage team to complete repairs from line damage in June. It is also possible that the outage could be extended if difficult circumstances are encountered.
We have arranged with campus facilities to receive generator power for 288 and 290 Soda so that we can continue to provide IT services to the outside world while Soda is dark. As such, no interruptions to IRIS services outside Soda Hall are expected at this time. However, due to many electrical load variables it is possible that we may need to shut down some services without notice during the maintenance window.
Power to research server rooms 420A and 287 will be lost and the sysadmins of those rooms are coordinating a full system shutdown.
For sysadmins of machines in 340 Soda, you will also need to shut down your machines. We recommend doing so before midnight.
A power outage to Soda Hall which began around 12:30pm on June 21, 2017 has taken out many IRIS services. This backup website will be updated until our main website is working again.
Power has been restored to Soda Hall and staff are bringing up network equipment now. The power conditioner in the main Soda machine room is having problems recovering. Electricians are on site assisting.
This incident was fully resolved around 12:30am on June 22. See this IRIS news article for details.
The EECS networks became unreachable around 1:20pm on Monday, April 10. Staff are investigating.
Update (2:15pm): Staff investigation found the problem to be at or with one of the department border firewall devices. The secondary firewall device has been made primary now, and traffic is routing properly. Staff will further examine the problem firewall device, to narrow down the specific cause.
We ran into a number of difficulties on the electrical side which delayed restoration of services, but basic networking (wired and wireless) were brought back at approximately 12PM.
At this time we are experiencing some slowness in the LDAP system, and we have also encountered trouble in bringing the NetApp file servers online. As a result the department website and file storage, as well as other services which rely on them, are still offline.