The maintenance work was completed and all services are online as of approximately 8:30 PM. Due to issues commissioning our new UPS, that task was not completed and another (shorter) shutdown will be necessary some time in the future to bring the UPS online without risk to the servers.
On Saturday, June 16th, the campus high voltage team will be cutting power to Soda, Jacobs, Etcheverry, and North Gate Halls to complete critical power grid maintenance. The electrical shutdown is currently scheduled for 8AM to 1PM and these buildings will be closed during this time to all but critical staff related to the maintenance projects.
We are also taking this time to perform safety tests in our server and network rooms, as well as replacing the failed battery backup system in our primary server room. As a result, the outage window for most IT services (other than basic network connectivity) is 11PM 6/15 – 9PM 6/16.
Anybody with computers and electronics in Soda and Jacobs is STRONGLY RECOMMENDED to power off their noncritical systems when leaving the office on Friday, June 15. Critical systems MUST be powered off by 6AM on Saturday the 16th. This includes machines hosted in the 340 Soda server room, and a separate message will be sent to those system administrators next week.
As most of EECS’ IT infrastructure is hosted in Soda Hall, this will have some impacts which are detailed below. This will affect all EECS locations which are not undergoing electrical maintenance: Cory, Sutardja Dai and Blum Halls, the Calvin Lab, and BWRC as well as remote users.
- Network infrastructure (wired and wireless connections, including DHCP and DNS services) will be offline for a brief period between 6:00 and 6:30 AM in order to perform annual Emergency Power Off and Fire Alarm tests in our network core. Basic networking will be restored by 6:30 and should remain up for the remainder of the day for all locations outside of Soda and Jacobs Hall.
- The IRIS website and portions of the department website including personal homepages, as well as other IRIS services that depend on file storage, will be shut down starting at 11PM Fri. 6/15
- Storage services, including home and project file storage, will be shut down at 12 AM Sat. 6/16
- All services are scheduled to be restored by 9PM, but may be online earlier if maintenance is completed ahead of schedule.
As the IRIS website will be unavailable during this maintenance, https://status.eecs.berkeley.edu will be available for in-the-moment status updates.
Please contact firstname.lastname@example.org with any questions or concerns.
The EECS Network is currently disconnected from the rest of the Internet. The EECS Network team is currently investigating.
The network border is currently offline due to a protocol failure. We are investigating. This issue does not appear to be related to the previous border issues.
Update: The network was stabilized just before noon today.
We are currently experiencing an issue at the EECS network border which is causing intermittent outages. The connection affected is between EECS and the outside world.
This means that communication within the EECS network should be unaffected, and connectivity to campus or the Internet is intermittently unavailable.
Impact to the research computing cluster in Soda Hall is currently undetermined.
Network staff are coming onsite to better diagnose the issue, and an update will be posted here when we have more detail or an ETA.
Update 11:54 AM:
We are troubleshooting an issue where OSPF control traffic is intermittently dropped between our core and edge router, causing all data traffic to stop.
Traffic is affected between:
- EECS network and campus/Internet
- EECS network and Soda research compute network
Traffic is not affected:
- within the EECS network
- between Soda research compute and campus/Internet
Network Border Temporarily Disabled 2/15/18
Derek Calderon at 12:31 p.m. Feb 15, 2018
We are temporarily disabling network uplinks to the outside world in order to troubleshoot without impacting network security.
We need to determine whether the network firewall is implicated in this issue before engaging vendor support. In order to do this without making the network vulnerable from a security perspective, we must disable the links to campus and the Internet while this work is performed.
During this time no traffic will be able to enter or leave the department. We expect this will last for up to 30 minutes.
The Soda Hall machine room has lost power at approximately 11:50am on Dec 20, 2017, during what was supposed to be routine work on the room’s UPS. This has caused an interruption in many IRIS services. Staff are bringing services back as soon as they can.
The Soda Hall machine room has lost power at approximately 3:10pm on Dec 1, 2017, during what was supposed to be routine work on the room’s UPS. This has caused an interruption in many IRIS services. Staff are bringing services back as soon as they can.
There will be a power shutdown for Soda Hall on the early morning of Wednesday, November 29th, scheduled for 2 to 6 AM. The shutdown and short notice are due to an urgent need for the campus high voltage team to complete repairs from line damage in June. It is also possible that the outage could be extended if difficult circumstances are encountered.
We have arranged with campus facilities to receive generator power for 288 and 290 Soda so that we can continue to provide IT services to the outside world while Soda is dark. As such, no interruptions to IRIS services outside Soda Hall are expected at this time. However, due to many electrical load variables it is possible that we may need to shut down some services without notice during the maintenance window.
Power to research server rooms 420A and 287 will be lost and the sysadmins of those rooms are coordinating a full system shutdown.
For sysadmins of machines in 340 Soda, you will also need to shut down your machines. We recommend doing so before midnight.
News/ Campus losing power, IRIS services shutting down
BY ROB MCNICHOLAS
PUBLISHED 2:59 P.M. AUG 02, 2017
Due to a fire threatening our transformers, PG&E has advised campus we will be losing power.
IRIS is taking proactive steps to shut down as much equipment safely before we totally lose power.
See https://www.nixle.us/9HHZX for the warning from UCPD.
A power outage to Soda Hall which began around 12:30pm on June 21, 2017 has taken out many IRIS services. This backup website will be updated until our main website is working again.
Power has been restored to Soda Hall and staff are bringing up network equipment now. The power conditioner in the main Soda machine room is having problems recovering. Electricians are on site assisting.
This incident was fully resolved around 12:30am on June 22. See this IRIS news article for details.