As per campus announcement, PG&E is implementing a power outage affecting campus buildings due to fire danger. This outage will begin at about 8am Wed Oct 9, but various systems and services will be shutdown in advance of the loss of power.
Please find updates at https://iris.eecs.berkeley.edu/news/ongoing/potential-campus-power-outage/
See also the campus system status page at https://berkeley.statusdashboard.com/
NetApp filers are due for their 4 year refresh cycle. Work on this has commenced, with new NetApp hardware on site. New filers are going to be added and data migration will occur some weeks down the road. While most of the work will be non-disruptive, some elements will be disruptive.
The first of this will happen on 10th June Monday, and the scheduled downtime will be from 6-8pm, to incorporate some network changes. All NFS and CIFS access to the filer (home and project storage) will be down during that time.
On Saturday, June 16th, the campus high voltage team will be cutting power to Soda, Jacobs, Etcheverry, and North Gate Halls to complete critical power grid maintenance. The electrical shutdown is currently scheduled for 8AM to 1PM and these buildings will be closed during this time to all but critical staff related to the maintenance projects.
We are also taking this time to perform safety tests in our server and network rooms, as well as replacing the failed battery backup system in our primary server room. As a result, the outage window for most IT services (other than basic network connectivity) is 11PM 6/15 – 9PM 6/16.
Anybody with computers and electronics in Soda and Jacobs is STRONGLY RECOMMENDED to power off their noncritical systems when leaving the office on Friday, June 15. Critical systems MUST be powered off by 6AM on Saturday the 16th. This includes machines hosted in the 340 Soda server room, and a separate message will be sent to those system administrators next week.
As most of EECS’ IT infrastructure is hosted in Soda Hall, this will have some impacts which are detailed below. This will affect all EECS locations which are not undergoing electrical maintenance: Cory, Sutardja Dai and Blum Halls, the Calvin Lab, and BWRC as well as remote users.
- Network infrastructure (wired and wireless connections, including DHCP and DNS services) will be offline for a brief period between 6:00 and 6:30 AM in order to perform annual Emergency Power Off and Fire Alarm tests in our network core. Basic networking will be restored by 6:30 and should remain up for the remainder of the day for all locations outside of Soda and Jacobs Hall.
- The IRIS website and portions of the department website including personal homepages, as well as other IRIS services that depend on file storage, will be shut down starting at 11PM Fri. 6/15
- Storage services, including home and project file storage, will be shut down at 12 AM Sat. 6/16
- All services are scheduled to be restored by 9PM, but may be online earlier if maintenance is completed ahead of schedule.
As the IRIS website will be unavailable during this maintenance, https://status.eecs.berkeley.edu will be available for in-the-moment status updates.
Please contact firstname.lastname@example.org with any questions or concerns.
The network border is currently offline due to a protocol failure. We are investigating. This issue does not appear to be related to the previous border issues.
Update: The network was stabilized just before noon today.
We are currently experiencing an issue at the EECS network border which is causing intermittent outages. The connection affected is between EECS and the outside world.
This means that communication within the EECS network should be unaffected, and connectivity to campus or the Internet is intermittently unavailable.
Impact to the research computing cluster in Soda Hall is currently undetermined.
Network staff are coming onsite to better diagnose the issue, and an update will be posted here when we have more detail or an ETA.
Update 11:54 AM:
We are troubleshooting an issue where OSPF control traffic is intermittently dropped between our core and edge router, causing all data traffic to stop.
Traffic is affected between:
- EECS network and campus/Internet
- EECS network and Soda research compute network
Traffic is not affected:
- within the EECS network
- between Soda research compute and campus/Internet
We are temporarily disabling network uplinks to the outside world in order to troubleshoot without impacting network security.
We need to determine whether the network firewall is implicated in this issue before engaging vendor support. In order to do this without making the network vulnerable from a security perspective, we must disable the links to campus and the Internet while this work is performed.
During this time no traffic will be able to enter or leave the department. We expect this will last for up to 30 minutes.
The Soda Hall machine room has lost power at approximately 11:50am on Dec 20, 2017, during what was supposed to be routine work on the room’s UPS. This has caused an interruption in many IRIS services. Staff are bringing services back as soon as they can.
The Soda Hall machine room has lost power at approximately 3:10pm on Dec 1, 2017, during what was supposed to be routine work on the room’s UPS. This has caused an interruption in many IRIS services. Staff are bringing services back as soon as they can.
Services Affected Wired Networking and Wireless Networking Begins 2:00 a.m. Nov 29, 2017
There will be a power shutdown for Soda Hall on the early morning of Wednesday, November 29th, scheduled for 2 to 6 AM. The shutdown and short notice are due to an urgent need for the campus high voltage team to complete repairs from line damage in June. It is also possible that the outage could be extended if difficult circumstances are encountered.
We have arranged with campus facilities to receive generator power for 288 and 290 Soda so that we can continue to provide IT services to the outside world while Soda is dark. As such, no interruptions to IRIS services outside Soda Hall are expected at this time. However, due to many electrical load variables it is possible that we may need to shut down some services without notice during the maintenance window.
Power to research server rooms 420A and 287 will be lost and the sysadmins of those rooms are coordinating a full system shutdown.
For sysadmins of machines in 340 Soda, you will also need to shut down your machines. We recommend doing so before midnight.