Soda Hall Power Outage June 21 2017

A power outage to Soda Hall which began around 12:30pm on June 21, 2017 has taken out many IRIS services.  This backup website will be updated until our main website is working again.

Update 3:15pm

Power has been restored to Soda Hall and staff are bringing up network equipment now.  The power conditioner in the main Soda machine room is having problems recovering.  Electricians are on site assisting.

This incident was fully resolved around 12:30am on June 22.  See this IRIS news article for details.

Brief outages to various IRIS services, tonight Apr 12, 6pm-6:30pm

Routine RedHat Linux operating system patches have been released which require system reboots. Starting at 6pm Wednesday, April 12, IDSG will be patching and rebooting affected servers. The following services will experience brief outages of 10 minutes or less between 6pm and 6:30pm.

  • Department mailing list services (lists.eecs.berkeley.edu)
  • Department jabber/xmpp service (jabber.eecs.berkeley.edu)
  • IRIS website (iris.eecs.berkeley.edu)

EECS Network Outage

The EECS networks became unreachable around 1:20pm on Monday, April 10.  Staff are investigating.

Update (2:15pm):   Staff investigation found the problem to be at or with one of the department border firewall devices. The secondary firewall device has been made primary now, and traffic is routing properly. Staff will further examine the problem firewall device, to narrow down the specific cause.

UPDATE: EECS Network and Service Outage 12/22/16

We ran into a number of difficulties on the electrical side which delayed restoration of services, but basic networking (wired and wireless) were brought back at approximately 12PM.

At this time we are experiencing some slowness in the LDAP system, and we have also encountered trouble in bringing the NetApp file servers online. As a result the department website and file storage, as well as other services which rely on them, are still offline.

EECS Full Network and Service Outage, Thurs. 12/22/16 7:30AM – 1:00PM

Published 2:43 p.m. Nov 01, 2016

Scheduled Maintenance

Services Affected: Active Directory, DHCP, DNS, FTP Server, Home Directory Storage, IRIS Website, Mailing Lists, Project Storage, Repo Service, Unix Login Server, Wiki Hosting, Windows Terminal Server, Wired Networking, and Wireless Networking

Begins 7:30 a.m. Dec 22, 2016

We will have a full system shutdown on Thursday, December 22nd at 7:30 AM to perform necessary fire and emergency power testing in our server rooms. The entire EECS network and all hosted resources will be offline during this maintenance.

This will affect networking in the following locations:

Wired and wireless networks offline:

  • Soda Hall
  • Cory Hall
  • Sutardja Dai Hall
  • Jacobs Hall
  • BWRC

Wireless networks offline, wired remains online:

  • Blum Hall
  • Calvin Lab

Hosted resources such as the EECS website, file servers, etc. will be offline and globally inaccessible.

The facilities maintenance window is scheduled from 7:30 AM to 1 PM, but we anticipate the network and critical services should be back by 11AM if all goes well.

The intent is to perform these tests annually going forward; however we will learn from this year’s procedure and aim to keep more critical services online in the future.

If you have any questions about this maintenance, please contact help@eecs.berkeley.edu.

Fileserver maintenance, Nov 24 2016, 10am

Originally Published 11:56 a.m. Oct 26, 2016 on IRIS News

Scheduled Maintenance

Services Affected: Home Directory Storage, IRIS Website, Project Storage, Repo Service, and Unix Login Server

Begins 10:00 a.m. Nov 24, 2016

On the morning of Thursday, November 24 (Thanksgiving Day), we will be performing some disruptive maintenance on the department fileserver:

  • Recable to new network datacenter switches.
  • Upgrade cluster switches firmware.
  • Upgrade NetApp OS.

Department fileservice will be unavailable during the maintenance window, approximately 10am until 1pm. Many services that rely on access to department fileservice (home directories, webservice, repo service, etc), will either be offline or will fail to work during this window. All NFS and CIFS mounts will be affected.

We expect most, if not all, clients to recover automatically once the fileservers are back online. This news item will be updated to indicate when the maintenance is completed.

Systems still experiencing mount issues after the maintenance window, would need to remount or reboot.

COMPLETION UPDATE:

FILESERVER MAINTENANCE COMPLETED
Pathma Venasithamby at 12:37 p.m. Nov 24, 2016
Incident Resolved
Scheduled Maintenance Resolved
Systems on new datacenter network switches. All firmware updates completed. Netapp OS upgraded on all nodes. If there are any NFS or CIFS issues please reconnect and followup with helpdesk if there are outstanding issues.

Network outage at department border

Staff are investigating a network outage affecting all EECS network hosts. The outage began at about 6:25am September 22.

At this time, it’s unclear whether the outage is due to EECS network equipment, or upstream with campus networking.

 

UPDATE: Cause was due to a loss of default routes normally advertised by campus BGP. The issue was isolated and the campus NOC restored the routes shortly before 10AM.

See also https://iris.eecs.berkeley.edu/news/16634-eecs-internet-outage-92216