1.7 Downtime Policy - Center for High Performance Computing

Scheduled Maintenance

Historically, the physical requirements of the Komas Datacenter’s HVAC system dictated that CHPC have quarterly downtimes. CHPC utilized this mandatory work schedule as our ‘quarterly downtime' and extended it to equipment in our hardened SSB datacenter, and our smaller INSCC datacenter. However, as of 2014 when all of the CHPC hardware from the Komas and SSB datacenters was moved to the new downtown datacenter (DDC), this type of maintenance in not needed, and therefore we have revised our maintenance and downtime strategy.

Maintenance Policy

CHPC will schedule maintenance in one of three manners:
1. Rolling reboots
  1. Whenever possible CHPC will apply updates and do other maintenance in a rolling fashion in such a manner as to have either no or as little impact as possible to CHPC services.
2. Partial outages
  1. We will do these as needed but in a manner that impacts only some CHPC services at a time
  2. We will work with the appropriate users for the choice of the day/time.
3. Full outages
  1. These are outages that will affect all CHPC services, such as outages of core datacenter networking services or datacenter power maintenance.
  2. We will offer two windows a year, typically during the spring and fall break weeks. It should be noted that we do not anticipate that these outages will always be needed.
  3. CHPC's goal for these downtimes is to have them completed within one working day. However, unforeseen or unusual circumstances may result in a multiple-day downtime.
Notifications:
1. The description of which services will be down, the date, time duration and impact to users is detailed in email and flyers.
2. Email notifications are sent to chpc-hpc-users@lists.utah.edu, people-inscc@lists.utah.edu, chpc-network-outages@lists.utah.edu, rhn-users@lists.utah.edu, and “komas data center list”. The first notification is sent two weeks prior to downtime. A second reminder is sent a few days prior to actual downtime. These lists should have users 'official' U of U email address. Users are responsible for checking their official Umail account. Please notify helpdesk@chpc.utah.edu if you would like to use a different email address. You can see where these notifications are being sent by using the Profile link on the CHPC Web site. Note: you will need to authenticate with your uNID and campus password to access the Profile link.
3. News of the downtimes is posted on http://www.chpc.utah.edu when email notifications are sent.
4. The "message of the day" (motd) on the clusters is updated with dates as soon as they are set. You can see this when you login or any time by issuing the command:
  
  > cat /etc/motd
5. Flyers posted in INSCC, Sutton, WBB, Arapeen and HSEB
Scheduling Considerations
1. Downtime dates are set after considering services impacted, known needs, and academic schedules.
2. Please notify CHPC's Associate Director immediately if you have concerns or critical circumstances in conflict with the scheduled time frame
3. See 1.8 System Availability vs Staff Availability for more on availability policies.

“”

Unscheduled “EMERGENCY” Downtimes

Unscheduled downtimes can occur for any number of reasons, including:
1. Loss of cooling and/or power in the data center.
2. Loss of supporting infrastructure (i.e. hardware).
3. Critical need to make changes to hardware or software that negatively impacts performance or access.
4. Application of critical patches that can’t wait until the next scheduled maintenance.
5. For safety or security issues that require immediate action.
6. To make configuration “policy” changes that are deemed necessary immediately.
Notifications
1. The scope and impact of downtime detailed in emails.
2. Email to chpc-hpc-users@lists.utah.edu, people-inscc@lists.utah.edu, chpc-network-outages@lists.utah.edu, rhn-users@lists.utah.edu, and “komas data center list” sent out as information is available.
3. News is posted on www.chpc.utah.edu when email notifications are sent.