Protected Environment Frequently Asked Questions (FAQ) - Center for High Performance Computing

What is PHI (protected health information)?

This is a summary provided by HHS (health and human services). There is more detail at http://www.hhs.gov/ocr/privacy/hipaa/understanding/summary/index.html

The Privacy Rule protects all individually identifiable health information held or transmitted by a covered entity or its business associate, in any form or media, whether electronic, paper, or oral. The Privacy Rule calls this information protected health information (PHI). Individually identifiable health information is information, including demographic data, that:

relates to the individual’s past, present or future physical or mental health or condition;
relates to the provision of health care to the individual;
relates to the past, present, or future payment for the provision of health care to the individual;
identifies the individual or for which there is a reasonable basis to believe it can be used to identify the individual.

Individually identifiable health information includes many common identifiers (e.g., name, address, birth date, Social Security Number).

The Privacy Rule excludes from protected health information employment records that a covered entity maintains in its capacity as an employer and education and certain other records subject to, or defined in, the Family Educational Rights and Privacy Act, 20 U.S.C. §1232g.

Covered Entity chart: https://www.cms.gov/Regulations-and-Guidance/Administrative-Simplification/HIPAA-ACA/Downloads/CoveredEntitiesChart20160617.pdf

For UofU Privacy & HIPAA information: https://pulse.utah.edu/policies/FilteredFor/Information%20Privacy.aspx

What restrictions does de-indentified data have?

There are no restrictions on the use or disclosure of de-identified health information. De-identified health information does not provide a reasonable basis to identify an individual. There are two ways to de-identify information: (1) a formal determination by a qualified statistician; or (2) the removal of specified identifiers of the individual and of the individual’s relatives, household members, and employers is required, and is adequate only if the covered entity has no actual knowledge that the remaining information could be used to identify the individual.

What does it mean to be "HIPAA compliant?"

There are 169 Privacy, Security, and Breach rules that covered entities (CEs) and business associates (BAs) are evaluated against. As HIPAA is a risk-based (versus standards-based) assessment, there is not a simple list of criteria. NIST provides the HHS preferred criteria for security evaluation.

Any CE (us) and its BAs (vendors agreeing to handle a CE’s PHI) are evaluated against:

http://www.hhs.gov/ocr/privacy/hipaa/enforcement/audit/protocol.html

What resources are available in the CHPC Protected environment?

We have two main areas in this environment: 1) The High Performance Computing (HPC) analytical environment for computations, applications, etc. 2) The Virtual Machine (VM) environment for various development, application, database, and research services.

HPC Analytic Environment

This space is for computationally intensive needs such as data mining, machine learning, natural language processing, statistics, and other operations across large datasets. This requires large (and secure) storage, high network-bandwidth capacity, and high performance compute clusters. For access to the environment, we chose a double authentication mechanism. The first level of authentication uses the campus authentication and a second factor authentication. The second authentication level requires users to be listed in the CHPC Network Information Service (NIS) directory server (the windows machines pull in this NIS group info for windows group membership) in order to interact with the PE.

Access to the compute clusters is provided via front-end login servers. The login servers are restricted by router Access Control Lists to Remote Desktop Protocol for Windows hosts and Secure Shell for the Linux hosts. Public key, host, or RHOSTS-based authentications are not allowed. The network access control lists and login servers also employ firewall services to allow limited IP addresses.

All interactions between the login servers and the cluster file server, batch controller, and computing nodes take place on an isolated back-end, high-speed Infiniband network. All Linux hosts in the cluster are automatically updated on a monthly basis using the a Network update service, while the Windows hosts are updated using Microsoft update and use anti-virus software.

Virtual Machine Implementation

The VM environment is for scientists who need lightweight computing services that do not justify the expense or capabilities of a dedicated server. The VM Cluster consists of four servers (two VMware ESXs, one Windows, one Red Hat Linux) and a disk tray. One Windows server runs VMware vCenter Server that coordinates the load-balancing and failover functions of the two ESX servers. This server does not process any protected data. One Red Hat Linux server acts as an administrative access point and it also does not process any protected data. The two VMware servers host the actual guest VMs. These servers do process protected data but do not store it internally (i.e., all transactions are RAM-based). The disk tray provides shared storage to the two VMware servers. These disks store the actual VMs, and thus all sensitive data are in those VMs. We require all VMs to encrypt their disk. The VMs and applications are regularly scanned by the University ISO.

How is data stored in this environment?

For CHPC’s HPC protected environment, e.g, home and group/project services, see:

HPC Cluster (redwood)
Windows Statistics system (narwhal)
VM (Virtual Machine) Farm (prismatic)
PE Storage (mammoth)
- Scratch - /scratch/mammoth/serial
- Home - /uufs/chpc.utah.edu/common/HIPAA/UNID
- Project - /uufs/chpc.utah.edu/common/HIPAA/project
Archive Storage (elm)

For CHPC’s virtual machine protected environment, see: Protected VM Farm

How do I transfer files to/from narwhal?

To transfer files to/from narwhal, use WinSCP. (PLEASE remember to adhere to IRB protocol and HIPAA rules for storing protected health information) How to Use SCP/SFTP to Transfer files

How are backups handled?

Backups are restricted to one specific backup server and traffic is automatically encrypted on the Client side before traversing the network (BLOWFISH encryption). Backup media is stored in a locked cabinet in an access controlled/monitored data center. There is a two week data retention policy (unless otherwise arranged).

If PHI can be reasonable protected in a cluster environment, is it possibile to be HIPAA compliant in a shared environment?

In our setup we have the HIPAA HPC cluster environment isolated from our general HPC cluster with different IP space and access security controls. It is still a "shared environment" in the sense that users who have access to this space have a business need and have completed certain requirements. We save cost of hardware and administrative overhead by utilizing some existing/general infrastructure. We isolate the different groups/projects via unix ACLs and other controls.

Where is the hardware?

The physical hardware resides in a data center with 24 hour surveillance. Access is controlled via data center stewards, human issued key cards, and fingerprint biometric access.

If interactive nodes are segmented and configured in a typical HIPAA compliant fashion and the nodes are only capable of submitting jobs to the cluster (which is otherwise inaccessible), the jobs and relevant data are cleaned from the cluster once a job is complete, etc. Would that generally be compliant?

If we understand the scenario here (being that the interactive nodes are shared with HIPAA and non HIPAA compliant workflows), it would be difficult to justify to an auditor that it's a HIPAA compliant workspace. Maybe if the user had an interactive/login node specifically only for HIPAA use (for example, limited access by security groups and/or DUO/host list) and the user made a special queue that only those users could use, we could see how one could be HIPAA compliant, providing all authentication and data transmission (and at rest) are encrypted.

How is the environment secured and how is access allowed?

Instuctions for logging into the PE: https://www.chpc.utah.edu/resources/ProtectedEnvironment.php#logginintoPE

Is there any way you can summarize what you did to build a compliant cluster?

This is tough to condense down & summarize - we've snipped out some relevant sections from other documentation:

When building the protected environment we wanted to isolate it as much as possible, while still
utilizing some pieces of the core infrastructure to minimize costs. The network IP space we used for the protected
environments (i.e., the HPC and the VM environments) were assigned to separate logical subnets and vLANs from
both our existing services and from each other. We further segregated the individual Virtual Machines (VMs) to
project-specific groups. With this approach we were able to utilize most of our existing core services, like DUO,
DNS, NTP, and Kerberos.

Physical infrastructure: The physical hardware resides in a data center with controlled room access. These hosts are
racked in a locked cabinet and hosts have locked server bezels. Physical access to the data center is reviewed
biannually and documented on an access-controlled departmental log. Backups are restricted to one specific
backup server on one particular port. Backup data traffic is automatically encrypted (BLOWFISH encryption) at the
client side before traversing the network. Backup media are stored in locked cabinets in the access-restricted data
center. All employees and users who interact with the cluster have been through the University of Utah’s HIPAA
training, and many have completed the well-known CITI human subjects research training.

HPC analytic environment: The HPC environment was architected to require two authentication mechanisms.
The first level of authentication requires use of the University of Utah campus Central campus directory service in conjuction with Two factor security. The second authentication level requires users to be listed in the CHPC NIS directory server in order to interact with the
cluster. The access to both the Central campus directory service and the NIS server are set up only by explicit approval.

Access to the computational cluster servers is provided via front-end “login servers.” The login servers are restricted (limited to certain IP addresses) by router Access Control Lists (ACL) to Remote Desktop Protocol (RDP uses TCP port 3389) for the Windows hosts and Secure Shell (SSH v2 uses TCP port 22) for the Linux hosts. Public key, host, or RHOSTS-based authentication are not allowed. The login servers also employ firewall services to limit access to a specific set of IP ranges. All interactions between the front-end hosts and the cluster file server, batch controller, and computing nodes take place on an isolated back-end, high-speed Infiniband network. All Linux hosts in the cluster are automatically
updated on at least a monthly basis utilizing the Redhat Network update service, while the Windows hosts are
updated using Microsoft update and use anti-virus software that is also regularly updated. We have to provision a
dedicated fileserver for the HPC home directories in the protected environment, but we utilize the existing UNIX
application file systems in read-only mode to deliver executable programs. Login sessions are automatically logged out after
15 minutes of inactivity.

Administrative procedures: In order for a user to access the HIPAA environment at CHPC they must meet all the
following requirements:
• Have an active account in the University of Utah's directory service system, using the
University of Utah’s mechanisms. This extends to any external collaborators as well.
• Have an active CHPC departmental account, where sponsorship and approval of a Principal
Investigator (PI) is required.
• Have an active CHPC account created in the HIPAA environments isolated NIS (network
information system) and be a member of the ‘NIS group’ that is listed in the PAM security
access.conf file. This requires verification and completion of the on-line HIPAA privacy and
security training.
• Have a two factor security account.

Permission to use a given dataset is governed by the approval of the University's Institutional Review Board (IRB).
Researchers must submit a proposal to the IRB listing the data to be used and the people who will have access to it.
If the IRB approves the use of the data in question, the researcher is given an IRB number. In order to store the data
in CHPC's HIPAA environment, the researcher must provide CHPC with this number and a list of the users who will
be permitted to see the data. The list is independently verified with the IRB. Thereafter, the data may be transferred
to CHPC and only the IRB-approved users will be able to work with it.

Logical access is monitored by SYSLOG and process accounting (PSACCT). Logs are kept both locally and on a
remote SYSLOG server and routinely reviewed. Retention of logs on the SYSLOG server is currently kept
indefinitely. Logwatch reports are emailed daily to designated administrator accounts. These daily logwatch reports
show the last 24 hours of what accounts logged in (or failed to log in) and from what IP. Firewall configuration
prevents ‘brute force’ login attempts. If an account login is unsuccessful more than three times, it will lock/prevent the
account from authenticating for three minutes. Access to view or manipulate other users' or groups' data is enforced by
using UNIX file and directory permission(s). Two factor logs are available from the University of Utah’s IT (UIT)
Department. These logs show access and failures by date and time.

Account validation is periodically reviewed. User access to IRB data is periodically reviewed (scheduled
biannually). IRB projects are the authoritative source for who has access to HIPAA data. If a person is not listed on
an approved IRB project, then he or she is not allowed in the UNIX group for access to that project's data. Our IRB
reviews studies on annual basis.

Summary
We have been able to reuse a substantial part of the CHPC HPC infrastructure to develop a new protected
environment, increasing user productivity and compliance at the same time. For these researchers, in
addition to access to high performance computing power, other tangible benefits for researchers include CHPC
handling systems management issues, such as rapid response to electrical power issues, provision of reliable cooling
and heating; DUO support for a work-anywhere computing experience; and ensuring a hardened, secure environment
compared to office computers or departmental servers. For the institution, this resource allows much better
compliance and reduces the vulnerabilities of exposure of PHI data.

What is REDCap?

The Research Electronic Data Capture (REDCap) data management application is used to create web accessible forms, a secure database with continuous auditing, and a flexible reporting system. REDCap is hosted on a HIPAA aligned server farm within the CHPC area in the University of Utah Data Center. More info can be found at https://redcap01.brisc.utah.edu/ccts/redcap/index.php?action=training

REDCap can be 21 CFR Part 11 capable providing that all entities (discussed below) involved implement the required procedures and comply with the required responsibilities. Compliance depends on the setting, which includes both the technical aspects of the installation and maintenance, quality requirements [Implementation Quality (IQ) – Operational Quality (OQ) – Production Quality (PQ)], as well as the essential processes put in place by users.

CHPC is further exploring 21 CFR Part 11 requirements and working with appropriate leadership and will update as things develop.

Entities and required procedures can included:

1) The hosting environment (CHPC responsibility)

Involved as the CHPC provides a secure environment ( in the CHPC protected environment) for REDCap instances. Login accounts to these environments are provided and monitored by CHPC personnel; need to have administrative change management process in place for the hosting environment (VM-ESX).

2) The application itself (application stack, eg., REDCap etc) (BMIC responsibility)

Documentation of continued compliance and administrative standard operation procedures (SOPs) have to be in place.

3) Study management controls (PI responsibility)

Where audit trails are required - these (it seems) will be mostly the responsibility of the researchers to maintain documented audit trails SOP. The individual user or study team is responsible for ensuring compliance, including establishing appropriate standard operating procedures to protect and document data. Many of the activities encouraged by Part 11 are good practice in general, including explicit definition of study team roles and responsibilities, database change control and documentation, and record retention. Personnel training commensurate with responsibilities is also required. citiprogram.org offers a GCP good clinical practice course that is US FDA focused.

External Link with additional related information:

https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?CFRPart=11

Is there a HIPPA compliant MySQL or PostgreSQL server set up on CHPC? If so, how can I access that?

Yes. It's called mysql.apexarch.bridges. You will need to have access to either the HIPAA-secure Windows client narwhal or the Linux interactive node redwood, as this server is not publicly accessible. The server is running Mysql5.5 on RHEL6. For access, send a request to helpdesk@chpc.utah.edu. Include the name you would like for the database and we will provision it, creating a mysql user for you if necessary.

How do I access the protected environment and the home or project/group directories at CHPC?

When you are provisioned with access to the PE, you will be provided instructions of how to access.

Here's a link of how to get to home and group/project directories...

https://wiki.chpc.utah.edu/pages/viewpage.action?pageId=541163529

Here is a link (if you have access to the PE) for host access methods...

https://wiki.chpc.utah.edu/display/DOCS/Protected+Environment+%28CHPC-PE%29+service+host+roles+and+access

As always If you need assistance, please let us know.

Is it possible to map/mount the general CHPC fileservers (e.g., cottonwood) on narwhal or other hosts in the CHPC Protected Environment (PE) or vice versa ?

It is not possible to mount the General Home directory space (e.g., drycreek) from narwhal (or any machine located in the CHPC protected environment). This is, by design, to isolate protected data that may be stored in the CHPC PE.

How do I transfer files to/from Box from/to CHPC's protected environment (PE)?

One can use FastX or Microsoft Remote Desktop to access a host in the CHPC PE and then use a web browser (or box sync).

For more information about the UofU Box.com services please see... http://box.utah.edu/ & http://it.utah.edu/news/april_2014/node4_ubox.php

NOTE: box.com is currently the only UofU cloud storage service that is acceptable to store PHI data, please see "NOTE" at the following CHPC web page for more info... https://www.chpc.utah.edu/resources/ProtectedEnvironment.php

Is the information I put in the CHPC issue tracking system (ServiceNow) encrypted and secure?

NO! Never put Protected Heath Information (PHI), passwords/passphrases, or any sensitive information in the CHPC helpdesk ServiceNow issue tracking system!

Is a MRN (medical record number) considered ePHI?

YES. http://www.hipaa.com/hipaa-protected-health-information-what-does-phi-include/

See: (H) Medical record numbers;

As the MRN can be used to reidentify, it is considered protected. If you're moving data that contains an MRN, you need to consider it as PHI and encrypt/secure appropriately.

http://cphs.berkeley.edu/hipaa/hipaa18.html

“Also note, health information by itself without the 18 identifiers is not considered to be PHI. For example, a dataset of vital signs by themselves do not constitute protected health information. However, if the vital signs dataset includes medical record numbers, then the entire dataset must be protected since it contains an identifier.”

I have a DUO account and have a new phone, what do I need to do to get it registered or re-registered

Your DUO devices can be managed via the folowing link: https://ese.idm.utah.edu/duo-management/secure/user/enrollment#d

The link will prompt you to sign in to your CIS account. If DUO is no longer aligned to push you a notification, login using the "Enter a passcode" option and then select "Text me new codes". After logging in, you will be brought to the Terms and Conditions page.

After clicking 'Agree' to the Terms and Conditions, the site will automatically direct you to "Register a New DUO Device." If instead you wish to manage your devices that are already registered with DUO, click 'Cancel.' You will now be able to see a list of deviced registered with your DUO account.

See https://www.chpc.utah.edu/documentation/software/duo.php for further instructions on registering or re-activating your device(s) and for frequently asked questions regarding DUO security. Note: We recommend registering with multiple devices. ALSO, please note that if you have a major software update to your device (e.g., android 5.x to 6.x) you may need to go to reactivate your device using the above instructions.

What types of regulated data can CHPC provide services for?

CHPC is setup to host use cases for HIPAA regulated and NIH dbgap data sets. Here is the dbgap guidelines from NIH http://www.ncbi.nlm.nih.gov/projects/gap/pdf/dbgap_2b_security_procedures.pdf

What are the screen timeout values and account lockout values for the login hosts for this environment?

To prevents ‘brute force’ login attempts. If a unix account login is unsuccessful more than six (6) times it will be locked to prevent the account from authenticating (to that same host) for ten (10) minutes. There is a 30 minute inactivity screen timeout for sessions to PE access hosts (e.g., redwood/narwhal etc), if you leave an idle session for this time period it will lock your session.

DUO troubleshooting & device registration and management portal

Troubleshooting https://uofu.service-now.com/it?id=uu_kb_article&sys_id=5067fcf5db1f5644a9693c1ebf9619e8

device registration/management https://ese.idm.utah.edu/duo-management/index.htm#d

Here is a form that can be used for proof of device encryption

Encryption Attestation Form

https://pulse.utah.edu/policies/Lists/Policies/DispForm.aspx?ID=3537

The EDW has network allowances for some services from within the PE. If you need additional information feel free to send in a helpdesk ticket.