Skip to main content
Version: Current

HPC Campus Clusters

info

This page presents an overview of the High-Performance Computing Clusters at UC Merced.

As well as how to get access, logging in, file system, resource breakdown.

Currently, UC Merced has two clusters on site. They are maintained by the CIRT team. If you have any questions, feel free to contact us here.

note

The Pinnacles cluster located in the server facility (see Research Facility below) is available for all faculty projects at NO COST! The Pinnacles cluster runs with the Rocky (9.8) operating system, and employs the Slurm job scheduler and queueing system to manage job runs.

Facility Statement

UC Merced operates the Pinnacles cluster, a campus high-performance computing environment managed by the Office of Information Technology. Pinnacles supports research computing, AI/ML workflows, data-intensive analysis, training, and regional cyberinfrastructure activities. Technical support, documentation, and training are available through UC Merced Cyberinfrastructure and Research Technologies (CIRT) team.

The Pinnacles cluster includes:

  • 40 compute nodes, each with 2 × Intel 28-core Xeon Gold 6330 processors and 256 GB RAM;
  • 9 compute nodes, each with 2 × Intel 32-core Xeon Gold 6530 processors and 256 GB RAM;
  • 8 high-memory nodes, including 4 nodes with 2 × Intel 28-core Xeon Gold 6330 processors and 1 TB RAM, and 4 nodes with 2 × Intel 32-core Xeon Gold 6530 processors and 1 TB RAM;
  • 20 GPU nodes, including:
    • 8 nodes with 2 × NVIDIA A100 PCIe GPUs, 40 GB HBM2 each;
    • 8 nodes with 2 × NVIDIA L40S GPUs, 48 GB GDDR6 each;
    • 4 nodes with 2 × NVIDIA H200 NVL GPUs, 141 GB HBM3 each.

Central Valley Accessible Research and Computational Hub (CENVAL-ARC) compute resources, supported by NSF award #2346744, extend UC Merced's regional cyberinfrastructure capacity and support both campus-hosted and Open Science Grid (OSG)-accessible computing. CENVAL-ARC resources installed on the Pinnacles cluster include:

  • 9 compute nodes with dual Intel 32-core Xeon Gold 6430 or 6530 processors, 256 GB RAM, and local NVMe storage, configured with either 1 TB M.2 NVMe or 2 × 960 GB NVMe SSDs per node;
  • 4 high-memory nodes with dual Intel 32-core Xeon Gold 6430 processors, 1 TB RAM, and 1 TB M.2 NVMe local storage per node;
  • 8 GPU nodes with 2 × NVIDIA L40S GPUs, 48 GB GDDR6 each, dual Intel 32-core Xeon Gold 6530 processors, local NVMe storage, and HDR-100 InfiniBand connectivity;
  • 4 GPU nodes with 2 × NVIDIA H200 NVL GPUs, 141 GB HBM3 each, dual Intel 32-core Xeon Gold 6530 processors, 256 GB RAM, local NVMe storage, and HDR-100 InfiniBand connectivity.

CENVAL-ARC also provides OSG-accessible resources, including 4 compute nodes with dual Intel 32-core Xeon Gold 6430 or 6530 processors, 256 GB RAM, and local NVMe storage.

Pinnacles provides approximately 92 TB of NFS fast scratch storage and 1.5 PB of usable long-term storage. Compute nodes are interconnected through HDR InfiniBand with RDMA, supporting up to 100 Gb/s low-latency data transfer.

Availability and Support: The Pinnacles cluster is managed by CIRT. General access is available to UC Merced faculty projects at no cost, subject to allocation and scheduling policies. CIRT provides user support, consultation, onboarding, documentation, and training to support effective use of the cluster for research, training, and AI/data-intensive workflows.

How to cite

All Pinnacles users must agree to acknowledge the Pinnacles Cluster in talks, posters, manuscripts, and other forms of dissemination relying on results obtained from time on Pinnacles. An example acknowledgement section is:

This research [Part of this research] was conducted using the Pinnacles cluster, which is centrally funded by the University of California, Merced, and maintained by the Cyberinfrastructure and Research Technologies (CIRT) team at UC Merced.

From time to time the Committee on Research Computing (CoRC) may request a report of publications and presentations authored by Pinnacles users that have included results of calculations on Pinnacles. This information may be used by CoRC in advertising and report documents, future proposals, and/or other materials related to research computing at UC Merced.

CENVAL-ARC Node Use on Pinnacles

In addition, for those who also use cenval-arc nodes, please add the citation below to support NSF grant (#2346744). An example acknowledgement section is:

This research [Part of this research] was conducted using CENVAL-ARC compute resources on the Pinnacles cluster (NSF #2346744) at the Cyberinfrastructure and Research Technologies (CIRT) at University of California, Merced.

Research Facility

UC Merced's research computing infrastructure called the Borg Cube, is a 1,448-square-foot modular, data center-style research computing facility. The facility includes twenty 51U racks, providing up to 1,020U of rack capacity, and 400 kW of N+1 redundant power capacity at the rack bus. The facility is supported by redundant electrical distribution, including two 500 kVA power distribution units, dedicated 500 kVA uninterruptible power supply units, and two 800A Starline busways to support single- and dual-corded equipment.

The Borg Cube is cooled by three 60-ton indirect evaporative cooling units. Two units are sufficient to support the facility load, while the third provides operational redundancy for maintenance and continuity of service. The facility also includes clean-agent fire suppression, FM-200 tanks for each module, fire detection systems, and a very early smoke detection apparatus (VESDA) system. The facility is designed to support reliable operation of UC Merced's research computing resources and future expansion.

Cluster Hardware Configuration

Compute nodes: Compute nodes are where actual jobs run. There are three types of compute nodes on Pinnacles.

  • 49 Regular memory (RM) CPU nodes with 256GB RAM
  • 8 Big memory CPU nodes (bigmem) with 1TB RAM
  • 20 GPU Nodes
    • 8 GPU nodes with NVIDIA A100 GPUs
    • 8 GPU nodes with NVIDIA L40S GPUs
    • 4 GPU nodes with NVIDIA H200 NVL GPUs
CPU nodeRM nodebigmem node
Number of nodes404
CPU2 Intel 28 core Xeon Gold 63302 Intel 28 core Xeon Gold 6330
RAM256GB1TB
Node-local storage1TB NVMe Data Center Solid State Drive (SSD)1TB NVMe Data Center Solid State Drive (SSD)
NetworkConnectX-6 VPI adapter card, HDR 100 InfiniBand (100Gb/s) and 100GbE, single-port QSFP56, PCIe3/4 x16 SlotConnectX-6 VPI adapter card, HDR 100 InfiniBand (100Gb/s) and 100GbE, single-port QSFP56, PCIe3/4 x16 Slot
gpu GPU node
Number8
GPU per node2× NVIDIA A100 PCIe GPUs, 40 GB HBM2
CPU2x Intel 28-Core Xeon Gold 6330
RAM256GB
Node-local storage1TB M.2 NVMe Data Center Solid State Drive (110mm)
NetworkConnectX-6 VPI adapter card, HDR-100 IB (100Gb/s) and 100GbE, single-port QSFP56, PCIe3/4 x16 Slot
cenvalarc CPU Nodescenvalarc.compute - CPU Nodecenvalarc.bigmem - bigmem nodeOSG*
Number of nodes944
CPU2x Intel 32-Core Xeon Gold 6430 2.1GHz - 270W. Or 2× Intel Xeon Gold 6530, 32-Core, 2.3 GHz, 225 W.2x Intel 32-Core Xeon Gold 6430 2.1GHz - 270W2x Intel 32-Core Xeon Gold 6430 2.1GHz - 270W. Or 2× Intel Xeon Gold 6530, 32-Core, 2.3 GHz, 225 W.
RAM256GB1TB256GB
Node-local storage1TB M.2 NVMe Data Center Solid State Drive (110mm). Or 2× 960 GB 2.5″ NVMe PCIe SSDs per node1TB M.2 NVMe Data Center Solid State Drive (110mm)1TB M.2 NVMe Data Center Solid State Drive (110mm). Or 2× 960 GB 2.5″ NVMe PCIe SSDs per node
NetworkConnectX-6 VPI adapter card, HDR-100 IB (100Gb/s) and 100GbE, single-port QSFP56, PCIe3/4 x16 SlotConnectX-6 VPI adapter card, HDR-100 IB (100Gb/s) and 100GbE, single-port QSFP56, PCIe3/4 x16 SlotConnectX-6 VPI adapter card, HDR-100 IB (100Gb/s) and 100GbE, single-port QSFP56, PCIe3/4 x16 Slot
Assigned Nodesnode[074-076],node[081-086]hmnode[007-010]node[077-080]
note

* OSG Nodes are only accessable via test partition. To learn more information about the Open Science Grid (OSG) can be found here.

cenvalarc.gpu GPU nodeL40S NodesH200 Nodes
Number of nodes84
GPU per node2× NVIDIA L40S (48GB GDDR6)2× NVIDIA H200 NVL (141GB HBM3)
SLURM GRESgpu:l40s:2gpu:nvidia_h200_nvl:2
CPU2× Intel 32-Core Xeon Gold 6530 2.1GHz - 270W2× Intel 32-Core Xeon Gold 6530 2.1GHz - 270W
RAM256GB256GB
Node-local storage1TB M.2 NVMe Data Center SSD (110mm)1TB M.2 NVMe Data Center SSD (110mm)
NetworkConnectX-6 VPI, HDR-100 IB (100Gb/s), single-port QSFP56, PCIe3/4 x16ConnectX-6 VPI, HDR-100 IB (100Gb/s), single-port QSFP56, PCIe3/4 x16
constraint--gres=gpu:l40s:<number of gpu>--gres=gpu:nvidia_h200_nvl:<number of gpus>
Assigned Nodesgnode[017-024]gnode[026-029]

How to Request an Account

UC Merced Faculty Principal Investigators (PIs) can request access to Pinnacles cluster. All student user accounts on Pinnacles cluster must associate with UC Merced PIs.

UC Merced Principal Investigators (PIs) or other researchers request Pinnacles account here.

Click Here to View a Visual Guide for Creating an Account for Pinnacles

Requesting Access to Pinnacles Process.

  1. UC Merced Principal Investigators (PIs) or other researchers request Pinnacles account here.
    1. For new account group project applications, PIs please also make sure to complete the export control form, if the PI has not done one before.
    2. Once the form is completed, please attach the form to the request ticket scene in the following steps.
  2. Click Request Service Image of Request Service
  3. Begin to populate all the required information.
    1. At the question regarding PI Status. Typically only Professors are PIs, their students and post-docs would select No at this question. Image of PI Selection
  4. For selecting the system, from the drop-down, click pinnacles.ucmerced.edu (Free Cluster) Image of Pinnacles Selection
  5. Add any other additonal comments or information, you believe will be helpful for the requesting an account process.
  6. Click Request Service Submitting Ticket

Centralized login

Open OnDemand Login

For users seeking to access access Pinnacles and MERCED cluster via the web-based GUI, Open OnDemand. Please refer to this page here for accessing and making the most of the Open OnDemand Interface.

note

If connecting via SSH to the clusters from on campus, connect to eduroam or UCM CatNet. Otherwise ensure you are connected to the Campus VPN.

For users who want a traditional terminal experience, SSH login is the standard method to connect to Pinnacles.

SSH Login to Pinnacles

To login to Pinnacles using SSH, open a terminal and run ssh <username>@login.rc.ucmerced.edu, replacing <username> with your UC Merced NetID (UCMID). The full command is:

ssh <username>@login.rc.ucmerced.edu

Note: the command starts with ssh followed by your username and the hostname login.rc.ucmerced.edu.

The standard method for connecting to a remote machine is through Secure Shell (ssh) commands. Pinnacles and MERCED are accessed via a centralized login node at login.rc.ucmerced.edu. This means that once a user logs into one of the login nodes, they will be able to access both the MERCED and Pinnacles clusters. Users applying for a Pinnacles account can begin the application process here, and Pinnacles is FREE to use within the campus. However, to access the MERCED cluster, users must provide a COA account number and enter the number during the MERCED account application process.

Currently, we have three login nodes, and users can expect to be connected to either rclogin01, rclogin02, or rclogin03. Do not run computationally intensive processes on the login nodes. These nodes are appropriate for tasks such as file preparation/editing, compiling, simple analyses, and other low-computation activities. For more resource-intensive work, submit jobs to the cluster using the available queue system. Additionally, users can connect to a remote machine using an X-terminal (XQuarz or X11) forwarding (see example command below) to run graphics-based programs like gnuplot, gimp, etc.

Connect to the clusters

On Mac and Linux you can use the built-in terminal application; on Windows you can use MobaXterm to open a terminal, and type the following command, but replace <username> to your UCMID.

ssh <username>@login.rc.ucmerced.edu

X11 forwarding

tip

Mac OS:

Prerequisite: Install XQuartz Then open XQuartz through open -a XQuartz in the terminal or other CLI. Then type the following command.

ssh -X <username>@login.rc.ucmerced.edu
  • -X: Enables X11 forwarding.

Windows users

MobaXterm includes an integrated X11 server, so no additional installation of X11 software is needed.

  • Start a New SSH Session with X11 Forwarding
    • In the top-left corner, click on the "Session" button.
    • Choose "SSH" from the available options
  • Configure the SSH Session
    • In the Remote Host field, enter the address of the remote server (e.g., remote.server.com)
    • Ensure the "Specify username" box is checked, then enter your username for the remote server
    • Check the box that says "X11-forwarding". This option enables X11 forwarding for your session

File systems and storage

There are 2 folders (data and scratch) located in HOME that users will start with.

note

MERCED and Pinnacles have now been merged into a centralized system, allowing them to share the same file systems. We have also increased the quota for the data, scratch, and HOME directories. Please note that there is a 7-day grace period once the soft quota limit is reached.

Foldersoft quotahard quota
HOME70G75G
data500G512G
scratch500G512G
warning

The scratch folder is purged periodically when the overall system storage reaches 85% of capacity or higher. Please back-up your data to somewhere safe frequently.

Expanded Scratch or Data Allocation Pilot Program

Beginning August 2026, CIRT will launch a pilot program to provide additional scratch or data storage per user for research projects with demonstrated needs beyond the standard allocation levels.

Investigators may request additional storage for their research group by submitting a brief justification describing:

  • The research project or activity
  • The amount of additional storage requested. Requests up to 4 TB per user within a research group will be reviewed through the standard process.
  • How the storage will support the research effort

Approved allocations will remain in place through July 31, 2027.

Requests should be submitted through the CIRT Expanded Storage Pilot Program request form. To be considered for the initial August 2026 allocation cycle, requests should be submitted through this form by July 1, 2026.

As part of this pilot, CIRT will review utilization, demand, and sustainability of expanded storage allocations. Investigators who continue to require additional storage beyond July 2027 may submit a request for renewal in May 2027 for consideration. As this is a pilot program, renewal is not guaranteed.

Allocation decisions will be based on research need, available capacity, and alignment with the intended use of the storage resources.

warning

Expanded allocations remain subject to all existing storage policies and expectations. Scratch storage is intended for active computational workflows and temporary research data and should not be relied upon for long-term retention, archival purposes, or backup. CIRT reserves the right to review, adjust, or discontinue pilot allocations if storage is not being actively used for the stated research purpose, if capacity constraints arise, or as the pilot program evolves.

Queue Information

Pinnacles Cluster is the default cluster that is free and accessible to all users and has 6 public queues.

Public Queues(Available to all users)Max Wall TimeDefault TimeMax Nodes per JobMax # of jobs that can be submitted
^test1 hour5 min.2 nodes1
bigmem3 days1 hrs2 nodes2
gpu3 days1 hrs2 nodes4
*short6 hours1 hrs4 nodes12
medium1 day6 hrs4 nodes6
long3 days1 day4 nodes3
cenvalarc.compute3 day1 day4 nodes3
cenvalarc.bigmem3 day1 day2 nodes2
cenvalarc.gpu3 day1 day2 nodes4
tip

short queue is the default queue for all jobs submitted without specifying which queue job must run on

^test queue has access to all node types use constraints to test on specific types. Ex:

 #SBATCH --constraint=gpu,bigmem

Access to GPUs also requires

#SBATCH --gres=gpu:X

Global Modules on Pinnacles and MERCED

Pinnacles and MERCED already come with a collection of global modules or softwares that do not need to be individually installed by the user. The module system allows for the loading and unloading of a specific module. Users will make use of avail, load, list, unload, and swap. A table describing each of these Modules options is given below.

tip

A complete guide to using modules can be found via man module.

CommandDescription
module availThis command lists all available modules
module load <mod_name>This command loads the environment corresponding to <mod_name>
module listThis command provides a list of all modules currently loaded into the user environment
module unload <mod_name>This command unloads the environment corresponding to <mod_name>
module swap <mod_1> <mod_2>This command unloads the environment corresponding to <mod_1> and loads to <mod_2>
Click Here to Expand to View the List.
   admin/0.0.1                            gaussian/gdv-20170407-i10+             mpfr/4.2.0                               r-biobase/2.50.0
amber/20-devel gaussian/gdv-20210302-j15 (D) mpich/3.4.2-gcc-8.4.1 r-ctc/1.64.0
amber/20 (D) gcc/8.5.0 mpich/3.4.2-intel-2021.4.0 r-deseq2/1.30.0
anaconda3/2021.05 gcc/11.2.0 mpich/3.4.2-nvidiahpc-21.9-0 (D) r-edger/3.32.1
anaconda3/2023.09-0 (D) gcc/12.2.0 (D) multiqc/1.7 r-fastcluster/1.1.25
angsd/0.940 git/2.37.0 multiwfn/3.8 r-glimma/2.0.0
apbs/3.4.1 glpk/4.65 mvapich2/2.3.6-gcc-8.4.1 r-goplot/1.0.2
awscli/1.16.308 gmp/6.2.1 mvapich2/2.3.6-intel-2021.4.0 (D) r-goseq/1.42.0
bamtools/2.5.1 gnuplot/5.4.2 ncbi-blast+/2.12.0 r-gplots/3.1.1
bamutil/1.0.15 grace/5.1.25 netlib-lapack/3.9.1 r-qvalue/2.22.0
bbmap/39.06 gromacs/2021.3 netlib-xblas/1.0.248 r-rots/1.18.0
bcftools/1.12 gromacs/2022.3 nvidiahpc/21.9-0 r-sm/2.2-5.6
bcftools/1.14 (D) gromacs/2023.1 (D) octopus/13.0 r-tidyverse/1.3.0
bcl2fastq2/2.20.0.422 gsl/2.7 octopus/14.1 (D) r/4.1.1
beast/1.10.4 gurobi/9.5.0 onnx/1.10.1 r/4.2.2 (D)
beast2/2.6.4 hdf5/1.10.7-intel-2021.4.0 openbabel/3.0.0 raxml-ng/1.2.0
bedtools2/2.30.0 hdf5/1.14.1-2 (D) openblas/0.3.18 rclone/1.59.1
berkeleygw/3.0.1-intel-mvapich2 ibamr/0.8.0-testing openblas/0.3.21 (D) repeatmodeler/1.0.11
berkeleygw/3.0.1-intel-2021.4.0 ibamr/0.12.0-debug opencarp/8.1 rsem/1.3.1
berkeleygw/3.0.1 ibamr/0.12.0-opt openjdk/1.8.0_265-b01 salmon/1.4.0
berkeleygw/4.0-mvapich2-oneapi (D) ibamr/0.13.0-debug openjdk/11.0.20 samtools/1.13
blast-plus/2.12.0 ibamr/0.13.0-opt (D) openjdk/17.0.5_8 (D) scalapack/2.1.0
bowtie/1.3.0 intel/oneapi openmpi/3.1.3-gcc schrodinger/2022-1
bowtie2/2.4.2 interproscan/5.55-88.0 openmpi/3.1.6-gcc-8.4.1 schrodinger/2022-3 (D)
braker/2.1.6 ior/3.3.0 openmpi/3.1.6-intel-2021.4.0 sickle/1.33
butterflypack/2.0.0 iq-tree/2.1.3 openmpi/3.1.6-nvidiahpc-21.9-0 singularity/3.8.3
bwa-mem2/2.2.1 jellyfish/2.2.7 openmpi/4.0-merced-test smalt/0.7.6
bwa/0.7.17 julia/1.7.3 openmpi/4.0.6-gcc-8.4.1 sombrero/2021-08-16
casacore/3.4.0 julia/10.1.1 (D) openmpi/4.0.6-intel-2021.4.0 spiral/8.2.0
cgal/5.0.3 kallisto/0.46.2 openmpi/4.0.6-nvidiahpc-21.9-0 srilm/1.7.3
cmake/3.21.4 lammps/20210310+kokkos+cuda openmpi/4.1.1-gcc-8.4.1 stacks/2.53
cmaq/5.3.1 lammps/20210310+user-omp+kokkos openmpi/4.1.1-intel-2021.4.0 star/2.7.11b
collier/1.2.5 lammps/20210310 openmpi/4.1.4-gcc-12.2.0+cuda stata/17
cuda/10.2.89 lammps/20220107+ml-quip openmpi/4.1.4-gcc-12.2.0 (D) stata/18 (D)
cuda/11.0.3 lammps/20220107 orca/5.0.1 stringtie/2.2.3
cuda/11.4.0 lammps/20230208 (D) orthofinder/2.5.2 subversion/1.14.1
cuda/11.5.0 latte/1.2.2 perl-db-file/1.840 suite-sparse/5.13.0
cuda/11.8.0 lftp/4.9.2 perl-uri/1.72 tcl/8.5.19-gcc-8.5.0
cuda/12.3.0 (D) libraries perl/5.34.0 terachem/1.95
dakota/6.12 libtirpc/1.1.4 phyluce/1.6.7 tk/8.5.19-gcc-8.5.0
dalton/2020.0 libxc/5.2.3-gcc-12.2.0 picard/2.26.2 toolchain/scientificstack-11.2.0
elpa/2021.11.001 likwid/5.2.2+cuda pigz/2.7 transdecoder/5.5.0
emacs/27.2 likwid/5.2.2 (D) plink/1.90-beta-7.1 trimmomatic/0.39
express/1.5.2 localcolabfold/1.5.1 protobuf/3.18.0 trinity/2.12.0
fastqc/0.11.9 mathematica/12.3.1 py-numpy/1.21.4 trinity/2.15.1 (D)
ffmpeg/4.3.2 mathematica/14.0.0 (D) python/3.8.12 user-modules
fftw/3.3.10-gcc-8.5.0 matlab/r2021b python/3.11.0 (D) vcftools/0.1.14
fftw/3.3.10-intel-2021.4.0 (D) matlab/r2023a quantum-espresso/6.7-intel-test vmd/1.9.1
gate/9.0 matlab/r2024a (D) quantum-espresso/7.1 vmd/1.9.3 (D)
gatk/4.2.6 metis/5.1.0 quantum-espresso/7.2-gcc-openblas (D) wannier90/3.1.0
gaussian/g09-d01 minimap2/2.14 r-ape/5.4-1 xcrysden/1.6.2
gaussian/g16-b01 molden/6.7 r-argparse/2.0.3

Where:
D: Default Module



Checking disk quota and usage

To look at your current usage amounts of HOME, data or scratch use the following command

quota -vs 

This will output, in sections, the filesystem, current space usage, quota, hard limit, and other relevant information in a more readable format.

tip

To convert the outputted megabytes to gigabytes = space(MB) divided by 1024

Checking the size of directories and content

To check the size of the current directory or any directories in it use the du command.

note

du command alone will output all directories, hidden as well, in real time so it will take a few moments to finish. It is recommended to execute the command with some of the following options to make the process more clear and concise.

OptionDescription
-hDisplays storage values in human-readable format (e.g., KB, MB, GB)
-sSummarizes the size of the whole current directory
-shShows the size of the specified sub-directory
--max-depth=NShows the sub-directories up to depth N, where N is a number representing the max depth
--allWrites counts for all files, not just directories
--helpDisplays all other options for the du command

Example usage of du command:

du -h -s <directory name>
warning

Users who submit jobs to MERCED and use the unified storage are expecting slower network communications.

  • Home - Shared over 10G network from Pinnacles to Merced, connected over IB on pinnacles.
  • Data - Shared over 10G network from Pinnacles to Merced, connected over IB on pinnacles.
  • Scratch - Shared over 10G network from Pinnacles to Merced, connected over IB on pinnacles

Please avoid writing files directly to /tmp on the head node, as this can fill up disk space and cause issues for all users. Instead, use your personal scratch directory for temporary files. Some programs may default to using /tmp, so ensure that the appropriate scratch directory is properly configured for your code.

note

Disclaimer: Users are responsible for backing up all data stored on the clusters and are fully accountable for its availability. CIRT is not liable for any data loss in the event of accidents