GSE Server

GSE Server#

General Info#

Access: If you would like to get access, please write an email to gse-server-citg@lists.tudelft.nl, and we will add you to the users. For any questions regarding the server get in touch with Joost, Jan, or Denis. A good place to ask questions is also the #Computational channel in the GSE-Official Teams.

Please note that the GSE Server is NOT a replacement for DelftBlue and serves another purpose. Do continue to use DelftBlue for large (distributed) jobs that run for long times. Our servers are meant for interactive computing (no queuing system), testing out things, trying things with heavy hardware requirements (e.g., many-core runs in one application), and so on.

DelftBlue has an excellent documentation. Many things in their documentation also applies to our own server: https://doc.dhpc.tudelft.nl/delftblue.

Hardware Configuration#

  • 2 CPU only servers:

    • maui.citg.tudelft.nl

    • palmyra.citg.tudelft.nl

  • 2 CPU + GPU servers:

    • oahu.citg.tudelft.nl

    • samoa.citg.tudelft.nl

  • Each server consists of 2 AMD Rome 7742 processor each with 64 cores: in total 128 cores (256 logical)

  • Each server has 1024 GB of memory

  • Each server has 50 TB directly attached storage at /{servername}/data/
    => This directory is not backed-up, but uses RAID 6 and can survive 2 disk failures. Therefore, move your critical results/files to your project directory, which you can find mounted at /tudelft.net/.

  • The GPU servers have each 2 additional Nvidia A100 80GB GPU’s included

  • Login name is your NetID and password, home directory is your NFS mounted home directory. This home directory has limited space (8GB), use the /{servername}/data/{username} directories to install your stuff.

  • Speed of disks:

    $HOME (/home/nfs/NETID/
write : 166 MB/s
read  : 257 MB/s

Direct attached: /palmyra/data/jthorbecke
write : 656 MB/s
read  : 2.2 GB/s

local /scratch (samoa/oahu)
write : 1.2 GB/s
read  : 1.7 GB/s

local /scratch (palmyra/maui)
write : 5.3 GB/s
read  : 4.9 GB/s

NFS attached: /palmyra/data/jthorbecke from maui
write : 272 MB/s
read :  1.2 GB/s

Software#

  • Applications are installed with spack

    • spack list for available packages

    • request the admins (gse-server-citg@lists.tudelft.nl) to install additional packages, or if the package is very specific install on your own space /{servername}/data/{username}/

  • Applications are loaded with modules:

    • module avail for current installed modules.

    • modules are based on Lua and is hierarchical: for example module loading gcc/13.1.0 will also show additional packages that depend on and are compiled with the latest GNU compiler version 13.1.0. This is currently the only hierarchy we have build in.

  • NOTE: At the moment spack is only installed on palmyra, once we have nfs shared the attached storage to the other servers it will become available to all 4 servers

  • Feel free to install your own software on /{servername}/data/{username}/

Rules#

  • You can create your own data directory on /{servername}/data/{username}/
    Use /{servername}/data/{username}/ as your regular working directory

  • If IO is important in your program use /scratch/{username}/ for local fast storage that you program needs to read in or write out.

  • Store important files (small) / scripts / source code on your NFS mounted $HOME

  • There are no rules and use the system as you like, but be nice to the users of the system

    • OMP_NUM_THREADS is default set you 8.

    • Start your jobs with nice, e.g., $nice your-commands-here to lower the priority of your job (particularly if you use many cores).

  • Avoid, if possible, connecting to the cluster through VS Code.

  • For computational intensive non-development jobs should can use DelftBlue

    => for example: modelling seismic shots, a long reservoir modelling, processing data, …, things that take several days to complete.

Guidelines#

  • The first time you login (each time after not using it for a while) it can take 3-4 seconds before your are logged in. This is because the NFS auto mounter has to mount your home directory to the server.

  • The NFS mounted $HOME directory has a storage limit of 8 GB (quota command to check your storage)

  • In /tudelft.net/ you can find project and staff-* directories

Contents