Scrollable/Clickable Transcript
And we can go ahead and get started. And I'm sure it's going to be a great presentation. The links are here and you can reach out to xsede. We're committed to providing training events that foster inclusion and show respect for all. And with that, I will go ahead and hand it off to Mary Thomas, who will be sort of the manager of proceedings take away, Mary. . Mary you're muted. I am muted. If not, I will share. Show you guys what you are going to have to do a bit differently on Expanse in there are some differences. we're now using CentOS, but we're using SLURM. Move on to the next slide, please. And I'm not sure that it got posted to the website yet, but we'll start with Mahidhar he'll talk about an overview of Expanse to talk about SLURM. And then we'll have a short break and then we'll talk about modules and another break and then job charging and using the Expanse portal, a new a new application. Running on Expanse and interactive computing. We'll have people that are working and supporting the speakers to answer any questions you might have. Start by sharing and hopefully the right desktop. Can everyone see the slide. It looks good. So yeah, I'll start off with a Expand system overview and Then go into in the second presentation, I'll go into how to run jobs on Expanse So the outline of the talk, essentially. Just introducing the system to look at the system architecture specifically look at the AMD EPYC processor architecture of the hardware details the NUMA options and Basically how how to use them and then Look at some of the innovative features in Expanse itself. And then talk about allocations and Summarize what we look at so Expanse basically is a result of an NSF award. A category one capacity system. The PI of the project is Mike Norman and a lot of us co-PIs there. And then Basically, the primary vendor for the system is Dell, Aeon for the storage. The Intel processors on the GPU node. So it's a System that's made up of 13 scalable units and I'll go into a little detail on each scalable unit in the next few slides. There are 728 standard compute nodes, 52 GPU nodes, and each GPU node has four GPUs. We have four live memory nodes. In terms of the data we have a 12 petabyte Lustre file system, which can do about 140 gigabytes per second on Rights and 200 kilo 200 K io operations per second on the metric data side. Actually, it's much bigger than that and then As we've heard on a lot of the SDSC systems. For fast IO local storage. We're essentially going to restructure some existing storage. So quick overview of the compute side and the system itself. And based on the benchmarks. Close to I mean more than 2X throughput over Comet. Most of the applications will see a per core improvement or a Haswell, we see anywhere between Even performance with per core going out to almost 1. And then given the 2X core counts. The throughputs easily want to go 2X, we expect a smooth transition going from Intel to the AMD processors, we've compiled and run most of the common software packages. Lot of benchmarking and it seems good so far. Running and compiling and Right now, the system is in early user period, as I mentioned, we expect product production in November. Which I mean, should be pretty early November. And the operations scheduled for five years. So let's go into the rack-based scalable compute unit so As I mentioned, basically 13 scalable compute units over all we're looking So let me go to the next slide. So each SSCU you is essentially designed To have full bisection bandwidth within the rack. We also will have for GPU nodes in on SSCU and all of its connected through with HDR switch HDR switch which has It's a 200 gig HDR switch. So what that does is gives you the 60 connections down to the nodes and then 20 and then 200 gigs going up. They are 128 Zen 2 CPU cores essentially And It features PCI Jen for 256 gigs of RAM on each node and That's DDR for 3. Something to keep in mind when you're running is to use the local scratch if you if you have an AI ops heavy workload. On the GPU nodes. 1. And then going out to the network diagram you can see basically you have 56 compute nodes, four GPU nodes, and then the uplink of 10X 200 gigs. Overall the network plugs into a lot of the external fast networks listed out there so Which is similar to what we had on Expanse but we have a 25 gig connection on each compute node, so you can get quite a bit of external connectivity. So next let's look at the AMD EPYC processor architecture. So it has eight core complex dies. And there is a IO die in the center, which is at 40 nanometers and CCDs or seven nanometer. Pack in a lot more cores and And I through the IO die there are multiple ways of kind of booting up this node in terms of NUMA domains. So each of those core complex dies. Picture on the next slide is one of these green CCD CCDs in the diagram. There's a lot of L3 cache on this machine insights 256 megabytes of memory cache total Each core also has private L2 cache of 512 kilobytes, and So, so now you can kind of see that there is a hierarchy setup. CPUs split off into the CCDs and then each CCD has two core complex dies with the I mean with the shared L3 caches. You can essentially do NUMA domains on each socket. And the way this works is the processor is partition into for NUMA domain. This lets you if you do the binding right you can kind of get scaling out to all for like the full socket. Because of the memory channel has been split this way. The PCI devices will be local to one of these NUMA domains and So the network card could be on one of them. So the main thing to keep in mind is that there is some complexity in the NUMA domains. So there are several compilers available on the system. Right now, we don't have any defaults because what we found is depending on the applications different sets of compilers work better for some and worse for others, kind of, so you know you should like test out things For the applications we are installing ourselves we primarily stuck to the GCC compilers and then used AOCC, where appropriate, where it was better performing basically This is a moving target as what I would say because there's a lot of optimizations going on these processes and we are involved with a lot of different sites which are running AMD processors and also with the AMD themselves developing application recipes. So as I mentioned, at runtime. And we are actually developing some have some tools to help out. Especially for pthreads codes and for hybrid MPI open and being cases. So the type of applications we run on Expanse, we've gone through quite a few benchmarking efforts basically run ChromeX, NAMD, Neuron, Openform. Quantum espresso RAXML. And so you can see there's a broad range of applications that we have tested and kind of different application areas, all kinds of combinations. On the GPU side we have Benchmarks on the AMD. MD code codes like NAMD and Amber and also a lot of the machine learning kind of codes. What we've seen is greater than 1. So that actually brings me to the GPU node architecture. The host node itself as basically 384 gigabytes of RAM and 1. And if you look at the topology. The numbering is already even in terms of all the even numbers are on one socket and the odd numbers on others. Numbering there and then You have NVLink between the GPUs. CPU and through across, but now with NVLink should have much better performance. So in terms of software stack will support a pretty broad application based we are trying to replicate what's on Comet with a lot of the packages, we Commonly used packages like bioinformatics, molecular dynamics, machine learning. You may not have your favorite package on day one, but we can definitely work to install as people come online. Today's talk, basically we are aimed at informing you about like basically the difference is going between common to the Expanse application environment Marty's going to talk a lot about Modules, and the setup. For those who are curious. Port and develop a lot of HPC software stack and AMD has its own Fork of the Spack base and they are putting in performance. Application. You know packages into that. We will continue to support singularity based containerization on Expanse, just like on Comet. So some other Expanse, in addition to the standard way of You know, standard compute nodes and GPU nodes and the SLURM based scheduling also has a few innovative features. It of course needs a lot of work. But we have demonstrated this with work on Comet and we're going to replicate that on On Expanse basically you will be able to use SLURM to submit your job and then there's a whole infrastructure to make that work seamlessly so that you end up running the job on a cloud resource. Early work was kind of used for Running CIPRES jobs from the gateway. This approach is cloud agnostic and kind of aimed to support most of the cloud providers. So the So there are options behind the back end. Lots of different things to look at. Similarly, we are also looking at composible systems. You may have some pieces that are available on Expanse some pieces that are available on a public cloud, or a different resource. And what we what we are trying to do on Expanse is to be able to carve out some of the nodes using and set up a kubernetes cluster on that and and Basically federate with other communities. Resources outside of the system. needing like a little more advanced support. As part of an XRAC request and also make sure That there is a case to do it. we can definitely make this happen. So then in terms of the rest of the support mechanism. You can have the support coming through the XSEDE ticketing system, as you had on Comet In terms of transitioning, we have an overlap of six months between Comet and Expanse or rather it's five months, but quite a bit of overlap. And today's workshop is essentially the first in a set that basically will help you transition from Comet to Expanse As I mentioned, you have advanced support available for cloud integration and composible systems. And we'll support. such projects. So in terms of allocations. Expanse will show up as a resource right now and XSEDE XRAC window. There's the Expanse CPU, basically, which is the allocations on the compute part of the system Expanse GPU, which is basically allocations on the GPU we 100 parts of the system. These are locations on the Lustre file system essentially and the file system is available on all the nodes of the system. That we can do on the project side is At least at this point is around 5 petabytes and note that we will have a per project limit of 50 terabytes. You know, So in terms of,. for basically files that you can expect to need for the duration of a project. I have a few snapshots of the allocations page basically how you can log in and where the Expanse options are basically they will see the Expanse Dell cluster with AMD Rome and then the GPU and the storage piece. So to summarize, basically Expanse will provide a huge increase in performance and throughput compared to the Comet machine and it's essentially an evolution of the common design. Some innovations in cloud integration and composible systems and some of that work was started earlier and kind of continued to evolve into the production side of Expanse essentially. We will have integration with open science grid. And we looked at the, the architecture of the system basically the 728 compute nodes and 52 GPU nodes and the HDR100 interconnect on the nodes. As I mentioned, it's going to go production in a couple of weeks, essentially, and you can follow all things Expanse at this website. Almost 11"30 so we can probably do a few minutes of questions. Let me see if I can pick out the chat. I'm going to stop sharing and look at the chat. And Marty and Trevor have been handling it except for the last question. So Ron you had the thing that your projects, but the directory is different and another Yeah, I think the question about the project storage is it's possible. So, okay, it's possible if you're if you're already on the system. Of the early user Access Program. And I think Paul. And we have enabled sharing on that now in on specific directories. So Mahidhar. a Comet research allocation right now transfers to Expanse, I mean, I would just assume that like within a couple of weeks. Is that right. They're people have actually asked for Expanse explicitly in their allocations and because this has been available in an allocation cycle for a while. Comet users who who have an allocation that goes beyond the end date of Comet which is true for a lot of people. Request a transfer from Comet to Expanse, if you like to use Expanse and that is totally fine. I see in it. But we are not automatic transmission in every Comet allocation too expensive. Okay, good, good. So it's like a regular XSEDE transfer between resources. Okay, great. So I think there was a question of what whether there's a purge cycle for scratch. It's and the project spaces where you would keep something that you need for the longer term that is for the duration of Allocation I should say both the locations are not backed up. Alright so I think I will transition to the next talk which is running jobs on Expanse. Go to the front of this thing and start sharing again. So again, for completeness. again in this slide deck, just in case somebody picks up this slide deck offline. So I have a system overview and then we'll just look at the login info and after that what I'm essentially focusing on in this talk is looking at the SLURM scheduler. Then we'll look at just running jobs examples for MPI openMP and hybrid cases and GPU jobs. I'll go over the file systems, a little bit and also give you some example script. To use local scratch usage. So that was architecture thing I was talking about. quickly go over it in case somebody just joined. And then 52 GPU nodes with four V100s connected with NVLink. It's for a total of 208 V100 GPUs there is HDR 100 connectivity into the node and then the switches on the top of the rack for each of these racks rights that are basically HDR 200 switches. So let me skip through that. The allocations are different for CPU and GPU resources, just like on Comet so like when you get an XSEDE allocation, you would have a separate application for each however the login is the same so you can log in to login. So if you have an SSH client. You can just directly SSH using basically your username. I actually, I think I missed one question on the previous talk where somebody asked about if the allocation IDs will be the same. If your project is the same. Get information on What your allocations are so anyway and your user. So you can Directly log in and use your XSEDE portal password or And this will basically put you on one of the two login nodes login01-expanse. You can also log in via the single XSEDE single sign on hosts or login. You should be able to log in there and then GSA SSH to login. One other new thing on On Expanse is we will have an Expanse user portal. Basically submit jobs from that user portal. Programs through that and there's more details coming in Subha's talk there. You can use SSH keys. To do essentially enable access from Authorized hosts without having to enter your password but Make sure that you have a strong pass phrase on the private key on your local machine and don't copy that key around keep keep it on your local machine. You can use SSH agent to Avoid repeatedly typing the private key password on your end, but Whatever you do, don't do a passwordless thing because that's a huge security problem and And if you are For some reason, like to have a programmatic thing that connects Via SSH very frequently. Email us so we can give you an option that lets you avoid the problem and there is a security webinar that Scott did a little bit ago that kind of goes into some of these things from a Comet perspective what most of that applies on Expanse to So the first thing is the appropriate use of login nodes. Copying some small files. Remotely that's okay, but The main thing is you should use using minimal compute resources. Jobs on the login or even if they're meant to be short, because if you do that not enough to slow the login not down so please avoid running on the login node. And then all Computationally demanding jobs, even if they're interactively run should be going through the queuing system. So don't use it for anything computationally intensive Don't use them to run some intensive workflow management tool because that's going to cause problems too. And essentially If you're doing some data transfer. And do not run Jupyter notebooks on the login node and Mary's going to talk a little bit about how to Run those appropriately and also don't set up servers on the login node that provide services to the outside world, because that could lead to issues to. The one other big difference, I think, between Comet and Expanse is The GPU nodes. Since this whole CPU is different. So that's another reason to essentially get a GPU node allocation and then do the compiles there. So in terms of scheduling so Expanse uses SLURM, just like on Comet. limits and so on. But the usual SLURM commands will work. I encourage people to use the -u option so that you're not querying everyone's jobs at the same time. That thing slows things down and also causes an additional load on the scheduler, it's easier for it to deal with specific requests. And if you want to cancel a job, you can use the scancel command. So let's look at what similar to Comet and what's different from Comet in terms of the partitions. So in terms of what's common with Comet, like you have the compute partition, which is essentially CPU nodes exclusive access One real thing to keep in mind is If you make a mistake and put a shared Like two core job on a compute node there's a huge cost to it. Make a mistake and run on the compute partition. It can lead to A much bigger penalty in this case because of the higher core count And we obviously can't refund that because the jobs, the jobs are blocking those resources when you do that. In terms of shared partition. And similarly, on the GPU side we have the GPU nodes with exclusive access in the GPU partition and then the GPU shared partition. And just like Comet, we have a large shared partition for the large memory nodes. And a debug partition for short tests and quick access So what's new on Expanse. So we do have a GPU debug partition. We also have preempt partitions, which basically discount Job a discount SUs used and so that he can run on the nodes if they're free. Minute, can I jump in here for a moment. Y'all, there's been a lot of activity on the chat and it can be a challenge for some of our own tracking both at the same time. Let me go back. I might have to stop sharing and go back to the thing. Let's see. So I think there was a comment about Switching to node hours on the SU definition. Because then you would end up with fractional SUs which I don't think we have set up to deal with right now. And then So there's a question about memory usage being enforced for shared nodes as it is, and I'm going to talk about that a little bit further down So there was a question about The transfers basically going between Comet to Expanse So if, if you're intending to transfer basically in the March timeframe. time on Expanse, since we are not automatically moving the Comet time right away. So yeah, I'm not sure how much more time, you need to spend on that machine taking a bit of for people to get caught up on. Oh yeah, I think hopefully everyone's caught up and yeah And Marty and Trevor have been doing a great job. So back to the partitions so So this slide basically and Nicole is going to cover this again when she does the charging aspect of it. Have To share your slides. Oh, Thanks. Alright, good. Looks good. Yes. Okay. So yeah, so this is a quick overview of basically the limits. So some of the Limits are big. The size of the jobs that typically run and we kind of put those in But they are flexible and if it turns out that some of these might need to change. And so that's quite a bit more than what you had on Comet in terms of number of cores per job that you can run And so you can have maximum of 64 nNodes running your particular groups jobs and the reason we have these limits is because 64 nodes is essentially nearly 10% of the system. So so similarly on the shared compute side we have the equivalent of 32 nodes essentially if you run once core or So you want single core jobs, essentially. For a GPU job. Remember, these are the V100 nodes which are quite a bit faster than what we had on Comet so equal to 8 nodes on the Comet side. How the queue at times look like. So, these, these will be busy I think. And then on the debug partition. We Deliberately kind of stuck a lower run limit on the number of jobs because We have noticed people kind of trying to game the debug queue, a little bit and us and roll through a lot of jobs will be our lower limit. And as I mentioned, we have the GPU debug partition now. So a few other things, basically, on the Expanse Scheduler side. We have a few required party parameters. You will need anumber of nodes and the number of tasks per node or the number of N tasks, right, one of the two needs to be specified. Specify that And The wall clock time It should be specified Explicitly and then are two other things. On Expanse, and then on the GPU side if you remember on Comet, we were setting --gres So on Expanse it so --gpu, which is the total number of GPUs needed by the job. One other critical thing to note and this goes to I think it's a question on memory. So --manda shirt CPU or GPU one of these parameters, they are not required, however, the default is quite low. per core. So in this version of SLURM is quite Strict with all the I mean the way of your configured. Like the topology of your request. So I'll start off with the interactive jobs and you can do is run just as you did on Comet and that will give you interactive access to the compute nodes and then you can run through the scheduler, whatever you need to So again, one thing that I added in red here is basically you need to specify the account. So let's go into some sample scripts to see if anything's different essentially compared to Comet, so simple Hello World MPI job here so If you look at the parameters. Most of them are pretty standard right you have the job name, you have the output file. Name of the first primary compute node so that it's easier to track down issues later for us, like so, we recommend doing that because if you have a problem with a job. It's nice. And then Yeah, you have the partition. Number of nodes and the task per node, in this case, we could do ntask if you want And I highlighted the memory per CPU. Per task and per CPU. I'm not going to go into the environment piece of it so much, but since Marty is going to talk about it. I think you should get in the habit of basically starting off clean and loading exactly what you need. So do this in your job script instead of, instead of your . The GPU nodes and CPU nodes are different stacks and you can't really load both of them in the . So, so in this case, basically. And that is needed if you want to use a command like srun in your MPI job, for example. And also for MPI runs because they are integrated with the scheduler. So you would need module load slurm, and then you the module load cpu here. So as I mentioned, it's different on both sides. So we are doing srun here to launch this MPI job. But one thing I should not. MPI launch mechanisms that are integrated with the scheduler. This is because on a shared node, if we do an SSH based thing it could get real complicated so sorry. The scheduler. That's integrated So the one other thing is, basically. So you can do partition equals shared, nodes equals one and task per node one and CPUs 16 know here I should caution. You could have done ntask per node 16 and then launched an openMP job and it would pick up the 16 cores, but on Expanse, you probably will end up bound to one core. So explicitly ask for what you need. Task, and then it is 116 threads. And then I'm basically doing a similar thing. Setting up the environment and then setting the openMP threads to 16 that matches the CPU task ask and then run the code. So for hybrid openMP and MPI jobs. As, as I mentioned, you're basically partitioning into for new model no domains in there. And the memory is interleaved on the two channels there. And we are developing an ibrun script which be available when machine goes into production. With option. scripts that You're developing so because they could make a pretty big performance difference if you cross NUMA domains in this architecture. So I have a couple of hybrid MPI openMP Examples. Correct. Okay, and you're doing fine. Okay, I think we're doing okay. Okay, so this is an example of a hybrid case. And so I do the export here and I'm doing a basically a pin domain on Option with the Intel MPI, which lets you say that, hey, you have an open MP set up here with the compact. Pinning That we want to do so that lets you kind of Lay out the tasks the right way. At least the ones we have tested so far is that it's good to align with the L3 cache that's been shared on the four cores. So if you can Do your openMP threads so that the four of those threads are on one CCX that will work well. This is another example, basically similar setup, but now I'm using OpenMPI so I'm doing a different thing. So the next, next thing I want to look at is GPU nodes. And then compare it to what we were doing. Which is different from Comet where you would have done --gres-gpus equals one And I'll talk about the multi node thing where the difference shows up. OpenSSC job on the GPU node. So the multi node job is a little different. Not necessary to do this. And then I'm basically setting the --gpus equals 16 so this is different. Something equals four. And then the rest of it is similar. So, The last piece I want to kind of go into is the file systems so When you log in, you're on your home file system. Basically $HOME will also get you there. The home directory is limited in space, like I said, it should only be used for like source code binaries and maybe small input files and You can have your job script here, but make sure that the job script is about that. You have a core of hundred gigs on the home file system. And can't stress this enough, don't run any intensive IO to or from the home file system. Like I said, you could have like simple, like, you know, standard out kind of output go there. In our file per core IO that does large scale stuff. So talking of Lustre. There are two locations on this one file system. The /expanse/lustre/scratch/$user, which we will purge for files older than 90 days based on create date and the Lustre projects location, which is the /expanse/lustre/projects/groupid/$user. So, What we are doing here is, since it's the same file system. And The other thing to keep in mind as we are limiting the number and this is true on Comet too or we are limiting the inodes to 2 million per user and This this limit is basically in place. Millions of files, Lustre is probably not the right location for it and you should look into changing the workflow to using the NVME that's sitting on the files on each node. And if your workflow does need intensive extensive small block IO like lots of lots of small files, you should contact us if you need help with using the local scratch. I mean, this particular Lustre file system has metadata on NDT and can handle a lot more Iops, so it's it's a little more better equipped to deal with the small file I O, but we still recommend using NVME wherever you can So that brings me to the local NVME best scratch file system, all the Expanse nodes have NVME local scratch storage that this sizes vary based on the nodes. That's OK. The GPU nodes have 1. This is excellent for like Iops intensive workloads, it's also pretty good and performance with large files too as long as they fit and the storage that is there. So if you have codes are generated a lot of files on a per task basis for example, openform us to do that. Generate five per core per variable. And just like on Comet this Directory gets created when your job starts. Key thing is this location is purged at the end of a job. Make sure if you're copying out and it's a million files. Or you can tar directly into Lustre. Probably not finish in the time that you have And will cause problems on those systems, right. So, This is an example. Your scratch directory /scratch/$USER/job_$SLURM_JOBID . The other thing to keep in mind is the /tmp is not that large on these nodes. Keys off the tmp directory and writes to the tmp directory, please set your tmp dir, TMPDIR variable to this location so that you pick up the NVME scratch. So I think I'm about five minutes out. All the jobs have to go through the scheduler don't run on the login nodes. We support both node exclusive, and shared partition, just like Comet. The account information is required and job scripts. You should Put in memory requirements explicitly also And then we looked at the file system options. As I mentioned, we are going in production soon so you will all get to get on and try this once, once the machines available in production. Thanks. unanswered questions on the I think the mighty. See Can you say something about dev sham and the size and speed. Let's see. Memory so that message. Let's see. memory bandwidth total on the node I think compared to About 110 on Comet or there's a big, big increase. You could end up with lower bandwidth per course or you really depends on what's going on your field and I shared on but on a regular compute node. In terms of size, well it's a 256 gigabyte RAM. Let's see, Marty. I don't think so. So instead of a signal to the job. I love to check on that. This shouldn't be. I mean, I mean, I think I can encourage you did. With with the systems group. So, so this is Trevor, I don't see the question to Cuba is the question about a signal for a printing Yeah, kind of like, though. So there is the ability for the user to submit their job and ask for specific signal to be sent when the job is going to be preempted. But there are limits to how early that can happen before present time occurs because The preempt grace, time is only five minutes, so you can't ask for a 10 minute notification because we only know INSERM that it's five minutes until preamp time so You know, you got to look at the settings on that and and figure out whether or not you can basically do all your work that you need to do to To check on your job within the time limit that you have, you basically think of it as probably About three minutes of time limit from that signal until you could just have the note disappear on your feet. Um, but you can you can definitely register your job when you submit it to receive a signal if it's going to be preempted. And that will come as soon as firm decides it wants that node and then you'll get some some subsequent signals again later on. Doable I'd say I don't think many people are using preamp right now and probably aren't even using that feature because the users that are using presenting early user don't really care if the know disappears. They've got workflow managers that deal with all that kind of stuff. Hey, folks. Break. Where Marty Kandes is going to talk about modules compiling and basic optimization for CPU and then GPU. One quick addition to answer. There was a gentleman said to half of the total memory so 128 gigs. Okay, thanks. I have a couple comments, then Okay, I started the recording. Okay everybody, welcome back. We typically post the chats. At the end of the talks with that out. Alright. Full screen here. You Can go ahead and get started figure this out in a second. wanted to go over what the module environment on Expanse looks like and how it's different from Comet. apologize in advance that I have not completely fleshed out the, the, all the material for the talk today. What module what the module environment does for you and how it's useful and sort of give you some best practices of how to use it because In my experience some users are a little inexperienced with setting up their environment. Sort of basic material that you can, you know, if you're not familiar with what the module environments doing it'll give you some better context of why we're suggesting you use it in a certain way and how to go about compiling code and then also running code as well so So I'll go over the module environment. material and then sort of use the second half of the talk to do some interactive demos and show you how, you know, In real time how you can, you know, take say some code that you've compiled on Comet if you have certain, you know, if you've Documented sort of your build process there and how you can, you know, modify it to then compile your code on Expanse, and how the different sets of software stacks that Mahi talked about are presented to you in the module environment itself. So let me close this chat so I can pay attention to the slides here. So I want to start with some basic definitions, though, for, for people who who might not be familiar, so When you log into Expanse or Comet right the program that's actually running when you log into your terminal is the shell program right and so This is the fundamental interface that you use to interact with the system. To the system to get your work done right. Your bash shell and this is what's used by, I would say, majority of users on the system. Would be, then you need to contact us via the ticketing system with the change request essentially this default needs to be set on the system side for each user So when you log in and your shell starts up, you have a bunch of built in commands that you're probably familiar with. You know, the CD command to change directories that make directory to create directories, you know, the list commands the ls command to list the files and directories. In your home directory or whatever directory that you're working in. And when you are working on Expanse or Comet you know many people often have multiple Terminal windows open up and these are both, for example, I'm running two different terminal windows here on my local desktop in this image. Where these are these different shell sessions are individual instances of the shell program running That has you know its own each, each one has its own state and environment that is set right so these although these two windows look completely same right there might be differences between the two shell environments, depending on what commands. Now there are it when you're working with the shell. String that you can attach a value to and they can have different types. When compiling code. Two separate shell windows open and you've defined a variable in one and not in the other while you're sort of maybe doing some deep interactive debugging or compiling your code. Those environments might be different. And so, for example, one of the things that show environments we usually use for is for, you know, very temporary data that might change pretty quickly. One example of this is the present working directory environment variable. I'm just, you know, Echo, I'm logged into Expanse I'm echoing the preent working directory. Directory for my my username, then the president working directory changes to that right, these, these variables are used to, you know, track the changes that are going on where you are in On the system and so that the shell knows you know when you run an ls command. This is what some of these definitions that you should be aware of. Now environment variables are sort of the key to what the module environment is going to be controlling for you. And environmental environment variables are just shell variables that are sort of export into what we call the shells environment. Other processes use or commands that you are running through the shell program to see these environment variables. The shell variables are only visible to the shell program itself, environment variables are then visible to the other you know applications binaries. Other commands that you might be running from the shell to, right and so Effectively also one of the main uses is the environment variables are used to sort of configure how your shell responds to the commands that you issue. And you know I would say in general environment variables are unlike regular shell variables, environment variables are used to store. More persistent data that isn't going to change much during your shell session. Or whatever program you're running, you're probably going to want to know where your home directory is on the system. Now the when the shell starts up there and you log into this when you log in the system, the shell and it goes this sort of initialization phase on startup right And it's usually a multi step process that actually there's some configuration files that some of you are probably familiar with that are looked at by the shell and use to configure your environment at start right so there's going to be a default set of environment variables and shell variables that are in your in your shelves environment when you start up And I won't probably go through all of the different initialization files that you can use, but I'll cover the high level ones that we Are most commonly used by people. So these are launched basically by by the operating or the shell looks from these configuration files. We control for you and you know may put things in there from time to time to simplify sort of your default environment on the system. Control what your particular shell environment looks like by modifying some of these other configuration files or creating these configuration files in your home directory on the system. So the two most common ones that people would modifies either the . users use on Comet and systems in general. The thing about the different. When you're running on the system essentially right so this . Non interactive bash shell starts. Your . Definitions and provide the context. Through these configuration files, you have to be aware of when they're going to be executed. And getting into trouble when for example they take you know they're doing one type of workflow for a certain set of Calculations or simulations and then they switch to a different code, the different application for a completely different set of work. How their . Even though I'm going over this material. To not use these configuration files. This is sort of a little bit more information that didn't change the title, the slides, but this is As I mentioned, there's different types of shells and when the different configuration files are executed depend upon whether it's a non login or login shell or an interactive versus non interactive shell And so I'll clean up these notes for you online. The, the key one is for when you're running a batch job script or a bash shell script in general is a shell script is always running in a non interactive shell. Right, and that's why this modifications to your . If you aren't if you're going to run multiple applications with different that require different environments on the system really don't want to modify Your . Overall, Perspective on the issues I see with with users using their . Now the module and environment. You'll see is mostly modifying both environment variables that are required for a certain piece of software. But the two most important things that is modifying is your path environment variable. Right, this is what is allowing when you run a command in your shell in your terminal right the operating system needs to know where to look for that command. So if you've compiled your application and you have the binary sitting in some directory in your home directory. Program about where to look for that primary it won't be found. The, the module environment. And this is probably the second most Modified variable that you'll see that the module environment is doing for you. The, the shell's going to look for when you have to, when you go to run an application. Again, just like just like the operating system needs to know where to look for the binaries. It's going to call out to right and so it's, again, this is what the module environment is actually doing for you, under the hood. You know as easily as possible to get you a simplified user interface to running the applications that you want. Now on Comet what we used was the environment module system. So this module system is one of the module systems that are out there that allow you to set up a nice command line driven interface to make all of these changes to your shells environment right And it's particularly you'll many people who are familiar with running HPC systems are running on HPC systems, you probably use environment module before. This is the environment module system that we've we've been using there and it's a little bit of an older version of what's been used before. But if you're curious about, you know the details right there's the link to the URL for the projects. And so the module environment allow it gives you the module system essentially gives you a set of command line tools to You know, navigate the software that's been installed on the system and is Manageable by the module system itself. Allows you to look at what modules are available or what modules are currently loaded on the system module avail can be run with or without this package input to look for what's available in the system, module purge allows you to sort of clear any modules that you've loaded And again, Which is the next commands. And then a module show is basically to get more information about the package itself. How the how that application may be compiled for example. So I have examples in here. And log into Expanse. on Expanse at least the ones that are going to be exactly the same as you ran on Comet. This Right. The only one, you really need You know, is the CPU one both which we'll talk about in a second. So the next one in my list was Module avail So for example, if you want to know what compilers are available on the system. And it'll give you a list of all of the GCC compilers or perhaps if GCC here in this example is shown in the name of the module name. It'll also appear here so you can see here we have four different versions of GCC shown here and two are actually the same version. In a bit here. Okay, the next one was probably module purge on my list. Right. So now I have a clean environment that has none of the other environment variables that may have been set by those other modules in the shell session. And so let's go through a quick exercise and show you how the module environment is, how the how the module system is actually making changes to some of those environment variables that I mentioned, like PATH and LD path, right. And my LD_library. Which has nothing right now. And so if I want to say compile code for For say the AMD compute nodes. Is Maybe let's do maybe a module show while we're here, I don't know if Trevor's actually put a description on here. 1. meta, meta module that you'll load to get access to all the compilers MPIs, math libraries that you should use for Expanse AMD compute nodes. And if you do a module avail, you can then see What packages are available. Packages on Comet for users. So the module load CPU 1. core set of packages. System Manager that we're using to sort of deploy all the system software. For the most part, for example, this is where when I did the module load GCC or module avail GCC. So one was coming from the Spack installed version, which you should use if you want to use GCC 9. You could use, but this is really for the system administrators at this point right now. And in the CM shared module environments. You'll also if you're using singularity, singularity is visible here in the in the CM local module environment. There's also in the user share, we have if you're using Globus, this is one that will likely stay in sort of visible module shown to you. Expanse user guide, there's also the, if you want to check your accounting in allocations. The sdsc/1. So personally, I think it's still a little confusing for some people but this is sort of if you're getting on if you're on the system. Some of these other modules and just want to indicate not to get confused if you're if you're trying to compile your application for either the AMD compute nodes or the Intel based Nvidia GPU nodes. You the two packages that you really want to focus on first loading is either the CPU 1. Okay, so you can see here that we have the CPU 1. Module loaded. The GCC 10. equals native it should hopefully pick that up that these are AMD compute nodes and use Zen2 nodes and use that as the Architecture compile against right so Just kind of clean this up. So as I said, right. CPU 1. And what we're going to do now is module load GCC 10. Right, it's that version of GCC is the first GCC found in your path. For example, if I do a module purge And then figure out which GCC right you'll see that you get the system GCC so GCC 8. Right. And so that is you know that's that's what the module environment is doing for you. If you have different applications. And you're going to compile them in different ways and then run them, you really don't want to put the modules load commands. In your in your . Or when you go run the code, right, those module loads have to be the same. Right, otherwise you're going to probably run into some headaches, if you, you know, forget to change your . Right. But maybe I'll stop here for any if there's any questions. No. I guess. They don't see the chat so Yeah. Trevor and you're doing a great job answering them. Okay. Good. Showing some examples of building code on Expanse and interacting with the module environment to do that in the changes that I would have to make this other package other piece of software that I've compiled before The last, the key thing that we need to point out is That the module environment or the model of system used on Expanse is different from Comet right so on Comet, we use the environment module system that I had a slide on before briefly. Which is, you know, one version of a module system that you can use to manage all these paths changes and access to software. Lmod is a newer system that was developed at TACC actually so And one of the things it does nicely, is it tries to help the users avoid mistakes that could be made with The environment module system that was available on Comet and, in particular, the way they do this is through a hierarchical module system essentially right where essentially if you want to compile a code with GCC tended to like we just showed. And it's an MPI code you need to then load an MPI module. You know, you could load a compiler Module one version of a compiler module and an MPI module that weren't compatible necessarily right So usually what we do on these HPC systems and once we cross compile we compile multiple versions of the MPI for a particular compiler. Intel MPI and MVAPICH2 to you really have to compile a version of each of those for each compiler that's available on the system. But the one problem with modules environment. Say that version of MPI that you wanted to use. Users would load multiple versions of MPI in there in the . Allows Uses a hierarchical system to avoid these types of problems where you can only load the types of MPI that are available for that particular compiler. Right. Now, The one I would say You know, the one thing that is going to be less familiar to you is, is how that happens like in practice, maybe I think a lot of people probably use Lmod on different systems. Open my terminal again. Okay, so And So what do I have loaded. Let me go ahead and do this right So if we do a module. Right, you'll see that there is no Spack modules anymore. And so that's why these sort of meta packages exist in the in in the management modules, essentially, right. That's what gives you access to the Spack based installed modules, right. So we see the meta packages load the correct Spack environment that you want for either the CPU or GPU nodes, depending on what you want to do. And then let's say we want to build an application with GCC 10. That these packages were built with GCC 10. A fair amount already of different applications in here because this as I mentioned GCC 10. Take advantage of. So let's go ahead and say this is an applications API application now to build and I want to use let's say OpenMPI Do a module avail and then there is another sort of nested environment, shown here. That there are already certain applications that were built here with both GCC 10. Right. Applications that are available to you from that combination of compiler and MPI. In using the parallel options to use MPI and it's being it's compatible with the GCC 10. One of the questions you'll probably have if you're not familiar with Lmod is Okay, yeah, this is great. But how am I supposed to know that that version of FFTw is available with, you know, GCC 10. Like, is it a bit like for example, your question might be isn't available for MVAPICH2 yet. The new module commands that isn't available in the old environment module system. This hierarchical structure for the packages, you need to load to get to the packages, you want So let's go ahead and run that example that I just mentioned of FFT W. If I clear this and do a module spider. FFTW You'll see a list of the different modules that are available. FFTW modules that are available and what modules, you have to successively load to get to it. In on the AMD compute nodes we have an FFTw a serialized version. And so that's how this. Okay, what software is available. Because it's going to mean you don't want to just start loading random compilers and my and MPI is to figure out which you know what Sort of mass libraries or other dependency, you might need to build your software, right. The figures out for you. Okay. Lmod because it's gonna be the one that you'll get the most bang for your buck right off the bat. Okay, so I'll stop here for a second. Know if anybody is using the chat. So Marty, just about now you should be looking at GPU. GPU. That you've allocated for each one, just Yeah, I mean, I didn't really stick to the original plan of you separate ones. Yeah, it's up to you. Um, OK. One of the things you might want to do is you take some code that you've had on Comet built on Comet and then transfer it over to Expanse. Now I'm gonna pick some examples that I have from Comet right now and show you how I modify or you will show you how I compile code in general and how You might want to think about doing something similar for sort of reproducibility aspects in the future, but also just you know how to then use the module environment that we just explored and to compile code on Expanse. So the first thing I want to say and I think Mahi mentioned during his talk is Unlike on Comet where we sort of didn't Didn't monitor this as much as maybe the system administrators would like us to and we really do encourage you to compile code on the type of compute node that you're running on. Especially if you're going to be using GPU code right the login nodes are AMD. But even for CPU code. Figure out how to build the piece of software that you want to Grab an interactive session in do that in sort of develop that build process and In it. And and in particular on the GPU sides, you really should do that because, again, as you know, depending on how you sort of structure your build process, right, if you're using, say, the m-arch native kind of setting right you need the compiler to pick up on those optimizations and you need to Sort of, you know, if you want it to be simpler. You can cross compile, but we've run into problems ourselves with that a bit. compilation process. And that would affect other users on the login notes. I'll do the GPU example. Just grab Commands. I guess I'll do 16 gigabytes one GPU. And I'm sure I have less than 30 minutes I'll just do this. So this is again. SLURM is not loaded in your default environment so Okay. So I basically started an interactive session on one of Expanses GPU nodes and now I need to correct Me to Login with my credentials correctly. So I can pull some files from Comet. Okay, so we're on an interactive compute node. Oh, it's probably not. Just do it here first. So I have some software sitting on Comet. Some build scripts that I've used in the past. grab these Scripts Gromacs is a molecular dynamics code that's widely used on Comet. MPI openMP and CUDA itself. Okay, so let me go open up this build script. The different dependencies that are needed and how you sort of configure that what I do is I create a build script essentially That sort of codifies all the changes that I made to the environment itself to compile the code. But as I mentioned, you also have to load those same modules. When you run the code, right. So I can compile the code in batch form as well. But it really codifies what was done to compile the code because you need to know exactly what modules were loaded When you compile the code to then run it. And so that's why you know trying to codify things in a build script like this is I think a good practice and it helps us when you run into problems. You know, oh, this is what I did. We trying to be reduced the text here a bit Of this build scripts. What time does this session. Yes. Okay. Log into Expanse here. And just so we can have another screen here for the What modules, I might want to load. Right, you can see that the path and the module environment for Spack is different than the one on the AMD compute nodes which makes sense because this is these are Intel based CPUs and NVIDIA GPUs. And you'll also notice there isn't. For most of the builds that we're doing. On the Intel compute nodes so and so that's what I'm going to use right now and so In this case, I don't really have a module. Module that I'm loading For the compiler itself because basically all of the defaults here. Are being built with this GCC 8. that's available. To the module environment on the GPU side like you have to sort of keep in your head that these packages in the Spack environment were built with GCC 831 so if you want to use this OpenMPI 404 you're going to you know your by default, you're using the GCC 831. So I'm going to change my build script here to use that OpenMPI for for and let's just double check to make sure that if I do a module load There's also a cmake in there. It's already there. So right now, at least the CUDA modules that are available are actually in the system manage module environment. You'll see the cuda 10. Right, and so the default one to load, I believe, is just the toolkit and I don't think I've run into any problems with having to actually load the rest of them. But I believe everything that I've done testing wise if I just use the main tool kit one works so But that's obviously dependent on the codes and you might be compiling against. So if all the paths that are in the toolkit module are there and really point to most of the libraries in CUDA that you're compiling against it's probably fine. I'll change this. And essentially, this is what my build scripts look like I sort of declare some environment variables that set what modules I want so I can use them later. To describe where to install things and create paths and things like that. It'll just be GCC By default, but you know you, then you know I always do a module purge before compiling and then I load the modules that are needed. To create the environment that I want to then compiled program. Creating some directories and then running cmake which is the build them build program used for Gromacs here, you know, there's different settings that you can set for your obviously for your code and These are sort of just default ones we use on Comet forever. And start compiling the code. Was an error here my GPU. So yeah, I actually did this earlier when I was testing the examples. So yeah, I did it again. Expanse GPU module. Right, because basically when I do the module purge i've you know removed the GPU stack from my environment, right. That should then load all the packages that I'm expecting to see, and then I can load those subsequently So yeah, so that's what we usually do when we're trying to help you compile software as we construct these build scripts that you know sort of You know, cement in what was done. Obviously, sometimes you want to do this interactively and just get a sense of, like, Oh, is this going to work. Which is fine, but as you work through it. Very easily able to switch over to Gromacs say 20 from 19 would have that for about three to 20. Right, I just changed the variables that we're pointing to different module names and the version of the package itself. Right. You know, you can also set these up as batch job scripts, because, for example, there wasn't someone who asked, Well, if I'm doing this and debug and it's going to take longer to compile My package. Or you can also, you know, if you have a build script like this. You can, you know, submit it as a job to the scheduler itself. You know, you get a build out, I mean you have the complete output file from the build itself. Right. I definitely encourage you to think about, you know, building code. And I think that's about out of time and If there's any questions. Hey, Marty. Yeah. It's a good question. I can put them on today. Yeah, maybe we should keep Yeah, yeah, basically, I will put you somewhere. In on Expanse In the examples directories that we have. And if you're currently on the system in terms of early user program, you know, If you, if you want. Access to some of these examples I can you know ping us with a ticket and I can show you where they are on Comet and you can sort of play around with them. or copy. Okay, well, if there's no more questions. 135 where Nicole Wolter will talk about job charging followed by Subha she'll go over the Expanse portal and then we'll do interactive computing. Going so we stay on time, we, we have Nicole Wolter will be talking about job charging followed by Subha who will talk about the OOD portal and then I'll talk about interactive computing This will run till 2:15 and then we'll have a final session on data management. All right. So, Can everyone see my screen or Tell me, you can all see my screen so I'm, I'm going to talk about Expanse managing allocations and charging and we've touched on this. A little bit in the previous presentations and there were a lot of discussion. So for them, so some of this might be repetitive, but there are some some little tidbits in here that are going to be different for Comet and that we haven't touched on yet so I hope it's helpful and one of one of the goals that we can get out of this. Your proposals because you know you know how the resources are going to be used and how you can use them and it will also help you when you're you're thinking about how to set up your workflows. So I'm going to start with the upbeat that job charging a super simple, at least on the user's behalf. So on the top level as Mahi mentioned before Expanse and Expanse GPU are completely separate resources and at SDSC and Expanse for, for I mean At XSEDE for that matter all resources are allocated in what we call service units or SU's but just because they're, they're called the same thing does not mean that they are the same at all. Allocated in core hour where GPU Expanse GPU SU is going to be a GPU hour a GPU hour I'm in addition you need to know that even if they are both allocated as core hours where Expanse and Comet were both allocated as core hours. Um, some of the things that. You're not just requesting the core hour. Kind of the high level thought here is that jobs charged jobs are charged for the resources that you request. And that'll be in separate from, I'm sorry, the jobs charged are for the resources you request, not the resources that you use. Another important thing to note is that the minimum charge for all jobs is one SU so if you run a On one core for a couple seconds is going to get charged one SU but if you run on one core for an entire hour that will also get charged one SU So when you're thinking about your workflows here, it's, it's something that you should consider into bundling lots of small jobs together in one job. So on the high level. It or is going to be dependent on the partition. We should be familiar with most of these. And GPU debug are all going to have a charge factor of one, but with Expanse, we added these preempt and the GPU preempt So these are discounted jobs that you can run on knows that aren't previously allocated, but as discussed before, if another job comes into the queue and needs to use those. A couple things to keep in consideration. And, the other thing to note is that in preempt It's like a compute or a GPU request, you will be charged for the entire core or the entire node. So we just we allocated in full nodes. So GPU charging you can think of it kind of like a Tetris game. When we're when we're scheduling. And one of the reasons we we do core hours is because we feel that with the shared queue, we can fill in all those little back parts a little bit better. So one Expanse node has 128 cores and 256 gigs of memory. So when you take the fraction of the smallest unit that we can allocate then is one Expanse SU or one core hour is going to be one core and up to two gigabytes of memory, whereas on Comet. Because we had fewer cores, even though we had less memory, per core you got a little bit more. Something that's really, really important to note at this point is that on in all partitions unless you specify memory, you're going to get a default of one gig of memory, that's across all the partitions. Your in shared and you want one or one core By default it is going to put in one gig. So the charge here is fairly straightforward. One core and three gigabytes memory. The wall clock time or the duration is going to be in hours and then a charge factor, of course, is related to The partition that you use. GPU is very similar. GPUs is, the charging is going to be the exact same but again you need to note that one GPU on Expanse is different than what we had on Comet. On Comet we actually had two different types of GPUs available, but on Expanse one GPU node has four GPUs 40 cores and 384 gigabytes. Portion of that node, you can have up to one GPU, up to 10 cores and up to 96 gigabytes of memory. By default, if you request one GPU and don't add the other elements into your batch script. The equation here is similar equivalent GPUs job duration charge factor but again the max is going to be out of GPUs cores or gigabytes of memory. So we talked a little about the Our Marty actually mentioned about the Expanse client tool we have. Comet, which was show accounts and project details. And what this tool will do for you. Marty also mentioned that to be able to run this command, you have to load the SDSC module. The syntax for this is Expanse-client and then the command that comes behind it, the commands you can use are either users or projects. When you list projects, it will give it will also when you use Expanse-client project if you need to also specify the project that you want to view. This will show you who all is on the allocation, what has been used By each individual user what the total is available to all the users. So in our account management system, we can actually specify percentage of allocation that's available or cap users at this so that might be different depending. It actually everyone in this allocation has the same amount available to them. The other command that you can use is users. And user will show you specifically what you have access to so Expanse module client user will show you all the applications that you currently have available to you. And will show you your usage. That the our accounting updates into the database nightly so everything that's been run today. You will only see information for all the jobs that were run yesterday or before then you won't see what happened today because the jobs are uploaded at midnight, but with this tool. This will show you what is currently in the queue. So that actually reconciles quite frequently. So, I went I went really quick and then I can I can give some more things, but just in kind of a review the charges are based on the reserve resources that you reserve and you request, not, not how you're going to use it. The default is one gigabyte of memory per core GPU in all partitions. Minimum charge for any job is one SU, a thing to notice that we dropped the minimum the time limit from 10 seconds to one second. I wrote down here again the changes in the cores and what they are and that we have new partitions. Some of some of the partitions and preempt I think we're still considering how to best manage them and how to tweak them so that they are the most advantageous for users. That's all I have. Yeah, I think, I think one of the questions I was trying to answer that just now in the in the chat was in terms of maybe you can just review again the charging versus cores and memories. So depending so we can actually there's we have a little more complicated equation for that. Can request. or I'll take the example of CPUs is one core and with that core, you can use up to two gigabytes of memory. You can request five cores and two gigabytes memory. And in that case, the, the number of cores is more than the amount of memory that five cores would have. The cores in that case. At all. Yeah, I think so. You're going to be charged by the resource that you use more of over the amount that's available on a per quarter basis. If you want, you know, for every core there's effectively two gigabytes of memory available on the system. Memory instead because effectively you're taking that memory away from the other cores, essentially. Yeah, well, and I should also add why we Why we chose to set it at one gigabyte. In the case with the example I gave you have one core and three gigabytes. It makes more sense to take advantage of someone who's using a core and one gigabyte example again this is a Tetris example. If that makes sense. And say, Okay, if there aren't any more questions. Yes, I can share my slides now. Okay, I can get started. Yeah, looks good. Hi, my name is Subha Sivagnanam, as Mahidhar mentioned in this talk earlier today, we are developing a user portal for Expanse users. So the goal of the Expanse user portal is to provide an integrated web based environment for doing file management and job submission operations on Expanse Using an easy to use interface the portal is also being developed to serve as a gateway for launching interactive applications such as, you know, MATLAB and R studio So we did not build this you know we're not building this portal from scratch. So the, the portal provides graphical file management capability where users can view and operate on files in their home directory or On Lustre scratch directory. Moving, renaming and uploading files, users can also edit files without needing a shell. Edit submit jobs and also monitor jobs. We currently have example templates for jobs like MPI and openMP and we'll be adding more but however if you don't see a job template that you think would be widely used for a specific application or for research community. And you want that to be provided, please do let us know. So, and without, you know, needing to install and local X server. user request we may add more interactive applications, and for those users who need to access a you know who need a terminal. So now who can use the Expanse portal is anyone any PI or users who have a valid Expanse allocation and have the XSEDE base credentials. They can only access the Expanse portal and what that means is, once your application is approved, and you need to have your home directory and everything set up on the back end before you could use the Expanse portal. And though this is obvious any policy like you know scheduler application usage security policies that's been set on Expanse resource. Or any guidelines that has been established is automatically applicable when you're using the portal as well. So the portal is being developed. Prototype for Comet that's available if you're interested in checking it out. Yes. So let me so So as I mentioned, you need to have XSEDE portal username and password to login so you'll be automatically redirected to the authentication page. Where you enter in your details and then you will be redirected to our portal. You know, it's pretty explanatory, but it's just you can access your files or you can go for the job composer area. Interactive apps to launch your interactive job on the cluster. View files I can edit files I can you know move and rename or create a new file or directory. Show you. And then I'm sorry the Zoom bar. Have to go back So as you can see now. The edits would be saved and I can view the changes here and I can submit this job using the job submission interface. So under the jobs that there's an active job in a job composer tabs, the active jobs is where. Come here to monitor the status. The job composer is where you would actually create your job scripts and submitted to the cluster. And if you're interested in using the templates. Through the button and right now we have a basic hello world. And if you want to use any of these existing templates to start the work you can just click on creating new job and that's created and you can see the job script itself over here. If you want to add. Ran yesterday. So in addition to creating submitting jobs. Here you can choose your queue the number of hours what account you want to use on where you want to launch the job from and I will. Lustre scratch directory and I'm just running for one hour. There is a tab where you can go view all your live interactive sessions in under my interactive session right now. For some reason they say you're debugging with a colleague or you want to you want them to see what you're doing. It's failed to connect, so Let me see. I'm sorry, I'm not sure what's going on. And interactive terminal when the Matlab will be launched and you can plot and display and open your Matlab files and And use the interactive application there. So other than that. So this is the portal, in a nutshell, we will be adding more features such as you know your information about your job and information about your applications will also be displayed here and we will also be adding more interactive applications as there is a demand. So with that, I will end my talk and see if you have any questions. So, Subha there was one in the chat about whether there's a REST API for OOD. OOD. Okay, thanks, Subha. And let's see if I can make this Work. There we go. Everybody see The talk. Okay. Go ahead and turn off my video. So Subha just talked about the portal and there's a lot of different definitions for interactive computing. Subha was saying, there'll be other interactive tools brought on to Expanse So once interactive computing. But it's a little bit different, as you've gathered and if you've worked on HPC systems. A high performance computer, and in particular on the compute nodes, you have to get them allocated, you have to get your hands on them and you have to be able to work on them. work being done in the field of interactive computing on HPC systems. Is being held. So with that, here are some scenarios for HPC interactive HPC MATLAB, parallel MATLAB like Subha just showed you and And on Expanse. Some scientific application, looking at airflow monitoring so anytime you're running an application and you're accessing a cluster node its interactive HPC computing So to get an interactive node, you have to run a particular command. And you'll be able to run commands. On the login node. So this is some of the arguments. To SLURM are a lot more exacting, so you have to give a little bit more information. Let us know, and so I'm going to focus on Jupyter notebooks and letting them on Expanse. Jupyter notebooks and Jupyter lab and they're actually considered services and I'll show you why. And understanding that will help you realize some of a key aspect we're trying to focus on, which is the security of these notebooks. So we have a tutorial on running notebooks on Comet that will be ready for Expanse as well. But go to that tutorial. notebooks are popular in a lot of different scenarios, especially machine learning, you can run parallel notebooks, you need to use something like Spark. And there's a lot of applications that are using them. So what we recommend is that you use the mini-anaconda environment called conda and so you can go to the tutorials and Find out the more detailed explanations or go to the conda site to install it, but the power of conda installing it is it gives you your own local environment. So you can install the version of Python, like Marty was showing you with modules, it's very similar. It was originally created for Python and so you can you can also develop your own conda environments and then have different versions of Python in those environments. So it gives you a lot of flexibility and control for what you're doing. We're not trying to keep different versions of conda environments maintained for 10,000 users the users will teach you to install conda. So you want to you can also create these virtual environments. And you can look on the conda site. And off you go. If you want to use Jupyter notebooks, you need to install the notebooks and you need to install Jupyter lab is pretty straightforward. So, and then you want to make sure that the environment is working. So a key issue that we're addressing it at SDSC is the security of Jupyter notebooks. taught about this, but most of the time when you run a notebook they're not secure it's HTTP. There are some ways to make it more secure. The team spent the last year, improving the security model. So what, why is this a problem that connection by default is not secure, you can connect over SSH tunneling it's secure and we have a little tutorial and we have no problem with that. Your connection may timeout and you lose your notebook. Optimal and we came up with this idea of a reverse proxy service. So we allow Jupyter services. So what's wrong with the default Jupyter notebook. The server whichever it could be the login node, which we don't want you to do, or even the remote node that you get with an interactive command. The Jupyter service launches and it's a web server basically running for you on that node, but it's HTTP. So you have a secure SSH connection. And so we're trying to overcome that by making this a little more secure SSH tunneling works this way. Using the port you set up a port on your laptop, a port on the Jupyter notebook and then an SSH connection between that port on your laptop. And that port on the notebook and everything communicated here it's secure. So what's more secure is what we've architected in called the reverse proxy service. queue. In in Repeating what's in that diagram the start Jupyter script is what you're going to run you'll check it out and clone it from the repository. And you have arguments. Do you want to use notebook or Jupyter lab right now. A little more explicitly but we're working on improving that part of the user interface. As you as we build the script and submitted. And it gets installed and then you come down into the directory and you'll see this script here. Service so that it can work for systems that have SLURM or twerk, which means that works on Comet Expanse TSCC Stratus and And we can just add all we have to do is develop the right notebook. Configuration for the particular service. Here's an example of starting it you'll get some output and one of the things that comes out is your notebook is here. Paste it into a web browser and you'll be given a monitoring window and when it's finally launched and that can take some time, because the queuing system might be very busy, you'll get your notebook or your, your browser and this is over here is just a little example of The, the script, I'm going the wrong way and This is an example of using the Expanse portal to launch the notebook. A terminal window and say what you get is a shell and there's a link on this portal. And you're in your shell. So you can do it through the portal or you can do this through the command line, whichever you want And I think in my time. That's it. Thanks, Mary. I'm sorry, I just saw that I apologize. I'm going to do a quick overview of the storage we have available and the policies that are associated with it. And so I think it's important that we drive it home to you that it's coming. And then I'll talk about some various options and tools that are going to help you manage how how to manage the data you have Alright, so like on Comet we have the home file system and then we have the parallel global file system. Similar to Comet all users will get one, you will have 100 gigs of data there. Static stuff that you want to keep forever. So instead we we recommend you use the parallel file systems. On, Comet we recommended that it should be less than three months or less than 90 days on Expanse, we're gonna we're going to help remind you that this is scratch and we're going to automate that purge to help everyone and it's going to be 90 days from file creation. The other option we have. Projects is actually allocated space, but all projects will, by default, get 500 gigs of memory. Or if you need additional space as Mahi mentioned you're going to need to justify it, and either put it in your initial proposal or request a supplement The other really really important thing is that the Lustre file system does not have backups. Problem of that I've lost all my data. Soon, since we know that we're not supposed to use login nodes we shouldn't use login nodes for Any compute or IO intensive and the IO intensive is, of course, including moving large amounts of data. So we have for data mover nodes and they're there specifically to help you guys transfer data back and forth, um, the data mover nodes can be used with all the tools that I'm going to discuss further When you access or you can access data mover nodes with oasis-dm. So there's of the tools we have available this kind of a list, we have out there. When you're deciding on what tool to use. The size of the individual files and the number of files that you're moving Other things that you kind of want to take into consideration is your workflow. For larger for larger things as Mahi mentioned the globus toolkit. In addition, we're kind of adding two new tools that into our into our idea here because we're getting into the cloud, a little bit more. So we'll start with a small file. SFTP, as Marty gave us a great example earlier is secure file transfer protocol. Um, with all these tools there are lots of options of how you can use. I recommend on the system you do SCP help or command help. A local site to a remote site from a remote site back to your local site or between two remote sites. As well. The difference. The next three tools I'm going to talk about rely on the globus toolkit, the globus toolkit is collection of clients and middleware, that is designed specifically for enabling technology for grid computing. It's a command line. command line tool that supports the standard SSH and grid. So you can use this in the Single Sign On but that's that's what is using us gsissh to interface with globus. With Globus the transfer tools that are in there are gsiscp and gsiftp and they're essentially the same as a SCP and SFTP The one thing with all the globus tools is that you're going to need an XSEDE, an active, an active, that's important, XSEDE allocation. If you are a local user who only has a local allocation, feel free to contact us. XSEDE cuts you off. But in case that's still the case you should contact us either through the XSEDE ticketing system or the SDSC local ticketing system and will help facilitate it so that you can move your data as needed. If you're uncomfortable with the command line, you can use Globus via GUI you'll log into www. The endpoints are important. Another thing to keep in mind is that the past and little bit or on the data mover nodes Are a little different than what is actually on the system itself, so check in the in the user guide to to find how those paths, how they correlate between them. And the next one I'm going to talk about is globus-url-copy. command line interface for high speed data transfers. If you have a large number of really small files it is highly recommended that you tar them up and send them as one large file, rather than lots of little files. So to use Globus URL copy once you're on the data nodes you need to load the globus module, you need to check to make sure you have an active user proxy. The command is grid-proxy-info, it can tell you if it's available, how long, how long is available by default. The command my-proxy-logon to create a new certificate for yourself. To help you improve performance and to make this as efficient as possible. Than using SCP for large files and like the terabytes and stuff. So now I'm going to talk about two tools that I am by no means Fully versed in all the capabilities. And that the tool or the command that we use is as ascp it uses a fast, fast technologies and to transfer and share files, it does a whole lot more. And again, this can be used for the cloud stuff as well. Um, I have a simple example here that that's just kind of the high level. IBM Sparrow copy tool is available is currently available on the data mover nodes. So the AWS command line interface is not available yet, you would need to download it to use it. Instructions and is super simple. It's an open source tool to interact with the AWS services using commands in your command line shell, it'll use Linux. Um, again, it's a very, very robust tool. So start with backup all your data. And then kind of an in review. You should always avoid moving a lot of small files. There's an automated purge on the Lustre file system. And finally, we haven't met. Which means it's file system will go away in mid April. So we strongly encourage you, if you have a lot of data to start thinking about where that data. Something else that I should should mention, that's a difference between Comet and all the resources before and Expanse is that Comet loaded have the same home directory and actually loaded. We aren't doing that with Expanse, so anything in your home directory. If it was on Comet, you're going to need to copy it over. So, and I think that's all I have. stop sharing and check the Check. Yes. Hey everybody, I guess we're done. But we'll have everything uploaded as PDFs in this directory. Again in the spring before Comet shuts down, send us questions, send questions to help that xsede. I just want to add that this being a three hour recording means it'll take a day or two at least To clean up transcript and get it online. So thanks very much everyone for hanging in there. Yeah. Thanks.
Transcript
And we can go ahead and get started.
Search the Transcript
No search results yet...
Skip to navigation