9.1.7 is a patch release with an unusually wide reach. It ships the slurm-shim with the product, lets execution hosts register themselves, teaches Qontrol to snapshot and diff a cluster configuration, extends resource map requests, brings AMD Instinct GPUs to qgpu, and makes TLS available in Open Cluster Scheduler. Here is what matters.
Run your Slurm scripts unchanged
Many clusters have years of Slurm in them: #SBATCH headers in job scripts, sbatch calls in pipelines, tooling that parses squeue output. Rewriting all of that is the part of a migration nobody wants to budget for.
The slurm-shim takes that work off the table. It provides the Slurm command line tools your users and applications already expect and translates them onto an Open or Gridware Cluster Scheduler cluster, so scripts and integrations keep working while the scheduler underneath changes. Users who know Slurm stay productive from day one; nothing has to be retrained or rewritten before the switch.
Starting with 9.1.7 the shim is delivered with the product rather than fetched separately: a self-contained package next to the OCS and GCS packages, with the binaries for all supported architectures, a README, and its own license information and SBOM.
The shim also gained a feature Slurm does not have: hot-spare nodes. A training or inference job can be submitted with a number of spare nodes held in reserve, so that it survives the loss of up to k nodes without being restarted from scratch. For long-running multi-node AI workloads this is the difference between a hiccup and a lost day.
And because the shim has a configuration of its own, Qontrol’s Configuration Files section now covers it too.
(Available in Open and Gridware Cluster Scheduler.)
Hosts that join and leave on their own
Until now a host had to be configured as an administrative host and as an execution host before its execution daemon was allowed to register. That ordering is fine for a rack that is installed once. It is a nuisance for anything that comes and goes — nodes booted from an image, cloud instances, a pool that grows under load and shrinks at night — because something has to touch the cluster configuration before every new host can report for duty.
Two new qmaster_params turn that into a rule. AUTO_ADD_HOSTS lists who may add themselves, and optionally the host group each one lands in:
qconf -mconf
...
qmaster_params AUTO_ADD_HOSTS=10.42.0.0/16:@cloud;gpu-*.example.com:@gpu
A matcher with a slash is an IPv4 network in CIDR notation; anything else is an fnmatch(3) pattern tested against the host name and its address. The first matching rule wins. Where a rule names a host group, the host is added to it (the group is created if needed), so queue instances appear wherever a cluster queue already uses that group. A prepared cluster queue on @cloud is then the entire configuration a new instance needs. Slots are set to the number of processors the host reports, exactly as the execution host installer would do, and a value an administrator sets later is left alone.
The host is added as an execution host and nothing more: it does not become an admin or submit host, and only the execution daemon of the host itself can add it.
AUTO_REMOVE_HOSTS_AFTER covers the way out. A host that added itself and has then been unknown to the master for that long is removed from its host groups and from the execution host list, taking its queue instances with it:
qmaster_params AUTO_ADD_HOSTS=10.42.0.0/16:@cloud,AUTO_REMOVE_HOSTS_AFTER=0:30:00
That is what keeps a reclaimed cloud instance from haunting qhost and qstat forever. Hosts an administrator added are never removed, and a host that still has jobs running is kept until they are gone. Both parameters are documented in sge_conf(5).
(Available in Open and Gridware Cluster Scheduler.)
Qontrol: snapshots, diffs, and advance reservations
Qontrol, the web console replacing qmon, picks up two things administrators have been asking for.
Configuration snapshots with comparison. Take a snapshot of the whole cluster configuration by hand or on a schedule, and diff any two of them. Want to know what changed between Friday and the Monday the queue stopped accepting jobs? Compare the snapshots. Want a daily record of configuration drift across the clusters Qontrol manages? Schedule it. Together with the operations log from 9.1.1, this closes the “who changed what, and when” question for good.
Advance reservations in the UI. Advance reservations can now be created and managed in Qontrol, including the resource map instances they hold since 9.1.6. A new qrsub -par switch on the command line lets you choose the PE allocation rule of a reservation as well.
And as mentioned above, the Configuration Files section now knows about the slurm-shim.
Ask for the GPU you mean
9.1.6 let a job say that the instances it receives from a resource map must agree with one another (same=id, same=numa_node). 9.1.7 lets a job say which instances it wants, and pick them by what the administrator recorded about them rather than by their names.
The new id= parameter takes an expression over the identifiers of the map — shell wildcards combined with !, & and |:
qsub -l 'gpu=1[id=gpu0]' job.sh
qsub -l 'gpu=1[id=gpu1*]' job.sh
qsub -l 'gpu=2[id=(gpu1*|gpu2*)&!*b]' job.sh
Any parameter whose name is not reserved is read as a complex and matched against that characteristic of the instance, using the relation operator of the complex. With gpu_memory defined <=, this means “at least 40 G”:
qsub -l 'gpu=1[gpu_memory=40G]' job.sh
Parameters combine, so a job can constrain the device and the properties of its shares at once — four shares of one card, and only of a card with at least 80 G on board:
qsub -l 'gpu=4[same=id,gpu_memory=80G]' job.sh
Finally, scope=job extends a same= constraint across every host of a parallel job, which is what you want where a map is named consistently across the cluster:
qsub -pe mpi 2 -l 'gpu=1[same=id,scope=job]' job.sh
Under the hood, the scheduler now decides which instances a job gets while matching a host, not afterwards while booking them, so a reservation reflects the instances the job will actually receive. The full grammar lives in sge_rsmap(5).
(Available in Open and Gridware Cluster Scheduler; characteristics and request parameters in Gridware Cluster Scheduler only.)
AMD Instinct joins qgpu
qgpu 1.2.0 supports AMD Instinct GPUs end to end: load sensor, installer, device selection in prolog and epilog, per-job accounting and a quick health check. The vendor is detected per host and per grant, so one queue configuration and one set of wrapper scripts serve NVIDIA and AMD hosts in the same cluster.
An AMD host reports amd_gpu_count, amd_gpu_backend and per-device amd_gpu_<N>_* load values, manages its devices in an AMD_GPUS resource map, and hands each job ROCR_VISIBLE_DEVICES for the devices it was granted (plus AMD_VISIBLE_DEVICES for container runtimes). qgpu reads the amdgpu kernel driver interfaces directly — KFD topology, sysfs, hwmon — so it needs neither a ROCm installation nor a vendor library. Per-job accounting reads the driver’s hardware counters at job start and end and shows up as amd_* values in qacct -j on MI300 and MI350 class GPUs. Partitioned GPUs appear as GPUs in their own right, each with its own index, render node and resource map identifier.
(Available in Gridware Cluster Scheduler only.)
Encrypt and authenticate — in Open Cluster Scheduler too
TLS encryption of component communication, introduced in 9.1.0 for Gridware Cluster Scheduler, is now part of Open Cluster Scheduler as well. inst_sge -tls and security_mode tls in the bootstrap file give an open source cluster encrypted communication, certificate generation and automatic renewal, just like the enterprise product.
We strongly encourage you to install every new cluster, Open or Gridware, with both switches:
./inst_sge -m -tls -munge
Munge authenticates who is talking; TLS protects what they say. If you remember the old CSP mode – the hand-run certificate authority, the per-user certificates, the renewals nobody wanted to own – this is a different experience: the installer generates the certificates, the daemons renew them on their own, and Munge takes a single shared key. There is very little to configure and almost nothing to maintain, so there is no longer a reason to leave either switch off. Both need OpenSSL 3 and Munge on every host, and switching an existing cluster over means restarting all components, so plan it for a maintenance window. See the Installation Guide chapters “Enabling TLS Encryption” and “Installing Munge”.
(Available in Open and Gridware Cluster Scheduler.)
Fixes all round
The release also hardens a number of corners. The systemd integration survives a reset or stalled D-Bus connection instead of aborting the execution daemon or hanging it, and no longer floods the messages file when a connection is dead. sge_qmaster recovers its PostgreSQL connections after a database restart. qrdel with an exact reservation name now deletes only that reservation, not every user’s. Calendar windows starting at midnight on a future day switch the queue as intended, resource reservations are made before a calendar-disabled period, and a user can no longer widen the device isolation of their own job through an environment variable. Builds now produce CycloneDX 1.7 SBOMs across the board as part of the CRA compliance work.
Upgrade is drop-in for any 9.1.x installation. If you are coming from 9.1.4 or earlier, check the upgrade notes on the TLS key location and the LimitNOFILE setting in the qmaster unit file.
We would love to hear how the slurm-shim works for your migration, and how far AUTO_ADD_HOSTS gets you before you touch a configuration by hand.
Download Open Cluster Scheduler from our new download portal: https://open.clusterscheduler.io.
