Over the last two years, the focus in AI has shifted towards spending more resources on test-time compute. Test-time compute is the deliberate allocation of extra computational resources during a model’s inference phase rather than during training [1][2]. The reasons vary: raising the output quality of small local models, verifying outputs, pulling in additional information during inference, and more.

Agents exploit this massively. The core of the agent loop was described in the ReAct paper: the model reasons, acts by calling a tool, observes the result, and repeats [3]. So an agent run has two halves. The reasoning is the model thinking, which runs on GPUs. The acting is everything else: tool calls, builds, tests, simulations, and the verification of results. 

The acting half is not small. A study of a coding agent across 144 tasks found that tool calls and environment setup account for 56 to 74 percent of the end-to-end task time, with model reasoning making up the rest [4].

Agents are outgrowing the single node

Agents now outgrow single machines. They need to run securely, and their resources need to be managed in a very flexible way.

Workload managers that are built around CPU cores expect requirements to be expressed as rigid placement instructions: “I want to run 4 tasks on 80 nodes, each with 4 GPUs.” That is the right model for AI training and fine-tuning. It is not the right model for AI agents.

Agents look different. The same study measured an average CPU utilization of only 7.6 to 13.2 percent, with memory use arriving in bursts around tool calls [4]. An agent mostly waits: for the model, for I/O, or for other resources such a license. What it needs is:

• • access to arbitrary resources, not just cores and GPUs

• • I/O

• • fast job allocation

• • license-aware scheduling

• • fair-share within the cluster

• • quotas on arbitrary resources

A queue and slot scheduler, not a CPU scheduler

That list is the DNA of Open Cluster Scheduler (OCS) and Gridware Cluster Scheduler (GCS). It is Grid Engine DNA, and it is ours: we developed core parts of Sun Grid Engine and started Univa Grid Engine, and OCS and GCS continue that work. Both are centered around queues and slots rather than CPU cores, combined with a very flexible resource model.

• • Slots are not cores. A slot is a unit of work a queue is willing to run. A host can offer more slots than it has cores, which fits processes that sit at around 10 percent CPU. Memory stays a managed resource, so the bursts are accounted for.

• • Any resource can be scheduled. Licenses, GPUs, ports, tokens, or anything else you can count is defined as a resource and requested by a job [5]. Since 9.0.9, GCS discovers and tracks FlexLM licenses natively, and a running job can give back an expensive resource such as a license as soon as it no longer needs it [6]. Agents rarely know upfront what they will need, so this matters.

• • Quotas and fair-share on anything. Resource quota sets limit any resource per user, project, queue, or host [7]. Share-tree and functional policies decide who goes first. This is how agent runs are aligned with business priorities instead of with whoever started the most agents.

• • Built for huge job counts. OCS is designed to handle millions of independent jobs per day [8], and recent releases raised the submission rate by a further 50 percent [9].

• • Flexible accounting. Arbitrary per-job metrics can be written into the accounting record and show up in qacct next to the standard usage data [10]. For agents that can be token counts, tool calls, model names, or cost.

Industries such as chip design and bioinformatics have optimized for job throughput, I/O, and license usage rather than CPU usage for a long time. Their workloads are millions of small, I/O-bound verification and analysis jobs. From the scheduler’s point of view, an agent’s subtasks look the same.

I call this domain test-time compute orchestration: scheduling both halves, the inference and the agent’s work, under one set of policies.

Agents need to run where the data is created

A separate platform for agents is yet another cluster. It has its own storage, its own users, its own policies, and no connection to the license servers. The data the agents need is not there, and the results they produce have to be moved back.

Agents should run where the simulation jobs process their data, where the chip design files are located, and where the license servers are connected. With GCS and OCS, simulations, EDA regressions, and bioinformatics jobs run along with AI training, inference, and agents in one single, easy to manage cluster, under one set of quotas and one accounting. AI workloads that come with Slurm command lines run there as well: since 9.1.7 the slurm-shim accepts sbatch, srun, squeue, and scancel unchanged [14]

Your cluster resources are too valuable to run just one job type.

Secure agents with NVIDIA OpenShell

Running agents next to valuable data only works if the agent’s access is controlled. Secure agents run in environments like NVIDIA’s OpenShell, an open-source runtime that sandboxes an agent and enforces declarative policies for filesystem, network, and process access, including a policy prover that verifies those policies [11].

Our OpenShell integration for Gridware Cluster Scheduler spawns OpenShell instances for agents as jobs. The scheduler decides where and when an agent runs and what it may consume. OpenShell decides what it may touch.

There is still room for improvement in traditional HPC environments, where shared POSIX filesystems such as NFS decide access by UID and GID. An OpenShell sandbox does not run as the user who started it: it runs under one generic account for all agents [12] or under a subordinate UID [13].

A real run: 410 agents, 96 slots

We ran this on a cloud cluster on 2026-11-09: GCS 9.1.7 with 32 execution hosts of 4 cores each, 24 of them for agents and 8 for classic jobs. Every agent is an OpenShell sandbox, and every sandbox is a scheduler job. In this baseline run we bound each agent to one core instead of oversubscribing the hosts, so that the timings below are not distorted by agents competing for CPU. The 96 cores also match the 96 model API seats we had.

• • A coding swarm. 154 agents worked on the 77 sympy issues of SWE-bench Lite, two attempts each, 96 at a time. Each agent got a copy of the code, the issue, one core, and no network except the model API. The patches were scored in sandboxes too, because a patch is code an agent wrote. Result: 51 of 77 issues resolved in 10 minutes 46 seconds for $5.61 in API cost. This is our re-implementation of the SWE-bench test step, not an official evaluation.

• • A classic job with agents on its outcome. A LAMMPS parameter sweep ran as one array job of 256 tasks on the classic hosts. Then 256 agents each read one output and wrote a verdict: 2 minutes 7 seconds, $0.85. Checked against a deterministic reference, 93 percent of the verdicts agreed, and all 47 crashed runs were found.

• • Two teams, one cluster. Research and engineering together wanted 410 agents. The cluster ran 96. The share tree split them between the teams based on what each had already used, and an urgent agent submitted in the middle waited 0 seconds while about 300 others queued.

The policies behind this are a few lines of ordinary scheduler configuration. A quota caps the agents, and the model API is a consumable with a seat limit, exactly like a software license:

$ qconf -srqs agent_cap | grep limit; qconf -se global | grep complex_values
   limit        queues agent.q to slots=96
complex_values        anthropic_api=96

The bill per team comes from qacct -P, as for any other job: 29,591 seconds of wall clock for engineering and 16,266 for research.

Fast job allocation was on the list of what agents need, so we measured where the time of every agent goes. These are the medians for the 256 review agents:

All 256 sandboxes were in the scheduler after 5.2 seconds with pure command line job submission (not using the much faster DRMAA API), a rate of about 2,950 submissions per minute, and the first agent was running after 13.4 seconds. The queue wait is policy, not overhead: 256 agents wanted 96 slots. The largest fixed cost is the container runtime starting the sandbox.

The scheduling, quota, license, and accounting features described above are available in Gridware Cluster Scheduler and Open Cluster Scheduler today. If you are interested in how the OpenShell integration works, ping me.

Daniel Gruber – dgruber @ hpc-gridware.com

References

1. 1. What is Test-time Scaling?
   2. The Art of Scaling Test-Time Compute for Large Language Models
   3. ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al., 2022)
   4. AgentCgroup: Understanding and Controlling OS Resources of AI Agents
   5. Add Any Resource to Gridware Cluster Scheduler / Open Cluster Scheduler
   6. Gridware Cluster Scheduler 9.0.9: Smarter License Management and Enhanced Debugging
   7. sge_resource_quota man page
   8. Open Cluster Scheduler: The Future of Open Source Workload Management
   9. Gridware Cluster Scheduler Upgrades: Faster, Smarter, and Ready for AI Workloads
   10. Introducing Flexible Accounting in Gridware Cluster Scheduler: Collect Arbitrary Job Metrics
   11. Overview of NVIDIA OpenShell
   12. OpenShell Sandbox Runtimes
   13. OpenShell issue 2336: bind-mount ownership and rootless subuid mapping
   14. slurm-shim: Slurm command compatibility for Open Cluster Scheduler