Cpuset (cset) Tutorial

To set up a shield of 3 CPUs with 1 CPU left for low priority system processing, issue the following command.

This command does a number of things. First, a user cpuset is created with what’s called a CPUSPEC (CPU specification) from the -c/--cpu option. This CPUSPEC specifies to use CPUs 1 through 3 inclusively. Next, the command creates a system cpuset with a CPUSPEC that is the inverse of the -c option for the current machine. On this machine that cpuset will only contain the first CPU, CPU0. Next, all userspace processes running in the root cpuset are transfered to the system cpuset. This makes all those processes run only on CPU0. The effect of this is that the shield consists of CPUs 1 through 3 and they are now idling.

Note that the command did not move the kernel threads that are running in the root cpuset to the system cpuset. This is because you may want these kernel threads to use all available CPUs. If you do not, the you can use the -k/--kthread option as described below.

The shield setup command above outputs the information of which cpusets were created and how many tasks are running on each. If you want to see the current status of the shield again, issue this command:

Which shows us that the shield is set up and that 176 tasks are running in the system cpuset—the "unshielded" cpuset.

It is important to move all possible tasks from the root cpuset to the unshielded system cpuset because a task’s cpuset property is inherited by its children. Since we’ve moved all running tasks (including init) to the unshielded system cpuset, that means that any new tasks that are spawned will also run in the unshielded system cpuset.

Some kernel threads can be moved into the unshielded system cpuset as well. These are the threads that are not bound to specific CPUs. If a kernel thread is bound to a specific CPU, then it is generally not a good idea to move that thread to the system set because at worst it may hang the system and at best it will slow the system down significantly. These threads are usually the IRQ threads on a real time Linux kernel, for example, and you may want to not move these kernel threads into system. If you leave them in the root cpuset, then they will have access to all CPUs.

However, if your application demands an even "quieter" shield, then you can move all movable kernel threads into the unshielded system set with the following command.

You can see that this moved an additional 70 tasks to the unshielded system cpuset. Note that the -k/--kthread on parameter can be given at the shield creation time as well and you do not need to perform these two steps separately if you know that you will want kernel thread shielding as well. Executing cset shield again shows us the current state of the shield.

You can get a detailed listing of what is running in the shield by specifying either -s/--shield or -u/--unshield to the shield subcommand and using the verbose flag. You will get output similar to the following.

Note that I abbreviated the listing; we do have 251 tasks running in the system set. The output is self-explanatory; however, the "SPPr" field may need a little explanation. "SPPr" stands for State, Policy and Priority. You can see that the initial two tasks are Stopped and running in timeshare priority, marked as "oth" (for "other"). The [IRQ-9] task is also stopped, but marked at real time FIFO policy with a priority of 50. The last task in the listing is the cset command itself and is marked as running. Also note that adding a second -v/--verbose option will not restrict the output to fit into an 80 character screen.

Tear down of the shield, stopping the shield in other words, is done with the -r/--reset option to the shield subcommand. When this command is issued, both the system and user cpusets are deleted and any tasks that are running in both of those cpusets are moved to the root cpuset. Once so moved, all tasks will have access to all resources on the system. For example:

Now that we have a shield running, the objective is to run our "important" processes in that shield. These processes can be anything, but usually they are directly related to the purpose of the machine. There are two ways to run tasks in the shield:

The basic syntax of set for cpuset creation is:

This creates a cpuset named "my_cpuset1" with a CPUSPEC of CPU1, CPU2 and CPU3. The CPUSPEC is the same concept as described in the "Setup and Teardown of the Shield" section above. The set subcommand also takes a -m/--mem option that lets you specify the memory nodes the set will use as well as flags to make the CPUs and MEMs exclusive to the cpuset. If you are on a non-NUMA machine, just leave the -m option out and the default memory node 0 will be used.

Just like with shield, you can adjust the CPUs and MEMs with subsequent calls to set. If, for example, you wish to adjust the "my_cpuset1" cpuset to only use CPUs 1 and 3 (and omit CPU2), then issue the following command.

cset will then adjust the CPUs that are assigned to the "my_cpuset1" set to only use CPU1 and CPU3.

To rename a cpuset, use the -n/--newname option. For example:

Renames the cpuset called "my_cpuset1" to "super_set".

To destroy a cpuset, use the -d/--destroy option as follows.

This command destroys the newly created cpuset called "super_set". When a cpuset is destroyed, all the tasks running in it are moved to the parent cpuset. The root cpuset, which always exists and always contains all CPUs, can not be destroyed. You may also give the --destroy option a list of cpusets to destroy.

If you want to create a cpuset hierarchy, then you must give a path to the cset set subcommand. This path will always begin with the root cpuset, for which the path is /. For example.

These commands created two cpusets: top_set and sub_set. The top_set uses CPU1 and CPU3. It has a subset of sub_set which only uses CPU3. Once you have created a subset with a path, then if the name is unique, you do not have to specify the path in order to affect it. If the name is not unique, then cset will complain and ask you to use the path. For example:

This command adds CPU1 to the sub_set cpuset for it’s use. Note that using the path in this case is optional.

If you attempt to destroy a cpuset which has sub-cpusets, cset will complain and not do it unless you use the -r/--recurse and the --force options. If you do use --force, then all the tasks running in all subsets of the deletion target cpuset will be moved to the target’s parent cpuset and all cpusets.

Moving a cpuset from under a certain cpuset to a different location is currently not implemented and is slated for a later release of cset.

To list cpusets, use the set subcommand with the -l/--list option. For example:

This shows that there is currently one cpuset present called one. (Of course that there is also the root set, which is always present.) The output shows that the one cpuset has no tasks running in it. The root cpuset has 320 tasks running. The "-X" for "CPUs" and "MEMs" fields denotes whether the CPUs and MEMs in the cpusets are marked exclusive to those cpusets. Note that the one cpuset has subsets as indicated by a 1 in the Subs field. You can specify a cpuset to list with the set subcommand as follows.

This output shows that there is a cpuset called two in cpuset one and it also has subset. You can also ask for a recursive listing as follows.

This command lists all cpusets existing on the system since it asks for a recursive listing beginning at the root cpuset. Incidentally, should you need to specify the root cpuset you can use either root or / to specify it explicitely—just remember that the root cpuset cannot be deleted or modified.

Operation of the proc subcommand follows the same model as the set subcommand. For example, to list tasks in a cpuset, you need to use the -l/--list option and specify the cpuset by name or, if the name exists multiple times in the cpuset hierarchy, by path. For example:

This output shows us that the cpuset called two has CPU2 only attached to it and is running three tasks: two shells and the python command to list it. Note that cpusets are inherited so that if a process is contained in a cpuset, then any children it spawns also run within that set. In this case, the python command to list set two was run from a shell already running in set two. This can be seen by the PPID (parent process ID) of the python command matching the PID of the shell.

Additionally, the "SPPr" field needs explanation. "SPPr" stands for State, Policy and Priority. You can see that the initial two tasks are Stopped and running in timeshare priority, marked as "oth" (for "other"). The last task is marked as running, "R" and also at timeshare priority, "oth." If any of these tasks would have been at real time priority, then the policy would be shown as "f" for FIFO or "r" for round robin, and the priority would be a number from 1 to 99. See below for an example.

This output shows the first few tasks in the root cpuset. Note that both init and [kthread] are running at timeshare; however, the [migration/0] and [posix_cpu_timer] kernel threads are running at real time policy of FIFO and priority of 99. Incidentally, this output is from a system running the real time Linux kernel which runs some kernel threads at real time priorities. And finally, note that you can of course use cset as any other Linux tool and include it in pipelines as in the example above.

Taking a peek into the third cpuset called three, we see:

This output shows that a lot of beagled tasks are running in this cpuset and it also shows an ellipsis (…) at the end of their listings. If you see this ellipsis, that means that the command was too long to fit onto an 80 character screen. To see the entire commandline, use the -v/--verbose flag, as per following.

To exec a task into a cpuset, the proc subcommand needs to be employed with the -e/--exec option. Let’s exec a shell into the cpuset named two in our set. First we check to see what is running that set:

You can see that initially, two had nothing running in it. After the completion of the second command, we list two again and see that there are two tasks running: the shell which we execed and the python cset command that is listing the cpuset. The reason for the second task is that the cpuset property of a running task is inherited by all its children. Since we executed the listing command from the new shell which was bound to cpuset two, the resulting process for the listing is also bound to cpuset two. Let’s test that by just running a new shell with no prefixed cset command.

Here again we see that the second shell, PID 21118, has a parent PID of 20955 which is the first shell. Both shells as well as the listing command are running in the two cpuset.

Finally, if you misspell the command to be execed, the result may be puzzling. For example:

The result is no new process even though a new PID is output. The reason for the message is of course that the cset process forked in preparation for exec, but the command blah-blah was not found in order to exec it.

Although the ability to exec a task into a cpuset is fundamental, you will most likely be moving tasks between cpusets more often. Moving tasks is accomplished with the -m/--move and -p/--pid options to the proc subcommand of cset. The move option tells the proc subcommand that a task move is requested. The -p/--pid option takes an argument called a PIDSPEC (PID Specification). The PIDSPEC defines which tasks get operated on.

The PIDSPEC can be a single process ID, a list of process IDs separated by commas, and a list of process ID ranges also separated by commas. For example:

In the following example, we move the current shell into the cpuset named two with a range PIDSPEC and back out to the root cpuset with the bash variable for the current PID.

Use of the appropriate PIDSPEC can thus be handy to move tasks and groups of tasks. Additionally, there is one more option that can help with multi-threaded processes, and that is the --threads flag. If this flag is present in a proc move command with a PIDSPEC and if any of the task IDs in the PIDSPEC belongs to a thread in a process container, then all the sibling threads in that process container will also get moved. This flag provides an easy mechanism to move all threads of a process by simply specifying one thread in that process. In the following example, we move all the threads running in cpuset three to cpuset two by using the --threads flag.

There actually is no cset subcommand or option to destroy tasks—it’s not really needed. Tasks exist and are accessible on the system as normal, even if they happen to be running in one cpuset or another. To destroy tasks, use the usual ^C method or by using the kill(1) command.