SUID-less NixOS / NoNewPrivileges

Links / Prior work

Introduction to SUID/SGID, Ambient Capabilities

What is SUID/SGID?

SUID and SGID are short for “Set User-ID” and “Set Group-ID” respectively. SUID bits and SGID bits are part of the UNIX permission bits which can be attached to files and directories, and do very different things depending on whether a directory or an executable has SUID/SGID bits set. Any program which has the SUID bit set will be executed as the user owning the executable, regardless of which user actually invoked execve. Similarly, any program with SGID bit will be executed as the group owning the executable. Any program with SGID/SUID permission bit set is commonly called an SUID/SGID binary, in which case SUID/SGID is a property executables on disk may have.

A quite common example is the sudo command. sudo always executes as root. It is on the program itself to then be reasonably secure. In the case of sudo, this includes complicated password and permission checks against the sudoers file. For sudo, SUID makes sense: After all, sudo is explicitly intended to elevate privileges in a controlled way. There are, however, alternative ways to gain elevation, without the use of SUID binaries.

Executables owned by root user with the SUID bit set are called “SUID root”. Analogously, SGID executables owned by root group will be called “SGID root”.

What are Capabilities?

Capabilities in Linux allow specific privileged actions, without executing an entire program as root user (as would be the case with SUID root). This is an improvement over SUID/SGID, because only specific permissions are granted.

However, some of the permissions granted by capabilities are basically identical to root, including, but not limited to:

NixOS: /run/wrappers/bin

The Nix store must not contain files with SUID/SGID bits or capabilities, though current NixOS does rely on these functionalities. This is solved by setting up a special directory /run/wrappers/bin during system activation. This directory contains small shims (=“wrappers”) with their capabilities/suid/sgid defined. These wrappers are small binaries which just exec into the “real” binary in the nix store, inheriting the cabailities and user/group ownign the process. This is performed by the suid-sgid-wrappers.service.

This can be disabled using security.enableWrappers = false;. On current (“normal”) NixOS installations, be advised this will break a lot of things, including authentication via PAM which is needed to log in.

Systemd: Ambient Capabilities

Systemd understands three sets of capabilities: ambient, bounding, and “current”. The CapabilityBoundingSet does what it sounds like: any capabilities not in this set will not be granted to the service processes. AmbientCapabilitySet is the inverse: capabilities in this set will always be granted to service processes, without capabilities on the actual exectuable file being required.

Systemd has a NoNewPrivileges option, both global and for individual units. Any child of a unit with NoNewPrivileges cannot gain new privileges (such as ambient capabilities) if those were not explicitly present in the parent. This restricts more than just capabilities. The ultimate goal is to enable this option for an entire NixOS system.

Excursion: Experimenting with Capabilities

Systemd allows spawning shells with ambient capabilities:

$ run0 -u $(whoami) --property=AmbientCapabilities=CAP_DAC_OVERRIDE

Be advised, depending on the capability you pass in, this is basically a root shell with extra steps.

So why is SUID/SGID bad??

SUID root executables run as root. The program is itself solely responsible for user authentication and privilege dropping if necessary. This is bad design: If any of the various SUID binaries on a normal system has some flaw, you have LPE. The attack surface is massive. Don’t forget that many of these are ancient C programs, which need to do string operations (e.g. sudo fetching and checking a password). This is scary and has the potential to go catastrophically wrong.

One such example of things going wrong is a collection of screen CVEs from last year. The TL;DR here is: screen supports SUID/SGID for some privileged actions, such as lingering after logout. However, it did not aggressively drop permissions, resulting in a massive attack surface and multiple local privilege escalations.

In the case of recent cache corruption issues in the kernel, SUID root binaries were the easiest to attack by hijacking the file contents on disk, but be aware the same exploits could have been used to attack shadow/passwd instead, not requiring SUID root for LPE.

Sadly, sometimes capabilities and SUID/SGID are given to programs that have no valid need for them. While I personally believe we should to get rid of SUID/SGID and as much of the capability things as we can, some programs are just stupid. Cleaning this up is tedious. To give an example: https://github.com/NixOS/nixpkgs/pull/516307.

Working towards an SUID-less future

Both the kernel and systemd support a NoNewPrivileges flag. The goal is to enable that flag early, and only drop privileges with exec calls. This requires some preparation and changes to core system components.

Sockets and Daemons

We do not want SUID/SGID, nor do we want capabilities on executables. This directly implies any privileged operation needs to be performed by a privileged process or service. The service doing the privileged operations can be forked from an early privileged service in systemd, and remain as a daemon.

One such example for a daemon approach would be ssh. ssh root@::1 is an entirely valid way to gain a root shell without any SUID root binaries being involved. The “trick” happening here is simple: sshd.service is started during boot, and remains around. Whenever a ssh connection is established, it forks into a new session. This means we are only dropping permissions, we do not invoke any process which has more permissions than the parent process. Often, people will claim ssh as root is insecure. I believe the opposite is the case: ssh is one of the most battle-tested pieces of software out there, I’d rather trust ssh with a hardware key than trusting some SUID root binary to properly validate my password without any errors.

The daemon approach requires a root-owned process to linger, waiting for some less privileged entity to make contact. This means slightly more resource usage, as well as potential attacks against already-running processes. An alternative is systemd socket activated services. These essentially work by exposing a Unix domain socket as a systemd unit, and starting the associated daemon only when the socket is being opened from somewhere else. Comparing this to ssh, this means less network code involved, and no lingering daemon process running. Systemd uses varlink as protocol for its socket services. varlink is essentially just json over unix domain socket, with a somewhat rigid API. Socket services using unix domain sockets also allows passing open file descriptors through the socket, something the ssh root@::1 approach can never imitate. While the daemon approach does a good job getting rid of SUID/SGID root, socket services essentially do the same but better. The cost is rewriting components to support these socket services.

sudo/su/sg (solved: run0)

sudo (and su) may look intimidating, because those are among the best known SUID root binaries on most systems. Migrating those is not that hard, however. We already saw ssh can be used to imitate sudo. Systemd itself has a tool too, called run0. run0 is a small shim around systemd-run. systemd-run is intended to debug systemd units interactively, by running units without first moving them to the service manager directories and rebooting. run0 abstracts away the service internals, and provides an API somewhat similar to sudo, using PAM/polkit for authentication. Some details are missing, such as passing passwords over a file descriptor or an equivalent to sudoedit, though work here is happening.

Some legacy scripts explicitly expect sudo CLI syntax. run0-sudo-shim is my own attempt at creating a shim which behaves as close to sudo as possible while actually just deferring the complex tasks to run0. This will always be “best effort”, ultimately scripts need to stop expecting sudo to be universal.

run0 is always available on NixOS. Disabling sudo is simple:

security.sudo.enable = false;

Polkit (partially solved: socket service)

Polkit upstream started supporting systemd varlink with their askpass helpers in version 127. NixOS: https://github.com/NixOS/nixpkgs/pull/473068 Upstream: https://github.com/polkit-org/polkit/pull/501

This removed the polkit-agent-helper-1 wrapper, cleanly replacing it with polkit-agent-helper.socket systemd unit. Apparently this caused some issues with Cosmic and howdy, but for most users this change has been basically invisible.

pkexec still exists as SUID root. With systemd run0, we already have a PAM/polkit based elevator, it should be viable for most users to just disable the pkexec wrapper:

security.wrappers.pkexec.enable = false;

People over at Fedora experimented with this a while ago: https://discussion.fedoraproject.org/t/replacing-sudo-pkexec-with-run0-experiments-and-issues/137035. An attempt at removing pkexec by default in NixOS was made in https://github.com/NixOS/nixpkgs/pull/485657, however currently too much software still expects pkexec to exist.

PAM/shadow (proposed solution: account-utils)

This includes various SUID/SGID binaries:

Removing all these binaries basically requires reimplementing most of shadow and a good chunk of PAM with support for socket activated services doing the privileged actions. One such implementation is account-utils. This effort is spear-headed by Thorsten Kukuk, who is mostly active on the SUSE security team, doing a lot of PAM-adjacent things there.

account-utils is based on sockets:

These sockets are available to be called via varlink interface from any unprivileged process, which means the replacements to the previous SUID wrappers just become small frontends for the varlink API imitating the behavior of the traditional SUID equivalents on CLI. For shadow, this means we can simply give account-utils a stronger meta.priority, so it takes priority over shadow in $PATH. Then we can remove the wrappers from shadow entirely. Be aware shadow also has wrappers for su/sg not covered by account-utils, I assume those are replaced with something run0-shaped.

PAM itself does also use a SUID root binary: unix_chkpwd. Working around this is significantly more complex. account-utils accomodates this by implementing an entirely new pam_unix_ng.so, which calls the socket services instead of relying on unix_chkpwd. PAM tooling in NixOS is awful, and we (=anyone touching it in the past three years) have agreed it needs a big overhaul eventually. Short of that overhaul, the only reasonable option I could think of was adding an internal config option to define the pam_unix module path. Arguably this is a hack, but until our PAM is less cursed there is nothing more permanent than a temporary solution.

A pull request for an account-utils module is up and awaiting reviews: https://github.com/NixOS/nixpkgs/pull/453557

account-utils is probably not mature enough to actually use in a production environment. However, I believe the idea is important enough to warrant experimenting with it, and maybe it’ll stick.

Fuse 2 (proposed solution: drop)

https://github.com/NixOS/nixpkgs/pull/521536

This will take a a while, mostly because we need to migrate or remove the existing users of fuse 2. Help would be appreciated. If we can drop all users of libfuse (version 2 does not have a suffix, version 3 is libfuse3), nothing in the nix package tree can build a fuse2 file system that would require fusermount, at which point we can safely also remove the fusermount SUID wrapper.

Fuse 3 (unsolved)

We cannot reasonably drop libfuse3. Various software depends on this, such as flatpak and portals. That said, this should also not be default-enabled on literally every NixOS install. Ideally it’d be default-disabled, with each module requiring fuse (services.flatpak.enable, xdg.portal.enable, and probably many more) explicitly enabling fusermount3. This change is going to be breaking an awful lot of things, but I believe that change to be necessary.

CAP_SYS_NICE (partially solved: rtkit)

rtkit is an ancient piece of software historically used to do CPU scheduling for sound things, using a daemon approach. If the goal is to give some random game higher CPU priority (=lower niceness), rtkit can do that without a capability. An example here is the programs.gamemode.enableRenice option: This should just be a call to rtkit imo. Currently it installs a CAP_SYS_NICE wrapper.

Be aware rtkit was unmaintained for a long time, but was recently picked up by pipewire upstream (again for audio scheduling things):

Following the upstream change, rtkit also received a maintainer in nixpkgs and a general package recipe makeover: https://github.com/NixOS/nixpkgs/pull/470633.

rtkit does not allow you to change scheduling of arbitrary processes as CAP_SYS_NICE does. Personally, I would argue changing scheduling of more privileged processes should not be necessary. Overloading a system and throttling a logger would for example impact auditability.

Mount (replace with udisks/udev)

From mount(8):

   Non-superuser mounts
     Normally, only the superuser can mount filesystems. However, when fstab contains
     the user option on a line, anybody can mount the corresponding filesystem.

     Thus, given a line

        /dev/cdrom /cd iso9660 ro,user,noauto,unhide

     any user can mount the iso9660 filesystem found on an inserted CDROM using the command:

        mount /cd
  ...

To some degree, this can be emulated using udisks and/or udev. A 1:1 replacement likely needs a bespoke tool.

Screen (replace with tmux, don’t add SUID bits)

As shown above, screen with SUID/SGID root had real security issues. The easiest way to avoid those is to just not install it as SUID binary. Personal note: tmux is more modern and less terribly designed, there is little reason to remain using screen.

Honorary Mentions

These are tools that exist, ideas to work towards less dependence on SUID/SGID. I cannnot be familiar with literally everything out there. This list may be edited as people make me aware of other pieces of software in this space.

Switching on NoNewPrivileges

This has been a dream of many people, including many systemd developers upstream. As it stands, PAM/shadow and fuse3 are the only major components which do not yet have a viable replacement in nixpkgs. The addition of the account-utils module includes a NixOS VM test for interactive authentication via a full PAM login stack, with NoNewPrivileges enabled. Admittedly, this is a toy example. I do however believe flipping that switch may not be far away for non-interactive systems. On “normal” desktop systems, I suspect the dependency on fusermount3 is currently too strong, though progress is steadily being made.

Credits