CHERIoT: A Study in CHERI

Authors: Tony Chen, Nathaniel Wesley Filardo, Kunyan Liu, Robert Norton-Wright, Yucong Tao

CHERIoT (Capability Hardware Extension to RISC-V for Internet of Things) is a 32-bit RISC-V extension optimized for IoT and embedded applications.  Although CHERIoT began as a Microsoft Research project that we shared in our initial blog post, first release announcement, and early blue-teaming, the platform is now a thriving multi-vendor open-source platform. It includes a formal specification of the instruction set architecture (ISA), an LLVM toolchain, real-time operating system (RTOS), and a reference implementation of a RISC-V core supporting the extension.  It has been recently contributed to the RISC-V standardization effort.

CHERIoT is designed to bring the best of CHERI research to targets that traditionally have higher security challenges, such as microcontrollers. Microcontroller security has become increasingly important as those controllers can be exposed in IoT devices. The CHERIoT platform provides a showcase of what is possible on a system that can assume CHERI. It builds on top of Microsoft’s CHERI research to provide object-granularity temporal safety and lightweight compartmentalisation.

CHERIoT also highlights the value of the RISC-V extension model. It is possible to build extensions that are aggressively optimised for a specific range of microarchitectures. In our case, CHERIoT is optimised for microcontroller-scale cores, in low core counts, with memory sizes ranging from tens of KiBs to tens of MiBs.  It can make design trade-offs that provide enormous wins in this space, but which would not scale to massive multicore server systems. This can coexist in the RISC-V ecosystem with other extensions that are optimised for high-throughput servers, but which would have unacceptable power and area costs in a microcontroller.

CHERIoT was designed as a coherent platform spanning hardware and software. The hardware provides low-level guarantees, such as unforgeable pointers, the ability to ensure that pointers to marked memory regions cannot be used, and a lightweight way of making pointers into opaque tamper-proof tokens. Software uses these to enforce object-granularity memory safety, which is used to build a compartment model that allows fine-grained sharing. You can communicate with compartments simply by calling functions that they expose and share data with them by simply passing pointers as function arguments. The hardware guarantees are used to build a rich compartmentalization model that allows read-only sharing or sharing for the duration of a single function call.

The capability guarantees and the compartment model then build security properties for system integrators, making it possible to audit exactly which devices any compartment can access, which entry points it can call from other compartments, and a host of other properties. 

This whole-stack security solution highlights the benefits of RISC-V, where an extensible ISA can be tailored to provide increased benefit in specialised markets.

History of CHERIoT

The CHERIoT project began in 2019, to explore how to scale CHERI ideas down to very small systems. Most CHERI research has focused on application cores, ranging from mobile phones up to large servers (see CHERI ISAv9 and Arm’s Morello prototype). CHERIoT instead targets systems several orders of magnitude smaller than the smallest such system. Our goal was to provide stronger security guarantees than anything else currently available with a total area overhead of under 10% relative to a baseline microcontroller, an objective achieved and surpassed by a significant margin, as we will discuss.

The project demonstrated spatial and temporal memory safety for C/C++/assembly, with lightweight compartmentalisation (supporting both hierarchical and mutual distrust), with a silicon-area overhead of around 3%.  There is no fixed limit on the number of mutually distrusting compartments, adding another compartment costs less than 100 bytes of memory, and memory can be shared by simply passing pointers as function arguments.

CHERIoT could meet these lofty goals only through a holistic hardware-software co-design approach. The ISA extension, compartmentalisation model, C/C++ language extensions, and RTOS were designed together. This enables properties to be enforced at whichever layer in the system makes the most sense, in a way that is not possible if any layer of the stack is treated as an immutable object. The RISC-V extension model and the modularity and flexibility of the LLVM toolchain were central to our ability to work across so many layers at once.

Principles of CHERIoT

CHERI provides a hardware-enforced data type, a CHERI capability, that can be used to represent pointers. These carry bounds and permissions, in addition to an address. A CHERI capability can be stored in memory or in registers and is protected by guarded manipulation: Operations may reduce the rights that a capability conveys, but not increase them. Any attempt to tamper with a capability (for example, overwriting part of it in memory) clears a non-addressable tag bit. The tag bit is a simple attestation from the hardware that says that this capability was created by following these rules, starting with one of the all-powerful capabilities provided at boot and then constrained by layers of software in the system. For example, on a larger CHERI system with conventional UNIX-like OS, a kernel may hold a capability to the entire userspace portion of an address space. On an application request to map memory, the kernel may provide a capability with restricted permissions, bounded to the requested memory range. A userspace memory allocator, such as malloc, may then provide a bounded capability to a range within that mapping.  If this is used to store an array of structures, an iterator over the array may further bind the capability to a single structure, and a function may then provide a callee with a read-only capability to a single field in that structure.  Each of these refinements is a fast register-register operation.

CHERIoT honors the two core principles that drove the CHERI project:

• The principle of least privilege
• The principles of intentional use

Nothing in a system should have the rights to access anything that it doesn’t need to perform its task, and nothing exercises rights accidentally. Any CHERI system provides these guarantees for memory accesses. You cannot access memory unless the load, store, or jump instruction takes a capability authorising it as an explicit operand.

The CHERIoT architecture is co-designed with a compartmentalisation model and RTOS that expands these principles throughout the software stack.

Many of the software compartmentalisation abstractions are built on top of sealing. Sealing is one of the most important features of CHERI that differentiates it from other approaches to foundational memory safety. Sealing a CHERI capability turns it into an opaque token. You can pass it around as you would any other pointer, but you cannot dereference or modify it.  Any attempt to modify the capability will clear its tag bit.  The seal operation takes another capability with a permit-seal permission and embeds the value of that capability in the object type of the sealed capability. You can then unseal with a permit-unseal capability that has the matching type.  As shown in Figure 1, this gives an easy way of providing type-safe opaque pointers to untrusted code.

Each sealing type allows you to define one type-safe opaque type. In 64-bit CHERI, there’s space for a lot of these in the hardware capability format. In the 32-bit CHERIoT encoding, there is space for only three bits. Even this reduced quantity still lets us differentiate, among others, three kinds of types needed for the core of the security model and leaves one sealing type available for a virtualised sealing mechanism.

The sealing model is also used for sentry capabilities, which provide control-flow integrity guarantees. A small number of object types are reserved for executable capabilities to define immutable entry points. This respects the principles of intentionality and least privilege: The loader creates these capabilities to define a static control-flow graph and then destroys the capability that authorises their creation, preventing cross-library function-pointer forgery. 

CHERIoT provides a layered security model. At the core, CHERIoT provides object-granularity (spatial and temporal) memory safety. Any out-of-bounds access to an object (including attempting to access an object with the wrong pointer) will trap. Similarly, attempts to access an object after it has been freed will trap.

On top of this, CHERIoT provides a rich set of compartmentalisation abstractions, including privilege separation for the core of the RTOS. The switcher, which handles transitions between threads and between compartments, has access to one reserved register and is the only thing that has sufficient rights to break compartment or thread isolation. Despite the trusted nature of the switcher, its authority is still limited, unlike similar components in traditional, privilege-ring-based systems; for example, the switcher is still constrained by CHERI capabilities’ guarded manipulation rules. It is also very small and fast: around 300 instructions (a similar size to the amount of unverified code in seL4, and a current subject for formal verification). The scheduler is the equivalent of the kernel in a conventional RTOS. It chooses the next thread to run and so is trusted for availability, but (using the sealing mechanism) it has no access to interrupted threads’ state and cannot tamper with any thread or compartment’s data. 

The shared heap is provided by a heap allocator that runs in a separate compartment and has access to the whole heap, but it does not run in a special privilege mode (in fact, CHERIoT needs only M mode). Bugs here could break memory safety for the heap, but not for any other memory. Compartments may allocate memory only if they have an allocator capability, a sealed pointer (sealed with the correct type) to an object that encapsulates a quota. Sealed objects like this can be baked into the firmware image (as allocator capabilities must be, for bootstrapping) or allocated dynamically. This lets us express things like message queue endpoints that can be accessed directly only within the compartment that provides message queues, or socket handles that can be accessed only within the TCP/IP stack. Compartments can expose rich capability abstractions, respecting the principle of intentionality for much higher-level abstractions.

The capability model then extends into operational security. CHERIoT was designed to permit fearless code reuse. CHERIoT can run unmodified embedded software in a compartment and, at firmware-link time, run the cheriot-audit tool to ensure that it respects a compartmentalisation policy. This lets firmware developers and integrators reason about (and write policies to constrain) which compartments may call which entry points, which may allocate memory (and how much), which may make network connections (and to which hosts, on which ports), and so on.  Even a supply-chain attacker who has arbitrary-code execution abilities within a compartment is constrained in the damage that they can do to the overall system.

Auditability was a key design goal for the CHERIoT platform.  This extends to availability considerations as well as confidentiality and integrity.  In conventional embedded systems, any code can disable interrupts, which may cause another component to miss its real-time targets. In CHERIoT, the permission to arbitrarily toggle interrupt status is restricted to the switcher but there is another mechanism that allows interrupts to be disabled: sentries. As discussed above, sentries are a special form of sealed capabilities that can be used as jump targets, but not for any other purpose. CHERIoT adds variants of these that enable or disable interrupts on jump. Jump and link captures the original state in the link register, so interrupts can be disabled with structured-programming flows that you can reason about. Importantly, the loader must provide a sentry of the correct type, which means that this shows up in the auditing reports.  It’s possible to have an allow list of functions that may run with interrupts disabled and audit them at integration time. This approach would be difficult to implement on very large out-of-order cores, but RISC-V makes it possible to tune extensions for their intended targets; CHERIoT aims to support 2-7 stage single- or dual-issue pipelines, and not scale up to large application cores.

The CHERIoT capability format is the result of careful co-design with the software stack and analysis of the requirements. Most CHERIoT systems are likely to have under 1 MiB of RAM and so CHERIoT’s capabilities’ bounds representation is tuned to be fast to decode even in simple pipelines and to give good coverage of representable sizes within the most common ranges for objects and sub-objects.  CHERIoT has a rich set of permissions with an encoding that allows 12 permissions to be encoded in 6 bits. With this model, it can express shallow and deep immutability, and shallow and deep no-capture enforcement for pointers passed across compartment boundaries. When the TLS layer in the CHERIoT network stack wants to read data from the network, it passes a write-only, no-capture, bounded buffer that may not be used to store pointers to the TCP/IP layer. The platform, via a mixture of hardware and software constructions, ensures that the TCP/IP layer does not keep a copy of that pointer, cannot read any stale data (such as decrypted plaintext) from that buffer, and cannot store out of the region that was passed to it. This call in the TCP/IP layer also takes a sealed capability to the socket, which ensures that the TLS layer cannot tamper with a socket or read data from a socket that it is not authorised to access.

In the original 64-bit CHERI design, all capabilities were derived from a single fully-permissive capability provided at boot. By contrast, CHERIoT has a multi-rooted capability hierarchy, which improves the encoding efficiency and also provides some assurances by construction. To start with, capabilities that refer to the object-type space for sealing and those that refer to memory are separate, so there is never confusion over whether the value in a capability is a memory address or a sealing type. In addition, there are separate writeable and executable capabilities. This improves the permission encoding efficiency because some permissions are dependent permissions: they make sense only if you have another permission. For example, there is a permission that allows access to certain control and status registers (CSRs) when a capability with that permission is installed as the program counter capability. This permission makes sense only for capabilities with execute permission. Removing writable and executable capabilities also makes accidentally modifying code impossible. It is possible to do on-device code generation by holding both writeable and executable capabilities to the same memory, requiring a small code change in JIT compilers, but this is a much smaller change than would be required to support a new ISA extension. This change is sadly unlikely to be possible on a 64-bit CHERI system that runs conventional desktop or server operating systems, because APIs such as POSIX’s mmap or Windows’ VirtualAlloc assume that they can create writeable and executable objects.

Most importantly, CHERIoT has been able to remove a lot of complexity. CHERI was designed to support incremental deployment on large desktop-class operating systems.  Most of the 64-bit CHERI works to date (MIPS, RISC-V, Arm’s Morello) have mode switches to run unmodified binaries, something essential for adoption. In contrast, typically all code running on a microcontroller is compiled for that precise target and so CHERIoT can remove this, reducing complexity and validation costs.  Similarly, the RISC-V PMP provides a subset of the protections of a CHERI system and so it, too, can be removed.  CHERIoT is designed to enable mutual distrust and so hierarchical privilege modes are unnecessary, so CHERIoT CPUs support only Machine Mode. All of this helps reduce the power, area, and DV costs for an implementation.

CHERIoT Status

The CHERIoT platform is a complete hardware-software stack, with an LLVM port providing C/C++/assembly compilation and linking, an RTOS, a formal model of the ISA extension (built on the Sail RISC-V specification), and a reference implementation in the form of a modified Ibex microcontroller. All of these components are permissively licensed to enable widespread adoption.

The CHERIoT platform has also been the focus of formal verification efforts. Thomas Melham’s team at the University of Oxford has been verifying the implementation of CHERIoT Ibex (CHERI and non-CHERI parts) against the Sail specification (see his presentation at CHERITech’24: video, slides). Researchers at Google have been working on formally verifying the isolation properties of the switcher and have been instrumental in some recent improvements to CHERIoT. Formal verification of the core components helps to give us confidence in the security claims of the platform.  

lowRISC CIC, the maintainers of the original Ibex core that has been extended for the CHERIoT reference implementation, is also working to enable the CHERIoT ecosystem. The Sunburst Project is creating two FPGA boards, Symphony and Sonata, designed to run a CHERIoT core and set of peripherals. The board designs are also open source, allowing custom modifications for specific use cases. Microsoft has also contracted with lowRISC to verify the CHERIoT Ibex core.

SCI Semiconductor, the startup collaborating closely with Microsoft on the evolution of the CHERIoT platform, has announced development of the first commercially available CHERIoT devices.

The future of CHERIoT

With the CHERIoT ISA at a stable 1.0 release with long-term support for the compiler, we will begin upstreaming the support to LLVM with the goal of adoption as an official RISC-V specification, as the recommended microcontroller profile for CHERI RISC-V systems.

CHERIoT 1.0 is a useful point in the design space but there is, as always, still room for research and possible improvements.  The Sail definition’s issue tracker already holds discussions of several possible changes for a future edition of the architecture.

As CHERIoT progresses as a collective effort, we look forward to working with the community on other hardware implementations, including exploring multicore systems. On the software side, we would like to explore running embedded Linux applications in a compartment and working alongside other teams to expand CHERIoT’s focus from C and C++ to support other languages, for example, Rust and SPARK Ada.

We have recompiled large, embedded components such as the FreeRTOS+TCP network stack, mBedTLS, BearSSL, the Microvium JavaScript interpreter, and the TPM reference code, to get object-granularity memory safety from CHERIoT. We don’t recommend rewriting mature codebases in a safe language because doing so comes with a large opportunity cost and is likely to introduce new bugs (even if it eliminates memory-safety bugs). We do encourage writing new code in safe languages, because they provide compile-time guarantees that improve reliability. Safe languages alone however do not provide the same degree of supply-chain protection as a CHERIoT system because most safe languages have ways of escaping from their safety guarantees (and these are necessary for various kinds of embedded programming). A supply-chain attacker can easily compromise other safe-language code running in the same privilege domain, while CHERIoT makes it easy to isolate third-party components for fearless code reuse.

Getting Involved

The co-designed CHERIoT architecture and RTOS offer a new, open-source foundation for building secure embedded systems.  Read more at CHERIoT.org and please get in touch on our discussion forum.