By Erhu Feng (Shanghai Jiao Tong University), Qingyu Shang (Shanghai Jiao Tong University), Yu-Chien Lin (Andes), Che-Chia Chang (Andes), Bing Gui (Nuclei)

Introduction

There has been a surge of interest in employing Trusted Execution Environments (TEEs) to host security-critical applications across a spectrum of platforms—from IoT devices to cloud computing—aiming to minimize reliance on the trustworthiness of software and hardware components. While several TEE solutions for RISC-V exist, such as Penglai, Keystone, and CoVE, they continue to face significant challenges in supporting commercial Trusted Applications (TAs).

We have designed and implemented a comprehensive end-to-end solution to run OP-TEE on RISC-V platforms, utilizing the dynamic domain feature in OpenSBI—a key component derived from the Penglai-Zone system. This solution has been successfully deployed on real hardware boards from Andes and Nuclei, enabling commercial TAs using OP-TEE to operate within the RISC-V environment.

Next, we will introduce the fundamental background of RISC-V TEE systems. Then, we will present our solution for supporting TEE operating systems like OP-TEE based on the dynamic domain feature—now a key component of OpenSBI. Following that, we will show two real-world case studies demonstrating how this system is employed in Andes and Nuclei to enhance the TEE ecosystem.

Background: Trusted Execution Environment (TEE) in Current RISC-V

Penglai[1,2]: A User-Mode Enclave System for the RISC-V Architecture

Penglai leverages hardware-based security extensions like PMP (Physical Memory Protection) and IOPMP (Input/Output Physical Memory Protection) to isolate hardware resources among enclaves, operating systems, and other applications. To enhance the scalability and flexibility of secure hardware resources, Penglai introduces several optional hardware extensions such as sPMP[5], Guarded Page Tables[1], and sIOPMP[4]. In terms of the software ecosystem, Penglai integrates user SDKs like secGear, musl libc, and GlobalPlatform APIs, enabling developers to easily deploy various TAs—including key management, TLS servers, machine learning inference, etc.

OP-TEE[3]: A Leading TEE-OS in Mobile Systems

OP-TEE (Open Portable Trusted Execution Environment) is an open-source TEE implementation that provides a secure execution environment for applications that require confidentiality and integrity. It operates in conjunction with a Rich Execution Environment (REE), such as Linux, ensuring that sensitive operations can be performed securely without interference from potentially malicious software running in the REE. OP-TEE is designed to comply with the GlobalPlatform TEE Internal Core API and TEE Client API, making it suitable for various applications, including secure storage, digital rights management (DRM), payment clients, and attestation services.

The underlying support for RISC-V OP-TEE: Dynamic secure domain

In 2024, the RISE (RISC-V Software Ecosystem) launched a project to advance the maturity of the OP-TEE ecosystem on RISC-V, propelled by other projects developed under the firmware working group.

1. Domain context switching, which enhances the OpenSBI domain to achieve context switching between domain instances.
2. RPMI specification and SBI MPXY extension, which makes it possible to define TEE-specific data transfer protocols between operating systems and firmware.

These improvements have addressed the primary obstacle in supporting the OP-TEE secure payload dispatcher within OpenSBI.

3.1. Current limitation in OpenSBI

Using the RISC-V PMP mechanism, the upper-layer software can be restricted from accessing memory and IO resources, and an isolation domain can be created. The OpenSBI Domain mechanism includes a comprehensive definition and implementation of the isolation domain, covering hardware resources, boot protocols, system privileges, and more. As a fundamental feature of OpenSBI, each upper-layer software operates within its respective domain, the reference architecture shown below.

However, the previous Domain mechanism only supported static partitioning of CPU cores (HART) on the platform, which resulted in low utilization of physical CPU resources and also limited the number of isolation domains.

3.2. Dynamic domain and secure payload dispatcher

The objective of the dynamic domain is to design and implement a domain HART context manager. This context manager allows for efficient HART context switching between different isolation domains, enabling the utilization of CPU resources in scenarios including UEFI StandaloneMm, TEE-OS, and other system runtime services that require security isolation. For example, a system service domain might proactively suspend its execution after initialization, yielding CPU resources, and then be scheduled to execute on-demand when other domains request its services. It makes domains suitable for running multiple system services that follow the principles of least privilege and mutual isolation.

The reference architecture shown below illustrates the scenario of employing enhanced secure domains to support runtime services, with the context manager providing synchronous context switching when a service is called.

At runtime, the context manager offers two interfaces: sbi_domain_context_enter and sbi_domain_context_exit, both performing domain context switching on the current HART. The crucial difference is that sbi_domain_context_exit does not specify a target domain when called and simply signifies that the caller wishes to exit and suspend the current context. It returns to the previous caller domain which entered the current domain by sbi_domain_context_enter, or if there is no previous caller, start up the next domain(selected by a fixed rule).

The implementation of context management is based on the entity struct sbi_context, which contains trap states and CSR status. Save and restore the corresponding structure allows for suspension or resumption of execution for the specified domain HART. Each domain (including the ROOT domain) contains multiple sbi_context, indexed by HART_id, so when a HART enters a domain, HART context switches to that domain’s context on that HART, and when a HART exits from a domain, HART context switches to the next domain’s context on the same HART. If that context doesn’t initialized, using HSM mechanism to init it.

In the domain switching process, the context manager will simultaneously consider the switching of domain ID, PMP, and IOPMP states to ensure domain isolation. Dynamic domain context management have been upstreamed to OpenSBI, Refer to OpenSBI code location: sbi/sbi_domain_context.c for more.

Based on dynamic domain, secure payload dispatcher (a concept in Arm Trusted Firmware) can be implemented in OpenSBI to provide a unified calling interface for normal OS accessing services in different secure domains. For reference implementation, see RPMI MPXY below.

3.3. Dynamic domain user guidance

In order to make it easier for users to use, the declaration form of dynamic isolation domains is exactly the same as the original domain mechanism in OpenSBI, configured as a special device tree node, which can be referred to in the Domain Support document. In other words, the current domain mechanism (integrated with dynamic domains) supports domain switching by default without changing the declaration method, allowing the declaration and startup of isolation domains with more than the number of HARTs.

A sample device tree looks like below:

chosen {

...

opensbi-domains {

compatible = "opensbi,domain,config";

 

tmem: tmem {

compatible = "opensbi,domain,memregion";

base = <0x0 0x80C00000>;

order = <22>;   // 4M

};

allmem: allmem {

compatible = "opensbi,domain,memregion";

base = <0x0 0x0>;

order = <64>;

};

 

tdomain: trusted-domain {

compatible = "opensbi,domain,instance";

regions = <&allmem 0x3f>;

possible-harts = <&cpu0>, <&cpu1>;

next-arg1 = <0x0 0x81F80000>;

next-addr = <0x0 0x80C00000>;

next-mode = <0x1>;

};

udomain: untrusted-domain {

compatible = "opensbi,domain,instance";

regions = <&tmem 0x0>, <&allmem 0x3f>;

possible-harts = <&cpu0>, <&cpu1>;

boot-hart = <&cpu0>;

next-arg1 = <0x0 0xbfe00000>;

next-addr = <0x0 0x80200000>;

next-mode = <0x1>;

};

};

};




cpus {

...

cpu0: cpu@0 {

phandle = <0x03>;

device_type = "cpu";

reg = <0x00>;

status = "okay";

compatible = "riscv";

opensbi-domain = <&tdomain>;

...

};

 

cpu1: cpu@1 {

phandle = <0x01>;

device_type = "cpu";

reg = <0x01>;

status = "okay";

compatible = "riscv";

opensbi-domain = <&tdomain>;

...

};

...

};

It is worth noting that for the additional information required by dynamic domains, the possible_harts field in the isolation domain declaration now corresponds to the actual HART contained in each domain, and the declaration order of domain nodes is the startup order of domains. The opensbi-domain field of the cpu node now represents which domain the HART is assigned to when the platform is started, so the HART of the domain to be started first should be declared as assigned to that domain.

In the example declaration above, cpu0 and cpu1 are pre-assigned to tdomain. When tdomain is started and exited, the subsequent udomain will be started.

For more clarity, there are two sample applications demonstrating secure/non-secure domain switching on a dual-core Qemu platform, one is a simple prototype with bare-metal application on both sides, and the other shows the non-secure linux calls secure services, both providing device tree configuration and the test payload source code. Please refer to https://github.com/Shang-QY/test_context_switch/blob/master/README.md

Case Study-1: Practice of RISC-V OP-TEE in Andes Technology

About Andes: Andes, a founding premier member of RISC-V International, is publicly-listed company and a leading supplier of high-performance/low-power 32/64-bit embedded and application processor IP solutions. Andes offers the AndeSentry collaborative framework, which integrates Andes’ own solutions with those from the ecosystem and partners to meet customer security needs. As a RISE member, Andes Technology is committed to advancing RISC-V integration on OP-TEE, further strengthening secure computing environments.

1. How to boot the OP-TEE in the Andes RISC-V chip. 

OP-TEE is widely utilized in IoT and mobile devices, we are leveraging Physical Memory Protection (PMP) as a fundamental hardware requirement to achieve domain isolation. Below is an example of how trusted and untrusted domain instances are specified in the device tree. (For more details about OpenSBI domains, you can refer to the OpenSBI documentation here)

opensbi-domains {
 compatible = "opensbi,domain,config";

 tmem: tmem {
 compatible = "opensbi,domain,memregion";
 base = <0x0 0xF1000000>;
 order = <24>; /* 16MiB reserved for OP-TEE */
 };

 allmem: allmem {
 compatible = "opensbi,domain,memregion";
 base = <0x0 0x0>;
 order = <64>;
 };

tdomain: trusted-domain {
 compatible = "opensbi,domain,instance";
 regions = <&allmem 0x3f>;
 possible-harts = <&cpu0 &cpu1>;
 next-addr = <0x0 0xF1000000>; /* optee_os: CFG_TDDRAM_START */
 next-mode = <0x1>;
 };

 udomain: untrusted-domain {
 compatible = "opensbi,domain,instance";
 regions = <&tmem 0x0>, <&allmem 0x3f>;
 possible-harts = <&cpu0 &cpu1>;
 boot-hart = <&cpu0>;
 next-addr = <0x0 0x81200000>; /* u-boot: CONFIG_TEXT_BASE */
 next-mode = <0x1>;
 };
};

OpenSBI can subsequently establish exclusive access to a trusted memory (tmem) region where the OP-TEE OS and Trusted Applications (TAs) reside. This configuration enhances the security of the system by ensuring that sensitive operations and data are protected from untrusted domains, thereby maintaining the integrity and confidentiality of the execution environment.

OP-TEE Boot Process

The diagram illustrates a boot process for OP-TEE OS initialization.

During the boot process, the OP-TEE OS automatically modifies the device-tree for 1) reserved-memory node to isolate the PMP protected trusted memory from the Linux address space, and 2) the OP-TEE firmware node that is used to activate the OP-TEE kernel driver.

1. Interrupt Handling in OP-TEE OS

Interrupt handling is a very important feature. In general, TEE architecture has two kinds of interrupts: non-secure interrupts and secure interrupts. The non-secure OS is responsible for handling the non-secure interrupts, while the secure OS or underlying secure monitor is

responsible for handling the secure interrupts.

• Native Interrupts and Foreign Interrupts

From the OP-TEE OS perspective, it roughly divides all the interrupts into “native interrupts” which are handled by OP-TEE OS, and “foreign interrupts” which are not handled by OP-TEE OS. For example: an interrupt from a secure peripheral can be regarded as a native interrupt. An interrupt from a non-secure peripheral can be regarded as a foreign interrupt.

If a foreign interrupt is taken during execution of OP-TEE OS, OP-TEE OS can suspend the current running thread, and yield its execution to other system components to handle this interrupt. See OP-TEE documentation for more detail.

• RISC-V Standard Interrupts

Because the RISC-V’s OP-TEE OS is still in an early stage, we first focus on the 

three major interrupt sources in S-mode:

• supervisor-level external interrupts (sip.SEIP)
• supervisor-level timer interrupts (sip.STIP)
• supervisor-level software interrupts (sip.SSIP)

• Supervisor-level Timer Interrupts

In our PoC, we deployed a non-secure Linux and an OP-TEE OS. The machine timer is shared between two OSes. Linux needs supervisor-level timer interrupts for task scheduling, while OP-TEE OS does not implement any thread scheduler and the trusted threads are scheduled with Linux client caller context (see Trusted thread scheduling). Thus, OP-TEE OS does not use supervisor-level timer interrupts. If OP-TEE OS receives any supervisor-level timer interrupts during execution, the timer interrupts are treated as foreign interrupts. OP-TEE OS will suspend the current thread, and yield its execution to Linux to handle this supervisor-level timer interrupt.

• Supervisor-level Software Interrupts

In our PoC, the handling of supervisor-level software interrupts in OP-TEE OS is similar to the handling of supervisor-level timer interrupts. While Linux uses supervisor-level software interrupts for rescheduling or inter-processor synchronization, OP-TEE OS does not utilize these interrupts. Like supervisor-level timer interrupts, if OP-TEE OS receives any supervisor-level software interrupts during execution, they are treated as foreign interrupts. Consequently, OP-TEE OS will yield its execution to Linux upon receiving any supervisor-level software interrupts.

• Supervisor-level External Interrupts

The external interrupts are more complicated, because OP-TEE OS may manage the secure peripherals and handle their interrupts. Thus, external interrupts cannot be treated in the same way as supervisor-level timer interrupts and supervisor-level software interrupts, which were both regarded as foreign interrupts.

ARM’s GICv3 delivers the native external interrupts via IRQ signal to OP-TEE OS, and the foreign external interrupts via FIQ signal to OP-TEE OS. As a result, OP-TEE OS on ARM architecture can efficiently distinguish between native external interrupts and foreign external interrupts. Currently, RISC-V only has sip.SEIP signal, which is clearly insufficient to distinguish between native external interrupts and foreign external interrupts.

Some new RISC-V specifications, such as the Supervisor Domains specification, are being discussed to potentially address these issues. Since the specifications have not been ratified yet, our OP-TEE OS PoC currently treats all external interrupts received by OP-TEE OS as foreign interrupts.

Case Study-2: Practice of RISC-V OP-TEE in Nuclei System Technology

About Nuclei:  Established in 2018, Nuclei System Technology is a leading RISC-V CPU IP and solution vendor. Starting from scratch, Nuclei has developed 4 general product lines (N/U and NX/UX), and 3 vertical product lines (NS, NA and NI). As of today, Nuclei’s CPU IP has been deployed to various markets, covering AI, automotive electronics, 5G communication, network security, storage, industrial control, MCU, IoT, etc. with more than 200 official customers.

1. Secure boot flow in Nuclei chip

• HSM BootROM: loads hsmboot from flash to HSM ILM, verifies the signature, and decrypts it.
• HSM Boot: loads SPL from flash to Host CLM, verifies the signature, decrypts it, and runs NSBS to await Host requests for crypto services.
• SPL: initializes DDR, loads OpenSBI/OP-TEE/U-Boot from flash to DDR, verifies the signatures, and decrypts them.
• U-Boot: loads the kernel and rootfs from the SD card to DDR, verifies the signatures, and decrypts them.

 

2. Isolation mechanisms for CPU, memory, and interrupt

• Memory isolation: using the PMP mechanism which separates the memory address space between secure and non-secure systems. 
• CPU isolation: using the M mode monitor manages the CPU’s secure state context, responsible for secure state context switching and address space switching.

By integrating Nuclei Secure features, the CPU’s secure state has hardware support, and the BUS, Cache, and TLB also distinguish between hardware secure states.

• Interrupt isolation: 
  • Delegate all Interrupts triggered by the current world to the S-Mode
  • Monitor manages interrupt enable modes. For example, before entering the non-secure world, configure interrupts from secure peripherals to be enabled in M-Mode, and interrupts from non-secure peripherals to be enabled in S-Mode. To avoid race conditions, once in the secure world, non-secure world interrupts are not responded to, but secure interrupts can interrupt the execution of the non-secure world.
  • Nuclei PLIC Hardware Handling: Interrupts enabled in M-Mode cannot have their related registers modified in S-Mode. The CPU can modify PLIC interrupt pending registers only in M-Mode.

Conclusion

This article presents the end-to-end solution supporting OP-TEE in RISC-V with dynamic domain (key techniques from Penglai). It’s a significant step towards the generic RISC-V TEE ecosystem that we can deploy and run commercial trusted Apps on RISC-V devices directly. Our practices in Andes and Nuclei also confirm the practicability.

Acknowledgements: We sincerely thank Dong Du, Yubin Xia, Haibo Chen from SJTU for advising the Penglai project. We thank Yong Li (Intel), the team of RISE project, and Anup Patel for the support of the dynamic domain in OpenSBI. We thank all collaborators from Nuclei and Andes (Huaqi Fang, and others).

Reference

1. “Scalable Memory Protection in the PENGLAI Enclave” Erhu Feng, Xu Lu, Dong Du， Bicheng Yang, Xueqiang Jiang, Yubin Xia, Binyu Zang, and Haibo Chen. In 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21) Jul 2021.
2. Penglai repo, https://github.com/Penglai-Enclave
3. OP-TEE, https://www.trustedfirmware.org/projects/op-tee/
4. ”sIOPMP: Scalable and Efficient I/O Protection for TEEs“ Erhu Feng, Dahu Feng, Dong Du, Yubin Xia, Wenbin Zheng, Siqi Zhao, and Haibo Chen. 2024. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS ’24), Vol. 2. Association for Computing Machinery, New York, NY, USA, 1061–1076. https://doi.org/10.1145/3620665.3640378
5. sPMP, https://github.com/riscv/riscv-spmp
6. Nuclei OpTEE complete solution introduction, https://github.com/OP-TEE/optee_os/issues/6173 
7. OpenSBI Domain: https://static.sched.com/hosted_files/riscvsecurityforum2021/84/OpenSBI_domain_support_v4.pdf?_gl=1*8tk6us*_gcl_au*MjM5OTIzNzYyLjE3MjczMTY3NjI.*FPAU*MjM5OTIzNzYyLjE3MjczMTY3NjI