In a previous post, we considered how you could create an optimized ISA for a domain-specific processor core by profiling software and experimenting with adding/removing instructions. Using the open RISC-V ISA can be a great starting point for a processor that combines application-specific capabilities and access to portable software.
The old-fashioned way to modify the instruction set would be to firstly manually modify the instruction set simulator (ISS) to change the ISA, and then to update the SDK to reflect the new set of target instructions. This requires an extensive amount of manual work with associated technical risks. The resulting SDK will almost certainly need to make any custom instructions available as intrinsics or as inline assembler code. The alternative of modifying and verifying the compiler is costly in effort, but the end result is much better for the software developers. Similarly, if a processor is extended, traditionally it would be necessary to modify the microarchitecture by editing the RTL and then verify it against the ISS as the golden reference.