While reading the OSTEP book, I had a question.
From the book:
To specify the exact system call, a system-call number is usually assigned to each system call. The user code is thus responsible for placing the desired system-call number in a register or at a specified location on the stack; the OS, when handling the system call inside the trap handler, examines this number, ensures it is valid, and, if it is, executes the corresponding code. This level of indirection serves as a form of protection; user code cannot specify an exact address to jump to, but rather must request a particular service via number.
I’m trying to better understand how system call numbers work within the context of system calls and trap handling. Here’s what I understand so far:
- The system call number is used like a key or index in a trap table, allowing the CPU to jump to the correct trap handler.
- Inside the trap handler, the operating system validates the system call number.
- The C library’s system call API stores the system call number in a register or on the stack before invoking the trap.
Is my understanding accurate? Could someone clarify or provide additional details, especially regarding the role of the C library and the validation step?
11
- The system call number is used like a key or index in a trap table, allowing the CPU to jump to the correct trap handler.
No.
A “trap table” can be found in CPU architectures that support a TRAP N instruction where N is the trap number. In such architectures, the trap table maps trap numbers to addresses of trap handlers. An operating system typically defines one trap number for offering system calls.
Suppose we have a hypothetical CPU which supports multiple trap handlers, and an operating system which uses TRAP 42 for system calls.
The system call number is something you have to set up before invoking the operating system trap. For example:
LOAD R1, system_call_number
TRAP 42
So, the CPU uses that 42 to look up the trap handler in the trap table, and jumps to it. This will be the operating system trap handler. Within that handler, the operating system uses the value of R1 to look up, in another table, (call it the “system call” table,) the address of the system function, and jump to it.
Note that the process is very similar if the CPU supports a parameterless instruction for invoking the operating system:
LOAD R1, system_call_number
SYSCALL
In this case, there is only one handler, and no trap table. However, it is still possible to make many different system calls, so the mechanism with system call numbers, resolvable via a system call table, is still there.
- Inside the trap handler, the operating system validates the system call number.
Uh, sure. If the system call number does not correspond to an actual valid system function, the operating system will probably do bad things to your process.
- The C library’s system call API stores the system call number in a register or on the stack before invoking the trap.
I do not know what you mean by “The C library’s system call API”. In any case, anyone interested in invoking system calls must store the system call number somewhere before invoking the operating system. The operating system defines where the number should be stored. It is usually in a register.
3
Refer to @MikeNakis’s answer for most details. I just want to respond to this point:
The C library’s system call API stores the system call number in a register or on the stack before invoking the trap.
That’s a bit fraught. Perhaps it’s just a poor choice of words, but in case there is a genuine misunderstanding here:
The system call design described in your book has nothing directly to do with any userspace libraries. In principle, any userspace program can request a system call by the appropriate OS-specific version of the mechanism described by your quotation, without resorting to a library function.
However, if your system offers a hosted C implementation, then many of the functions in the C standard library it provides will indeed perform system calls as described. Additionally, you may be thinking in particular of syscall wrapper functions provided as extensions by some systems’ C implementations. These do little other than providing a convenient interface to a particular system call. Also, some systems’ C standard libraries provide a generic syscall wrapper that requires the caller to specify the syscall number and to provide appropriate argument in the form expected by the raw syscall. Except possibly for the last, I don’t think any of that is well characterized as a “system call API”.
1