System Calls

A system call is the only door from a user program into the kernel, and everything behind it is hostile territory: the arguments are chosen by code the kernel does not trust. In this lesson your program is the kernel side of the Linux x86-64 system call interface. It dispatches by number through a table, validates user pointers before touching them, and returns results and errors in the register the real ABI uses.

The calling convention

User code puts the system call number in rax and up to six arguments in rdi, rsi, rdx, r10, r8, r9, then executes syscall. The CPU switches to ring 0 at the address in a model-specific register (set by the kernel at boot), and the kernel's entry stub saves the user registers and calls the handler. On return, rax holds the result.

Errors are reported in the same register as negative errno values: a return value between -4095 and -1 means failure, and C library wrappers turn it into -1 with errno set. The ones this lesson uses:

The first numbers of the table you will implement:

Real Linux has more than 300 entries (2 is open, 57 is fork, 59 is execve). Running strace on any program prints exactly this: number, arguments, result.

Dispatch

The handlers live in an array indexed by the number, so a call costs one bounds check and one indirect jump. A number past the end of the table, or a slot that holds nothing, returns -ENOSYS: the same answer for a call that never existed and one that was removed.

Never trust a pointer

write(1, buf, 5) passes the address buf. The kernel runs with full privilege, so if it simply read from that address a user could pass a kernel address and make the kernel copy its own secrets into a file or a pipe. Every pointer must be checked to lie entirely inside the user part of the address space before any byte is copied (copy_from_user and copy_to_user in Linux), and a failed check returns -EFAULT.

The check is buf >= user_start and buf + count <= user_end, and the addition is the trap. With 64-bit arithmetic, buf = 0xffffffffffffff00 and count = 0x200 give buf + count = 0x100 after wrap-around, which is a perfectly fine looking small number. A naive buf + count <= user_end passes it, and the kernel then copies from an enormous range. The correct check also rejects any sum that wrapped past buf.

A worked trace

User memory is [0x1000, 0x2000) and holds the bytes 68 69 ("hi") at 0x1100.

Common mistakes

• Checking only the start of the buffer, not the end, or not guarding the sum against wrap-around.
• Reading user memory before validating the range.
• Returning errors as positive numbers: the convention is negative.
• Treating arguments as signed. They are 64-bit unsigned register values.

Your exercise

The starter parses commands and prints results. You write the range check and the dispatcher with its handlers.