The mmap System Call & Interpreters, JIT Compilers

Disclaimer

I’m just pursuing my curiosity with these posts, so it’s not impossible for some information provided to be inaccurate. If you come across something like this, please let me know, and I’ll correct it – thanks in advance!

Introduction

At the risk of grossly oversimplifying the process, when one writes code in a compiled language like C the compiler converts this source code into an executable for the targeted machine.

Modern(-ish?) languages like Python and Julia, on the other hand, are interpreted (and JIT compiled, at least in the case of a language like Julia), which means source code is read, converted to machine code and executed on the fly.

NOTE: Interpreters can be implemented in a manner that doesn’t require machine code generation. An example of such a model is the virtual machine model which is a program which consumes byte-code (an intermediate form of code) and emulates a physical machine executing the byte-code instructions. The JVM is a popular example.

This process, while conceptually similar to the first, always left me with one simple question:

How does a program (the interpreter in this case) run code that it generates on the fly?

The key here is the mmap (Linux) system call.

The mmap System Call

The man page description for mmap:

mmap, munmap - map or unmap files or devices into memory

This is quite abstract, but for the purposes of this post, we can think of mmap as a way to ask the kernel directly for memory, instead of using user-space allocators like glibc’s malloc. As a result, a lot of the niceties provided by malloc (e.g. tracking the size of allocated memory beginning at a given address) are traded off for control.

As the man page hints, there’s many other interesting applications for mmap, but I’ll focus on its utility in (JIT) interpreters: to execute regions of memory which contain generated code for a script being interpreted.

Let’s look at the signature of mmap (from the man page):

void *mmap(void *addr, size_t length, int prot, int flags,
          int fd, off_t offset);

Compared to malloc, there are 5 more parameters we need to supply. The most important one (in this case) is the prot parameter which allows the programmer to control the protection bits for the allocated memory (i.e. the Read, Write, eXecute bits).

Aside: mprotect

Memory returned by malloc does not have the eXecute bit set, so it can’t be used to run code - one would most probably end up with a segmentation fault. Strictly speaking, this is not true since one can use the mprotect system call to alter the permissions of the region of memory, but this is not advised because as stated in the man page for mprotect:

The behavior of this function is unspecified if the mapping was not established by a call to mmap().

This is, in part, due to page alignment: mmap returns a page aligned address, and the implementation of mprotect may require that the address is a multiple of the page size (from the man page of mprotect).

Executing Shell Code

I want to make a program exit with the code 123 by running the code to do so stored in a mmap allocated buffer. First, we need the shell code corresponding to this. To do so, we can get some help from the compiler.

int main() {
  __asm__(
    "mov $123,%rbx \n"
    "mov $1,%eax   \n"
    "int $0x80     \n"
  );
}

The above C code has inline assembly which just calls the exit system call with an argument of 123. We can compile this to a binary a.out and use objdump -d a.out to get the shell code corresponding to the main function:

0000000000001119 <main>:
    1119:   55                      push   %rbp
    111a:   48 89 e5                mov    %rsp,%rbp
    111d:   48 c7 c3 7b 00 00 00    mov    $0x7b,%rbx
    1124:   b8 01 00 00 00          mov    $0x1,%eax
    1129:   cd 80                   int    $0x80
    112b:   b8 00 00 00 00          mov    $0x0,%eax
    1130:   5d                      pop    %rbp
    1131:   c3                      ret

In this dump, the lines 111d to 1129 correspond to the inline assembly in the main function. We can just use the displayed shell code as the data to be written into the mmap allocated executable buffer. The following code does just that:

#include <sys/mman.h>

#define NULL (void *)0

const unsigned char shellcode[] = {
  0x48, 0xc7, 0xc3, 0x7b, 0x00, 0x00, 0x00, // mov    $0x7b,%rbx
  0xb8, 0x01, 0x00, 0x00, 0x00,             // mov    $0x1,%eax
  0xcd, 0x80                                // int    $0x80
};

int main(int argc, char **argv) {
  // allocate a buffer same size as shellcode, with rwx permissions
  char* buf = mmap(NULL, sizeof(shellcode),
                    PROT_READ | PROT_WRITE | PROT_EXEC,
                    MAP_PRIVATE | MAP_ANONYMOUS,
                    -1, 0);
  if (buf == MAP_FAILED)
    return 1;

  // copy payload over to buffer
  for (int i = 0; i < sizeof(shellcode); ++i)
    buf[i] = shellcode[i];

  // execute the buffer
  ((void (*)(void))buf)();
}

The exit code of this program can be checked using echo $? in Bash-like shells; the result will be 123, so we have successfully run some shell code.