I am writing a JIT compiler for an extension of Brainfuck that has functions. Functions are referred through a function pointer that you can increment/decrement and call it.

My approach is that it starts interpreting the code and counting function calls, and if a function is being called X times, it compiles it and starts invoking the compiled function instead.

I am using llvmlite to compile to LLVM and execute by passing a few parameters like the tape state.
Each function is compiled into an LLVM function. The compiled code has (a) a function pointer array that holds all currently compiled functions, and (b) a function index that should be within the range of the above array.

A simple compiled example for two functions is:

function_array = [f1, f2]
function_index = 0
tape = [1..100]
i = 12 // the tape pointer index

void f1(){
     function_index++;
     function_array[function_index]();
}
void f2(){
    tape[i]++;
    print(tape[i]);
}

Complication is when a function attempts to call a function not yet compiled – and there is no way to detect this before running the function. How can I pause execution, compile the callee code, call this callee function, and then resume execution in the caller.

One solution I have is that I’ll put a label before each call instruction (in the compiled code). Then before calling, I do a run-time array length check to see if the function_index is pointing to a valid function. If it doesn’t (meaning it wasn’t compiled), I return this function_index back to the interpreter. Next the interpreter will thus compile the callee and run it. When the callee returns, it calls the caller with some flag such that it jumps to the call, and I also pass in the tape and other states.

But is there a better way to do this?