Context

My goal is to catch SIGSEGVs from native code, since I want to prevent Minecraft (Java) from crashing whenever a mod written in my compiled modding language contains accidental infinite recursion.

I’ve been able to successfully do this for other programs (written in C/C++/Python) with sigaction() and sigsetjmp(), but it isn’t as easy in Java, as described by this Stack Overflow answer:

The Java VM (at least Oracles implementation, which also includes the OpenJDK) uses POSIX signals for internal communication, so it installs signal handlers e.g. for SIGSEGV. That means, to work correctly, the Java VM must get and examine any SIGSEGV which happens in the process, to distinguish between “communication” SIGSEGVs and real ones which would be program errors.

But signal handlers are a global resource, within a process any native code can install signal handlers and replace the ones the Java VM installed.

To solve this problem (user-installed signal handlers replace the JavaVM signal handlers) and to accomodate user code which may have a reason to install signal handlers, Java VM uses “signal chainging”, which basically means that the signal handlers (VM and Users) are chained behind each other: the JavaVM signal handlers run first; if they think the signal is of no interest to the VM, it hands down the signal to the user handler.

The libjsig.so is the answer for this problem: it replaces the system signal APIs (sigaction() etc) with its own versions, and any user code attempting to install a signal handler will not replace the global signal handler but the user handler will be chained behind the (already installed) Java VM signal handler.

It’s incredibly easy to accidentally have undefined behavior when sigsetjmp()ing out of a signal handler, as described in the APPLICATION USAGE section of this page, which is why my modding language blocks (using sigprocmask) and disables the signal handler at strategic moments. For the below minimal reproducible example I’m not being as careful, since the stack trace seems unrelated to undefined behavior.

Minimal reproducible example

Main.java:

class Main {
    private native void init();
    private native void foo();

    public static void main(String[] args) {
        System.loadLibrary("foo");

        new Main().run();
    }

    public void run() {
        System.out.println("Running in Java...");

        init();

        while (true) {
            foo();
        }
    }
}

foo.c:

#include <jni.h>
#include <setjmp.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>

jmp_buf jmp_buffer;

static void segv_handler(int sig) {
    (void)sig;

    siglongjmp(jmp_buffer, 1);
}

JNIEXPORT void JNICALL Java_Main_init(JNIEnv *env, jobject obj) {
    (void)env;
    (void)obj;

    printf("Initializing...n");

    static struct sigaction sigsegv_sa = {
        .sa_handler = segv_handler,
        .sa_flags = SA_ONSTACK, // SA_ONSTACK gives SIGSEGV its own stack
    };

    // Handle stack overflow
    // See /a/7342398/13279557
    static char stack[SIGSTKSZ];
    stack_t ss = {
        .ss_size = SIGSTKSZ,
        .ss_sp = stack,
    };

    if (sigaltstack(&ss, NULL) == -1) {
        perror("sigaltstack");
        exit(EXIT_FAILURE);
    }

    if (sigfillset(&sigsegv_sa.sa_mask) == -1) {
        perror("sigfillset");
        exit(EXIT_FAILURE);
    }

    if (sigaction(SIGSEGV, &sigsegv_sa, NULL) == -1) {
        perror("sigaction");
        exit(EXIT_FAILURE);
    }
}

void recurse() {
    recurse();
}

JNIEXPORT void JNICALL Java_Main_foo(JNIEnv *env, jobject obj) {
    (void)env;
    (void)obj;

    if (sigsetjmp(jmp_buffer, 1)) {
        fprintf(stderr, "Jumpedn");
        return;
    }

    printf("Recursing...n");

    recurse();
}

Compiling foo.so:

gcc foo.c -o libfoo.so -shared -fPIC -g -Wall -Wextra -Wpedantic -Werror -Wfatal-errors -Wno-infinite-recursion -I/usr/lib/jvm/jdk-23.0.1-oracle-x64/include -I/usr/lib/jvm/jdk-23.0.1-oracle-x64/include/linux

Running Main.java, with libjsig.so:

LD_PRELOAD=/usr/lib/jvm/jdk-23.0.1-oracle-x64/lib/server/libjsig.so java -Xcheck:jni -Djava.library.path=. Main.java

prints this:

Running in Java...
Initializing...
Recursing...
Segmentation fault (core dumped)

where running coredumpctl debug shows that the segfault wasn’t caught by my SIGSEGV handler, since the top of the stack trace just contains many calls to recurse().

If I leave the LD_PRELOAD=/usr/lib/jvm/jdk-23.0.1-oracle-x64/lib/server/libjsig.so out when running Main.java, it complains that it wants me to use jsig, which is expected. The important thing here is that this does run forever, as I desired (I’ve added comments to the log), but it of course stomps on JVM’s own SIGSEGV handler, so it’s no good in reality:

Running in Java...
Initializing...
Recursing... # Repeated many times
Jumped # Repeated many times
Warning: SIGSEGV handler modified!
Signal Handlers:
   SIGSEGV: segv_handler in libfoo.so, mask=11111111011111111101111111111110, flags=SA_ONSTACK, unblocked
  *** Handler was modified!
  *** Expected: Jumped
Recursing... # Repeated many times
Jumped # Repeated many times
Recursing...
javaSignalHandler in libjvm.so, mask=11100100110111111111111111111110, flags=SA_RESTART|SA_SIGINFO
    SIGBUS: javaSignalHandler in libjvm.so, mask=11100100010111111101111111111110, flags=SA_RESTART|SA_SIGINFO, unblocked
    SIGFPE: Jumped
Recursing...
javaSignalHandler in libjvm.so, mask=11100100010111111101111111111110, flags=SA_RESTART|SA_SIGINFO, unblocked
   SIGPIPE: javaSignalHandler in libjvm.so, mask=11100100010111111101111111111110, flags=SA_RESTART|SA_SIGINFO, unblocked
   SIGXFSZ: javaSignalHandler in libjvm.so, mask=11100100010111111101111111111110, flags=SA_RESTART|SA_SIGINFO, unblockedJumped

Recursing...
    SIGILL: javaSignalHandler in libjvm.so, mask=11100100010111111101111111111110, flags=SA_RESTART|SA_SIGINFO, unblocked
   SIGUSR2: SR_handler in libjvm.so, mask=00000000000000000000000000000000, flags=SA_RESTART|SA_SIGINFO, unblocked
    SIGHUP: UserHandler in libjvm.so, mask=Jumped
11100100010111111101111111111110, flags=Recursing...
SA_RESTART|SA_SIGINFO, unblocked
    SIGINT: UserHandler in libjvm.so, mask=11100100010111111101111111111110, flags=SA_RESTART|SA_SIGINFO, unblocked
   SIGTERM: UserHandler in libjvm.so, mask=11100100010111111101111111111110, flags=SA_RESTART|SA_SIGINFO, unblocked
   SIGQUIT: UserHandler in libjvm.soJumped
, mask=11100100010111111101111111111110Recursing...
, flags=SA_RESTART|SA_SIGINFO, blocked
   SIGTRAP: SIG_DFL, mask=00000000000000000000000000000000, flags=none, unblocked
Consider using jsig library.
Jumped
Recursing...
# Runs forever

Running java --version prints this, and I’m on Ubuntu 24.04.1, in case it’s of use:

java 23.0.1 2024-10-15
Java(TM) SE Runtime Environment (build 23.0.1+11-39)
Java HotSpot(TM) 64-Bit Server VM (build 23.0.1+11-39, mixed mode, sharing)