I’m working on a GPU picking set up for instances in a 3D space using react three fiber – I’ve seen this done with threeJS, so essentially porting over.

I have set up a main scene with my instaces in it, I have also copied those instances into a pickingScene and then using useFBO to render that to it’s own renderTarget (I’ve set the width and height of this renderTarget to my screen dimensions…i’m not sure if this is necessary yet?).

Then, the instances in my picking scene, I am assigning a unique vertex colour too, based on the instace index. For debugging, i’m returning a full screen plane, with the renderTarget mapped to it. If i push this back a little into my actual scene, then i can perfectly see an overlay of my actual scene spheres with the picking spheres, all in different colours. So, that confirms that the renderTarget is doing what it should do.

Now the next step is to try and read the pixel colour under my mouse in the pickingScene, in order to identify which sphere i’m hovering over in the real scene – but, I can’t seem to get this setup to work, and i’m not too sure what I’m missing. My renderTarget is set up as follows:

const pickingRenderTarget = useFBO({
    width: viewport.width,
    height: viewport.height
});

and then I have a useFrame to get my mouse position and use readRenderTargetPixels to try and return the colour under my mouse in the pickingRenderTarget, it looks like this:

useFrame(({ mouse, camera }) => {
    if (!meshRef.current) return;
    gl.setRenderTarget(pickingRenderTarget);
    gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT);
    gl.render(pickingScene, camera);

    const x = Math.floor(((mouse.x + 1) / 2) * pickingRenderTarget.width);
    const y = Math.floor(((1 - mouse.y) / 2) * pickingRenderTarget.height);
    const pixelBuffer = new Uint8Array(4);
    gl.readRenderTargetPixels(pickingRenderTarget, x, y, 1, 1, gl.RGBA, gl.UNSIGNED_BYTE, pixelBuffer);

    console.log(`Pixel RGBA at (${x}, ${y}):`, pixelBuffer);

    gl.setRenderTarget(null);

    if (debugMeshRef.current) {
        debugMeshRef.current.material.map = pickingRenderTarget.texture;
    }
});

To see what I mean, I have created a code sandbox here:
https://codesandbox.io/p/sandbox/wispy-fast-v8y6zt

You’ll be able to see in console, whlist I can log my mouse position, I can never log the output of the pixelBuffer