I have a simple ARKit/SceneKit app that draws a cube in front of the camera. Using the depth map, I want to hide the cube pixels that are occluded by an object. RealityKit supports occlusion, but that’s only on objects it is trained to recognize, such as human bodies and planes, and since I’m looking to detect occlusion on a drill, it doesn’t work for me.

Here’s how I’m approaching the solution:

• Scene Setup: When the app loads, it creates a SCNBox and adds it to the scene.
• Shader Implementation: I apply a material with a .fragment shaderModifier to the box. This shader uses the lidar depth map to decide whether to render each pixel.
• Depth Comparison:
  1. Obtain screen coordinates of the fragment.
  2. Fetch the physical depth from the depth map.
  3. Compare it with the virtual depth.
  4. Render the pixel only if the virtual object is closer than the physical object.

Unfortunately I can’t get it to work. I believe the main problem is getting the x/y screen position to access the physical depth from the depth map. I’ve simplified everything and posted it up. All the important code is at https://github.com/StevePotter/ARKitOcclusion/blob/main/ARKitOcclusion/ARController.swift

The shader code, which is also here:

#pragma arguments
texture2d<float, access::sample> depthMap;
float screenWidth;
float screenHeight;

#pragma body
constexpr sampler depthSampler(coord::pixel);

float4 modelSpacePosition = scn_node.modelViewTransform * float4(_surface.position, 1);
// the depth, in meters, of the fragment from the camera
float modelDepth = -modelSpacePosition.z;

// screenPosition IS DEFINITELY WRONG.  I think it will require some math with scn_node.inverseModelViewTransform or scn_node.modelViewProjectionTransform
float2 screenPosition = _surface.diffuseTexcoord * float2(screenWidth, screenHeight); // x and y pixel coordinates of this fragment on the screen
float physicalDepth = depthMap.sample(depthSampler, screenPosition).r;

// if the depth map detects something before this fragment, don't paint
if (physicalDepth <= modelDepth) {
   discard_fragment();
}

I’ve scoured the web and couldn’t find something that worked. Hoping one of you geniuses can help solve it. This is really important to me, so I’m offering a nice reward for the person who can get it working. Thanks so much everyone!