How are you doing?

I am developing a project for Android using AR. The main goal of this project is to identify people’s faces and place a virtual representation of their headsat the same place as the actual head. The virtual mesh is generated based on a series of images and download from a cloud into the device that’s running the application. For that, I am using Google’s MediaPipe, which identify some parts of the real face (like eyes, nose and mouth, among others) and place the virtual head based on them.

The application is being developed on Unity. There is a script that reads what is being captured by the device’s back camera and display it into a image object nested within a Canvas. Then, the actual virtual head object is placed, like in front of the canvas, based on the math done by the MediaPipe’s algorithm.

Everything is pretty decent, But the problem that is happening is that, irrespective from the device that is being used (and from the camera), the virtual head placement and synchronization stays a little off, and does not match entirely with the real-word head.

Image 1

In red, we have the virtual have generated, and behind it the capture running in the Unity’s canvas. From this image, we can already see that is off: the virtual object seems bigger

If we change the angle, it gets worse.

Image 2

I figure that this has something to do with lens distortion, and the actual focal length and field of view. But I was not able to find a solution.

What I tried was changing the canvas to a plane, in order to check if had anything to do with that. Also tried to change Unity’s camera to a physical camera and it’s focal length into a better value. The closest that I got was the following:

Image 3

Which is still off, but better. The problem still happens with a different angle, though.

Image 4

Also tried using ARCore for that, but the problem is that it only allows detection with the selfie camera, and that’s not what I need.

Any tips ou solutions for that?
Thanks 🙂