I am working on a project where I need to visualize a 3D object mesh using a virtual camera in VTK, aiming to match the real-world camera setup. I use SolvePNP to obtain the rotation vector (rvec) and translation vector (tvec) of the object, and apply these to the VTK camera. Additionally, I use camera intrinsics obtained from calibration.

The camera setup works well but is not as accurate as I need it to be. Specifically, the synthetic view appears slightly larger than the real image. To compensate, I manually increase the FOV along the Y-axis, which I would prefer to avoid.

Undistorting the image using the distortion coefficients from the camera calibration did not seem to help and only made it worse. I assume this is not the issue considering rvec and tvec are extracted from the distorted image by manually clicking points, and they correspond only to those points, which has nothing to do with the distortion.

The issue seems to be related to the size of the synthetic view (Size of the mesh) rather than alignment.

Here is the relevant portion of my code:

    def setup_camera(rvec, tvec, intrinsic, image_width, image_height):
    camera = vtk.vtkCamera()

    R, _ = cv2.Rodrigues(rvec)
    R_inv = R.T

    t_inv = -R_inv @ tvec
    
    camera.SetPosition(t_inv.flatten())
    
    # focal point (assuming the camera looks along the Z axis in its local coordinate system)
    focal_point = t_inv.flatten() + R_inv @ np.array([0, 0, 1])
    camera.SetFocalPoint(focal_point)
    
    up_vector = R_inv @ np.array([0, -1, 0]) 
    camera.SetViewUp(up_vector)
    
    fov_y = 2 * np.arctan((image_height / 2) / intrinsic[1, 1]) * (180 / np.pi)
    fov_y *= 1.025 # Manual adjustment (I want to avoid this)
    camera.SetViewAngle(fov_y)
    
    camera.SetClippingRange(0.1, 1000)
    return camera

Here are examples on how they match when overlaid – drawn in Paint (Black square – Real object, Red square – Synthetic view):

FOV_Y NOT MODIFIED

FOV_Y MODIFIED (FOV_Y*=1.025) Manual adjustment for a slightly better match – I want to avoid this

Are there any steps I might be missing in translating the camera parameters from SolvePNP and calibration to VTK?
Any advice or suggestions on how to enhance the accuracy of this virtual camera setup would be greatly appreciated.