The camera created from an OpenCV camera pose does not project the vertices the same way as MeshRenderer
:
import numpy as np
import matplotlib.pyplot as plt
import torch
from pytorch3d.structures import Meshes
from pytorch3d.renderer import MeshRenderer, MeshRasterizer, RasterizationSettings, SoftPhongShader
from pytorch3d.renderer import TexturesVertex
from pytorch3d.utils import cameras_from_opencv_projection
# Camera parameters
R = np.eye(3)
t = np.array([0, 0, 5])
f = 1731.2994092344502
w, h = 1280, 720
K = np.array([
[f, 0, w/2],
[0, f, h/2],
[0, 0, 1],
])
# Mesh parameters
verts = np.array([
[0.0, 0.0, 0.0],
[0.5, 0.5, 0.0],
[0.5, 0.0, 0.0],
])
faces = torch.tensor([[[0, 1, 2]]])
verts = torch.FloatTensor(verts).unsqueeze(0)
tex = TexturesVertex(torch.ones_like(verts))
# Create camera
camera = cameras_from_opencv_projection(
torch.FloatTensor(R).unsqueeze(0),
torch.FloatTensor(t).unsqueeze(0),
torch.FloatTensor(K).unsqueeze(0),
torch.Tensor([w, h]).unsqueeze(0),
)
# Create mesh and renderer
mesh = Meshes(verts, faces, tex)
renderer = MeshRenderer(
rasterizer=MeshRasterizer(
raster_settings=RasterizationSettings(image_size=(h, w)),
cameras=camera,
),
shader=SoftPhongShader(cameras=camera),
)
# Render image
mesh_img = renderer(mesh)
mesh_img = mesh_img.squeeze().cpu().numpy()
# Project vertices
points = camera.transform_points_screen(mesh.verts_packed())
points = points.squeeze().cpu().numpy()[:, :2]
# Plot
plt.figure(figsize=(10, 5))
plt.imshow(mesh_img)
for pt in points:
plt.scatter(pt[0], pt[1], color='red', s=50)
plt.show()
That code produces the following image:
I was expecting the vertices projected by PerspectiveCamera.transform_points_screen
would align with the image rendered by MeshRenderer, but they clearly do not. The points generated by PerspectiveCamera.transform_points_screen
match my expectations, but why does the image created by MeshRenderer
not line up?
New contributor
shmic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.