I have an image with grayscale changing with cos(wx), and I simulate the fisheye effect on it using the following python code, where K and D are intrinsic matrix and distortion parameter respectively.

def fisheye_effect(original_img):
    h, w = original_img.shape[:2]
    K = np.array([[w, 0, w / 2],
                  [0, h , h / 2],
                  [0, 0, 1]])
    D = np.array([2.0, 1.0, 0, 0])        
    map1, map2 = cv2.fisheye.initUndistortRectifyMap(K, D, None, K, (w, h), cv2.CV_16SC2)
    fisheye_img = cv2.remap(original_img, map1, map2, interpolation=cv2.INTER_LINEAR, borderMode=cv2.BORDER_CONSTANT)
    cv2.imwrite(fisheye_path, fisheye_img)

Here’s the original image

And here’s the distorted fisheye image

Now, my question is that how to undistort the fisheye image based on my self-defined K and D?

I attempted to do it using the following code and it can somehow undistort the image, but the black borders remain.

def undistort(fisheye_img):
    h, w = fisheye_img.shape[:2]
    new_D = np.array([0.3, -0.05, 0, 0])
    new_K = cv2.fisheye.estimateNewCameraMatrixForUndistortRectify(K, new_D, (w, h), np.eye(3), balance=0.0)
    map1, map2 = cv2.fisheye.initUndistortRectifyMap(K, new_D, None, new_K, (w, h), cv2.CV_32F)
    undistorted_img = cv2.remap(fisheye_img, map1, map2, interpolation=cv2.INTER_LINEAR, borderMode=cv2.BORDER_CONSTANT)

Here’s the undistortion result.
Anyone knows how I can undistort the fisheye image and let it look like the original one without cropping out the black border?