I have the following code which is supposed to draw an image at specific region on a red background. At first I did not draw any image, just making the background red:

import matplotlib.pyplot as plt
import numpy as np


papersize = (8.5, 11)

fig = plt.figure(figsize=papersize)
ax = fig.add_axes(rect=(0, 0, 1, 1))
ax.add_patch(plt.Rectangle((0, 0), 1, 1, color="red", zorder=-1))

# this row changes the layount of the image
# ax.imshow(np.asarray([[0, 1], [1, 0]]), cmap="gray", extent=(0.25, 0.75, 0.25, 0.75))

fig.savefig("test.png", dpi=200)

Which creates this nice red A4-sized image

But when I uncomment the line with imshow, instead of drawing a simple image in the middle, all hell breaks lose, resulting in the following image that spans the full width and other mess:

I have no idea why this happens or how to prevent it. Please tell me what am I missing.
The goal is to be able to draw multiple images at locations specified by the extent, while NOT breaking the layout of everything else.

I want to have a figure of the specified size, have the axes span the whole figure and just have a drawing canvas for me to put images on.