I’d like to plot an inset map in the top left corner of another map using package patchwork. Yet, when using coord_sf(), the alignment does not work anymore. How to solve this to ensure the inset map is placed in the top left corner and aligns with the borders of the big map?

library(ggplot2)
library(patchwork)

p1 <- ggplot(iris) +
  geom_point(aes(x = Sepal.Length, y = Sepal.Width)) +
  coord_sf()

p2 <- ggplot(iris) +
  geom_histogram(aes(x = Petal.Width)) +
  coord_sf()

p1 + inset_element(p2, 0, 0.6, 0.4, 1, clip = TRUE, align_to = "panel")

Without coord_sf(), the plots are aligned correctly: