I want to add two arrows with corresponding text to the outside of the Y axis in a ggplot2 plot, but the X axis is a categorical variable, and I have only found solutions (implemented below) when both variables are numerical. Is there a way to do this when one or both of them are categorical?

Here is a minimal reprex using the iris dataset and the clip = "off" argument in coord_cartesian():

library(tidyverse)

iris |> 
  ggplot(aes(x = Species, y = Sepal.Length)) +
  geom_boxplot(aes(fill = Species)) +
  coord_cartesian(clip = "off") +
  annotate(
    "text",
    x = 1, xend = 1, y = 7.5, yend = 8,
    label = "Longer sepal",
    angle = 90
  ) +
  annotate(
    "text",
    x = 1, xend = 1, y = 5.5, yend = 6,
    label = "Shorter sepal",
    angle = 90
  ) +

(I have not added the code to add the arrows to keep the example short, but a solution for the text annotation would also work instantly for the arrows). If use negative values for the x coordinates, it just extends the grey plot area to the left, but still places the text annotation inside the plot, and not to the left of the Y axis.

However, I want to achieve something similar to this mockup: