When I sorted the rectangles and lines on the two axes of the relative alluvial plot, I found that using factor can only adjust one axis, while the other cannot. What should I do?

The data is as follows:

data <- data.frame(
  Start = c(1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 5, 5, 5, 5),
  End = c(1, 2, 3, 4, 1, 2, 3, 4, 5, 6, 1, 2, 3, 4, 5, 6, 2, 3, 4, 5, 6, 3, 4, 5, 6),
  Area = c(10, 15, 20, 25, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10)
) 

I hope the factor order in the start axis is 2,1, 3, 4, 5, and in the end axis is 3,2,1,4,5,6,
So , I used factor as follows:

data$Start <- factor(data$Start, levels = c("2", "1", "3","4", "5"))
data$End <- factor(data$End, levels = c("3", "2", "1","4", "5","6"))

then,My drawing code is as follows:

library(ggplot2)
library(ggalluvial)
ggplot(data = data,
       aes(axis1 = Start, axis2 = End, y = Area)) +
  geom_alluvium(lode.guidance = "forward",aes(fill = Start),width = 0.1,alpha = 0.8,knot.pos = 0,curve_type = "quintic") +
  geom_stratum(width = 1/7) + 
  geom_text(stat = "stratum", aes(label = after_stat(stratum))) +
  scale_x_discrete(limits = c("1960s", "2020s"), expand = c(0.15, 0.05)) +
  theme_void() +
  scale_fill_manual(values = c("1" = "#4a69bd",
                               "2" = "#54a0ff",
                               "3" = "#82ccdd",
                               "4" = "#b8e994",
                      "5" = "#ff9ff3",
                      "6" = '#e55039'))+
  labs(y = "", fill = "") +
  theme(legend.position = "none")

But always only Start can be sorted