If we use ggplot2, and ggExtra, on a scatterplot, the ggMarginal will add marginal histograms, but only for X and Y dimensions.

I want to add a histogram onto the color legend. Say we have 3 dimensions, x,y,z. The scatterplot displayes x,y as location, and then z as color. We then have a legend concerning the color of points. Is there a way to add a histogram to the legend, so that the viewer can see relative frequency beyond just guessing from visual interpretation of color?

Starting point:

x = c(rnorm(100, 2, 2), rnorm(100, -1, 3))
y = c(rnorm(100, -1, 3), rnorm(100, 2, 1))
z = c(rnorm(50, -2, 2), rnorm(100, 1, 1), rnorm(50, 3, 2))

d = data.frame(x, y, z)
library(ggplot2)
ggplot(d, aes(x = x, y = y, color = z)) + geom_point() + scale_color_viridis_c()

I want to add a histogram to the legend for z. Any idea how?