It looks like spark sql unable to do this kind of optimization. Here’s my sample code:

I’m trying to find specific documentation about how Spark’s Catalyst optimizer works in detail because I plan on customizing it in a way that’s tailored to reduce resource usage rather than computational time.