I was reading about spark structured streaming joins and came across the supported types of joins for static to streaming dataset. I have a question, why it is not supported since as far as I understand at a given point in time the streaming dataframe behaves like static and when we join static with streaming it should work. If it supports following
- Left outer join when the left side is a streaming DataFrame
- Right outer join when the right side is a streaming DataFrame
If we change the direction of the dataframes it is not supported and full outer join is also not supported. Let me know what i’m missing here. Essentially I want to understand what is the reason it is not supported what exactly happens which prevents spark from supporting it.