CLUSTER BY is succesfully executed at Spark version3.4.
But, It failed when I ran the same code at Spark version3.5.
%%sql
SELECT SalesOrderLineNumber, SalesOrderNumber,'' as CustomerName, SUM(Quantity), SUM(UnitPrice), SUM(TaxAmount)
FROM sales_df
GROUP BY SalesOrderLineNumber, SalesOrderNumber
UNION ALL
SELECT SalesOrderLineNumber,'' as SalesOrderNumber, CustomerName, SUM(Quantity), SUM(UnitPrice), SUM(TaxAmount)
FROM sales_df
GROUP BY SalesOrderLineNumber, CustomerName
UNION ALL
SELECT '' asSalesOrderLineNumber, SalesOrderNumber, CustomerName, SUM(Quantity), SUM(UnitPrice), SUM(TaxAmount)
FROM sales_df
GROUP BY SalesOrderNumber, CustomerName
CLUSTER BY (SalesOrderLineNumber, SalesOrderNumber)
Does Spark 3.5 support CLUSTER BY function?
New contributor
a-tkim is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.