I wish to create an ID column in Spark SQL which is auto-incrementing. Since it is not directly possible, I tried to search a method which is similar to Postgres’s RETURNING ID

I tried going with the monotonically_increasing_id(), but it is only applicable after a table has been created and then assigning the ID to the table.

I wish to have a method which either gives me the latest ID of the column and then I increment it while inserting data; or I have a method to auto-increment the ID column.