I am doing the following in spark sql:
spark.sql("""
SELECT
data.data.location.geometry.coordinates[0]
FROM df""")
This works fine, however I do not want to use raw SQL, I use dataframe API like so:
df.select("data.data.location.geometry.coordinates[0]")
Unfortunately this does not work:
AnalysisException: [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve "data.data.location.geometry.coordinates[0]" due to data type mismatch: Parameter 2 requires the "INTEGRAL" type, however "0" has the type "STRING".;
'Project [data#680.data.location.geometry.coordinates[0] AS 0#697]
+- Relation [data#680,id#681,idempotencykey#682,source#683,specversion#684,type#685] json
I now that I can use the F.col api and go with a getItem(0), but is there a built-in way to have the shortcut of getItem?
‘.’ is the shortcut of getField is there one for array slicing?
Thank you for your insight