I have spark df created in AWS Glue with no header and columns with values of ‘0001-01-01-00.00.00.000000’ or ‘0001-01-01-00’.
sample data without header
0001-01-01-00 | 2022-07-09-13.24.60.000000 | yyy
2024-06-01-02 | 0001-01-01-00.00.00.000000 | zzz
i created sample df successfully with below code, and when i tried to add if condition with df.take(1) it is giving me error “year 0 is out of range”.
df.spark.read.format(“txt”).option(“delimiter”, “|”).option(“inferSchema”, True).option(“header”, False).load(“”)
if len(df.take(1)) >0:
True
Note : i used count to validate empty df check but trying to understand why it is giving error for take().