I am trying to read a nested JSON file using simple spark.read.json and it is giving me expected result with proper schema definition.

But when am trying to do the same using databricks Autoloader code, it is only giving me the root elements and not the nested elements.

#Working code
    df = spark.read.json(f'abfss://[email protected]/bronze/trend/", multiline=True)
    df.printSchema()

#Not working code
     df1 = (spark.readStream
       .format("cloudFiles")
       .option("cloudFiles.format", "json")
.option("cloudFiles.schemaLocation",'abfss://[email protected]/checkpoint/bronze_to_silver/trend')
       .option("multiline", True)
       .load('abfss://[email protected]/bronze/trend/'))

    df1.printSchema()