i have this piece of code where i am loading a file from a stream and then adding a column. how would i move the column to the front of the dataframe? i want to prepend the column, i do not want to reorder after.
<code>@dlt.table(name="DEMO")
def table():
return (
spark.readStream.format("cloudFiles")
.option("cloudFiles.Format", "PARQUET")
.load("abfss://...")
.withColumn("DEMO", helper.transform(col("MYCOLUMN_NAME")))
)
</code>
<code>@dlt.table(name="DEMO")
def table():
return (
spark.readStream.format("cloudFiles")
.option("cloudFiles.Format", "PARQUET")
.load("abfss://...")
.withColumn("DEMO", helper.transform(col("MYCOLUMN_NAME")))
)
</code>
@dlt.table(name="DEMO")
def table():
return (
spark.readStream.format("cloudFiles")
.option("cloudFiles.Format", "PARQUET")
.load("abfss://...")
.withColumn("DEMO", helper.transform(col("MYCOLUMN_NAME")))
)
i want something close to this,
<code>@dlt.table(name="DEMO")
def table():
return (
spark.readStream.format("cloudFiles")
.option("cloudFiles.Format", "PARQUET")
.withColumn("DEMO", helper.transform(col("MYCOLUMN_NAME")))
.load("abfss://...")
)
</code>
<code>@dlt.table(name="DEMO")
def table():
return (
spark.readStream.format("cloudFiles")
.option("cloudFiles.Format", "PARQUET")
.withColumn("DEMO", helper.transform(col("MYCOLUMN_NAME")))
.load("abfss://...")
)
</code>
@dlt.table(name="DEMO")
def table():
return (
spark.readStream.format("cloudFiles")
.option("cloudFiles.Format", "PARQUET")
.withColumn("DEMO", helper.transform(col("MYCOLUMN_NAME")))
.load("abfss://...")
)
But this obviously should not work. Note that the new column depends on record metadata.