Getting a Pyspark reference error here (full code below): ‘if fi.stat().st_ctime >= today_midnight:’. Working in Palantir Foundry, using a transform to rename a json file. Any idea what is causing this? Thx!!
Full Error: There is an incorrect call to a Column object in your code. Please review your code.
curr_date = date.today()
mytime = dt.datetime.strptime('0000','%H%M').time()
today_midnight = dt.datetime.combine(curr_date, mytime)
@transform(
asset_input = Input('ri.foundry.main.dataset.000000000000000000000000000'),
asset_rename_output= Output('/Folder/OtherFolder/Output'),
)
def rename_func(asset_input, asset_rename_output):
new_filename='test_newname'
json_files = asset_input.filesystem().files('**/*.json')
for fi in json_files:
if fi.stat().st_ctime >= today_midnight:
shutil.move(fi, new_filename)
asset_rename_output.write_dataframe(new_filename)```