We are writing data frame to a directory like below in pySpark code.
Its writing with file name as “part-****”, is there anyway to get the name of the file in code after its written?
In our case there are other diff jobs will be writing file to same target directory so want to know which specific file is generated by this job code.
df.write
.format(file_type)
.option(strings.str_header_lc, header)
.option(strings.str_delimiter_lc, delimiter)
.mode(mode)
.save(path)
Thanks,
Mahendra