I have to add new rows to the csv(if it exists) from a dataframe. For this I read the file into a dataframe and use pd.concat to concat the new dataframe and then remove any duplicates and sort it by date and finally write it to the file. I have a problem saving it in right format.

My original data is like

datetime,symbol,open,high,low,close,volume
2023-08-01 09:15:00,XYZ,2550.25,2559.0,2538.05,2542.15,413767.0      
2023-08-01 09:30:00,XYZ,2542.75,2544.5,2532.05,2536.35,261710.0      
2023-08-01 09:45:00,XYZ,2536.25,2540.65,2536.15,2538.0,135671.0      
2023-08-01 10:00:00,XYZ,2538.15,2538.3,2535.0,2536.75,96318.0        
2023-08-01 10:15:00,XYZ,2536.6,2538.9,2529.1,2532.05,177226.0        
2023-08-01 10:30:00,XYZ,2532.7,2534.0,2525.0,2527.3,221328.0

I have this code to save the dataframe when the file doesn’t exist

path = str(f"{filename}.csv")
csv = Path(path)

if csv.exists():

    df = pd.read_csv(path)
    df = pd.concat([df, df1])
    df = df.drop_duplicates()
    df = df.sort_values(by=['datetime'], ascending=True)
    df.to_csv(path)

else:
    df1 = df1.sort_values(by=['datetime'],ascending=True)
    df1.to_csv(path)

`

But when I write to csv after I concat 2 dataframes, there’s a new column added to the old data (read from csv) and the new data without index and sorting fails as well.

,datetime,symbol,open,high,low,close,volume
0,2023-08-01 09:15:00,XYZ,2550.25,2559.0,2538.05,2542.15,413767.0
1,2023-08-01 09:30:00,XYZ,2542.75,2544.5,2532.05,2536.35,261710.0
2,2023-08-01 09:45:00,XYZ,2536.25,2540.65,2536.15,2538.0,135671.0
3,2023-08-01 10:00:00,XYZ,2538.15,2538.3,2535.0,2536.75,96318.0
.
.
.
2024-05-23 09:15:00,,XYZ,2919.0,2937.0,2910.0,2931.1,327859.0
2024-05-23 09:30:00,,XYZ,2930.95,2942.5,2930.1,2938.9,328784.0
2024-05-23 09:45:00,,XYZ,2938.85,2940.9,2935.4,2939.1,165822.0
2024-05-23 10:00:00,,XYZ,2939.35,2939.5,2932.15,2935.7,87516.0

I tried index_col=0 in read_csv but with that the sort throws error.

TypeError: '<' not supported between instances of 'Timestamp' and 'str'

How do I update the csv as in the original format?