I am trying to download CSV files from this website:

https://www.misoenergy.org/markets-and-operations/real-time–market-data/market-reports#nt=%2FMarketReportType%3AHistorical%20LMP%2FMarketReportName%3ADay-Ahead%20Market%20ExPost%20LMPs%20(csv)&t=10&p=0&s=MarketReportPublished&sd=desc

and I want to download all the files from 2022/01/01, which is on the last page, all the way to the newest file, which comes out daily. From this screenshot of the website, it’s evident that there is a pattern to the naming of the files, such that all files include the year, month, and day (e.g. 20240525) and all end in _da_expost_lmp.csv.

Therefore, how could I write a Python code to download these files from the earliest date in that section to the most recent one updated daily? I understand that possibly I may need to use a loop of some kind, but any clarification would be useful as I am new to Python and don’t want to resort to ChatGPT for learning purposes.

This is my existing code:

import urllib3     

import pandas as pd     

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)     

df = pd.read_csv('https://docs.misoenergy.org/marketreports/20240524_da_expost_lmp.csv', skiprows = 4)     
df.head()  

How could I update this to my request? Any help would be appreciated; thank you.