I am retrieving data from an API which contains multiple pages.

The required fields from the initial page is added to a pandas dataframe – in my code this variable is originally defined as df.

From the original API response, I am able to to understand the total amount of pages in the full response, and hence I can create a loop to iterate through the pages and sequentially extract all the information.

Inside the loop I am concatenating each page to the original dataframe (df), but here is where I am receiving a “Future Warning“

Here is the code:

df = call_to_api(1)
# call_to_api returns the dataframe, the parameter is the page number

current_page = response['Pagination']['PageNumber']
total_pages = response['Pagination']['NumberOfPages']

while current_page < total_pages:
    current_page = current_page + 1
    df2 = call_to_api(current_page)
    df_list = [df, df2]
    df = pd.concat([df for df in df_list if not df.empty], ignore_index=True) #Future Error happens here

Here is the full Error I am getting:

FutureWarning: The behavior of DataFrame concatenation with empty or
all-NA entries is deprecated. In a future version, this will no longer
exclude empty or all-NA columns when determining the result dtypes. To
retain the old behavior, exclude the relevant entries before the
concat operation

What is the best way to write this code so that I dont get the Future Error message?