I’m working with a DataFrame in Python using pandas, and I’m trying to apply multiple conditions to filter rows based on temperature values from multiple columns. However, after applying my conditions and using dropna(), I end up with zero rows even though I expect some data to meet these conditions.

The goal is compare with Ambient temp+40 C and if the value is more than this, replace it with NaN. Otherwise, keep the original value“

Here’s a sample of my DataFrame and the conditions I’m applying:

Sample DataFrame:

                          Temp1   Temp2   Temp3   Temp4   Temp5   Temp6   AmbientTemp   
Datetime                                                                                                                                                     
2022-08-04 15:06:00       53.4    57.8    59.0    46.7    52.8    50.1    29.0  
2022-08-04 15:07:00       54.3    57.0    58.8    47.1    53.1    50.5    28.8  
2022-08-04 15:08:00       53.7    57.0    58.7    46.9    53.0    50.3    28.6  
2022-08-04 15:09:00       54.3    57.2    59.1    46.9    53.1    50.3    28.7  
2022-08-04 15:10:00       55.4    57.5    59.7    47.3    53.4    50.6    28.9  

Code:

temp_cols = ['Temp1', 'Temp2', 'Temp3', 'Temp4', 'Temp5', 'Temp6']
ambient_col = 'AmbientTemp'

condition = (df1[temp_cols].lt(df1[ambient_col] + 40, axis=0))

filtered_df = df1[condition].dropna()
print(filtered_df.shape)

Response:

(0, 99)

Problem:

Despite expecting valid data that meets the conditions, the resulting DataFrame is empty after applying the filter and dropping NaN values. What could be causing this issue, and how can I correct it?