I have a dataframe with lists of nested dictionaries that want to unpack.
I need to get the date and price from the priceHistory
and the items listed in both WaterConservation
and EnergyEfficient
. The sample below is only two rows of a much larger dataframe where there are not the same number of dictionary items per dataframe row.
df = pd.DataFrame(
[[19, [{'priceChangeRate': 0, 'date': '2015-05-29', 'source': 'Public Record', 'postingIsRental': False, 'time': 1432857600000, 'sellerAgent': None, 'showCountyLink': False, 'attributeSource': {'infoString2': 'Public Record', 'infoString3': None, 'infoString1': None}, 'pricePerSquareFoot': 275, 'buyerAgent': None, 'event': 'Sold', 'price': 877205}], ['Low flow commode', 'Low flow fixtures', 'Water-Smart Landscaping'],''],
[89, [{'priceChangeRate': 0.090909090909091, 'date': '2023-07-14', 'source': 'Public Record', 'postingIsRental': False, 'time': 1689292800000, 'sellerAgent': {'name': 'seller1', 'photo': {'url': 'https://sellerphoto1.jpg'}, 'profileUrl': '/profile/sellerprofile1/'}, 'showCountyLink': False, 'attributeSource': {'infoString2': 'Public Record', 'infoString3': None, 'infoString1': None}, 'pricePerSquareFoot': 308, 'buyerAgent': {'name': 'buyer1', 'photo': {'url': 'https://buyerphoto1.jpg'}, 'profileUrl': '/profile/buyerprofile1/'}, 'event': 'Sold', 'price': 1200000}, {'priceChangeRate': 0, 'date': '2015-08-20', 'source': 'Public Record', 'postingIsRental': False, 'time': 1440028800000, 'sellerAgent': None, 'showCountyLink': False, 'attributeSource': {'infoString2': 'Public Record', 'infoString3': None, 'infoString1': None}, 'pricePerSquareFoot': 50, 'buyerAgent': None, 'event': 'Sold', 'price': 195000}],'', ['Windows', 'Insulation', 'HVAC', 'Appliances', 'Lighting']]],
columns=['id', 'priceHistory', 'WaterConservation', 'EnergyEfficient'])
I have tried too many things to list here, but this seems to be the most efficient (just to get priceHistory
) (source):
df = pd.concat(
[
df,
df.pop("priceHistory").apply(
lambda x: pd.Series({k: v for d in x for k, v in d.items()})
),
],
axis=1,
)
print(df)
But I get this error:
TypeError: ‘float’ object is not iterable