I’m using faust
to process some data, the example model looks like this:
class MyModel(faust.Record):
entry_id: uuid.uuid4
metadata: json.loads
timestamp: lambda x: pd.Timestamp(x, tz=None)
_offset: int
# ... some other int, str, bool fields
Note I’m using functions as field descriptors to process incoming fields.
Now, I need to pre-fetch some rows first from database:
# cols is a list of declared 'fields' for a type, cls = MyModel
cols = get_class_variables(cls)
# this returns a DataFrame with given columns
df = client.select(
f'SELECT {", ".join(cols)} FROM ... WHERE ...', columns=cols)
# here I want values to be list[MyModel]
values = [??? for row in df.to_dict(orient='records')]
I tried the following for ???
cls(**row)
cls.loads(row)
cls.from_data(row)
none of them seems to work, it just “casts” the dictionary into the type (i.e. metadata is still string, not parsed json). Any idea what function should I use for the dictionary fields to be processed according to the model? I want it to be as generic as possible since I’m using the same code for many models.