I have a data file like the following (simplified, I have more columns):

And an object like this (again simplified):

class Gaze:
    def __init__(self, ts, frame_idx, gaze2D, gaze_dir3D=None):
        self.ts = ts
        self.frame_idx = frame_idx
        self.gaze2D = gaze2D
        self.gaze_dir3D = gaze_dir3D

where gaze2D is a numpy array containing [gaze_pos_x, gaze_pos_y] and gaze_dir3D is a numpy array containing [gaze_dir_x, gaze_dir_y, gaze_dir_z].

I want to efficiently load in the data file and make one Gaze object per row. I have implemented the below, but this is very slow:

def readDataFromFile(fileName):
    gazes   = []
    data    = pd.read_csv(str(fileName), delimiter='t', index_col=False, dtype=defaultdict(lambda: float, frame_idx=int))
    allCols = tuple([c for c in data.columns if col in c] for col in (
        'gaze_pos','gaze_dir'))
    # allCols -> ([gaze_pos_x, gaze_pos_y],[gaze_dir_x, gaze_dir_y, gaze_dir_z]), a list can be empty if a set of columns is missing (gaze_dir is optional)

    # run through all rows
    for _, row in data.iterrows():
        frame_idx = int(row['frame_idx'])  # must cast to int as pd.Series seems to lose typing of dataframe.... :s
        ts        = row['timestamp']

        # get all values (None if columns not present)
        # again need to cast to float despite all items in the series being a float, because the dtype of the series is object... :s
        args = tuple(row[c].astype('float').to_numpy() if c else None for c in allCols)
        gazes.append(Gaze(ts, frame_idx, *args))
    return gazes

As said, this is very slow, the row iteration takes forever, it is prohibitively slow for my use case. Is there a more efficient way of doing this? Using a similar read-in function using a csv.DictReader is a little faster but still way too slow.