I have a TypeScript function subscribe that is used to subscribe to specific event types extracted from a union type. The function definition and type declarations are as follows:

type Notification = {
    readonly type: "new_photo";
} | {
    readonly type: "liked_photo";
    readonly id: string & Brand<"PhotoId">;
} | {
    readonly type: "disliked_photo";
    readonly id: string & Brand<"PhotoId">;
}

type EventType = "new_photo" | "liked_photo" | "disliked_photo"


type Subscribers = {
  [K in event.EventType]?: Set<
    (message: Extract<event.Notification, { type: K }>) => void
  >;
};

const subscribers: Subscribers = {};

export function subscribe<T extends event.EventType>(
  type: T,
  callback: (message: Extract<event.Notification, { type: T }>) => void,
) {
  if (!subscribers[type]) {
    subscribers[type] = new Set(); // error comes here
  }
  subscribers[type]?.add(callback);
  return () => {
    subscribers[type]?.delete(callback);
  };
}

I am encountering the following error in the subscribe function:

Type 'Set<(message: Extract<{ readonly type: "new_photo"; }, { type: T; }> | Extract<{ readonly type: "liked_photo"; readonly id: string & Brand<"PhotoId">; }, { type: T; }> | ... 5 more ... | Extract<...>) => void>' is not assignable to type 'Subscribers[T]'.
  Type 'Set<(message: Extract<{ readonly type: "new_photo"; }, { type: T; }> | Extract<{ readonly type: "liked_photo"; readonly id: string & Brand<"PhotoId">; }, { type: T; }> | ... 5 more ... | Extract<...>) => void>' is not assignable to type 'Set<(message: { readonly type: "new_photo"; }) => void> & Set<(message: { readonly type: "liked_photo"; readonly id: string & Brand<"PhotoId">; }) => void> & Set<...> & ... 8 more ... & Set<...>'.
    Type 'Set<(message: Extract<{ readonly type: "new_photo"; }, { type: T; }> | Extract<{ readonly type: "liked_photo"; readonly id: string & Brand<"PhotoId">; }, { type: T; }> | ... 5 more ... | Extract<...>) => void>' is not assignable to type 'Set<(message: { readonly type: "new_photo"; }) => void>'.
      Type '(message: Extract<{ readonly type: "new_photo"; }, { type: T; }> | Extract<{ readonly type: "liked_photo"; readonly id: string & Brand<"PhotoId">; }, { type: T; }> | ... 5 more ... | Extract<...>) => void' is not assignable to type '(message: { readonly type: "new_photo"; }) => void'.
        Types of parameters 'message' and 'message' are incompatible.
          Type '{ readonly type: "new_photo"; }' is not assignable to type 'Extract<{ readonly type: "new_photo"; }, { type: T; }> | Extract<{ readonly type: "liked_photo"; readonly id: string & Brand<"PhotoId">; }, { type: T; }> | ... 5 more ... | Extract<...>'.ts(2322)

I tried adjusting the type definitions and casting the set, but nothing is properly narrowing down the callback to be of the right type.
Something very similar happens when I try to call the callback like this:

            if (subscribers[m.payload.type]) {
              subscribers[m.payload.type]?.forEach((cb) => cb(m.payload));

Where the payload is properly typed to the right event

Any help or suggestions would be greatly appreciated. Thank you!