I know video files can (and do) contain frames that are encoded out of order, which is why dts and pts exist, but can you send your frames in an arbitrary order, with their correct pts, and expect the video file to play?

Specifically I’m wondering about concatenating video files, can I decode both inputs in parallel and send the frames with the correct merged pts as they get decoded? And if so, what would the playback performance implications be?