I’ve been working on a stream proxy – the idea is that instead of streaming directly to Twitch, OBS streams to a custom RTMP server running on the same machine. The server decodes flv from the rtmp stream into rawvideo using ffmpeg, modifies pixels, and encodes back into flv, streaming the result to twitch. Again, using ffmpeg.

However, I was not able to make this setup work reliably – I always run into buffering issues on Twitch. Even if ffmpeg shows a stable bitrate and 60fps, twitch slowly loses buffer size, then pauses to buffer, and then slowly loses buffer again… This results in endlessly growing delays and frequent pauses.

I simplified this setup, removing the rawvideo part together with frame modification. A simplified setup accepts the rtmp stream, and dumps it into FFmpeg, which sends it to Twitch with minimal overhead (I hope).
But even with this setup, Twitch still increases latency, although considerably slower.

The connection between rtmp server and ffmpeg is done with TCP sockets.
I tried using stdin, but it works even worse.
I also tried using windows named pipes but ran into a bottleneck – writing rawvideo from ffmpeg and reading it from script worked fine, as well as writing from a script and reading from ffmpeg. However, running both simultaneously in two different pipes slowed down.

Initially, all of this was written in python, but I also tried using go, hoping that rtmp server realisation in python was the problem.

Am I missing something fundamental here? Is this idea possible at all?