I am working on custom HLS Live transcoding implementation in GCP with k8s pods. I have tried s3fs, rclone, GCS-FUSE with sidecar solutions to sync data into s3/GCS. But getter very high latency to sync data to respective storage origin. Can someone suggest how i can reduce this latency?
System info: K8s Pod in GCP
Transcoding Tools: FFmpeg
Storage Origin: GCS, S3
Feed Type: live rtmp
Output: HLS with .m3u8 manifest & 4 sec .ts chunks
- GCS-Fuse sidecar:
with this transcoding pipeline is not able to use full cpu and it limit ffmpeg cpu used to 0.4 core - Temp output directory for ffmpeg output & rclone ffmpeg output with GCS-Fuse sidecar mounted folder:
This implantation allowed ffmpeg to consume cpu till 10 core but rclone does not work as intended. It syncs data randomly. Tried a couple rsync optimisation and checksum options. but did not got expected result. - s3fs:
This takes admin privileges. did not able to excute it for now. But i am assuming it will give similar result to GCS-Fuse