This post might seem a little vague but I do intend on clarifying what specifically I’m looking for. I’m attempting to design (from scratch, for learning purposes) a very rudimentary backend service + frontend app that functions similarly to SoundCloud.

From what I understand, I’m trying to design a streaming service. As a result, I’ve looked into various protocols that could serve as the backbone of the architecture, that being the HLS protocol. This is all good and fine, and this would cover the communication between the backend and frontend apps. What I’m more looking to figure out though, is how I’m meant to manage a temporary storage of the media I’m planning to serve. All of the descriptions of these that I’ve seen discuss that storage is done in some cloud service like Amazon S3, and I can completely understand that, but they never actually discuss how the backend server serves these files? So I’ve basically been trying to homebrew a solution, as this seems like an easy enough task.

Because of my lack of experience with these types of apps, my thoughts here are extremely rudimentary, and hence why I’m seeking help here. My first initial thought was that I could simply use the same filesystem that my backend code is stored in, within a specific media directory. This would result in the following file structure (for the backend code (this is a Python-based backend)):

venv/
media/
|- audio1.mp3
+- audio2.mp3
src/
|- __main__.py
+- etc

The backend API would then manage these files, deleting any files that were unused for a specific amount of time, and serve them to the frontend through some HTTP requests, probably with an audio/mpeg Content-Type(?).

Now, this seems like an absolutely terrible idea, and the only reason I could come up with is that if the backend server were particularly overloaded with requests, it would fill up with files…? But I don’t know how else this would be done.

Am I on the right track here, and if not, what should I look into to try to design this system?