Folks, I am in middle of writing a web application (Python/Flask) where home page has user profile image in the navbar which is coming from a database (blob), I am wondering if this is a good practice. Here I have following choices:
- Currently it is stored image in the backend db in a table in a blob
field. - Store the image disk location in the table instead of storing the image itself in the blob field, this will refer to the disk file
where to load the image from instead of transferring the image file
from a blob field. - Avoid hitting the db altogether when accessing the images and use a local disk file location instead, this location can be mapped to
nginx directly which means web caching will ensure fast image
retrieval.
Wondering if this is a big issue and how you guys are handing it anyway, what is the industry best practices here? Please note the database will be hit anyway to load various other items like user first_name and possibly other bits too but I don’t want to lose the choice of storing images which can be cached.
Any help here will be highly apprecaited.
5
The industry best practice is to never store images (and other big blobs) in database directly. And only store “references” to those images. The reason for that is that such images overload database heavily for no reason. And database will often be the most important resource for you. It is the biggest bottleneck most of the time.
Databases, especially ACID databases are notoriously hard to scale. But images and other static blobs, are not hard to scale. In fact this problem is already solved through Content Delivery Networks, which also handle other issues, like load balancing, geospatial latency, caching, etc. While you might not need full CDN power from the beginning, it is a good idea to be prepared for it.
Consistency is rarely an issue. You just need a background worker that will periodically scan for orphaned files and remove them. That’s likely to be the only consistency issue.
- Currently it is stored image in the backend db in a table in a blob field.
Bad. Scales badly, overloads database.
- Store the image disk location in the table instead of storing the image itself in the blob field, this will refer to the disk file where to load the image from instead of transferring the image file from a blob field.
- Avoid hitting the db altogether when accessing the images and use a local disk file location instead, this location can be mapped to nginx directly which means web caching will ensure fast image retrieval.
These two sound very similar. I assume that the difference is that in 3 you use nginx and in 2 you use the web application and serve the file manually? That is secondary to be honest. Nginx will do that in a more efficient way, so sure, why not. But yes, storing files locally, and serving through a separate mechanism is a good start.
There are few improvements that you may want to consider.
- You can store full URLs in the database, and serve them to the client. That way it will be possible for the client to load the image from anywhere, even outside your application. This will be especially useful once you want to switch to CDN (which is a separate service, with its own domain).
- Security is a concern. Should those images be publicly accessible? If not, then you will have to put some authorization layer. Most CDNs support some form of URL signing, which is an efficient form of authorization, and often good enough. But with this concern I don’t think nginx is an option (unless there’s some plugin that supports it, which I’m not aware of). And also signed URLs means that you can’t store URLs in the database, they have to be generated on the fly. You will have to store some kind of identifier/path, but not a static URL. You could make it very flexible and store
(kind, reference)pairs in the database, where kind can be URL or signed URL, or something else. - Uploading is a separate issue. Going through the web server is an option, and the only reasonable at the beginning. But there’s a risk involved, what if too many uploads happen? What if a malicious user tries to DDOS you? Many people don’t bother with those questions, and that is fine most of the time. If you ever need to deal with it, you will need a proper upload service, like AWS S3. Let maintainers of those services deal with those issues. Most of those services support notifications after the upload, so that your web application can react once it finishes (to do some post processing for example, or database update, or whatever).
Since images are mostly static content, usually a CDN (Content Delivery Network) is used to deliver them as mentioned by @freakish.
Consider a global website. One user is in California, the other in Germany. The big advantage with a CDN is that content can be delivered and cached from an edge location, so that users in California get images from an edge location on the west coast of the United States and users in Germany get images from an edge location in Europe. This maximizes network utilization.
If your application is a single set of users in just one location, maybe a CDN is overkill, but any sort of global web site can benefit greatly from delivering static content in this fashion.
I once worked on an application that served images from a database and many customers called in complained how much bandwidth the application consumed on their network. So, it wasn’t the greatest design since the application served lots of images. We offloaded the images to a CDN and customers were much happier after that as network utilization dropped considerably.