My requirement is to store a large number of files within a single file.The files stored could be anything like images, videos or simple text files as well. I want some ideas to implement the same. I am thinking of implementing a file system within a file, but am not sure if its a good idea.
Adding in more details as requested : Platform to be developed on is Android. The idea initially was to store all the data using sqlite and provide encryption on it, but I think it will eventually lead to a slow down as the file size increases. The file is going to increase in size with time.
The main area of concern here is the access time. Also I want to provide encryption for this single file.
Any suggestions are welcome.
8
Note: if you stated the purpose of your file container more clearly, describing access patterns, desired platforms, and the problem you’re solving in general, the answers might be better.
Your description looks awfully similar to a game resource file, a renowned file type. These files are not intended to be updated frequently (if ever), but are optimized for fast seeking and reading.
There are several known implementations: for instance, iD Software used WAD files and PAK files, but finally came to use ZIP files.
Many applications use various derivatives of IFF format, built from self-describing chunks, and, with some care, efficiently updatable.
In a chunked file, or a zip file (consider zero compression), you can encrypt each entry independently, before writing it into the file. Provided that you use a block cipher like AES256, your data does not change size, except for aligning it to block boundary. You definitely want your decryption key stored somewhere else 🙂
Writing a file system to provide encryption is not only possible, but has actually been done many times. For instance, Linux has encfs, and Windows has encrypted folders. TrueCrypt is an advanced virtual encrypted FS available on many platforms.
Please note that while zip files provide for a sort of native encryption, this encryption is relatively weak. Same applies to rar files, for all I know.
Google confirms it is good idea in some cases: https://www.google.ru/images/nav_logo114.png
In this example Google optimises with reducing http requests number. User downloads one image file containing few navigation sprites and then browser splits and places it with CSS and JS.
0
but am not sure if its a good idea.
It seems to be! You don’t say what operating system you’re working with. In Unix-based operating systems, a directory already is a kind of file. So, two things to consider:
-
You may be able to just mark this requirement ‘done’ and move on.
-
If you decide to go ahead with the project, you’ll need to think carefully about how your implementation can be faster than existing ones.
Have you considered jar format? It’s java-specific but then it would easily port to multiple operating systems. Drawback is your users would need to have java on their machines to do anything with the files after downloading and you would probably have to do work to make that automatic/transparent to them. You would also have to handle the encryption/decryption in some way.
3