I’m creating application in C++ and found out that storing information in XML file was very prodigally. Despite the possibility to read it without any specific application, random access via DOM or VTD library and possibility to backup by simple copying XML file, it’s very RAM-expensive (even while using VTD it’s necessary 17GB of RAM). Now, there are 12000 entries and I’m planning to store up to million ones. Each entry contains 37 fields (now, maybe more in future) with different types: string, double, float and 128-bit float. These fields are distributed into 5 groups (hierarchy which is possible by XML tags). I tried to find something more productive with the same benefits. Unfortunately, googling didn’t help me because there are so many DBs and DBMSs that I’m totally confused.
The XML file structure:
<paient>
<analysis name="">
<result type="">some_data</result>
</analysis>
<diagnosis>
<preliminary></preliminary>
<final></final>
<diagnosis>
...
<patient>
So could advise me a solution for my problem?
12
If your application is local (doesn’t support remote access) you could use an embedded DB engine.
It gives you the easy installation and independence from other installed programs. So you need to choose the appropriate library. What to look for when choosing?
-
As I said, the library should be embedded. It lets you create the single executable file without any difficult configurations.
-
The library should support C/C++ to integrate with your existing code.
-
The library should be well-known and widely-used. It ensures the most terrible errors have been discovered and fixed already.
-
It’s optional, but it would be cool if you could write SQL queries.
-
It’s optional, but… open-source!
Well, what’s the choice?
I propose to pay attention to SQLite and Berkeley DB. Both are embedded, both are open-source, both supports C/C++ (also Java, Python, etc.)
SQLite is a relational DB engine, since Berkeley DB is not. My opinion, SQLite is quite easier to learn, but I could be wrong.
Try both. Use the one that will work earlier.
First of all. Do not bother much about it. When you separate your concerns well, retrieve data, from processing and displaying it, you will always have the chance to switch to a “better” system.
Sizes of about 10,000 entries is nothing a database will complain about, even million sets aren’t a big deal. Your database will have to grow as your application/system grows. Think about the most common usecases and which data is presented together and which data is used together. This should lead you, whether to use a column-database, a document database, a graph database or simply a relational database.
Do not think too long about performance issues or theoretical problems which might occur, define what’s important and select a system, which you can get most support for it. Remember to decouple your data(model) from your application(logic) and presentation(view), then you can always make a better informed guess next time. When making your application open source a decoupled design will propably encourage others to provide a database implementation when required which might be better suited the (or your) needs.
There is much knowledge around – Think first of good enough. Be pragmatic.
Please check the link i shared in the comment section. This is a nice overview about pro and cons about the type of database you might need.
First of all, where you see hierarchy, a database expert sees relations. Take some time understanding the relational database model and make a simple model that fits your data. This classical model is tried and tested, based on mathematical foundations and is the basis for virtually every large system out there. Do not be misled by people claiming a graph database or NoSQL database will solve your immediate problems. Most often they are oblivious of proper database design and medium to long-term maintenance. Graph databases and NoSQL databases shine in very specific niches of software engineering, but the choice to use them should generally be made after conventional solutions prove insufficient.
Performance should clearly be taken into account as one million records might already pose a problem when dealing with certain request-response scenario’s. E.g., you might want to deliver a response within 200 milliseconds, which would likely entail the use of indices.
To choose a DBMS, you should start with PostgreSQL. It has a very good track record, comes with a performant query optimizer, is reasonably SQL standard and is used in many (larger) production settings. I’ve personally designed and implemented PostgreSQL settings containing tens of billions of records, hundreds of tables and probably around thousands of indices, driving SaaS applications with very respectable performance characteristics.
1