I’m developing a WPF application whose core functionality involves creating a large object graph (often tens of thousands of entities) which the user can modify certain parts of, then save to a database. A graph can also be subsequently retrieved, modified and saved again. It’s feasible that a user can create dozens of these graphs per day.
The app also provides the user with different ways to search the entities in the database (predominantly text searches on various fields), presenting the search results to the user and allowing the user to select one, which will result in the relevant entity graph being retrieved in full and presented for the user to view and modify as above.
Currently I’m using Entity Framework and SQL Express, but I’m uneasy about certain aspects of the architecture and design, and the client isn’t keen on having to install SS. I’ve only recently come across the concept of NoSql databases, and sounds like they might be a good fit for what I’m doing, but I have a couple of questions.
First, I’m assuming performance wouldn’t be any worse than EF when reading or writing one of these object graphs?
What about my app’s search functionality? Will a NoSql db support this sort of thing, and what would the performance be like, bearing in mind the size and quantity of “documents” that I’m likely to have.
What about lookup (reference) data? Would I duplicate such data in each NoSQL document, or would I keep it all in a single NoSql document and store their IDs in the main documents?
Finally, any recommendations for a product? MongoDb and RavenDb seem to be the main OSS contenders for Windows.
2
This is simply the wrong question to be asking.
“NoSQL” doesn’t refer to a specific database, it refers to an entire superclass of databases including document databases, distributed key-value stores, graph databases, and object databases.
Speed is generally the least important factor in making decisions about data storage, no matter what some people might tell you. A SQL Server table with a billion rows can do key and index lookups at about the same rate as a MongoDB collection with a billion documents or a db4o database with a billion objects. The exception of course is if you can do sharding, in which case you’ll want a product that supports it, but if your users are antsy about installing SQL Express then rest assured they’ll run for the hills if you tell them to grab 200 old desktop PCs and shard an HBase instance across all of them.
Do you need full-text search? The industry standard for that is Lucene. It’s not really a database by itself, it’s something that’s bolted onto other databases and sometimes packaged more nicely by tools like Elastic Search or Solr. Most SQL databases also have some form of full-text search; it’s generally slower and inferior to Lucene. Some NoSQL databases have full-text search (RavenDB, for example, actually uses Lucene) but most have either no support or are in a very primitive stage.
Do you need to store very deep hierarchies of objects all at once, hierarchies that always have a single well-known aggregate root? If so, then document databases like MongoDB or CouchDB would work well for you. But if you ever need to change your hierarchy, or find that you need transactional consistency (and not the “eventual” kind) then you’re in for a world of hurt.
Does your data consistent of multiple related entities that each have independent lifetimes? If so, then a relational database like SQL Server or mysql is by far the best choice. Relational databases allow you to defer or ignore many difficult modeling decisions and in general you’ll only have to model once, as compared with document databases or key-value stores where you may have to frequently change your model or maintain multiple parallel models in order to solve various different use cases. If you want to keep things simple, you definitely want to stick with SQL.
If you need an embedded database, consider SQLite. It’s not as powerful as SQL Server but it’s fast and easy to use and easy to deploy, and you’ll find the syntax to be mostly familiar.
Incidentally, if you’re really that worried about speed (and by the sounds of it, your needs are way too small to justify any such worry) then you might want to look at the benchmark done some time ago by the ServiceStack guys. Entity Framework comes in dead last, and by a pretty wide margin too. NHibernate is probably the best choice when you balance out the mutually exclusive requirements of compatibility and performance. Although personally I prefer not to use any ORM at all, and you definitely won’t be able to use a pretty little ORM if you switch to NoSQL. Good luck learning the eleventy billion Redis commands if you’ve got no prior experience with the product.
4
There is no such thing as noSQL. There is only a whole bunch of new database technologies with completely different philosophies and use-cases, and all they have in common are things they also have in common with SQL databases. That means when you plan a project and are unsure about the database technology, you need to evaluate each noSQL database individually.
When your data is based on large graphs, it sounds like a perfect use-case for a graph-oriented database like Neo4j.
Document-oriented databases like MongoDB are generally not a good fit for graphs, because they don’t support connections between documents very well.
2
Design First, Shop Later
You should design your system before you start shopping for specific implementations. Specifically:
- Design your data structures based on how you plan to search and retrieve data.
- Evaluate the efficiency and performance of comparable solutions that implement your design requirements.
There’s really no substitute for benchmarking various solutions on a representative corpus of data, and no one else but your organization can evaluate the various trade-offs for your particular project.
Graphing? Then Graph!
Choosing the right data structure to represent your data is essential to preventing premature aging in programmers. Selecting the right models for searching, storing, and retrieving your data is very context-dependent, but it’s the essential first step in designing a new system.
I’ve never found a practical need for it myself, but if your fundamental data structure is a graph, why not take a look at a graphing database solution? Neo4j seems like it’s designed to address your problem domain, but I can’t shill for it since I’ve never used it.
Even if Neo4j is not the right solution for your project, you should certainly investigate all the options that make it easy to work with your core data structures. A key goal is to reduce the number of transformations you need when converting data between input and output formats, so keep the focus on how you will use the solution.
1
Its quite likely that a NoSQL database will meet your needs better than an SQL database, but all that does is increase your choice options.
Pretty much every type of database can store your data set, but what really matters is what kind of searches you need to do. The different database types have very different abilities in this area, and that is what will help you identify the suitable types of database to use.
This link contains a good list of different NoSQL products grouped into their loose behaviours.
What I would say though, is that most of these products are not targeted to the windows environment. This means that you would need a linux database server and that you have a seperate application server hosting the windows application.