I am traditionally a MsSQL or MySQL developer – I have been prototyping a new application that involves storing data from many targets.
I have seen that the app appears to get slow after scaling to 10k targets after inputting around 300k results for each (64 bit ints) – especially once I start trying to average and/or graph a few targets at once.
Once the app is deployed I will need to add around 1k targets a month and store one data point from each for each and every minute for approx 2 years.
I have never used NoSql, but from reading, it sounds like a good fit – create a document for each target and keep adding fields for each target.
So, I was just wondering, is NoSQL the way to go for this project or should I be sticking with traditional SQL – and/or can anyone offer any advice?
And the second part – If NoSQL is a solution can anyone recommend a fair comparison as I am getting lost in biased reviews that seem to say each solution is superior!?
3
to get the point if either one of the SQL DBs or one kind of NoSQL is right you should first analyze what causes your problem.
is it the amount of data itself,
is it the amount of parallel writes,
is it the amount of parallel reads,
or is it the amount of parallel reads and writes
as far as you have this, you should check how to solve this with SQL, (SQL system are betten then NoSQL systems tells about them)
if you figure out that also a big cluster of SQL with lots of different HD and partitioned data doesn’t help then check out depending on your problem which NoSQL systems are there.
as already mentioned there are the graph DBs, they are good for reading and writing lots of connected data. but connections for the data can’t be unlimited.
for fast reading there is mongodb, good community and also commercial support.
for fast writing, there is hypertable and hbase . where hypertable is the faster one, but its written in c, and java or php or… apis are only available as thrift libs, which needs a bit more reading how to use it. community is active and responding for both.
mongodb, hypertable and hbase, have all support to create a Map Reduce job.
this is usefull if you want todo stuff on your data which takes longer.
means if you dont need a answer to a query (sql speaking) right now, but during the next hour this is the right approach for you.
but for all NoSQL System, you need to start thinking from scratch, so if you start forgot everything from SQL, and only concentrate on the one NoSQL you wanna use.
and design your data around this system
one thing that most NoSQL have in common is, that data can be stored twice,
on SQL you create foreign keys joins and so.
on NoSQL you simple write the data twice (one step to thinking for such systems), (many system event dont have an update method )
2
graph databases are sub category of NoSQL. I would recommend you to use a specialized DB for your app. from the description of your app I think any graph/key-value store will do the job.
it seems like your have written and tested your app on SQL, moving to noSQl will require you to rewrite/remodel your app. InfiniteGraph, Neo4j and FlockDB are good graph DBs backed my strong community. people from SQL background have complete paradigms shift when working with NOSQL solution, so for the first dive go for the easiest solution. im not the fan boy of any of these, mostly noSQL DBs are free and open source it woldnt take that long to try out few DBs.