I want to publish a small web project that is supposed to contain some of my research results to present it to the scientific community. All my analysis I ran so far have been written in python already, thus I considered to use django web framework to provide the results. This is, because I plan to show some visualization on up-to-date data, so I can run my analysis on new data by a cron directly in django.
there are actually two questions on my mind so far, how I am going to start developing:
- How can I easily store
numpy arraysin sqlite to keep the structure of the array and which type of django database field suits best? - What is the best way to
json.dumpthenumpy 2d-arraysto be able handle them in d3.js?
As you can see it is quite about putting numpy + django + d3.js together. My first hacking was to store the numpy results as strings, but I think this is not quite the best, since I need to parse them again after pulling them from database. Concerning the second question I already found simplejson.dumps but unfortunately it does not really work on numpy arrays and thus, sometimes I needed to hack something like this
return HttpResponse('{"array1" : '+simplejson.dumps(list(array1))+'}')
which I actually don’t really like.
So to put it together again:
What are your techniques for storing numpy structures in django database fields and how do you serialize them for http?
the best solution depends on a number of further questions, e.g.:
-
do you need to query the DB for array-content?
if yes, you will probably need to store the arrays as strings and query for strings (regex).
if no, consider serializing with cPickle. cPickle is fast and less verbose than simple strings.
anyway, you’ll have to put your array in a text-field. -
do you need to read or modify the arrays in python after they are written to the DB?
if yes, cPickle is the way to go. you read it back from the db, recreate the numpy array, read or modify and then encode the array as json.
if no, (and your arrays are not prohibitively big) store the array directly as json and just write the db-record verbatim to the http-respose.