I have found a need for a standardization for storage and recording of data relating to diabetes information (I don’t think there’s any, but I may have missed it).
To that end I made this github project because I want to create one.. but I don’t really know how to go about it, or even for sure what I want to create is called.
Do I need to pick a language (xml, json) or a database type (mongo, sqlite, mysql), or how ‘unspecific’ is possible with something like this.
Is a data format as I describe here a thing? Are there other names for what I want to try to do? It’s a bit of a strange task to accomplish compared to what I’m used to.
3
Have you found a need for a standardized format, or just a format?
The need for a standardized format really only arises when you want to make data available to multiple clients, and want to enable other providers to provide data as well. In that case, you need to discuss this with all the interested organizations and mutually come up with a good, all-covering data specification.
Any data format sufficiently specified can be called a ‘standard’. However, it’s only when it becomes widely adopted that it is truly a standard.
Anyway, the data specification will need to be unambiguous, easy to implement, robust, possibly versioned, and probably more.
Then you need to lobby to have major organizations or lots of people use your format, forcing the rest of the world to also use the format.
Congratulations, you’ve introduced a standard!
Obligatory:
http://xkcd.com/927/
At first, you need to define an ontology: what are the “things” that need representation in your problem domain, what are their properties, how do they link together? If you’re trying to define a general standard for sharing data, then your ontology needs to cover any of the cases that could come up in a rather broad collection of scenarios.
Once you’ve got that semantic bit out of the way, the next part is comparatively simple: just define a data format that can express all of the above :-). Seriously, choosing the technology to represent the problem domain is going to be much less problematic than unambiguously expressing the problem in the first place.
This is definitely “the hard bit” with medical data, where multiple existing practices and techniques already exist: where someone has recorded that treatment was started, have they given you the calendar date, or the number of days after diagnosis, or the number of days after symptoms were first presented? Which of those is important? Can you “standardise” that into a single representation in all cases?
Don’t use XML or Mr. Atwood will find and destroy you. All kidding aside, JSON could be used, and can be loaded, manipulated, and saved out from a variety of tech these days, such as with Javascript (JQuery) on web pages. JQuery can load a file hosted on a remote URL, the result from a web service API URI, parse it into Javascript objects, save it back out to disk or send results off to other places. Ubermensch mentions above that MongoDB can load from JSON, but you could just as easily create your own web service (REST or SOAP) that takes JSON via HTTP POST and then stores it in any database system you want. He also mentions that you can create a data schema, but I’m not sure if there exists something as easy to validate with as an XML Schema Definition (XSD) and the various more robust tools around that older tech. So one advantage to using XML would be to create an XSD and publish it, and various editing tools could load that and understand how to give you “code-completion” (Visual Studio’s IntelliSense, for example) and in-line validation. Another technology to consider to making accessing the data standardized would be OData, so if you did have a web service in front of your data warehouse you could expose it to people for querying in a secure fashion.
3