I am implementing a REST API endpoint that returns lists of objects of different types. The list will have roughly the following shape:
{
"type1": [
{ "prop1": "value1a", ... },
{ "prop1": "value1b", ... },
...
],
"type2": [
{ "prop2": "value2a", ... },
...
],
"type3": [
{ "prop3": "value3a", ... },
...
]
}
As the list of objects can get quite long, I would like to emit it as a stream, so that neither the server nor the client have to keep the whole list in memory and the client can already start processing the data before all of it has arrived. The JSON object would be streamed in chunks like this:
{
"type1": [
{ "prop1": "value1a", ... },
{ "prop1": "value2b", ... }
],
"type2": [
...
So far, so good.
To make it easier for people to use my REST API, I want to provide a TypeScript library that provides methods to call the various API endpoints. My question is: How should this TypeScript library represent the stream containing a nested object structure? My problem is that JavaScript streams are made for flat structures, but I’m trying to find a way to stream a nested structure.
I have two ideas:
Idea 1: Return a nested stream for each object type (ReadableStream<["type1", ReadableStream<Type1Object>] | ["type2", ReadableStream<Type2Object>] | ["type3", ReadableStream<Type3Object>]>
). The advantage of this approach seems to be that it resembles the underlying object structure much more and could be iterated over in the same way as the plain object could be. The disadvantage is that I have a feeling that this usage might not be intended by the Streams API and there might be drawbacks that I haven’t realized so far.
Idea 2: Flatten the stream into a stream of entries (ReadableStream<["type1", Type1Object] | ["type2", Type2Object] | ["type3", Type3Object]>
). The advantage that I see with this approach is that everything is returned as one stream, making it easier to work with. The disadvantage that I see is that the returned structure is quite inconsistent with what the REST API actually returns, and that a stream consumer might have to make an additional effort to unflatten the data again.
Are there any drawbacks with either of these approaches that I have missed? Is there another approach that might solve this problem in a better way?