I am working on an HTTP Server in Delphi which simply sends back a custom XML dataset. I am not following any type of standard formatting, such as SOAP. I have the system working seamlessly, except one small flaw: When I have a very large dataset to send back to the client, it might take up to 2 minutes for all the data to be transferred. The HTTP Server I’m building is essentially an XML Data based API around a database, implementing the common business rule – therefore, the requests are specific to the data behind the system.
When, for example, I fetch a large set of product data, I would like to break this down and send it back piece by piece. However, a single HTTP request calls for a single response. I can’t necessarily keep feeding the client with multiple different XML packets unless the client explicitly requests it.
I don’t have any session management, but rather an API Key. I know if I had sessions, I could keep-alive a dataset temporarily for a client, and they could request bits and pieces of it. However, without session management, I would have to execute the SQL query multiple times (for each chunk of data), and in the mean-time, if that data changes, the “pages” might get messed up, therefore causing items to show on the wrong pages, after navigating to a different page.
So how is this commonly handled? What’s the methodology behind breaking down a large XML dataset into chunks to save the load?
2
Decide on the maximum number of pages your user is expected to browse in 1 session. Do a client fetch that gets set of primary keys that satisfy your maximum criteria and return this set to your client. This process is performed only 1 time. Each time the user requests next or previous page, use the cashed set of keys to get the desired rows based on the page size. This method always retrieves at most n rows where n is the number of rows in your page (after the initial cash retrieval). When the user is done, flush the keys cash. This method is specially useful when you have a complex query where a simple SQL such as “SELECT * FROM … Where Key > lastKey” won’t work. The drawbacks of this approach are:
1 – This method ignores new and removed records after the user has requested initial browse request, however, this is usually acceptable in many types of LOB applications.
2 – This method requires fetching the keys in advance, however, if your max. number of pages is reasonable, this should not be a problem, specially when the query is well-qualified.
The solution depends on where the problem really is.
-
If the problem is in the time taken to generate the XML server-side, then you could consider optimizing that. In particular look for an alternative approach that doesn’t involve creating a DOM in memory and serializing it. (I’m not familiar with Delphi).
-
If the problem is in fetching the data from the database, then query optimization may help.
-
If the problem is in the time taken to transmit the data, then consider sending less; e.g. leave out stuff that isn’t going to be displayed immediately. You can do this by leaving out fields (that can be fetched later if required) or by “paging” the dataset. Unfortunately, both of these entail client-side changes as well as server-side changes.
If you scale this up, then paging is the only solution that can potentially scale indefinitely.
An entirely different approach is process the large dataset on the server side and only send a summary …
I’m going to implement session management, where I can keep a dataset open for a period of time until idle time expires. During that time, the client can fetch chunks of the results at a time.
You need to be careful with this. There is a potential for accidental or deliberate denial of service issues if clients open lots of datasets and allow them to expire:
-
The timeout should be short, and based on the time since the last use of the dataset handle, rather than the time since it was opened.
-
Consider an implementation that doesn’t require a long-running transaction for each dataset handle; e.g. use some schema-specific identifier that allows you to close one database result set and then issue a new query to resume the iteration.
2
While debating the concept of Session Management, I’ve decided that I will go ahead and implement it. Each session will have just one dataset, refreshed upon request, where the client can fetch chunks of it at a time. I’m confident that I won’t have excessive sessions and therefore I understand how to implement this via sessions. The structure is quite more than explained in the question, and I’ve concluded an API Key should trigger another level of authentication: Sessions/Cookies. Each session has its own dataset.