Suppose i have 4 large databases that contain records (approx. 2000-10000 records total between all dbs) about assets on an SQL server. These assets have multiple characteristics in the SQL db.
The only way to get these db results is to use very complex join and views as the databases contain sensitive information. I am tasked with the following:
-
I must improve the response time for querying each of these databases, as well as querying all of them combined. (Currently, if you try to query all, the request times out after 120 seconds or so, when you try to search for a specific asset, based on your filters, it takes up to 15 seconds or more).
-
I must create a web app that allows the user to search through each of the 4 databases, and shows all of the properties from all 4 databases on a map.
Note: I can’t change anything about the SQL databases, as i only have read access.
My possible solution:
- Migrate the necessary data from all databases into a unified JSON file that will be updated every 24hrs to ensure up to date data.
- Migrate each of the 4 databases into JSON files that will be updated every 24hrs to ensure up to date data.
- Use Avro-js to create a Binary Search Tree & use Kafka-JS for data processing these 5 JSON files (1 unified, 4 specific JSON files per db).
Note: I only need the name and address to plot the asset on the map, for detailed asset results, i must query further information about each asset. - Query each JSON file as a means of returning results based on the user’s search input & returning the data on the map, in the web app.
Is this solution viable?
Are there other ways to solve this slow DB problem?
Is it viable to migrate this data into a more modern, faster db? (Firestore etc)
Some insight on this would be appreciated, Thanks!
So far, ive only looked into using the above mentioned approach, but maybe there are better ways to overcome this problem?
Paul S is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.