I’m having a bit of a tough time at work trying to weigh up some design decisions, and I’m in a headspace where I feel, either I’m not being understood or I’m being a hapless moron.

My firm sunk a lot of cost into KDB, and we’re kinda using it as a data-lake. We have a real time in memory KDB instance, and we have a KDB instance that persists data to partitioned files.

There’s a whole team around maintaining schemas and writing data ingestors.

I find this set up extremely difficult to use. I can not fetch large datasets of historical data from this, it will just hang, because if you ask KDB for 1 million rows, it will buffer 1 million rows and I’ll kill KDB for everyone. What I’m told the solution is, is to write your aggregation in KDB, which is fine for people who do spot trading, but not so fine for my team that does derivatives trading where I have a lot of non-trivial functions that I’m not comfortable re-writing in Q and keeping them up to date with my C++ code.

We’re also at the limits of vertically scaling KDB, because it’s cheaper to pay for a magnum.di.ck.20.xl AWS instance than it is to buy a license for another KDB core.

Every time I suggest, using Parquet, Athena/Polars/Dask for this stuff no one listens, the response is:

The rest of the firm uses KDB and is invested in it. Other firms use KDB for this just fine.

But I just don’t understand how other firms manage this. What do they do when writing complex backtesting software in KDB? Like if I need to backtest a strategy on a time-series of euro-option order books, one of my inputs will be my own SVI-surface which is a non-trivial model needing a fair bit of code to translate into volatility.

Do they have a team of KDB quants who are separate from the non-KDB quants who’ll just duplicate the work?

Maybe I’m just misunderstanding the whole shebang, but I’d really appreciate if someone had some actual experience with derivatives pricing and KDB to tell me what they did.