Referring to: /questions/78760216/…
I got the problem that put large data into the ob-database increases the native-part of memory in the Android Studio Profiler, which after that isn’t freed up anymore.
So short description what I do:
I am parsing an xml-file from an url, creating EpgChannels (400) & EpgData (80.000) objects while parsing and save them in a separate list. After the parsing is finished I insert (put) the data into the database and I “fill” the toMany of each epg-channel with its related epgdata.
Following my current code for doing this:
private fun addEpgDataToDatabase() {
_epgProcessState.value = ExternEpgProcessState.EpgDataToDatabas
if (isUpdating) {
val existingDataQuery = epgDataBox.query().build()
val existingDataIds = existingDataQuery.find()
.map { it.idByAccountData }.toSet()
existingDataQuery.close()
val iterator = epgDataBatch.iterator()
while (iterator.hasNext()) {
val epg = iterator.next()
if (existingDataIds.contains(epg.idByAccountData)) {
iterator.remove()
}
}
}
epgDataBox.store.runInTx {
epgDataBatch.chunked(20000).forEach {
epgDataBox.put(it)
}
}
epgDataBox.closeThreadResources()
_epgProcessState.value = ExternEpgProcessState.EpgDataToDatabasFinished
addEpgDataToChannel()
}
private fun addEpgDataToChannel() {
_epgProcessState.value = ExternEpgProcessState.EpgChannelAndListToDatabase
epgChannelBox.store.runInTx {
epgChannelBatch.forEach { epgch ->
epgch.epgDataList.addAll(epgDataBatch.filter { it.epChId == epgch.chEpgId })
}
epgChannelBox.put(epgChannelBatch)
}
epgChannelBox.closeThreadResources()
_epgProcessState.value = ExternEpgProcessState.EpgChannelAndListToDatabaseFinished
epgChannelBatch.clear()
epgDataBatch.clear()
}
As I got some (partially huge) lags while doing this – especially when using the app directly on my TV I investigated the memory Profiler. And that showed me an increase of the native-part when putting the data (especially the epgdata) into the database, that later isn’t freed up anymore. I started to try some different batch-sizes, or if it’s working better when putting all data at once, but the “problem” wasn’t completely solved. Particullary that it the native-part isn’t freed up seemed strange to me.
It’s freed up only after restart of the app.
So is there something I am missing? Or can I ignore this, like the part: Others of the Profiler (as objectbox is managing the memory automatically)?
Because I recognised also that if I add other xml-files the Native-part stays more or less the same, so it isn’t increasing again. But what could cause the lags then? Putting to much data at once?
(I tried a smaller xml-file with only 15.000 epgdata and there I didn’t noticed this)
And is epgDataBox.closeThreadResources() & epgChannelBox.closeThreadResources() necessary in my code?
And another question: as there is no update/ignore conflict strategy (yet?), is there another way to put only new epgdata into the database or do I have to use something like the iterator in my code?