Recently, we made some optimization attempts in the Native code of a specific app in our company to address performance issues. We released a gray version containing the mentioned modifications to a subset of users for evaluating the effectiveness of the optimizations. However, after the release, we observed a significant number of crash reports, and the crashes were mainly limited to 64-bit smartphones of the VIVO brand, with a few similar issues reported on Samsung phones. The crash reports indicate that the libc’s memcpy function is the source of the error, while our modifications did not involve any changes to the calling path of the memcpy function but instead focused on modifying other business logic. Can anyone provide some troubleshooting ideas?

Below is the stack trace from one of the above mentioned crashes,

SIGSEGV(SEGV_ACCERR)

#00 pc 000000000004eebc /apex/com.android.runtime/lib64/bionic/libc.so (__memcpy+60) [arm64-v8a::3bac97c2012f8248ca610b4e558e17d1]
#01 pc 000000000014a4c8 /data/app/~~k3_ers-e6zmrSs_FANzwIg==/<OurAPPCorporationName>.<OurAPPTypeName>-yFrQvavwtQa6pbKyOTxrKA==/lib/arm64/<OurNativeSharedObjectName>.so [arm64-v8a::2b50f378ff7e1b8d1d2b6ba552079baf]
#02 pc 0000000000087aa8 /data/app/~~k3_ers-e6zmrSs_FANzwIg==/<OurAPPCorporationName>.<OurAPPTypeName>-yFrQvavwtQa6pbKyOTxrKA==/lib/arm64/<OurNativeSharedObjectName>.so [arm64-v8a::2b50f378ff7e1b8d1d2b6ba552079baf]
#03 pc 0000000000086c78 /data/app/~~k3_ers-e6zmrSs_FANzwIg==/<OurAPPCorporationName>.<OurAPPTypeName>-yFrQvavwtQa6pbKyOTxrKA==/lib/arm64/<OurNativeSharedObjectName>.so [arm64-v8a::2b50f378ff7e1b8d1d2b6ba552079baf]
#04 pc 000000000009e330 /data/app/~~k3_ers-e6zmrSs_FANzwIg==/<OurAPPCorporationName>.<OurAPPTypeName>-yFrQvavwtQa6pbKyOTxrKA==/lib/arm64/<OurNativeSharedObjectName>.so [arm64-v8a::2b50f378ff7e1b8d1d2b6ba552079baf]
#05 pc 000000000009bb20 /data/app/~~k3_ers-e6zmrSs_FANzwIg==/<OurAPPCorporationName>.<OurAPPTypeName>-yFrQvavwtQa6pbKyOTxrKA==/lib/arm64/<OurNativeSharedObjectName>.so [arm64-v8a::2b50f378ff7e1b8d1d2b6ba552079baf]
#06 pc 000000000008ab80 /data/app/~~k3_ers-e6zmrSs_FANzwIg==/<OurAPPCorporationName>.<OurAPPTypeName>-yFrQvavwtQa6pbKyOTxrKA==/lib/arm64/<OurNativeSharedObjectName>.so [arm64-v8a::2b50f378ff7e1b8d1d2b6ba552079baf]
#07 pc 00000000002072e4 /data/app/~~k3_ers-e6zmrSs_FANzwIg==/<OurAPPCorporationName>.<OurAPPTypeName>-yFrQvavwtQa6pbKyOTxrKA==/oat/arm64/base.odex [arm64-v8a::515535bd03fc04c8a3440ba0667ca07e]
java:
<OurAPPCorporationName>.engine.logic.<OurBusinessLogicModuleName>.F(SourceFile:1)
<OurAPPCorporationName>.engine.logic.PinyinLogic.h0(SourceFile:3)
<OurAPPCorporationName>.engine.logic.state.PredictionState.o(SourceFile:29)
<OurAPPCorporationName>.engine.logic.state.PredictionState.k(SourceFile:165)
<OurAPPCorporationName>.engine.logic.state.PredictionState.a(SourceFile:5)
<OurAPPCorporationName>.engine.logic.PyLogicStateContext.a(SourceFile:16)
<OurAPPCorporationName>.engine.logic.thread.LogicHandler.handleMessage(SourceFile:20)
android.os.Handler.dispatchMessage(Handler.java:106)
android.os.Looper.loopOnce(Looper.java:223)
android.os.Looper.loop(Looper.java:324)
android.os.HandlerThread.run(HandlerThread.java:67)

,and on other mobile phones that also crash at calll to memcpy, there are also crashes as below, with top of the call stack being like below, seemingly more platform-specific variant of memcpy named __memcpy_aarch64_simd:

#00 pc 000000000009d6e0 /apex/com.android.runtime/lib64/bionic/libc.so (__memcpy_aarch64_simd+160) [arm64-v8a::5ba801faadc5ce7796fe32eab5104a1e] 

We also attempted to reproduce the issue by using a large number of random inputs (100,000 records) on a local VIVO phone but did not experience any application crashes. By analyzing the calling path from our local business logic to the memcpy function, we were able to pinpoint the file and line number of each layer in the crash’s call stack. However, our modifications were not directly present in the calling path leading to the crash. This has left us confused and without a clear troubleshooting approach, as we lack direct clues or a specific direction for investigation.

The platform of the crashes are primarily distributed on (1)Android 13,level 33 and (2)Android 14,level 34

, as shown below from online crash reports

Android 13,level 33
Android 14,level 34
Android 14,level 34
Android 14,level 34
Android 13,level 33
Android 14,level 34
Android 14,level 34
Android 13,level 33
Android 14,level 34
Android 14,level 34
Android 13,level 33
Android 13,level 33
Android 13,level 33
Android 13,level 33
Android 13,level 33
Android 13,level 33
Android 13,level 33
Android 14,level 34