I am using Vuforia with Unity and I am developing for Android and iOS.
My Application is quite big but I will try to simplify it by using a simple Cube as an example (it portrais the problem well enough)

I have a working scenario where I point my Phone towards an Image Target and a Cube Appears 3-4 meters above the Target just fine. The problem is that I want to continue looking towards that Cube but when I look upwards for 2-3 seconds Vuforia loses sense of where the Ground is so the Cube gets hidden. If I point my phone downwards Vuforia recalibrates and finds the Ground and the Cube is visible once more. I am now looking for a way to freeze any other Environment Tracking (Ground Plane, Anchors, etc.) after the Marker is recognized and continue moving around and looking at the Cube using only data from the Phone’s sensors (Gyroscope, Accelerometer, etc.). I need to do it that way because (1) Users will mainly be looking upwards to see fancy animations/objects/etc. (not just a simple Cube) and (2) my Mobile Application will be used mainly at night (e.g. even if the Phone is not pointed upwards Vuforia exhibits the same behavior because it cannot find a Ground plane (because it is too dark) after the initial Shiny Marker is recognized.

So, what I am trying to achieve is “Scan Once -> Load the 3d World (pin objects at certain positions in the World) -> Instruct Vuforia to somehow rely only on Mobile Sensors to track User Movement, Camera Rotation, etc. instead of searching for Ground, etc.
Is there a way to Initialize Vuforia World on first Marker Detection and continue tracking only using Mobile Phone sensors (even if the Cube gets a bit jittery/unstable). Is this even a supported scenario by Vuforia? I feel like I am missing something simple – is Vuforia always expecting to constanly have a Ground Plane visible and/or a Marker?

Note: Using EXTENDED_TRACKING for the Observer keeps the Cube visible even if I am not looking towards it or the Marker (which is nice) but whenever the phone loses sense of where the Ground is (e.g. it is too dark in the scene, or I point the Phone upwards to look at the Cube) then the whole Scene is hidden (because of the same issues described above).

I have tried the Mid-Air and Gound Placement Vuforia Samples but in these samples, the World is again hidden whenever it is too Dark or there is no Ground Plane Detected. I also tried setting VuforiaBehaviour.Instance.DevicePoseBehaviour.enabled = false; after first Recognition but it made the AR experience even more unstable.