Situation:

I made a custom Android app that runs on a Temi Robot.
The Android app moves the robot through a room and takes pictures with the onboard camera.
These pictures get saved onto the internal storage.

Technical details:

• MinSdkVersion: 24
• Target SDK version 33
• sourceCompatibility JavaVersion.VERSION_1_8
• Using CameraX version 1.3.0.
• Kotlin version 1.8.22

Problem:
The app fails to take pictures at random positions/angles.

For example, when the robot is moving through a room it often faces a painted white wall. It seems that sometimes the camera fails to take a picture and the app crashes. I think this is because of a (auto)-focus problem of the white wall.

There are also other cases where this seems to happen, but they all have one thing in common. They are in difficult-to-focus positions. For example, a pure white wall or a close object that is in front of it.

Error
There is no clear error given. I posted the full error log below, but this does not seem to give any clue on how to fix it.

What I tried & other details
I first suspected memory issues, but after implementing garbage collection, and also taking over 50+ photos from different positions, this does not seem the problem. I also upgrade to the CameraX package version 1.3.0. Also without any result.

Interesting to note is that the DeferrableSurface message does not appear when the app succeeds in making a photo.

2024-04-22 18:47:45.033 23802-23828 DeferrableSurface       com.robotemi.sdk.sample              D  use count+1, useCount=2 androidx.camera.core.SurfaceRequest$2@617af66
2024-04-22 18:47:45.033 23802-23828 DeferrableSurface       com.robotemi.sdk.sample              D  New surface in use[total_surfaces=3, used_surfaces=2](androidx.camera.core.impl.ImmediateSurface@96c60c0}
2024-04-22 18:47:45.033 23802-23828 DeferrableSurface       com.robotemi.sdk.sample              D  use count+1, useCount=1 androidx.camera.core.impl.ImmediateSurface@96c60c0

Code

Starting camera

 private fun startCamera() {
        setUi()

        val cameraProviderFuture = ProcessCameraProvider.getInstance(this)
        cameraProviderFuture.addListener({
            val cameraProvider: ProcessCameraProvider = cameraProviderFuture.get()

            val preview = Preview.Builder().build().also {
                it.setSurfaceProvider(binding.previewView.surfaceProvider)
            }

            imageCapture = ImageCapture.Builder()
                .setCaptureMode(ImageCapture.CAPTURE_MODE_MAXIMIZE_QUALITY)
                .setFlashMode(ImageCapture.FLASH_MODE_OFF)
                .build()

            try {
                cameraProvider.unbindAll()

                val cameraSelector = CameraSelector.Builder()
                    .requireLensFacing(CameraSelector.LENS_FACING_FRONT)
                    .build()

                cameraProvider.bindToLifecycle(
                    this, cameraSelector, preview, imageCapture
                )
            } catch (exc: Exception) {
                Log.e(TAG, "Use case binding failed", exc)
            }

            binding.previewView.visibility = View.VISIBLE // Show camera preview
            binding.imageViewFace.visibility = View.GONE // Hide the photo view

        }, ContextCompat.getMainExecutor(this))
    }

Taking picture

private suspend fun takePhotoMan(): Boolean {
        val photoCaptureCompletion = CompletableDeferred<Boolean>()

        val outputdir = FileUtils.getOutputDirectory(this)
        val photoFile = File(
            outputdir,
            "$photoSessionId-$photoLocationsTaken-$photoNr.jpg"
        )

        val outputOptions = ImageCapture.OutputFileOptions.Builder(photoFile).build()

        Log.d(TAG, "Sleeping for 5 seconds before taking photo")
        delay(5000)
        Log.d(TAG, "Sleeping done for 5 seconds before taking photo")

        Log.d(TAG, "Taking photo $photoSessionId-$photoLocationsTaken-$photoNr")

        // cancel other photo requests
        imageCapture.camera?.cameraControl?.cancelFocusAndMetering()

        try {
            imageCapture.takePicture(
                outputOptions,
                ContextCompat.getMainExecutor(this),
                object : ImageCapture.OnImageSavedCallback {
                    override fun onError(exc: ImageCaptureException) {
                        Log.e(TAG, "Photo capture failed: ${exc.message}", exc)
                        photoCaptureCompletion.complete(false)
                    }

                    override fun onImageSaved(output: ImageCapture.OutputFileResults) {
                        val savedUri = Uri.fromFile(photoFile)

                        Log.d(TAG, "Resolution of the saved image is: - $savedUri")

                        // Get the resolution of the Bitmap

                        binding.imageViewFace.setImageURI(savedUri)

                        lastPhotoUri = savedUri.toString()

                        photoCaptureCompletion.complete(true)
                    }
                }
            )
        } catch (e: Exception) {
            Log.e(TAG, "Error taking photo: ${e.localizedMessage}")
            photoCaptureCompletion.complete(false)
        }

        return photoCaptureCompletion.await()
    }

Error message

2024-04-22 18:47:35.066 23802-23802 PhotoActivity           com.robotemi.sdk.sample              D  Sleeping for 5 seconds before taking photo
2024-04-22 18:47:40.072 23802-23802 PhotoActivity           com.robotemi.sdk.sample              D  Sleeping done for 5 seconds before taking photo
2024-04-22 18:47:40.072 23802-23802 PhotoActivity           com.robotemi.sdk.sample              D  Taking photo 1713804422007-path_1713782278852_9830-0-1
2024-04-22 18:47:40.072 23802-23802 ImageCapture            com.robotemi.sdk.sample              D  takePictureInternal
2024-04-22 18:47:40.073 23802-23802 CameraOrientationUtil   com.robotemi.sdk.sample              D  getRelativeImageRotation: destRotationDegrees=0, sourceRotationDegrees=0, isOppositeFacing=false, result=0
2024-04-22 18:47:40.073 23802-23802 TakePictureManager      com.robotemi.sdk.sample              D  Issue the next TakePictureRequest.
2024-04-22 18:47:40.102 23802-23828 Camera2CapturePipeline  com.robotemi.sdk.sample              D  TriggerAf? AF mode auto
2024-04-22 18:47:40.102 23802-23828 Camera2CapturePipeline  com.robotemi.sdk.sample              D  Trigger AF
2024-04-22 18:47:40.103 23802-23828 Camera2CameraImpl       com.robotemi.sdk.sample              D  {Camera@b00f8e1[id=0]} Issue capture request
2024-04-22 18:47:40.103 23802-23828 CaptureSession          com.robotemi.sdk.sample              D  Issuing capture request.
2024-04-22 18:47:40.105 23802-23828 Camera2Cap...estBuilder com.robotemi.sdk.sample              D  createCaptureRequest
2024-04-22 18:47:40.156 23802-23828 Camera2CapturePipeline  com.robotemi.sdk.sample              D  checkCaptureResult, AE=CONVERGED AF =INACTIVE AWB=CONVERGED
2024-04-22 18:47:40.251 23802-23828 Camera2CapturePipeline  com.robotemi.sdk.sample              D  checkCaptureResult, AE=CONVERGED AF =LOCKED_FOCUSED AWB=CONVERGED
2024-04-22 18:47:40.253 23802-23828 Camera2CameraImpl       com.robotemi.sdk.sample              D  {Camera@b00f8e1[id=0]} Issue capture request
2024-04-22 18:47:40.253 23802-23828 CaptureSession          com.robotemi.sdk.sample              D  Issuing capture request.
2024-04-22 18:47:40.258 23802-23828 Camera2Cap...estBuilder com.robotemi.sdk.sample              D  createCaptureRequest
2024-04-22 18:47:41.484 23802-23828 DeferrableSurface       com.robotemi.sdk.sample              D  use count-1,  useCount=1 closed=false androidx.camera.core.SurfaceRequest$2@617af66
2024-04-22 18:47:41.484 23802-23828 DeferrableSurface       com.robotemi.sdk.sample              D  use count-1,  useCount=0 closed=false androidx.camera.core.impl.ImmediateSurface@96c60c0
2024-04-22 18:47:41.484 23802-23828 DeferrableSurface       com.robotemi.sdk.sample              D  Surface no longer in use[total_surfaces=3, used_surfaces=1](androidx.camera.core.impl.ImmediateSurface@96c60c0}
2024-04-22 18:47:41.485 23802-23828 Camera2CameraImpl       com.robotemi.sdk.sample              D  CameraDevice.onError(): 0 failed with ERROR_CAMERA_DEVICE while in OPENED state. Will attempt recovering from error.
2024-04-22 18:47:41.486 23802-23828 Camera2CameraImpl       com.robotemi.sdk.sample              D  Attempt to reopen camera[0] after error[ERROR_CAMERA_DEVICE]
2024-04-22 18:47:41.489 23802-23828 Camera2CameraImpl       com.robotemi.sdk.sample              D  {Camera@b00f8e1[id=0]} Transitioning camera internal state: OPENED --> REOPENING
2024-04-22 18:47:41.492 23802-23828 CameraStateRegistry     com.robotemi.sdk.sample              D  Recalculating open cameras:
                                                                                                    Camera                                       State                 
                                                                                                    -------------------------------------------------------------------
                                                                                                    Camera@b00f8e1[id=0]                         OPENING               
                                                                                                    -------------------------------------------------------------------
                                                                                                    Open count: 1 (Max allowed: 1)
2024-04-22 18:47:41.493 23802-23828 CameraStateMachine      com.robotemi.sdk.sample              D  New public camera state CameraState{type=OPENING, error=StateError{code=3, cause=null}} from OPENING and StateError{code=3, cause=null}
2024-04-22 18:47:41.493 23802-23828 CameraStateMachine      com.robotemi.sdk.sample              D  Publishing new public camera state CameraState{type=OPENING, error=StateError{code=3, cause=null}}
2024-04-22 18:47:41.496 23802-23828 Camera2CameraImpl       com.robotemi.sdk.sample              D  {Camera@b00f8e1[id=0]} Resetting Capture Session
2024-04-22 18:47:41.503 23802-23828 CameraCaptureSession    com.robotemi.sdk.sample              E  Session 0: Exception while stopping repeating: 
                                                                                                    android.hardware.camera2.CameraAccessException: CAMERA_ERROR (3): The camera device has encountered a serious error
                                                                                                        at android.hardware.camera2.impl.CameraDeviceImpl.checkIfCameraClosedOrInError(CameraDeviceImpl.java:2231)
                                                                                                        at android.hardware.camera2.impl.CameraDeviceImpl.stopRepeating(CameraDeviceImpl.java:1241)
                                                                                                        at android.hardware.camera2.impl.CameraCaptureSessionImpl.close(CameraCaptureSessionImpl.java:578)
                                                                                                        at androidx.camera.camera2.internal.SynchronizedCaptureSessionBaseImpl.close(SynchronizedCaptureSessionBaseImpl.java:476)
                                                                                                        at androidx.camera.camera2.internal.CaptureSession.release(CaptureSession.java:526)
                                                                                                        at androidx.camera.camera2.internal.Camera2CameraImpl.releaseSession(Camera2CameraImpl.java:555)
                                                                                                        at androidx.camera.camera2.internal.Camera2CameraImpl.resetCaptureSession(Camera2CameraImpl.java:1329)
                                                                                                        at androidx.camera.camera2.internal.Camera2CameraImpl.closeCamera(Camera2CameraImpl.java:468)
                                                                                                        at androidx.camera.camera2.internal.Camera2CameraImpl$StateCallback.reopenCameraAfterError(Camera2CameraImpl.java:1827)
                                                                                                        at androidx.camera.camera2.internal.Camera2CameraImpl$StateCallback.handleErrorOnOpen(Camera2CameraImpl.java:1779)
                                                                                                        at androidx.camera.camera2.internal.Camera2CameraImpl$StateCallback.onError(Camera2CameraImpl.java:1754)
                                                                                                        at androidx.camera.camera2.internal.CameraDeviceStateCallbacks$ComboDeviceStateCallback.onError(CameraDeviceStateCallbacks.java:121)
                                                                                                        at android.hardware.camera2.impl.CameraDeviceImpl.notifyError(CameraDeviceImpl.java:1629)
                                                                                                        at android.hardware.camera2.impl.CameraDeviceImpl.lambda$oDs27OTfKFfK18rUW2nQxxkPdV0(Unknown Source:0)
                                                                                                        at android.hardware.camera2.impl.-$$Lambda$CameraDeviceImpl$oDs27OTfKFfK18rUW2nQxxkPdV0.accept(Unknown Source:8)
                                                                                                        at com.android.internal.util.function.pooled.PooledLambdaImpl.doInvoke(PooledLambdaImpl.java:278)
                                                                                                        at com.android.internal.util.function.pooled.PooledLambdaImpl.invoke(PooledLambdaImpl.java:201)
                                                                                                        at com.android.internal.util.function.pooled.OmniFunction.run(OmniFunction.java:97)
                                                                                                        at androidx.camera.core.impl.utils.executor.SequentialExecutor$1.run(SequentialExecutor.java:111)
                                                                                                        at androidx.camera.core.impl.utils.executor.SequentialExecutor$QueueWorker.workOnQueue(SequentialExecutor.java:231)
                                                                                                        at androidx.camera.core.impl.utils.executor.SequentialExecutor$QueueWorker.run(SequentialExecutor.java:173)
                                                                                                        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
                                                                                                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
                                                                                                        at java.lang.Thread.run(Thread.java:923)
2024-04-22 18:47:41.507 23802-23828 Camera2CameraImpl       com.robotemi.sdk.sample              D  {Camera@b00f8e1[id=0]} Releasing session in state REOPENING
2024-04-22 18:47:41.508 23802-23828 CaptureSession          com.robotemi.sdk.sample              D  onSessionFinished()
2024-04-22 18:47:42.150 23802-23802 PhotoActivity           com.robotemi.sdk.sample              D  position: Position(x=3.5866, y=-4.527, yaw=1.8288, tiltAngle=0)
2024-04-22 18:47:44.176 23802-23828 Camera2CameraImpl       com.robotemi.sdk.sample              D  {Camera@b00f8e1[id=0]} CameraDevice.onClosed()
2024-04-22 18:47:44.177 23802-23828 Camera2CameraImpl       com.robotemi.sdk.sample              D  {Camera@b00f8e1[id=0]} Camera closed due to error: ERROR_CAMERA_DEVICE
2024-04-22 18:47:44.181 23802-23828 Camera2CameraImpl       com.robotemi.sdk.sample              D  {Camera@b00f8e1[id=0]} Attempting camera re-open in 700ms: androidx.camera.camera2.internal.Camera2CameraImpl$StateCallback$ScheduledReopen@3e20a3c activeResuming = true
2024-04-22 18:47:44.924 23802-23828 Camera2CameraImpl       com.robotemi.sdk.sample              D  {Camera@b00f8e1[id=0]} Attempting to open the camera.
2024-04-22 18:47:44.925 23802-23828 CameraStateRegistry     com.robotemi.sdk.sample              D  tryOpenCamera(Camera@b00f8e1[id=0]) [Available Cameras: 0, Already Open: true (Previous state: OPENING)] --> SUCCESS
2024-04-22 18:47:44.926 23802-23828 CameraStateRegistry     com.robotemi.sdk.sample              D  Recalculating open cameras:
                                                                                                    Camera                                       State                 
                                                                                                    -------------------------------------------------------------------
                                                                                                    Camera@b00f8e1[id=0]                         OPENING               
                                                                                                    -------------------------------------------------------------------
                                                                                                    Open count: 1 (Max allowed: 1)
2024-04-22 18:47:44.927 23802-23828 Camera2CameraImpl       com.robotemi.sdk.sample              D  {Camera@b00f8e1[id=0]} Cancelling scheduled re-open: androidx.camera.camera2.internal.Camera2CameraImpl$StateCallback$ScheduledReopen@3e20a3c
2024-04-22 18:47:44.927 23802-23828 Camera2CameraImpl       com.robotemi.sdk.sample              D  {Camera@b00f8e1[id=0]} Opening camera.
2024-04-22 18:47:44.928 23802-23828 Camera2CameraImpl       com.robotemi.sdk.sample              D  {Camera@b00f8e1[id=0]} Transitioning camera internal state: REOPENING --> OPENING
2024-04-22 18:47:44.928 23802-23828 CameraStateMachine      com.robotemi.sdk.sample              D  New public camera state CameraState{type=OPENING, error=null} from OPENING and null
2024-04-22 18:47:44.928 23802-23828 CameraStateMachine      com.robotemi.sdk.sample              D  Publishing new public camera state CameraState{type=OPENING, error=null}
2024-04-22 18:47:44.933 23802-23828 UseCaseAttachState      com.robotemi.sdk.sample              D  All use case: [androidx.camera.core.ImageCapture-ae41370f-7c81-45db-9221-245993cff140215543379, androidx.camera.core.Preview-3742a4d1-5788-4c2d-ab91-dccb1ad4a51d80666178] for camera: 0
2024-04-22 18:47:45.020 23802-23828 Camera2CameraImpl       com.robotemi.sdk.sample              D  {Camera@b00f8e1[id=0]} CameraDevice.onOpened()
2024-04-22 18:47:45.021 23802-23828 Camera2CameraImpl       com.robotemi.sdk.sample              D  {Camera@b00f8e1[id=0]} Transitioning camera internal state: OPENING --> OPENED
2024-04-22 18:47:45.023 23802-23828 CameraStateRegistry     com.robotemi.sdk.sample              D  Recalculating open cameras:
                                                                                                    Camera                                       State                 
                                                                                                    -------------------------------------------------------------------
                                                                                                    Camera@b00f8e1[id=0]                         OPEN                  
                                                                                                    -------------------------------------------------------------------
                                                                                                    Open count: 1 (Max allowed: 1)
2024-04-22 18:47:45.023 23802-23828 CameraStateMachine      com.robotemi.sdk.sample              D  New public camera state CameraState{type=OPEN, error=null} from OPEN and null
2024-04-22 18:47:45.023 23802-23828 CameraStateMachine      com.robotemi.sdk.sample              D  Publishing new public camera state CameraState{type=OPEN, error=null}
2024-04-22 18:47:45.025 23802-23828 UseCaseAttachState      com.robotemi.sdk.sample              D  All use case: [androidx.camera.core.ImageCapture-ae41370f-7c81-45db-9221-245993cff140215543379, androidx.camera.core.Preview-3742a4d1-5788-4c2d-ab91-dccb1ad4a51d80666178] for camera: 0
2024-04-22 18:47:45.028 23802-23828 SyncCaptureSessionBase  com.robotemi.sdk.sample              D  [androidx.camera.camera2.internal.SynchronizedCaptureSessionBaseImpl@7e21c3] getSurface...done
2024-04-22 18:47:45.028 23802-23828 CaptureSession          com.robotemi.sdk.sample              D  Opening capture session.
2024-04-22 18:47:45.033 23802-23828 DeferrableSurface       com.robotemi.sdk.sample              D  use count+1, useCount=2 androidx.camera.core.SurfaceRequest$2@617af66
2024-04-22 18:47:45.033 23802-23828 DeferrableSurface       com.robotemi.sdk.sample              D  New surface in use[total_surfaces=3, used_surfaces=2](androidx.camera.core.impl.ImmediateSurface@96c60c0}
2024-04-22 18:47:45.033 23802-23828 DeferrableSurface       com.robotemi.sdk.sample              D  use count+1, useCount=1 androidx.camera.core.impl.ImmediateSurface@96c60c0
2024-04-22 18:47:45.402 23802-23828 CaptureSession          com.robotemi.sdk.sample              D  Attempting to send capture request onConfigured
2024-04-22 18:47:45.402 23802-23828 CaptureSession          com.robotemi.sdk.sample              D  Issuing request for session.
2024-04-22 18:47:45.402 23802-23828 Camera2Cap...estBuilder com.robotemi.sdk.sample              D  createCaptureRequest
2024-04-22 18:47:45.407 23802-23828 CaptureSession          com.robotemi.sdk.sample              D  CameraCaptureSession.onConfigured() mState=OPENED