Situation:
Made a custom Android app that runs on a Temi Robot.
The Android app moves the robot trough a room and takes pictures with the onboard camera.
These pictures get saved onto the internal storage.
Technical details:
- MinSdkVersion: 24
- Target SDK version 33
- sourceCompatibility JavaVersion.VERSION_1_8
- Using CameraX version 1.3.0.
- Kotlin version 1.8.22
Problem:
The app fails to make pictures at random positions/angles.
For example, when the robot is moving trough a room it often faces a painted white wall. It seems that sometimes times the camera fails to take a picture and the app crashes. I think this is because of a (auto)-focus problem of the white wall.
There are also other cases were this seem to happen, but they all have one thing in common. They are in difficult-to-focus positions. For example, a pure white wall or a close object that is in front of it.
Error
There is no clear error given. I post the full error log below, but this does not seem to give any clue on how to fix it.
What I tried & other details
I first suspected memory issues, but after implementing garbage collection, and also making over 50+ photos from different positions, this does not seem the problem. I also upgrade to the CameraX package version 1.3.0. Also without any result.
Code
Starting camera
private fun startCamera() {
setUi()
val cameraProviderFuture = ProcessCameraProvider.getInstance(this)
cameraProviderFuture.addListener({
val cameraProvider: ProcessCameraProvider = cameraProviderFuture.get()
val preview = Preview.Builder().build().also {
it.setSurfaceProvider(binding.previewView.surfaceProvider)
}
imageCapture = ImageCapture.Builder()
.setCaptureMode(ImageCapture.CAPTURE_MODE_MAXIMIZE_QUALITY)
.setFlashMode(ImageCapture.FLASH_MODE_OFF)
.build()
try {
cameraProvider.unbindAll()
val cameraSelector = CameraSelector.Builder()
.requireLensFacing(CameraSelector.LENS_FACING_FRONT)
.build()
cameraProvider.bindToLifecycle(
this, cameraSelector, preview, imageCapture
)
} catch (exc: Exception) {
Log.e(TAG, "Use case binding failed", exc)
}
binding.previewView.visibility = View.VISIBLE // Show camera preview
binding.imageViewFace.visibility = View.GONE // Hide the photo view
}, ContextCompat.getMainExecutor(this))
}
Taking picture
private suspend fun takePhotoMan(): Boolean {
val photoCaptureCompletion = CompletableDeferred<Boolean>()
val outputdir = FileUtils.getOutputDirectory(this)
val photoFile = File(
outputdir,
"$photoSessionId-$photoLocationsTaken-$photoNr.jpg"
)
val outputOptions = ImageCapture.OutputFileOptions.Builder(photoFile).build()
Log.d(TAG, "Sleeping for 5 seconds before taking photo")
delay(5000)
Log.d(TAG, "Sleeping done for 5 seconds before taking photo")
Log.d(TAG, "Taking photo $photoSessionId-$photoLocationsTaken-$photoNr")
// cancel other photo requests
imageCapture.camera?.cameraControl?.cancelFocusAndMetering()
try {
imageCapture.takePicture(
outputOptions,
ContextCompat.getMainExecutor(this),
object : ImageCapture.OnImageSavedCallback {
override fun onError(exc: ImageCaptureException) {
Log.e(TAG, "Photo capture failed: ${exc.message}", exc)
photoCaptureCompletion.complete(false)
}
override fun onImageSaved(output: ImageCapture.OutputFileResults) {
val savedUri = Uri.fromFile(photoFile)
Log.d(TAG, "Resolution of the saved image is: - $savedUri")
// Get the resolution of the Bitmap
binding.imageViewFace.setImageURI(savedUri)
lastPhotoUri = savedUri.toString()
photoCaptureCompletion.complete(true)
}
}
)
} catch (e: Exception) {
Log.e(TAG, "Error taking photo: ${e.localizedMessage}")
photoCaptureCompletion.complete(false)
}
return photoCaptureCompletion.await()
}
Error message
2024-04-22 18:47:35.066 23802-23802 PhotoActivity com.robotemi.sdk.sample D Sleeping for 5 seconds before taking photo
2024-04-22 18:47:40.072 23802-23802 PhotoActivity com.robotemi.sdk.sample D Sleeping done for 5 seconds before taking photo
2024-04-22 18:47:40.072 23802-23802 PhotoActivity com.robotemi.sdk.sample D Taking photo 1713804422007-path_1713782278852_9830-0-1
2024-04-22 18:47:40.072 23802-23802 ImageCapture com.robotemi.sdk.sample D takePictureInternal
2024-04-22 18:47:40.073 23802-23802 CameraOrientationUtil com.robotemi.sdk.sample D getRelativeImageRotation: destRotationDegrees=0, sourceRotationDegrees=0, isOppositeFacing=false, result=0
2024-04-22 18:47:40.073 23802-23802 TakePictureManager com.robotemi.sdk.sample D Issue the next TakePictureRequest.
2024-04-22 18:47:40.102 23802-23828 Camera2CapturePipeline com.robotemi.sdk.sample D TriggerAf? AF mode auto
2024-04-22 18:47:40.102 23802-23828 Camera2CapturePipeline com.robotemi.sdk.sample D Trigger AF
2024-04-22 18:47:40.103 23802-23828 Camera2CameraImpl com.robotemi.sdk.sample D {Camera@b00f8e1[id=0]} Issue capture request
2024-04-22 18:47:40.103 23802-23828 CaptureSession com.robotemi.sdk.sample D Issuing capture request.
2024-04-22 18:47:40.105 23802-23828 Camera2Cap...estBuilder com.robotemi.sdk.sample D createCaptureRequest
2024-04-22 18:47:40.156 23802-23828 Camera2CapturePipeline com.robotemi.sdk.sample D checkCaptureResult, AE=CONVERGED AF =INACTIVE AWB=CONVERGED
2024-04-22 18:47:40.251 23802-23828 Camera2CapturePipeline com.robotemi.sdk.sample D checkCaptureResult, AE=CONVERGED AF =LOCKED_FOCUSED AWB=CONVERGED
2024-04-22 18:47:40.253 23802-23828 Camera2CameraImpl com.robotemi.sdk.sample D {Camera@b00f8e1[id=0]} Issue capture request
2024-04-22 18:47:40.253 23802-23828 CaptureSession com.robotemi.sdk.sample D Issuing capture request.
2024-04-22 18:47:40.258 23802-23828 Camera2Cap...estBuilder com.robotemi.sdk.sample D createCaptureRequest
2024-04-22 18:47:41.484 23802-23828 DeferrableSurface com.robotemi.sdk.sample D use count-1, useCount=1 closed=false androidx.camera.core.SurfaceRequest$2@617af66
2024-04-22 18:47:41.484 23802-23828 DeferrableSurface com.robotemi.sdk.sample D use count-1, useCount=0 closed=false androidx.camera.core.impl.ImmediateSurface@96c60c0
2024-04-22 18:47:41.484 23802-23828 DeferrableSurface com.robotemi.sdk.sample D Surface no longer in use[total_surfaces=3, used_surfaces=1](androidx.camera.core.impl.ImmediateSurface@96c60c0}
2024-04-22 18:47:41.485 23802-23828 Camera2CameraImpl com.robotemi.sdk.sample D CameraDevice.onError(): 0 failed with ERROR_CAMERA_DEVICE while in OPENED state. Will attempt recovering from error.
2024-04-22 18:47:41.486 23802-23828 Camera2CameraImpl com.robotemi.sdk.sample D Attempt to reopen camera[0] after error[ERROR_CAMERA_DEVICE]
2024-04-22 18:47:41.489 23802-23828 Camera2CameraImpl com.robotemi.sdk.sample D {Camera@b00f8e1[id=0]} Transitioning camera internal state: OPENED --> REOPENING
2024-04-22 18:47:41.492 23802-23828 CameraStateRegistry com.robotemi.sdk.sample D Recalculating open cameras:
Camera State
-------------------------------------------------------------------
Camera@b00f8e1[id=0] OPENING
-------------------------------------------------------------------
Open count: 1 (Max allowed: 1)
2024-04-22 18:47:41.493 23802-23828 CameraStateMachine com.robotemi.sdk.sample D New public camera state CameraState{type=OPENING, error=StateError{code=3, cause=null}} from OPENING and StateError{code=3, cause=null}
2024-04-22 18:47:41.493 23802-23828 CameraStateMachine com.robotemi.sdk.sample D Publishing new public camera state CameraState{type=OPENING, error=StateError{code=3, cause=null}}
2024-04-22 18:47:41.496 23802-23828 Camera2CameraImpl com.robotemi.sdk.sample D {Camera@b00f8e1[id=0]} Resetting Capture Session
2024-04-22 18:47:41.503 23802-23828 CameraCaptureSession com.robotemi.sdk.sample E Session 0: Exception while stopping repeating:
android.hardware.camera2.CameraAccessException: CAMERA_ERROR (3): The camera device has encountered a serious error
at android.hardware.camera2.impl.CameraDeviceImpl.checkIfCameraClosedOrInError(CameraDeviceImpl.java:2231)
at android.hardware.camera2.impl.CameraDeviceImpl.stopRepeating(CameraDeviceImpl.java:1241)
at android.hardware.camera2.impl.CameraCaptureSessionImpl.close(CameraCaptureSessionImpl.java:578)
at androidx.camera.camera2.internal.SynchronizedCaptureSessionBaseImpl.close(SynchronizedCaptureSessionBaseImpl.java:476)
at androidx.camera.camera2.internal.CaptureSession.release(CaptureSession.java:526)
at androidx.camera.camera2.internal.Camera2CameraImpl.releaseSession(Camera2CameraImpl.java:555)
at androidx.camera.camera2.internal.Camera2CameraImpl.resetCaptureSession(Camera2CameraImpl.java:1329)
at androidx.camera.camera2.internal.Camera2CameraImpl.closeCamera(Camera2CameraImpl.java:468)
at androidx.camera.camera2.internal.Camera2CameraImpl$StateCallback.reopenCameraAfterError(Camera2CameraImpl.java:1827)
at androidx.camera.camera2.internal.Camera2CameraImpl$StateCallback.handleErrorOnOpen(Camera2CameraImpl.java:1779)
at androidx.camera.camera2.internal.Camera2CameraImpl$StateCallback.onError(Camera2CameraImpl.java:1754)
at androidx.camera.camera2.internal.CameraDeviceStateCallbacks$ComboDeviceStateCallback.onError(CameraDeviceStateCallbacks.java:121)
at android.hardware.camera2.impl.CameraDeviceImpl.notifyError(CameraDeviceImpl.java:1629)
at android.hardware.camera2.impl.CameraDeviceImpl.lambda$oDs27OTfKFfK18rUW2nQxxkPdV0(Unknown Source:0)
at android.hardware.camera2.impl.-$$Lambda$CameraDeviceImpl$oDs27OTfKFfK18rUW2nQxxkPdV0.accept(Unknown Source:8)
at com.android.internal.util.function.pooled.PooledLambdaImpl.doInvoke(PooledLambdaImpl.java:278)
at com.android.internal.util.function.pooled.PooledLambdaImpl.invoke(PooledLambdaImpl.java:201)
at com.android.internal.util.function.pooled.OmniFunction.run(OmniFunction.java:97)
at androidx.camera.core.impl.utils.executor.SequentialExecutor$1.run(SequentialExecutor.java:111)
at androidx.camera.core.impl.utils.executor.SequentialExecutor$QueueWorker.workOnQueue(SequentialExecutor.java:231)
at androidx.camera.core.impl.utils.executor.SequentialExecutor$QueueWorker.run(SequentialExecutor.java:173)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.lang.Thread.run(Thread.java:923)
2024-04-22 18:47:41.507 23802-23828 Camera2CameraImpl com.robotemi.sdk.sample D {Camera@b00f8e1[id=0]} Releasing session in state REOPENING
2024-04-22 18:47:41.508 23802-23828 CaptureSession com.robotemi.sdk.sample D onSessionFinished()
2024-04-22 18:47:42.150 23802-23802 PhotoActivity com.robotemi.sdk.sample D position: Position(x=3.5866, y=-4.527, yaw=1.8288, tiltAngle=0)
2024-04-22 18:47:44.176 23802-23828 Camera2CameraImpl com.robotemi.sdk.sample D {Camera@b00f8e1[id=0]} CameraDevice.onClosed()
2024-04-22 18:47:44.177 23802-23828 Camera2CameraImpl com.robotemi.sdk.sample D {Camera@b00f8e1[id=0]} Camera closed due to error: ERROR_CAMERA_DEVICE
2024-04-22 18:47:44.181 23802-23828 Camera2CameraImpl com.robotemi.sdk.sample D {Camera@b00f8e1[id=0]} Attempting camera re-open in 700ms: androidx.camera.camera2.internal.Camera2CameraImpl$StateCallback$ScheduledReopen@3e20a3c activeResuming = true
2024-04-22 18:47:44.924 23802-23828 Camera2CameraImpl com.robotemi.sdk.sample D {Camera@b00f8e1[id=0]} Attempting to open the camera.
2024-04-22 18:47:44.925 23802-23828 CameraStateRegistry com.robotemi.sdk.sample D tryOpenCamera(Camera@b00f8e1[id=0]) [Available Cameras: 0, Already Open: true (Previous state: OPENING)] --> SUCCESS
2024-04-22 18:47:44.926 23802-23828 CameraStateRegistry com.robotemi.sdk.sample D Recalculating open cameras:
Camera State
-------------------------------------------------------------------
Camera@b00f8e1[id=0] OPENING
-------------------------------------------------------------------
Open count: 1 (Max allowed: 1)
2024-04-22 18:47:44.927 23802-23828 Camera2CameraImpl com.robotemi.sdk.sample D {Camera@b00f8e1[id=0]} Cancelling scheduled re-open: androidx.camera.camera2.internal.Camera2CameraImpl$StateCallback$ScheduledReopen@3e20a3c
2024-04-22 18:47:44.927 23802-23828 Camera2CameraImpl com.robotemi.sdk.sample D {Camera@b00f8e1[id=0]} Opening camera.
2024-04-22 18:47:44.928 23802-23828 Camera2CameraImpl com.robotemi.sdk.sample D {Camera@b00f8e1[id=0]} Transitioning camera internal state: REOPENING --> OPENING
2024-04-22 18:47:44.928 23802-23828 CameraStateMachine com.robotemi.sdk.sample D New public camera state CameraState{type=OPENING, error=null} from OPENING and null
2024-04-22 18:47:44.928 23802-23828 CameraStateMachine com.robotemi.sdk.sample D Publishing new public camera state CameraState{type=OPENING, error=null}
2024-04-22 18:47:44.933 23802-23828 UseCaseAttachState com.robotemi.sdk.sample D All use case: [androidx.camera.core.ImageCapture-ae41370f-7c81-45db-9221-245993cff140215543379, androidx.camera.core.Preview-3742a4d1-5788-4c2d-ab91-dccb1ad4a51d80666178] for camera: 0
2024-04-22 18:47:45.020 23802-23828 Camera2CameraImpl com.robotemi.sdk.sample D {Camera@b00f8e1[id=0]} CameraDevice.onOpened()
2024-04-22 18:47:45.021 23802-23828 Camera2CameraImpl com.robotemi.sdk.sample D {Camera@b00f8e1[id=0]} Transitioning camera internal state: OPENING --> OPENED
2024-04-22 18:47:45.023 23802-23828 CameraStateRegistry com.robotemi.sdk.sample D Recalculating open cameras:
Camera State
-------------------------------------------------------------------
Camera@b00f8e1[id=0] OPEN
-------------------------------------------------------------------
Open count: 1 (Max allowed: 1)
2024-04-22 18:47:45.023 23802-23828 CameraStateMachine com.robotemi.sdk.sample D New public camera state CameraState{type=OPEN, error=null} from OPEN and null
2024-04-22 18:47:45.023 23802-23828 CameraStateMachine com.robotemi.sdk.sample D Publishing new public camera state CameraState{type=OPEN, error=null}
2024-04-22 18:47:45.025 23802-23828 UseCaseAttachState com.robotemi.sdk.sample D All use case: [androidx.camera.core.ImageCapture-ae41370f-7c81-45db-9221-245993cff140215543379, androidx.camera.core.Preview-3742a4d1-5788-4c2d-ab91-dccb1ad4a51d80666178] for camera: 0
2024-04-22 18:47:45.028 23802-23828 SyncCaptureSessionBase com.robotemi.sdk.sample D [androidx.camera.camera2.internal.SynchronizedCaptureSessionBaseImpl@7e21c3] getSurface...done
2024-04-22 18:47:45.028 23802-23828 CaptureSession com.robotemi.sdk.sample D Opening capture session.
2024-04-22 18:47:45.033 23802-23828 DeferrableSurface com.robotemi.sdk.sample D use count+1, useCount=2 androidx.camera.core.SurfaceRequest$2@617af66
2024-04-22 18:47:45.033 23802-23828 DeferrableSurface com.robotemi.sdk.sample D New surface in use[total_surfaces=3, used_surfaces=2](androidx.camera.core.impl.ImmediateSurface@96c60c0}
2024-04-22 18:47:45.033 23802-23828 DeferrableSurface com.robotemi.sdk.sample D use count+1, useCount=1 androidx.camera.core.impl.ImmediateSurface@96c60c0
2024-04-22 18:47:45.402 23802-23828 CaptureSession com.robotemi.sdk.sample D Attempting to send capture request onConfigured
2024-04-22 18:47:45.402 23802-23828 CaptureSession com.robotemi.sdk.sample D Issuing request for session.
2024-04-22 18:47:45.402 23802-23828 Camera2Cap...estBuilder com.robotemi.sdk.sample D createCaptureRequest
2024-04-22 18:47:45.407 23802-23828 CaptureSession com.robotemi.sdk.sample D CameraCaptureSession.onConfigured() mState=OPENED