I need to upload multiple mp4 files to Firebase Cloud Storage on an Android device within a Flutter app. These files are captured offline, then stored on external device storage. I’ve noticed after a number of uploads the app crashes due to an out of memory error. I’m interested if anyone might have any advice around this issue?
I’ve created a minimal example below that crashes an Android Pixel 3 Emulator. It has a single button that uploads all files from an external directory to Cloud Storage using putFile. I’ve copied 54 roughly 20s (50MB) mp4 files using adb push to the referenced external directory for testing. The app is connected to the Firebase storage emulator for local testing.
I’ve tested a sync (below) and non-sync version both seem to cause the issue. Note in my real implementation I use a Queue structure but for simplicity in this case I’m using the Completer (open to better ways to do this!). The code looks like the following:
import 'dart:async';
import 'dart:io';
import 'package:firebase_storage/firebase_storage.dart';
import 'package:flutter/material.dart';
import 'package:firebase_core/firebase_core.dart';
import 'package:path_provider/path_provider.dart';
import 'firebase_options.dart';
Future<void> main() async {
WidgetsFlutterBinding.ensureInitialized();
await Firebase.initializeApp(
options: DefaultFirebaseOptions.currentPlatform,
);
try {
await FirebaseStorage.instance.useStorageEmulator('10.0.2.2', 9199);
} catch (e) {
print(e);
}
runApp(const MyApp());
}
class MyApp extends StatelessWidget {
const MyApp({super.key});
@override
Widget build(BuildContext context) {
return MaterialApp(
home: Scaffold(
appBar: AppBar(
title: const Text('Firebase Upload Test'),
),
body: Center(
child: ElevatedButton(
onPressed: () => uploadFromExternalDirectory(),
child: const Text('Upload Assets'),
),
),
),
);
}
void uploadFromExternalDirectory() async {
try {
final directory =
await getExternalStorageDirectory();
if (directory == null) {
print('Error: Could not get external storage directory');
return;
}
Directory videos = directory;
final storage = FirebaseStorage.instance;
// Iterate through files in the directory (synchronously)
for (var entity in videos.listSync(recursive: false)) {
print('Processing $entity');
if (entity is File) {
String fileName = entity.path;
try {
UploadTask task = storage.ref('test/$fileName').putFile(entity);
// Use a Completer to create a Future that completes when the upload is done
final Completer<void> completer = Completer<void>();
final StreamSubscription<TaskSnapshot> streamSubscription =
task.snapshotEvents.listen((TaskSnapshot snapshot) {
double progress =
(snapshot.bytesTransferred / snapshot.totalBytes) * 100;
print('Upload progress for $fileName: $progress%');
}, onError: (error) {
completer
.completeError(error); // Complete the future with an error
}, onDone: () {
completer.complete(); // Complete the future successfully
});
// Await the Completer's future to wait for the upload to finish
await completer.future;
// Cancel the stream subscription after the task is complete
await streamSubscription.cancel();
print('Upload of $fileName complete!');
} on FirebaseException catch (e) {
print('Error uploading $fileName: $e');
}
}
}
} catch (e) {
// Handle any potential errors that might occur
print('An error occurred: $e');
}
}
}
The app crashes after roughly 15-19 of these files have been uploaded (see screenshot). The out of memory error looks like the following:
I/pload_issue_app(24622): Alloc concurrent copying GC freed 273(182KB) AllocSpace objects, 0(0B) LOS objects, 0% free, 191MB/192MB, paused 28us,137us total 9.132ms
I/pload_issue_app(24622): WaitForGcToComplete blocked Alloc on Alloc for 9.821ms
I/pload_issue_app(24622): Starting a blocking GC Alloc
I/pload_issue_app(24622): WaitForGcToComplete blocked Alloc on Alloc for 9.805ms
I/pload_issue_app(24622): Starting a blocking GC Alloc
I/pload_issue_app(24622): Waiting for a blocking GC Alloc
I/pload_issue_app(24622): Clamp target GC heap from 215MB to 192MB
W/pload_issue_app(24622): Throwing OutOfMemoryError "Failed to allocate a 59 byte allocation with 88776 free bytes and 86KB until OOM, target footprint 201326592, growth limit 201326592; giving up on allocation because <1% of heap free after GC." (VmSize 19423944 kB)
I/pload_issue_app(24622): WaitForGcToComplete blocked Background on Alloc for 19.661ms
I/pload_issue_app(24622): Waiting for a blocking GC Alloc
I/pload_issue_app(24622): Forcing collection of SoftReferences for 16B allocation
I/pload_issue_app(24622): Waiting for a blocking GC Alloc
E/AndroidRuntime(24622): FATAL EXCEPTION: main
E/AndroidRuntime(24622): Process: com.example.upload_issue_app, PID: 24622
E/AndroidRuntime(24622): java.lang.OutOfMemoryError: Failed to allocate a 59 byte allocation with 88776 free bytes and 86KB until OOM, target footprint 201326592, growth limit 201326592; giving up on allocation because <1% of heap free after GC.
E/AndroidRuntime(24622): at dalvik.system.VMRuntime.newNonMovableArray(Native Method)
E/AndroidRuntime(24622): at java.nio.DirectByteBuffer$MemoryRef.<init>(DirectByteBuffer.java:71)
E/AndroidRuntime(24622): at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:265)
E/AndroidRuntime(24622): at io.flutter.plugin.common.StandardMessageCodec.encodeMessage(StandardMessageCodec.java:77)
E/AndroidRuntime(24622): at io.flutter.plugin.common.BasicMessageChannel$IncomingMessageHandler$1.reply(BasicMessageChannel.java:266)
E/AndroidRuntime(24622): at io.flutter.plugins.firebase.storage.GeneratedAndroidFirebaseStorage$FirebaseStorageHostApi$14.success(GeneratedAndroidFirebaseStorage.java:1312)
E/AndroidRuntime(24622): at io.flutter.plugins.firebase.storage.GeneratedAndroidFirebaseStorage$FirebaseStorageHostApi$14.success(GeneratedAndroidFirebaseStorage.java:1309)
E/AndroidRuntime(24622): at io.flutter.plugins.firebase.storage.FlutterFirebaseStoragePlugin.referencePutFile(FlutterFirebaseStoragePlugin.java:509)
E/AndroidRuntime(24622): at io.flutter.plugins.firebase.storage.GeneratedAndroidFirebaseStorage$FirebaseStorageHostApi$-CC.lambda$setup$13(GeneratedAndroidFirebaseStorage.java:1321)
E/AndroidRuntime(24622): at io.flutter.plugins.firebase.storage.GeneratedAndroidFirebaseStorage$FirebaseStorageHostApi$$ExternalSyntheticLambda13.onMessage(Unknown Source:2)
E/AndroidRuntime(24622): at io.flutter.plugin.common.BasicMessageChannel$IncomingMessageHandler.onMessage(BasicMessageChannel.java:261)
E/AndroidRuntime(24622): at io.flutter.embedding.engine.dart.DartMessenger.invokeHandler(DartMessenger.java:295)
E/AndroidRuntime(24622): at io.flutter.embedding.engine.dart.DartMessenger.lambda$dispatchMessageToQueue$0$io-flutter-embedding-engine-dart-DartMessenger(DartMessenger.java:322)
E/AndroidRuntime(24622): at io.flutter.embedding.engine.dart.DartMessenger$$ExternalSyntheticLambda0.run(Unknown Source:12)
E/AndroidRuntime(24622): at android.os.Handler.handleCallback(Handler.java:958)
E/AndroidRuntime(24622): at android.os.Handler.dispatchMessage(Handler.java:99)
E/AndroidRuntime(24622): at android.os.Looper.loopOnce(Looper.java:205)
E/AndroidRuntime(24622): at android.os.Looper.loop(Looper.java:294)
E/AndroidRuntime(24622): at android.app.ActivityThread.main(ActivityThread.java:8177)
E/AndroidRuntime(24622): at java.lang.reflect.Method.invoke(Native Method)
E/AndroidRuntime(24622): at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:552)
E/AndroidRuntime(24622): at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:971)
I/pload_issue_app(24622): WaitForGcToComplete blocked Alloc on Background for 33.559ms
I/pload_issue_app(24622): Starting a blocking GC Alloc
I/pload_issue_app(24622): WaitForGcToComplete blocked Alloc on Background for 19.223ms
I/pload_issue_app(24622): Starting a blocking GC Alloc
I/pload_issue_app(24622): Clamp target GC heap from 215MB to 192MB
I/pload_issue_app(24622): Alloc concurrent copying GC freed 568(58KB) AllocSpace objects, 0(0B) LOS objects, 0% free, 191MB/192MB, paused 21us,11us total 10.624ms
I/pload_issue_app(24622): WaitForGcToComplete blocked Background on Alloc for 12.056ms
I/Process (24622): Sending signal. PID: 24622 SIG: 9
Lost connection to device.
Exited.
See screenshot of uploaded files on the emulator UI
The most obvious indication of memory increase seems to be related to the Flutter Profiler logs. The percentage free moves from about 49% to 0% in about 30 seconds then it sticks to 0% for another ~2-3mins before it eventually crashes. See example logs below.
# Before upload
I/pload_issue_app(28092): Explicit concurrent copying GC freed 2821(112KB) AllocSpace objects, 1(20KB) LOS objects, 49% free, 2794KB/5589KB, paused 13us,13us total 4.503ms
# At 0% just before crash
I/pload_issue_app(28092): Explicit concurrent copying GC freed 360(291KB) AllocSpace objects, 0(0B) LOS objects, 0% free, 191MB/192MB, paused 46us,42us total 59.649ms
I’ve also tried tracking the issues on the Flutter Memory Profiler and Android Memory Profiler. Visually, the RSS seems to increase on the Flutter Profiler. For the Android Profiler the overall memory seems increase initially due to Java then periodically for native. Please see screenshots below. Happy to expand on findings here if this is useful.
Overall Android Profiler Memory Chart
Flutter Profiler Chart Snippet
I tried an alternate method of using putData as follows. This increases Flutter’s memory usage significantly and it crashes in 5s. I guess that’s expected given that it’s loading the bytes directly into memory. Unfortunately, there is no putStream for the Flutter library like there is for Android native.
await ref.putData(entity.readAsBytesSync());