I have an app that stores photos locally, and I’m trying to create a way for users to send their photo library to another device.

I created a LibraryTransferPackage struct which can be successfully encoded to/decoded from JSON. This works well for small libraries but for large ones, it uses too much memory trying to hold the file at once and crashes.

I learned about JSON streaming, which seems like a good solution, but I’m having trouble. First it kept crashing because I couldn’t figure out the estimated file size. Then I landed on a way to do that using FileHandle, and it successfully creates an output file, but it seems corrupted (“the data couldn’t be read because it isn’t in the correct format” when I try to import and decode).

Can anyone help troubleshoot my streaming code, or if not suggest another way to do this?

        struct URLDataPair: Codable {
    
            var url: URL
            var data: Data
    
        }

        struct LibraryTransferPackage: Codable {
    
            var assets: [URLDataPair]
    
        }


        let package = LibraryTransferPackage(assets: libraryFiles)
        
    
        guard let outputFile = OutputStream(url: destinationURL, append: false) else {
                print("^^Failed to open output file")
                return
            }

        
        outputFile.open()
        
        let jsonEncoder = JSONEncoder()
        
        do {
            for filePath in libraryFiles.map({$0.url.path}) {
                // Open the files referenced by filePaths
                let fileHandle = FileHandle(forReadingAtPath: filePath)
                guard let fileHandle = fileHandle else {
                    print("Failed to open file:", filePath)
                    continue
                }
                
                defer {
                    fileHandle.closeFile()
                }
                
                // Get the size of the file
                let fileSize = fileHandle.seekToEndOfFile()
                fileHandle.seek(toFileOffset: 0)
                
                var offset: UInt64 = 0
                let chunkSize = 1024
                
                // Encode the file data into JSON incrementally
                while offset < fileSize {
                    let remainingBytes = fileSize - offset
                    let bytesToRead = min(UInt64(chunkSize), remainingBytes)
                    
                    let chunkData = fileHandle.readData(ofLength: Int(bytesToRead))
                    
                    let jsonData = try jsonEncoder.encode(chunkData)
                    
                    let bytesWritten = jsonData.withUnsafeBytes { bufferPointer in
                        return outputFile.write(bufferPointer.bindMemory(to: UInt8.self).baseAddress!, maxLength: jsonData.count)
                    }
                    
                    if bytesWritten < 0 {
                        print("Error writing JSON data to output file")
                        break
                    }
                    
                    offset += UInt64(bytesToRead)
                }
            }
        } catch {
            print("Error encoding JSON data:", error)
        }
        
    
        outputFile.close()