`I am having issues with my lambda not being able to getObject from s3 through a stream. When I check the lambda logs, there is no sign of an error what so ever. I see that is doing the getFileStream and logging each file… and then just stops.

Here is my code and I can answer any questions:

import { S3 } from "@aws-sdk/client-s3";
import * as stream from "stream";
import archiver from "archiver";
import type { Context } from "aws-lambda";
import { logger } from "../../utils/logger.js";
import { objectClient } from "../../adapters/object-client.js";
import type { ArtifactsEmailArgs } from "../../domain/ports/email-client.js";
import { emailClient } from "../../adapters/email-client.js";

export interface FilesZipperRequest {
  user_email: string;
  user_alt_email?: string;
  user_first_name: string;
  destination_bucket: string;
  destination_key: string;
  source_files: FilesZipperSourceFiles[];
}

export interface FilesZipperSourceFiles {
  source_bucket: string;
  source_key: string;
}

// Sample Request taken by this lambda
//{
//     "user_email": "[email protected]",
//     "user_first_name": "John",
//     "destination_bucket": "destination-bucket",
//     "destination_key": "zips/myzip.zip",
//     "source_files": [
//         {
//             "source_bucket": "source_bucket",
//             "source_key": "full_path_of_file"
//         }
//     ]
// }

export const handler = async (
  request: FilesZipperRequest,
  context: Context
) => {
  logger.info("Event=" + JSON.stringify(request));
  logger.info("Context=" + JSON.stringify(context));

  const userEmail = request.user_email;
  const userAltEmail = request.user_alt_email;
  const userFirstName = request.user_first_name;
  const bucketName = request.destination_bucket;
  const destinationKey = request.destination_key;
  const fileNames = request.source_files;

  const s3 = new S3({});
  const zipStream = new stream.PassThrough();
  const archive = archiver("zip");
  archive.pipe(zipStream);

  const missingFiles: string[] = [];

  logger.info("*** Obtain users and products for mapping from ids ***");

  try {
    for (const file of fileNames) {
    
      try {
        const fileStream = await getFileStream(
          file.source_bucket,
          file.source_key,
          s3
        );
        archive.append(fileStream, { name: file.source_key });
      } catch (error: any) {
        logger.error(
          `Error fetching file: ${file.source_key} from bucket: ${file.source_bucket}`,
          error
        );
        missingFiles.push(file.source_key);
      }
    }
    archive.finalize();

    const buffer = await streamToBuffer(zipStream);

    await emailArtifactPreSignedUrl(
      bucketName,
      destinationKey,
      userFirstName,
      userEmail,
      missingFiles,
      userAltEmail
    );

    await objectClient.upload(
      bucketName,
      destinationKey,
      buffer,
      "application/zip"
    );

    return;
  } catch (error: any) {
    logger.error("An error occurred during the zip process:", error);
    return {
      statusCode: 500,
      body: JSON.stringify({
        message: "Failed to create zip file",
        error: error.message,
      }),
    };
  }
};

/**
 * Fetches a file from the specified S3 bucket and key.
 *
 * @returns A promise that resolves to a readable stream of the file.
 * @throws Will throw an error if the file is not found or if another error occurs during the fetch.
 */
async function getFileStream(
  bucket: string,
  key: string,
  s3Client: S3
): Promise<stream.Readable> {
  const params = {
    Bucket: bucket,
    Key: key,
  };
  try {
    const data = await s3Client.getObject(params);
    logger.info(`Successfully fetched file: ${key} from bucket: ${bucket}`);
    return data.Body as stream.Readable;
  } catch (error) {
    if ((error as Error).name === "NoSuchKey") {
      throw new Error(`File not found: ${key} in bucket ${bucket}`);
    }
    throw error;
  }
}

/**
 * Converts a stream to a buffer.
 *
 * @param stream - The input stream to be converted.
 * @returns A promise that resolves to a buffer containing the concatenated data from the stream.
 */
function streamToBuffer(stream: stream.PassThrough): Promise<Buffer> {
  const chunks: Buffer[] = [];
  return new Promise((resolve, reject) => {
    stream.on("data", (chunk) => chunks.push(chunk));
    stream.on("error", reject);
    stream.on("end", () => {
      console.log("Zip file created.");
      resolve(Buffer.concat(chunks));
    });
  });
}

const emailArtifactPreSignedUrl = async (
  bucketName: string,
  keyName: string,
  firstName: string,
  email: string,
  missingArtifacts: string[],
  altEmail?: string
) => {
  const preSignedUrl: string = await objectClient.getGetPreSignedUrl(
    bucketName,
    keyName,
    60 * 60 // 1 hour expiry in seconds
  );
  logger.info(`PresignedUrl of zip file is: ${preSignedUrl}`);
  const args: ArtifactsEmailArgs = {
    to: [email],
    cc: altEmail ? [altEmail] : undefined,
    firstName,
    preSignedUrl,
    missingArtifacts,
  };
  logger.info("Emailing the artifact file with missing files information.");
  await emailClient.sendArtifactsEmail(args);
};

This code works perfectly fine when I am getting not that many files, but the moment I try to do this with a lot of files it fails. I am expecting the files to passed in, then we are checking to make sure we have them in the S3(if it isn’t there it’s okay), then we are adding the files one at a time to an S3 which we are using to create a downloadUrl.`