I have a create-llama app that works as expected against 100+ pages of PDF documents using SimpleDirectoryReader().loadData. However, when I attempt to load a large .csv file of (17566 tokens apparently) I receive the following error:

BadRequestError: 400 This model's maximum context length is 8192 tokens, however you requested 17566 tokens (17566 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.
at Function.generate (C:Codecanoenode_modulesopenaisrcerror.ts:70:14)
at OpenAI.makeStatusError (C:Codecanoenode_modulesopenaisrccore.ts:383:21)
at OpenAI.makeRequest (C:Codecanoenode_modulesopenaisrccore.ts:446:24)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async OpenAIEmbedding.getOpenAIEmbedding (C:Codecanoenode_modulesllamaindexdistcjsembeddingsOpenAIEmbedding.js:100:26)
at async OpenAIEmbedding.getTextEmbeddings (C:Codecanoenode_modulesllamaindexdistcjsembeddingsOpenAIEmbedding.js:111:16)
at async batchEmbeddings (C:Codecanoenode_modulesllamaindexdistcjsembeddingstypes.js:61:32)
at async OpenAIEmbedding.getTextEmbeddingsBatch (C:Codecanoenode_modulesllamaindexdistcjsembeddingstypes.js:43:16)
at async OpenAIEmbedding.transform (C:Codecanoenode_modulesllamaindexdistcjsembeddingstypes.js:47:28)
at async VectorStoreIndex.getNodeEmbeddingResults (C:Codecanoenode_modulesllamaindexdistcjsindicesvectorStoreindex.js:487:9)
at async VectorStoreIndex.insertNodes (C:Codecanoenode_modulesllamaindexdistcjsindicesvectorStoreindex.js:572:17)
at async VectorStoreIndex.buildIndexFromNodes (C:Codecanoenode_modulesllamaindexdistcjsindicesvectorStoreindex.js:497:9)
at async VectorStoreIndex.init (C:Codecanoenode_modulesllamaindexdistcjsindicesvectorStoreindex.js:445:13)
at async VectorStoreIndex.fromDocuments (C:Codecanoenode_modulesllamaindexdistcjsindicesvectorStoreindex.js:523:16)
at <anonymous> (C:Codecanoeappapichatenginegenerate.ts:27:5)
at getRuntime (C:Codecanoeappapichatenginegenerate.ts:14:3) {

I understand that I should be using a different strategy to load and query against this .csv file, but after combing the documentation for llamaIndex and asking Google, I can’t figure out what I should be doing instead. Smaller .csv files work as expected.

How can I load this large .csv using llamaIndex to effectively:

1. create a doc store, index store, and vector store without errors
2. use my create-llama app to query against the .csv data using OpenAI api within the token limit requirement

here is the basic code where the VectorStoreIndex creation is failing:

async function generateDatasource() {
  console.log(`Generating storage context...`);
  // Split documents, create embeddings and store them in the storage context
  const ms = await getRuntime(async () => {
    const storageContext = await storageContextFromDefaults({
      persistDir: STORAGE_CACHE_DIR,
    });
    const documents = await getDocuments();
    await VectorStoreIndex.fromDocuments(documents, {
      storageContext,
    });
  });
  console.log(`Storage context successfully generated in ${ms / 1000}s.`);
}

here are my .env vars

MODEL=gpt-4-turbo 
EMBEDDING_MODEL=text-embedding-3-large
EMBEDDING_DIM=1024