I am working with LangChain’s LLMChain to process large documents. I see quite some problems when the input text exceeds the token limit of the language model. For long input text, it has significant slowdowns and can even sometimes fail due to overflow of tokens.
This is the sample code,
from langchain.chains import LLMChain
from langchain.prompts import PromptTemplate
from langchain.llms import OpenAI
prompt = PromptTemplate("Analyze the following text and summarize it: {text}")
llm_chain = LLMChain(llm=OpenAI(model="gpt-3.5-turbo"), prompt=prompt)
long_text = "This is a very long document..." # Assume this text is extremely long
response = llm_chain.run(text=long_text)
print(response)
The problem is that when long_text is too long, the chain either fails for token limit reasons or works for an absurdly long time. I mean, models like GPT-3.5-turbo do have a token limit set, but still, I would not like to have to do this manually at each use of my function.
Can the input tokenization of LangChain be adjusted dynamically, so that it remains efficient while considering the handling of long texts?
DAR is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.