I have been trying to create a vector database for RAG model. But the Python script just gets stuck trying to add the documents
This is the code that adds the documents to the database
def add_to_chroma(chunks: list[Document]):
# Load the existing database.
db = Chroma(
persist_directory=CHROMA_PATH, embedding_function=get_embedding_function()
)
# Calculate Page IDs.
chunks_with_ids = calculate_chunk_ids(chunks)
# Add or Update the documents.
existing_items = db.get(include=[]) # IDs are always included by default
existing_ids = set(existing_items["ids"])
print(f"Number of existing documents in DB: {len(existing_ids)}")
# Only add documents that don't exist in the DB.
new_chunks = []
for chunk in chunks_with_ids:
if chunk.metadata["id"] not in existing_ids:
new_chunks.append(chunk)
if len(new_chunks):
print(f" Adding new documents: {len(new_chunks)}")
new_chunk_ids = [chunk.metadata["id"] for chunk in new_chunks]
db.add_documents(new_chunks, ids=new_chunk_ids) ## Code gets stuck here
db.persist()
else:
print(" No new documents to add")
And this is the error I get when I kill the process
Error
Iam using the Ollama embeddings
from langchain_community.embeddings.ollama import OllamaEmbeddings
def get_embedding_function():
embeddings = OllamaEmbeddings(model="nomic-embed-text")
return embeddings
And Ollama seems to be running fine
New contributor
Manish I is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.