What is the difference between using a persistent storage like S3 vs Using a Vector DB to store data in LLAMA INDEX?
In either cases I’m loading the index from a persistent storage and running a query on top of it.
Using S3 or Local disk storage :
# load documents
documents = SimpleDirectoryReader("./data/paul_graham/").load_data()
index = VectorStoreIndex.from_documents(documents)
# save index to disk
index.set_index_id("vector_index")
index.storage_context.persist("./storage")
# rebuild storage context
storage_context = StorageContext.from_defaults(persist_dir="storage")
# load index
index = load_index_from_storage(storage_context, index_id="vector_index")
Using FAISS or any other Vector Store :
# load documents
documents = SimpleDirectoryReader("./data/paul_graham/").load_data()
vector_store = FaissVectorStore(faiss_index=faiss_index)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(
documents, storage_context=storage_context
)
# save index to disk
index.storage_context.persist()
# load index from disk
vector_store = FaissVectorStore.from_persist_dir("./storage")
storage_context = StorageContext.from_defaults(
vector_store=vector_store, persist_dir="./storage"
)
index = load_index_from_storage(storage_context=storage_context)
Trying to understand how will one benefit over the other