I’m building a RAG application that will scrape ~10k articles from the internet and use those for a chatbot.

I’m wondering where to store the data in the interim.

The pipeline is essentially:

1. Scrape the data

2. Add metadata

3. Chunk/Embed

4. Add to vector database

5. Run queries

My question is where do I store the articles between steps 1 and 2/3.

These are some ideas I considered:

1. Throw it all into postgres, articles and everything

2. Throw it all into mongodb and add metadata as fields

3. Store the articles in s3 (each source would have a single file with all the articles) and then store the metadata in postgres along with an id for each article (probably the url?)

The other thing worth noting is that I plan to run queries over the article contents to generate the metadata. For example, I want to pass the article contents to an LLM and have the LLM return the topic of the article