I’m developing a system and could use some expert insights on its architecture and implementation.

Here’s a brief overview of the current flow:

1. User fills out an intake questionnaire with X questions
2. System references datasets/examples from existing system folders
3. Multiple parallel flows run with Claude API to generate texts based on guidelines and relevant intake responses

Now, I’m grappling with some key issues:

• Overall Architecture

1. Should I use Agentic AI frameworks? If so, which ones?
2. Or is there a more efficient approach for a system like this?

• Data Storage and Management

1. How to store intake responses for efficient AI flow access?
2. Is RAG (Retrieval-Augmented Generation) advisable for storage and retrieval? If not, what are the alternatives?

• Existing Dataset Integration

1. What’s the most efficient way to incorporate examples from existing folders into the flow?
2. Better to train a small model on the dataset or integrate it directly?

• Parallel Process Optimization
  How to efficiently manage and synchronize the parallel flows?

Important notes:

1. Output (generated texts) and number of intake questions remain constant
2. It’s a complex flow split into several sub-flows, not a single long prompt

Thanks in advance!

I am quite clueless since there are many options, I am expecting an experienced advice