I am busy building a demo application with Spring AI (Open AI as the LLM) and Redis as my vector database.

I have had trouble trying to set up multiple indexes in the database. Currently, I load many text documents and store them all under one index specified in my application.properties. However I would like certain documents to be stored in “index1” and some documents stored in “index2”.

I tried to manually configure the RedisVectorStore and set up multiple indexes but I struggled to achieve success here.

Here is an example of my service class which processes, chunkes, embeds and loads the data into my redis database. Along with my application.properties:

Data Loading:

  public void load() {
        logger.info("Starting to load documents");
        try {
            List<Document> allDocuments = new ArrayList<>();

            for (VectorStoreProperties.DocumentConfig docConfig : vectorStoreProperties.getDocumentsToLoad()) {
                List<Document> documents = processDocument(docConfig.getResource(), docConfig.getTag());
                allDocuments.addAll(documents);
            }

            logger.info("Processed {} document chunks in total", allDocuments.size());

            this.vectorStore.accept(allDocuments);

            logger.info("Finished loading documents into vector store");
        } catch (Exception e) {
            logger.error("Error loading documents", e);
        }
    }

    private List<Document> processDocument(Resource resource, String tag) throws IOException {
        String content = new String(resource.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
        var tokenTextSplitter = new TokenTextSplitter();
        List<Document> documents = tokenTextSplitter.apply(Collections.singletonList(new Document(content)));

        // Add metadata tag to each document
        return documents.stream()
                .map(doc -> {
                    Map<String, Object> metadata = new HashMap<>(doc.getMetadata());
                    metadata.put("tag", tag);
                    return new Document(doc.getId(), doc.getContent(), (List<Media>)doc.getMedia(), metadata);
                })
                .collect(Collectors.toList());
    }

Application.properties:

spring.ai.vectorstore.redis.index=index1
spring.ai.vectorstore.redis.prefix=idx1:
spring.ai.vectorstore.redis.initialize-schema=true

spring.data.redis.host=<>
spring.data.redis.port=<>

sfg.aiapp.documentsToLoad[0].resource=classpath:/Example.txt
sfg.aiapp.documentsToLoad[0].tag=example

`