I am integrating an AI model into a web app (this model needs to have some context in order to maintain a fluid conversation with the user) on a local deployment. The problem is the intrinsic structure of the thread.

I know how a pool thread works. And the problem is that, when doing multiple POST request (for chatting with the bot), there is possibily that another thread that has not been used responses to that request. Then, the context will be saved in the memory of that thread, not in the one that we have been previously using.

Firstly, I want to mention that none solution should not be either implemeting cookies or creating a file for saving the context. The idea is to assign a thread per session_token.

I have tried the following:

from fastapi import APIRouter, HTTPException
from pydantic import BaseModel
from dotenv import load_dotenv
from openai import AzureOpenAI
import os
from typing import Dict, List
from uuid import UUID, uuid4

load_dotenv()
router = APIRouter()

class Message(BaseModel):
    """Class representing the message of a conversation"""
    role: str  # either 'user' or 'assistant'
    content: str

class Prompt(BaseModel):
    """Class for sending or receiving messages"""
    session_id: UUID
    prompt: str

class NewSessionResponse(BaseModel):
    session_id: UUID
    
class ResetRequest(BaseModel):
    session_id: UUID

# Dictionary to store conversations
conversations: Dict[UUID, List[Message]] = {}

@router.post("/ai/chat")
async def chat(prompt: Prompt):
    """In charge of executing and obtaining the connection with the model"""
    api_key = os.getenv("key")
    api_url = os.getenv("endpoint url")
    
    if not api_key or not api_url:
        raise HTTPException(status_code=500, detail='API key or API endpoint not found! Try again')
    
    client = AzureOpenAI("here goes some parameters")
    
    # Retrieve the conversation history for the session
    session_id = prompt.session_id
    if session_id not in conversations:
        conversations[session_id] = []
    
    # Add the user's prompt to the conversation history
    conversations[session_id].append(Message(role="user", content=prompt.prompt))
    
    # Create the context for the API request
    context = [{'role': msg.role, 'content': msg.content} for msg in conversations[session_id]]
    
    # Request completion from the model
    response = client.chat.completions.create(
        model="gpt-35-turbo-4k-0613",
        messages=context
    )
    
    # Extract the model's response
    model_response = response.choices[0].message.content
    
    # Add the assistant's response to the conversation history
    conversations[session_id].append(Message(role='assistant', content=model_response))
    print(conversations)
    return {"response": model_response}

@router.post("/ai/new_session", response_model=NewSessionResponse)
async def newSession():
    """Creates a new conversation session"""
    session_id = uuid4()
    conversations[session_id] = []
    return NewSessionResponse(session_id=session_id)

I also try to implement threading when managing requests, but it did not work. I suppose that this happens because the Uvicorn threads and the threads created here are not the same.