I’m having a lot of issues here with trying to get Langserve Playground chat mode to work properly.

So when I started this project, I initially deployed a Langchain app on google cloud run by using the python code sample from from https://codelabs.developers.google.com/codelabs/build-and-deploy-a-langchain-app-on-cloud-run#0 . I also set the playground to default mode. When I type in a question, and a little bit later the bot responded with an answer. Everything works as intended, no issue there.

However, I want to use chat mode. I understand that I am supposed to use ‘dict’ as an input_type for my chain, otherwise I’ll get an ‘Expected content-type’ error as others have reported. So I did exactly that, but what happens is the “AI” line shows up in the chat, the bot thinks about it for a bit, and then leave the answer blank. See the screenshot below:

enter image description here

I tried several different ways to solve this, even creating my own custom input type like “class InputChat(BaseModel)” and set the chain’s input_type to InputChat but the end result is the same, the bot thinks for a few seconds but does not output any text response. I’m not sure if this is an issue with Langserve, or if I coded it wrong. I would very much appreciate anyone’s help with this.

Here is my server code that I deployed:

import os
from google.cloud.sql.connector import Connector
from langchain_google_vertexai import VertexAI
from langchain_google_vertexai import VertexAIEmbeddings
from langchain_core.runnables import RunnablePassthrough, RunnableParallel
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import PromptTemplate, ChatPromptTemplate, MessagesPlaceholder
from langchain_community.vectorstores.pgvector import PGVector

from typing import List, Union
from langchain.pydantic_v1 import BaseModel, Field
from langchain_core.messages import AIMessage, HumanMessage, SystemMessage
from langchain.chat_models import ChatOpenAI

from fastapi import FastAPI
from fastapi.responses import RedirectResponse
from langserve import add_routes

app = FastAPI()


@app.get("/")
async def redirect_root_to_docs():
    return RedirectResponse("/docs")


# (0) Initialize chat history table




# (1) Initialize VectorStore
connector = Connector()


def getconn() -> pg8000.dbapi.Connection:
    conn: pg8000.dbapi.Connection = connector.connect(
        os.getenv("DB_INSTANCE_NAME", ""),
        "pg8000",
        user=os.getenv("DB_USER", ""),
        password=os.getenv("DB_PASS", ""),
        db=os.getenv("DB_NAME", ""),
    )
    return conn


vectorstore = PGVector(
    connection_string="postgresql+pg8000://",
    use_jsonb=True,
    engine_args=dict(
        creator=getconn,
    ),
    embedding_function=VertexAIEmbeddings(
        model_name="textembedding-gecko@003"
    )
)

# (2) Build retriever


def concatenate_docs(docs):
    return "nn".join(doc.page_content for doc in docs)


reports_retriever = vectorstore.as_retriever() | concatenate_docs



# (3) Create prompt template

template =     """You are a helpful financial analyst answering questions from investors. Use the retrieved analyst reports on the company's quarterly earnings to answer the questions

Reports: {reports}

Here is your question: {query}
Your answer: """

prompt_template = ChatPromptTemplate.from_template(template)    


# (4) Initialize LLM
llm = VertexAI(
    model_name="gemini-1.0-pro-001",
    temperature=0.2,
    max_output_tokens=500,
    top_k=40,
    top_p=0.95
)


# (5) Chain everything together
chain = (
    RunnableParallel({
        "reports": reports_retriever,
        "query": RunnablePassthrough()
    })
    | prompt_template
    | llm
    | StrOutputParser()
)


# add_routes(app, NotImplemented)

add_routes(app,
    chain.with_types(input_type=dict,output_type=str),         
    playground_type="chat",
           )


if __name__ == "__main__":
    import uvicorn

    uvicorn.run(app, host="0.0.0.0", port=8000)