I need to display the generated text from my Ollama Endpoint as it arrives.
When i call:
curl http://localhost:11434/api/generate -d '{
"model": "llama3.1",
"prompt": "Why is the sky blue?"
}'
Then i get one json response after the other with a little of time in between like that:
{"model":"dolphin-llama3","created_at":"2024-09-25T01:55:49.628815773Z","response":"The","done":false}
{"model":"dolphin-llama3","created_at":"2024-09-25T01:55:49.91987057Z","response":" sky","done":false}
{"model":"dolphin-llama3","created_at":"2024-09-25T01:55:50.210919534Z","response":" appears","done":false}
...
But when i try to get the same response in flutter using this code:
final request = http.Request('POST', Uri.parse(apiEndpoint + '/generate'))
..headers['Content-Type'] = 'application/json'
..body = json.encode({
'model': 'dolphin-llama3',
'prompt': message,
'stream': true,
});
try {
// Send the request and get the streamed response
final streamedResponse = request.send().asStream().asyncExpand((response) => response.stream
.transform(utf8.decoder)
.transform(const LineSplitter()));
// Initialize an empty string to collect the bot response
String botResponse = '';
await for (var jsonLine in streamedResponse) {
// Parse each line of the stream as JSON
final parsedJson = jsonDecode(jsonLine);
if (parsedJson.containsKey('response')) {
setState(() {
botResponse += parsedJson['response'];
_messages.last['message'] = botResponse;
});
}
// If done, break out of stream.
if (parsedJson['done'] == true) {
break;
}
}
} catch (e) {
print('Error: $e');
// Optionally, add an error message to the chat if something goes wrong
setState(() {
_messages.add({
'role': 'bot',
'message': 'An error occurred while fetching the response.'
});
});
}
All the messages are displayed at the same time when the last one would have arrived.
What would i need to change to display each word right after it arrives?
I expected the json responses to behave equally regardless whatever i use curl or the http library in dart/flutter.
aiko929 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.