I need to insert records into multiple collections within the same MongoDB database. Each record can be inserted into a different collection. More than one structure is trying to do this operation at the same time and the wait queue is swelling.
And after then there are latency more than 5 secs about getting information wait queue and exceptions like MongoWaitQueueFullException or TimeoutException because of the wait queue.
I can get rid of WaitQueueFullException by increasing MaxConnectionPoolSize, but then I start to get timeout while getting information from wait queue, wait queue timeout is 5 seconds, I think even this is too much.
How can I prevent this wait time?
public class WriteOperation(IMongoCollection<BsonDocument> collection, IEnumerable<BsonDocument> documents)
{
private IMongoCollection<BsonDocument> Collection { get; } = collection;
private IEnumerable<BsonDocument> Documents { get; } = documents;
public async Task WriteAsync(InsertManyOptions options)
=> await Collection.InsertManyAsync(Documents, options);
}
public async Task WriteAsync(Dictionary<string, List<TTick>> groupedTicks)
{
var writeOperations = from groupedTick in groupedTicks
let collectionName = groupedTick.Key + "_ticks"
let collection = GetCollection(collectionName)
let documents = groupedTick.Value.Select(x => x.ToBsonDocument()).ToList()
select new WriteOperation(collection, documents);
var tasks = writeOperations.Select(op => op.WriteAsync(_insertingOptions)).ToArray();
await Task.WhenAll(tasks).ConfigureAwait(false);
}
private IMongoCollection<BsonDocument> GetCollection(string collectionName)
{
if (_collections.TryGetValue(collectionName, out var collection))
return collection;
collection = _database.GetCollection<BsonDocument>(collectionName);
_collections[collectionName] = collection;
return collection;
}
Halil is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.