Using OpenSearch: 2.12.0, connecting through @opensearch-project/opensearch client library from Node.js api service.

Facing high frequency(50-60%) of timeout errors randomly, which are not flowing to OpenSearch cluster from api-service. OpenSearch cluster is configured on vpc.

Error Details:

"err": {
    "type": "TimeoutError",
    "message": "Request timed out",
    "stack": "TimeoutError: Request timed outn    at ClientRequest.onTimeout (/home/node/code/node_modules/@opensearch-project/opensearch/lib/Connection.js:123:16)n    at /opt/dynatrace/oneagent/agent/bin/1.299.45.20240924-123410/any/nodejs/nodejsagent.js:2273:48n    at t.runInContext (/opt/dynatrace/oneagent/agent/bin/1.299.45.20240924-123410/any/nodejs/nodejsagent.js:2384:24)n    at ClientRequest.onTimeout (/opt/dynatrace/oneagent/agent/bin/1.299.45.20240924-123410/any/nodejs/nodejsagent.js:2273:18)n    at ClientRequest.emit (node:events:513:28)n    at ClientRequest.emit (node:domain:489:12)n    at TLSSocket.emitRequestTimeout (node:_http_client:848:9)n    at TLSSocket.l (/opt/dynatrace/oneagent/agent/bin/1.299.45.20240924-123410/any/nodejs/nodejsagent.js:5043:29)n    at /opt/dynatrace/oneagent/agent/bin/1.299.45.20240924-123410/any/nodejs/nodejsagent.js:2273:48n    at t.runInContext (/opt/dynatrace/oneagent/agent/bin/1.299.45.20240924-123410/any/nodejs/nodejsagent.js:2384:24)n    at TLSSocket.l (/opt/dynatrace/oneagent/agent/bin/1.299.45.20240924-123410/any/nodejs/nodejsagent.js:2273:18)n    at TLSSocket.emit (node:events:525:35)n    at TLSSocket.emit (node:domain:489:12)n    at TLSSocket.Socket._onTimeout (node:net:570:8)n    at listOnTimeout (node:internal/timers:569:17)n    at processTimers (node:internal/timers:512:7)",
    "name": "TimeoutError",
    "meta": {
      "body": null,
      "statusCode": null,
      "headers": null,
      "meta": {
        "context": null,
        "request": {
          "params": {
            "method": "POST",
            "path": "/{index}/_search",
            "body": "....",
            "querystring": "",
            "headers": {
              "user-agent": "opensearch-js/2.12.0 (linux 5.10.223-212.873.amzn2.x86_64-x64; Node.js v18.15.0)",
              "content-type": "application/json",
              "content-length": "386"
            },
            "timeout": 30000
          },
          "options": {},
          "id": 1
        },
        "name": "opensearch-js",
        "connection": {
          "url": "https://vpc-{...url...}.amazonaws.com/",
          "id": "https://vpc-{...url...}.amazonaws.com/",
          "headers": {},
          "deadCount": 4,
          "resurrectTimeout": 1727432936527,
          "_openRequests": 0,
          "status": "dead",
          "roles": {
            "data": true,
            "ingest": true
          }
        },
        "attempts": 3,
        "aborted": false
      }
    }

Tried to run scripts on 10-15 queries of (api service which calls OpenSearch service internally) and timeout happens on queries randomly without any entry to OpenSearch cluster. It’s not able to establish http connection to OpenSearch cluster.

1. Timeout’s are very random without any pattern on query and sometimes all of the queries work.
2. It’s an internal call from api-service to OpenSearch so the client address is never changing.

Questions:

1. Is there any limit on number of connections to OpenSearch through client library?
2. Anything to take care when hosting on vpc, because from analysis we came to know api-service is not able to establish connection to OpenSearch ELB from AWS fully managed service?