I am trying to fetch all the folders under AWS organizations recursively but getting following throttling exception intermittently.
com.amazonaws.services.organizations.model.TooManyRequestsException: AWS Organizations can't complete your request because another request is already in progress. Try again later. (Service: AWSOrganizations; Status Code: 400; Error Code: TooManyRequestsException; Request ID: f29b4e1c-0b1b-4259-abcc-8d9ea8ae78dc; Proxy: fproxy01.p04.phx01.qualys.com)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
<code>
com.amazonaws.services.organizations.model.TooManyRequestsException: AWS Organizations can't complete your request because another request is already in progress. Try again later. (Service: AWSOrganizations; Status Code: 400; Error Code: TooManyRequestsException; Request ID: f29b4e1c-0b1b-4259-abcc-8d9ea8ae78dc; Proxy: fproxy01.p04.phx01.qualys.com)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
... 40 frames truncated
</code>
com.amazonaws.services.organizations.model.TooManyRequestsException: AWS Organizations can't complete your request because another request is already in progress. Try again later. (Service: AWSOrganizations; Status Code: 400; Error Code: TooManyRequestsException; Request ID: f29b4e1c-0b1b-4259-abcc-8d9ea8ae78dc; Proxy: fproxy01.p04.phx01.qualys.com)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
... 40 frames truncated
I tried configuring retry while creating AWS client itself by referring AWS Docs andd added following to the code:
<code>PredefinedBackoffStrategies.EqualJitterBackoffStrategy jitterBackoffStrategy = new PredefinedBackoffStrategies.EqualJitterBackoffStrategy(
awsClientConfig.getBaseDelayBeforeNextRetryInMills(), awsClientConfig.getMaxBackoffTimeInMills());
RetryPolicy retryPolicy = new RetryPolicy
.withBackoffStrategy(jitterBackoffStrategy)
.withMaxErrorRetry(awsClientConfig.getMaxRetries())
.withHonorMaxErrorRetryInClientConfig(false)
ClientConfiguration clientConfiguration = new ClientConfiguration()
.withProtocol(Protocol.HTTPS)
.withRetryPolicy(retryPolicy)
.withProxyHost(proxyConfiguration.getHost())
.withProxyPort(proxyConfiguration.getPort())
.withConnectionTimeout(awsClientConfig.getConnectTimeoutInMills());
AWSOrganizations client = AWSOrganizationsClientBuilder.standard()
.withClientConfiguration(clientConfiguration)
.withCredentials(new AWSStaticCredentialsProvider(new BasicSessionCredentials(
Credentials().getAccessKeyId(),
Credentials().getSecretAccessKey(),
Credentials().getSessionToken())))
<code>PredefinedBackoffStrategies.EqualJitterBackoffStrategy jitterBackoffStrategy = new PredefinedBackoffStrategies.EqualJitterBackoffStrategy(
awsClientConfig.getBaseDelayBeforeNextRetryInMills(), awsClientConfig.getMaxBackoffTimeInMills());
RetryPolicy retryPolicy = new RetryPolicy
.RetryPolicyBuilder()
.withBackoffStrategy(jitterBackoffStrategy)
.withMaxErrorRetry(awsClientConfig.getMaxRetries())
.withHonorMaxErrorRetryInClientConfig(false)
.build();
ClientConfiguration clientConfiguration = new ClientConfiguration()
.withProtocol(Protocol.HTTPS)
.withRetryPolicy(retryPolicy)
.withProxyHost(proxyConfiguration.getHost())
.withProxyPort(proxyConfiguration.getPort())
.withConnectionTimeout(awsClientConfig.getConnectTimeoutInMills());
AWSOrganizations client = AWSOrganizationsClientBuilder.standard()
.withClientConfiguration(clientConfiguration)
.withCredentials(new AWSStaticCredentialsProvider(new BasicSessionCredentials(
Credentials().getAccessKeyId(),
Credentials().getSecretAccessKey(),
Credentials().getSessionToken())))
.withRegion(region)
.build();
</code>
PredefinedBackoffStrategies.EqualJitterBackoffStrategy jitterBackoffStrategy = new PredefinedBackoffStrategies.EqualJitterBackoffStrategy(
awsClientConfig.getBaseDelayBeforeNextRetryInMills(), awsClientConfig.getMaxBackoffTimeInMills());
RetryPolicy retryPolicy = new RetryPolicy
.RetryPolicyBuilder()
.withBackoffStrategy(jitterBackoffStrategy)
.withMaxErrorRetry(awsClientConfig.getMaxRetries())
.withHonorMaxErrorRetryInClientConfig(false)
.build();
ClientConfiguration clientConfiguration = new ClientConfiguration()
.withProtocol(Protocol.HTTPS)
.withRetryPolicy(retryPolicy)
.withProxyHost(proxyConfiguration.getHost())
.withProxyPort(proxyConfiguration.getPort())
.withConnectionTimeout(awsClientConfig.getConnectTimeoutInMills());
AWSOrganizations client = AWSOrganizationsClientBuilder.standard()
.withClientConfiguration(clientConfiguration)
.withCredentials(new AWSStaticCredentialsProvider(new BasicSessionCredentials(
Credentials().getAccessKeyId(),
Credentials().getSecretAccessKey(),
Credentials().getSessionToken())))
.withRegion(region)
.build();
But I am not able to figure out that while AWS client reties internally, will it throw above exception evert time. For instance, if I have set max_retry = 3
will the above exception (I am catching it when calling listRoot API of AWS SDK) be printed 3 times or SDK will throw it only once after all the reties are done.
Also is there any way we can log all the retries of AWS SDK.