This question is the second part of my debugging journey regarding retryable Datastore exceptions. The first part of my issue can be found here.

I have the following two stack traces (when encountering retryable datastore exceptions like the ones described in my other post) :

Trace A

  at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:140)
  at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:125)
  at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.runQuery(HttpDatastoreRpc.java:202)
  at com.google.cloud.datastore.DatastoreImpl.lambda$runQuery$0(DatastoreImpl.java:214)
  at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
  at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
  at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
  at com.google.cloud.datastore.DatastoreImpl.runQuery(DatastoreImpl.java:213)
  at com.google.cloud.datastore.QueryResultsImpl.sendRequest(QueryResultsImpl.java:79)
  at com.google.cloud.datastore.QueryResultsImpl.<init>(QueryResultsImpl.java:63)
  at com.google.cloud.datastore.DatastoreImpl.run(DatastoreImpl.java:196)
  at com.google.cloud.datastore.DatastoreImpl.run(DatastoreImpl.java:185)
  at com.googlecode.objectify.impl.AsyncDatastoreReaderWriterImpl.run(AsyncDatastoreReaderWriterImpl.java:62)
  at com.googlecode.objectify.impl.QueryEngine.queryNormal(QueryEngine.java:66)
  at com.googlecode.objectify.impl.QueryImpl.iterator(QueryImpl.java:329)
  at com.googlecode.objectify.impl.QueryImpl.list(QueryImpl.java:337)
  at com.XXXXXX.dao.StatsDAO.lambda$getStatsForAccount$0(StatsDAO.java:127)

Trace B

at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:140)
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:125)
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.runQuery(HttpDatastoreRpc.java:202)
at com.google.cloud.datastore.DatastoreImpl.lambda$runQuery$0(DatastoreImpl.java:214)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.datastore.DatastoreImpl.runQuery(DatastoreImpl.java:213)
at com.google.cloud.datastore.QueryResultsImpl.sendRequest(QueryResultsImpl.java:79)
at com.google.cloud.datastore.QueryResultsImpl.computeNext(QueryResultsImpl.java:98)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140)
at com.googlecode.objectify.impl.StuffingQueryResults.hasNext(StuffingQueryResults.java:32)
at com.googlecode.objectify.impl.KeyQueryResults.hasNext(KeyQueryResults.java:19)
at com.googlecode.objectify.impl.ResultWithCursor$Iterator.hasNext(ResultWithCursor.java:26)
at com.google.common.collect.Iterators$4.hasNext(Iterators.java:629)
at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46)
at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46)
at com.google.common.collect.Iterators$ConcatenatedIterator.getTopMetaIterator(Iterators.java:1379)
at com.google.common.collect.Iterators$ConcatenatedIterator.hasNext(Iterators.java:1395)
at com.google.common.collect.Iterators$5.computeNext(Iterators.java:671)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140)
at com.googlecode.objectify.impl.HybridQueryResults.hasNext(HybridQueryResults.java:93)
at com.google.common.collect.Iterators.addAll(Iterators.java:366)
at com.google.common.collect.Lists.newArrayList(Lists.java:146)
at com.googlecode.objectify.util.MakeListResult.translate(MakeListResult.java:22)
at com.googlecode.objectify.util.MakeListResult.translate(MakeListResult.java:12)
at com.googlecode.objectify.util.ResultTranslator.nowUncached(ResultTranslator.java:21)
at com.googlecode.objectify.util.ResultCache.now(ResultCache.java:30)
at com.googlecode.objectify.util.ResultProxy.invoke(ResultProxy.java:32)
at com.sun.proxy.$Proxy27.isEmpty(Unknown Source)
at com.XXXXXX.service.UserService.getUsersByEmail(UserService.java:339)
at com.XXXXXX.dao.UserDAO.getUsersByEmail(UserDAO.java:57)

Let me highlight that both queries are call list() to calculate and return the results and NOT iterator().
Based on the above logs (and other logs), it appears that Objectify (or Datastore) sometimes executes the query and returns the results directly, while other times it returns a proxy (indicating lazy loading). This inconsistency causes issues because I’ve implemented a retry mechanism in all the read methods across my DAO classes, which was quite a challenging task. The problem arises when exceptions are thrown in the service layer that calls the DAO. This can happen when iterating over the result set or calling the isEmpty() method on the returned list (which may be a proxy).

TL;DR; my retry mechanism successfully managed the exception for Trace A, but it encountered difficulty with Trace B due to an exception being thrown in the Service layer. To address this issue, I’ve added a call to get the size of the list to force evaluation, as shown below.

  public List<User> getUsersByEmail(
      String email) {
    return runWithRetries(
        () -> {
          List<User> users =
              load().filter("email", email).list();
          // Force evaluation by getting the size of the list
          int size = users.size();
          log.debug("runWithRetries forcing evaluation, retrieved {} entries", size);
          return users;
        });
  }

Am I missing something? How can Datastore be so unpredictable, returning actual results at times and a proxy at other times? I feel like this is the last step to getting retries to work properly, but I’m running out of patience with how random these exceptions seem. Does anyone else find this as frustrating as I do?