Going off docs/other posts online, you should cache() before checkpoint() because checkpoint() is done afterwards with a different action. However looking at spark query plan, this doesn’t seem to be true for what I’m doing:

    <bunch of spark transformations - filters, joins, column renaming>
    df_main = df_main.checkpoint()
    <bunch of spark transformations - filters, joins, column renaming>>
    df_main.write.mode("overwrite").parquet('s3://<location>')

The query plan for the final parquet jobs query:

You can see there’s a Scan ExistingRDD, which is the results from the previous query for the transformations up to the checkpoint. I was thinking maybe this is some special quirk when doing a write so I tried count() instead and got the same results.

I tried adding cache() to see the difference; all it did was change the checkpoint() query to have InMemoryRelation and InMemoryTableScan instead of AdaptiveSparkPlan, jobs still split the same.

I want to understand exactly whats happening so that I know how/if I should be using checkpoint+cache() together. Am I looking at the wrong thing? Is my understanding incorrect?

Thanks

ps: The one time I did find cache() useful, was when I branched off from before the checkpoint; in this case the write did use existing RDD, but the branch had recomputed everything.

    df_main = xxx
    df_branch = df_main.<transformations>
    df_main = df_main.checkpoint()
    df_main.union(df_branch)
    df_main.write....

Spark version I’m on: 3.3.0-amzn-1, using AWS/EKS

edit: I noticed that in a lot of the examples, people don’t reassign, ie they do df.checkpoint() not df = df.checkpoint(). When I do this, then both checkpoint+write will re-do all the transformations, and then it makes sense to cache(). This sort of begs the question then, what is the case I want to do df.checkpoint() without reassignment? When I just want to make a checkpoint but not to truncate lineage? Is it only useful for spark streaming?