I have MySQL to S3 replication on AWS DMS in CDC mode (target prep mode: do nothing, aka don’t drop existing data after (re)starting task).

After the CDC task inevitably fails and I need to restart it (restart, not resume) it truncates the partition list on the Glue table (partitioning by date).

To fix that, I run a Glue Crawler, which in turn crashes the DMS Task:

2024-04-28T12:41:09 [TARGET_APPLY    ]E:  Error executing data handler  {operation:COMMIT (7), txnId:48309804031215, streamPosition:mysql-bin-changelog.011248:11886116:-1:11886143:48309804031215:mysql-bin-changelog.011248:11885893} [1028202]  (streamcomponent.c:2048)

What to do?

DMS task settings

{
  "FullLoadSettings": {
    "MaxFullLoadSubTasks": 8,
    "TransactionConsistencyTimeout": 600,
    "CommitRate": 10000,
    "TargetTablePrepMode": "DO_NOTHING",
  },
  "ChangeProcessingDdlHandlingPolicy": {
    "HandleSourceTableDropped": false,
    "HandleSourceTableTruncated": false,
    "HandleSourceTableAltered": true
  },
  "TargetMetadata": {
    "SupportLobs": true,
    "FullLobMode": false,
    "BatchApplyEnabled": false,
    "LimitedSizeLobMode": true,
    "LobChunkSize": 64,
    "LobMaxSize": 10000,
    "TaskRecoveryTableEnabled": false
  },
  "ValidationSettings": {
    "EnableValidation": false
  },
  "Logging": {
    "EnableLogging": true,
    "EnableLogContext": true,
    "LogComponents": [
      {
        "Id": "SOURCE_UNLOAD",
        "Severity": "LOGGER_SEVERITY_DEFAULT"
      },
      {
        "Id": "TARGET_LOAD",
        "Severity": "LOGGER_SEVERITY_DEFAULT"
      },
      {
        "Id": "SOURCE_CAPTURE",
        "Severity": "LOGGER_SEVERITY_DEFAULT"
      },
      {
        "Id": "TARGET_APPLY",
        "Severity": "LOGGER_SEVERITY_DEFAULT"
      },
      {
        "Id": "TASK_MANAGER",
        "Severity": "LOGGER_SEVERITY_DEFAULT"
      }
    ]
  }
}

Glue Table

Serde parameters (1)

| Key                 | Value |
|---------------------|-------|
| serialization.format| 1     |

Table properties (12)

| Key                               | Value              |
|-----------------------------------|-------------------|
| sizeKey                           | 26970155771        |
| objectCount                       | 1666               |
| UPDATED_BY_CRAWLER                | my_project_crawler |
| EXTERNAL                          | TRUE               |
| record Count                      | 48271914           |
| CrawlerSchemaSerializerVersion    | 1.0                |
| transient_lastDdlTime             | 1714307853         |
| averageRecordSize                 | 5476               |
| compression Type                  | none               |
| CrawlerSchemaDeserializer Version | 1.0                |
| classification                    | parquet            |
| typeOfData                        | file               |