I have an Elasticsearch cluster configured in a docker compose file.
The cluster has three nodes, all running on the same host. In each node I have a docker volume (not pinned) that is mapped to /usr/share/elasticsearch/data folder inside the node.
I wanted to move the data of two of the nodes to a different drive on the host, so I copied the folders from /var/lib/docker/volumes to the different drive, and changed the volume defintion under the nodes to be defined insead of:
`volumes:
- data01:/usr/share/elasticsearch/data`
to:
`volumes:
- /media/some_drive/data01:/usr/share/elasticsearch/data`
When I started the cluster again, it failed to assign some of the indexes because their master shard couldn’t be allocated. So I tried running curl -X POST http://localhost:9200/_cluster/reroute?retry_failed=true
which failed bacuse it couldn’t find the indexes.
In hindsight, there was a _data
folder inside the folders I copied from the docker volumes folder, and I think this is the actual volume that was used by docker, so I had to map that to the volume and not the containing folder.
I decided to try to put everything back into stable state and rething my plan. So I copied the folders back to the docker volumes folder in the original place, reverted the volumes config in the docker compose file and started the cluster.
The cluster failed to start, when I looked at the logs, it said that elasticsearch failed to elect a master node because it was waiting for nodes with some IDs, but the existing nodes IDs are different.
The volumes’ size looks correct, so I hope the data is not lost.
Is there anything I can do?