At my company, I have just had a few devops related to scaling, clustering Elasticsearch so I should rewrite it in case we need to use it later. One task is to move data from the old single-node Elasticsearch to the new cluster . The old single-node Elasticsearch was on a private virtual server (VM) and deployed using docker-compose. The new Cluster Elasticsearch is deployed with Helm in Kubernetes and on another VM cluster.
A relatively simple way to do this is to create a snapshot backup on the old single-node and restore the snapshot on the new cluster. After a while google, I have closed the basic steps of using Kibana:
- Install repository-s3 plugin
- Configure AWS S3 credentials for Elasticsearch
- Register a repository with type S3 on the old single-node Elasticsearch
- Create a snapshot policy to automate snapshot capture and push to the S3 bucket from old Elasticsearch
- Register a similar repository on the new Elasticsearch cluster and restore the snapshot
Here we go into detail.
1. Install the repository-s3 plugin for Elasticsearch
1.1. Rebuild Elasticsearch docker image
Installing and activating the plugin requires restarting the Elasticsearch cluster. Therefore, we cannot install it directly, we need to edit the Elasticsearch image. Specifically, it is necessary to write a new Dockerfile to build a new image from the default elasticsearch image:
1 2 3 4 | <span class="token keyword">ARG</span> version=latest <span class="token keyword">FROM</span> docker.elastic.co/elasticsearch/elasticsearch <span class="token punctuation">:</span> $ <span class="token punctuation">{</span> version <span class="token punctuation">}</span> <span class="token keyword">RUN</span> bin/elasticsearch <span class="token punctuation">-</span> plugin install <span class="token punctuation">-</span> <span class="token punctuation">-</span> batch repository <span class="token punctuation">-</span> s3 |
Type the command build image as follows:
1 2 3 4 5 6 7 | <span class="token comment"># latest version</span> docker build --tag yourname/elasticsearch <span class="token builtin class-name">.</span> <span class="token comment"># specific version</span> docker build --build-arg <span class="token assign-left variable">version</span> <span class="token operator">=</span> <span class="token number">7.10</span> .1 --tag your-name/elasticsearch:7.10.1 <span class="token builtin class-name">.</span> <span class="token comment"># more fancy specific version command</span> <span class="token builtin class-name">export</span> <span class="token assign-left variable">VERSION</span> <span class="token operator">=</span> <span class="token number">7.10</span> .1 <span class="token operator">&&</span> docker build --build-arg <span class="token assign-left variable">version</span> <span class="token operator">=</span> <span class="token variable">$VERSION</span> --tag yourname/elasticsearch: <span class="token variable">$VERSION</span> -f elasticsearch.dockerfile <span class="token builtin class-name">.</span> |
If the build image goes well, we can see the output as follows:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | ... -> Installing repository-s3 -> Downloading repository-s3 from elastic @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.lang.RuntimePermission accessDeclaredMembers * java.lang.RuntimePermission getClassLoader * java.lang.reflect.ReflectPermission suppressAccessChecks * java.net.SocketPermission * connect,resolve * java.util.PropertyPermission es.allow_insecure_settings read,write See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. -> Installed repository-s3 ... |
After successfully building the image, push the image to somewhere that your Kubernetes cluster can pull, for example: docker hub , AWS ECR … If you push the image to a private registry, pay attention to configuring imagePullSecrets.
1.2. Deploy the old single-node Elasticsearch with the newly created image
Normally, old Elasticsearch will re-launch successfully, if something goes wrong, look at the log and try to google why… lol…
2. Configure AWS S3 credentials
To perform this step, we need to docker exec into the old Elasticsearch container and type:
1 2 3 | elasticsearch-keystore add s3.client.default.access_key elasticsearch-keystore add s3.client.default.secret_key |
The two commands above will ask you to enter (prompt) the information, but when entering, you will not see anything, just copy paste and enter. Kind of you will see a screen like this:
1 2 3 4 5 | [ <a class="__cf_email__" href="/cdn-cgi/l/email-protection">[email protected]</a> bin]# elasticsearch-keystore add s3.client.default.access_key Enter value for s3.client.default.access_key: [ <a class="__cf_email__" href="/cdn-cgi/l/email-protection">[email protected]</a> bin]# elasticsearch-keystore add s3.client.default.secret_key Enter value for s3.client.default.secret_key: |
To get access_key and secret_key from an IAM user with S3 bucket read and write permissions, go to AWS IAM Users> select a user with permissions on the S3 service> Security credentials> Create access key. After entering credentials, restart single-node Elasticsearch. After a successful reboot, open Kibana Dev Tools to reload_secure_settings:
1 2 3 | POST _nodes/reload_secure_settings {} |
3. Register a repository with type S3
3.1. Enter a name and select the type as S3 -> click Next
3.2. Enter a bucket name -> click Register
Buckets need to be initialized on the AWS S3 console .
3.3. (optional) Verify repo
After successful registration, the screen will display the newly created repo information. You will see the button “Verify repository” in the details panel on the right, after clicking this button Kibana will show “Verification status” as connected. 
If the status is “Not connected”, the error information will also be displayed in Details.
4. Create Policy to automate snapshot creation and deletion
We just need to switch to the policy tab, located right next to the repository. Click “Create a policy”. 
The steps to create the policy are quite simple and intuitive on the interface of Kibana. You can refer to the genuine document here: https://www.elastic.co/guide/en/kibana/current/snapshot-repositories.html#kib-snapshot-policy . 
5. Restore the snapshot on the new Elasticsearch cluster
5.1. Repeat steps 1.2 through 3.3 with the new Elasticsearch cluster
In the scope of this article, we only need to restore the snapshot on the new cluster, so there is no need to create a policy to backup.
5.2. Restore snapshot
Go to the Snapshots tab, choose the latest snapshot and click Restore. 
After the restore has started, you can monitor the status on the “Restore Status” tab. 
That’s it then. Thank you for reading here!