ElasticSearch or even Elastic as a whole, has a great detail documentation on performing a snapshot and restoring the backup. However in order to a simple snapshot; especially for the beginner it may be overwhelming.
In the below steps, I aim to achieve a simple snapshot and restore from Local directory.
Steps:
- Define the index, and populate data in the index
- Create Repository
- Define Snapshot Policy
- Run the snapshot
- Query Snapshots using DevTools
- Restoring the Snapshot
Define the index, and populate data in the index
Ensure the Elasticsearch and Kibana is started. All the below commands are ran under DevTools of Kibana you can route using below URL:
localhost:5601/app/kibana#/dev_tools/console
Lets run the below command in to create a index:
PUT /elastic_snapshot_demo
Lets also add some data,
POST /_bulk
{ "index" : { "_index" : "elastic_snapshot_demo"}}
{ "message": "Some data added to the list", "user": { "id": "Smit" }}
{ "index" : { "_index" : "elastic_snapshot_demo"}}
{ "message": "Some data 2 added to the list", "user": { "id": "Steven" }}
{ "index" : { "_index" : "elastic_snapshot_demo"}}
{ "message": "Some data 3 added to the list", "user": { "id": "Lia" }}
Once the data is added, we can search the index to validate:
GET /elastic_snapshot_demo/_search
It will be return all the data that we just added.
Create Repository
For more detail information about the repository, you may read more here
Before we could run a repository creation command, we need to append the path.repo in the elasticsearch.yml
path:
repo:
- /Users/Smit/Documents/Dev/ELK/elasticsearch-7.8.0/mount/backups
Here the path needs to be your local machine. For me it is under Documents > Dev > ELK > elasticsearch-7.8.0 I have created a folder mount and backups under mount
Before you restart the elasticsearch, ensure the folder mount and backups both has correct permissions for elasticsearch to write to the folder.
For Unix, you can use the below command to provide required permissions
chmod o+rx /Users/Smit/Documents/Dev/ELK/elasticsearch-7.8.0/mount
Once this setup, lets start the elasticsearch and run the following command in the devTools in Kibana to register the Snapshot registry:
PUT /_snapshot/elastic_snapshot_demo_fs_backup
{
"type": "fs",
"settings": {
"location": "/Users/Smit/Documents/Dev/ELK/elasticsearch-7.8.0/mount/backups/elastic_snapshot_demo_fs_backup",
"compress": true
}
}
Here the name elastic_snapshot_demo_fs_backup can be customised, based on the requirements.
After running, the response should look similar to below:
{
"acknowledged" : true
}
In order to also check if it was successful, you can go below URL to check if the Repository is registered successfully:
localhost:5601/app/kibana#/management/elast..
Define Snapshot Policy
Lets add the policy, for that lets navigate to
localhost:5601/app/kibana#/management/elast..
Lets add the following values:
Policy name: daily-elastic_snapshot_demo-snapshot
Snapshot name: <daily-snap-{now/d}>
Repository: elastic_snapshot_demo_fs_backup
Schedule: 0 0 0 ?
For the Policy Name, it needs to be unique, and best to have it in-line with the index that you are planning on backing up.
For Snapshot Name, please note of the angular brackets < >, those are required if you are planning on adding expression
For Repository, select the repository we created in the previous step.
For the schedule, Please note it is UTC time. So In the above express, it states it will run this policy every 12th hour in the morning i.e. 00:00
Once all the values are created, lets go Next
For the Indices section, un-toggle and unselect all the indices (if you have any)
Once done, select just the index you wish to snapshot:
Once done, rest leave it as default and move on to the next stage.
For the Snapshot retention, leave it as default as well, move on to the review stage:
Once all values are correctly verified, lets click create policy, once completed it should be show similar to the below screenshot:
Run the snapshot
As stated in the previous step, the policy will run every 24hr, but to test the policy you may run the policy ad-hoc as well.
In order to run ad-hoc, you can click on the play icon, as shown in the below:
It will take sometime for the snapshot to be created, especially if it is running for the first time.
After a while you can view your backups folder, you will see the snapshot folder is created:
That's all, you have finally created your very first snapshot of an index.
Query Snapshots using DevTools
Run the below command, to see all the snapshots you have
GET /_snapshot/_all
Restoring the Snapshot
For restore, we need to delete the index first to simulate the restore
DELETE /elastic_snapshot_demo
Once deleted, you can run the search command again, and it should return 404
GET /elastic_snapshot_demo/_search
Restoring can also be done using two methods, first is using the devTools, and another is using the Kibana Restore view
Using the Kibana DevTools
Run the following command
POST /_snapshot/elastic_snapshot_demo_fs_backup/daily-snap-2021.04.17-bkpbzxtjrfoywdu_bmqkkg/_restore
In the command there are few things you may have update
first is elastic_snapshot_demo_fs_backup is the name of the repository we created earlier,
secondly daily-snap-2021.04.17-bkpbzxtjrfoywdu_bmqkkg is the name created when the snapshot was created.
If this is hard to replicate, you can simply use the second method.
Using the Kibana Snapshot and Restore section
Select the restore icon for the snapshot that you would like to restore:
Once selected, it will prompt for few actions, simply select next, next and Restore Snapshot
Thats all needed to restore.
You can now run the command to query the same index, and see if the data is correctly restored:
GET /elastic_snapshot_demo/_search
The result of the above command to return three set of data under hits.
Thank you for reading, if you have any suggestions or improvements feel free to reach out to me.
If you liked, please share and like the article.
Part 2: shahsmit.hashnode.dev/elasticsearch-snapsho..
Part 3: shahsmit.hashnode.dev/migrating-from-older-..
PS: Next series of articles will include, Snapshotting at S3, migrating to a newer version of elasticsearch etc.