cool hit counter Playing around with Redis persistence_Intefrankly

Playing around with Redis persistence

Redis provides two ways for persistence.

  • RDB: Can take snapshots of your data for storage at specified intervals.
  • AOF: records each write operation to the server, and re-executes these commands when the server restarts to restore the original data.

In this article, we will present the following content in the hope of providing a more comprehensive and clear understanding of these two persistence methods, as well as understanding this way of thinking about saving data to apply to your own system design.

  • Persistent configuration
  • How RDB works with AOF persistence
  • How to recover data from persistence
  • Recommendations on performance and practice

Persistent configuration

In order to use the persistence feature, we need to know how to turn on persistence first.

RDB persistence configuration

 # Time strategy
save 900 1
save 300 10
save 60 10000

 # File name
dbfilename dump.rdb

 # File save path
dir /home/work/app/redis/data/

 # If there is a persistence error, does the master process stop writing
stop-writes-on-bgsave-error yes

 # Whether to compress
rdbcompression yes

 # Whether or not to check when importing
rdbchecksum yes

The configuration is actually very simple, here's what exactly the persistent time policy means.

  • save 900 1 This means that if there is 1 write command in 900s, it will trigger a snapshot, which can be interpreted as a backup.
  • save 300 10 means that 10 writes in 300s produce a snapshot

The following is similar, so why do you need to configure so many rules? Since Redis is certainly not balanced in terms of read and write requests per time slot, to balance performance with data security, we are free to customize when a backup is triggered. So here's where it makes sense to configure it based on your own Redis writes.

stop-writes-on-bgsave-error yes This configuration is also a very important one, this is to stop the main process from accepting new write operations when the backup process fails, it is to protect persistent data consistency issues. If your business has a well-developed monitoring system, you can disable this configuration. Otherwise, please turn it on.

About the configuration of the compression rdbcompression yes It is not necessary to turn it on, after all, Redis itself is a CPU-intensive server, and turning on compression again will bring more CPU consumption, which is worth more than the cost of the hard disk.

Of course if you want to disable the RDB configuration, it is very easy to do so, just write on the last line of the savesave ""

Configuration of AOF

 # Whether to turn on aof
appendonly yes

 # File name
appendfilename "appendonly.aof"

 # Synchronization methods
appendfsync everysec

 # whether aof is synchronized during rewrites
no-appendfsync-on-rewrite no

 # Rewrite the trigger configuration
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

 # What to do if there is an error when loading aof
aof-load-truncated yes

 # File rewrite policy
aof-rewrite-incremental-fsync yes

Still focusing on explaining some key configurations.

appendfsync everysec It actually has three modes :

  • always: sync every write command to aof immediately, very slow, but safe
  • everysec: one sync per second, a compromise
  • no: redis doesn't handle it to the OS, very fast but also the least secure

It is generally used everysec configuration, which allows for a balance of speed and security, with a maximum loss of 1s of data.

aof-load-truncated yes If this configuration is enabled, a log will be written to the client when the aof tail is found to be incorrect on load, but will continue to execute if set to no , the error is found and it stops and must be fixed before it can be reloaded.

Working Principle

For the principles section, we will mainly look at how RDB and AOF accomplish persistence and how their process works.

Before describing the principle, let's talk about Redis's internal timing mechanism. The frequency of timing task execution can be specified in the configuration file via hz 10 to set (this configuration indicates 10 executions in 1s, which means that the timed task is triggered every 100ms). This value can be set to a maximum of.500 However, it is not recommended to exceed.100 The higher the value, the more frequent and higher the execution frequency, which brings more CPU consumption and thus affects the read/write performance of the main process.

The timed tasks use Redis' own implementation of TimeEvent It will regularly call commands to complete timed tasks that may block the main process causing Redis performance to degrade. So when we configure Redis, we must consider some of the configurations that will trigger timed tasks as a whole and adapt them to the actual situation.

Principle of RDB

There are two types of triggers for RDB persistence in Redis: your own manual trigger and Redis timed trigger.

For the RDB approach to persistence, the manual trigger can be used.

  • save: will block the current Redis server until persistence is complete, and should be disabled on line.
  • bgsave: This trigger method forks a child process, which is responsible for the persistence process, so blocking will only occur when forking the child process.

but (not) auto-trigger The main scenarios are the following:

  • According to our save m n Automatic triggering of configuration rules.
  • When a slave node is fully replicated, the master node sends an rdb file to the slave node to complete the replication operation, and the master node triggers bgsave
  • execute debug reload Hours.
  • execute shutdownWhen aof is not turned on, it will also trigger.

thanks to save It's basically not going to be used, so let's focus on bgsave This command is how RDB persistence is accomplished.

The point to note here is that fork operations can block, causing Redis read and write performance to degrade. We can control the maximum memory of a single Redis instance to minimize the event consumption of Redis at fork time. As well as the frequency of auto-triggering mentioned above to reduce the number of forks, or use manual triggering to accomplish persistence based on your own mechanism.

Principle of AOF

The whole process of AOF can be broadly seen in two steps, one being the real-time writing of commands (in the case of appendfsync everysec configuration, there will be a 1s loss), and the second step is a rewrite of the aof file.

For the incremental append to file step the main flow is: command write = " append to aof_buf = " sync to aof disk. So why write buf first before syncing to disk here? Writing to disk in real time would introduce very high disk IO and affect overall performance.

The aof rewrite is to reduce the size of the aof file and can be triggered manually or automatically, see the configuration section above for rules on automatic triggering. The fork operation also occurs at this step of the rewrite, and it is also here that blocking occurs to the main process.

Manual trigger. bgrewriteaof auto-trigger It's just triggered based on the configuration rules, and of course the overall timing of the automatic triggering also has something to do with the frequency of Redis' timed tasks.

Here's a flowchart of one of the rewrites.

There are four key points to add to the above chart.

  1. During the rewrite, since the main process is still responding to commands, in order to ensure the integrity of the final backup; it still writes to the old AOF file, and is able to ensure that no data is lost if the rewrite fails.
  2. In order to write the writes responded to during the rewrite to the new file as well, a buf is therefore also reserved for the child process to prevent the newly written file from losing data.
  3. Rewriting is generating the corresponding commands directly from the current memory data and does not require reading the old AOF file for analysis and command merging.
  4. The text protocol used directly by the AOF file is mainly compatible, easy to append, and highly readable can be considered to modify the repair.

It can't be RDB or AOF both write to a temporary file first and then pass rename Complete the replacement of the document.

Recovering data from persistence

Once the data is backed up and persisted, how do we restore the data from these persistent files? If a server has both an RDB file and an AOF file, who should be loaded?

In fact, to recover data from these files, all you need to do is restart Redis. Let's go through the process still with the diagram.

The startup will first check if the AOF file exists and try to load the RDB if it doesn't. So why would AOF be loaded first? Because AOF preserves more complete data, we know from the above analysis that AOF basically loses at most 1s of data.

Performance and Practice

From the above analysis, we all know that RDB snapshots, AOF rewrites all require fork, which is a heavyweight operation that can cause blocking on Redis. So in order not to affect the Redis main process response, we need to keep blocking as low as possible.

  1. Reducing the frequency of forks, e.g. can be done manually to trigger RDB to generate snapshots, with AOF rewrites.
  2. Controlling the maximum memory used by Redis to prevent forks from taking too long.
  3. Use of more awesome hardware.
  4. Configure Linux's memory allocation policy appropriately to avoid fork failure due to lack of physical memory.

What exactly do we do online? I offer some of my own practical experience.

  1. If the data in Redis is not particularly sensitive or can be rewritten in other ways to generate data, persistence can be turned off and the data can be replaced by other means if it is lost.
  2. Develop their own policies to periodically check on Redis, which can then manually trigger backups, rewrites of data.
  3. Single machines, if multiple instances are deployed, to prevent multiple machines from running persistence and rewrite operations at the same time, preventing competition for memory, CPU and IO resources and making persistence serial.
  4. The possibility of adding master and slave machines, using one slave machine for backup processing and the other machines to respond normally to client commands.
  5. RDB persistence and AOF persistence can co-exist and be used in conjunction.

This article is mostly about some notes on ops, but we developers learn about them in a way that helps us find weird bugs at some point. Knowledge of master-slave replication and clustering for Redis will be covered next.

1、The Zone moves the world lights up the future
2、Sky mirror chess coco chess jianghanyou chess it turns out that there are hangers that control good cards
3、Chengdu hot pot big data the longest queue of more than 2000 number 400 am there are people eating hot pot
4、DBeaver Community Edition 511 released database management tool
5、East Lake Big Data and Chutian Cloud sign agreement to build a new ecology of Hubeis big data industry

    已推荐到看一看 和朋友分享想法
    最多200字,当前共 发送