This article Zhang, I want to talk to you Redis Best practices .

Your project may already use Redis It’s been a long time , But in use , You may also encounter the following problems more or less ：

• my Redis Why is memory growing so fast ？
  
• Why my Redis The operation delay is getting bigger ？
  
• How to reduce the Redis Frequency of failure ？
  
• Daily Operation Redis What to pay attention to ？
  
• Deploy Redis when , How to do a good job in resource planning ？
  
• Redis What are the key indicators to monitor ？
  

Especially when your projects become more and more dependent on Redis when , These issues become particularly important .

here , You urgently need a 「 Best practice guide 」.

This article , I will start from the following seven dimensions , Take you 「 comprehensive 」 analysis Redis Best practice optimization for ：

• Memory
  
• performance
  
• Highly reliable
  
• Daily Operation
  
• Resource Planning
  
• monitor
  
• Security
  

At the end of the article , I’ll also give you a complete list of best practices , Whether you’re a business developer , still DBA Operations staff , This list will help you to be more 「 grace 」 Make good use of land Redis.

This article is full of dry goods , I hope you can read it patiently .

How to use Redis More memory saving ？

First , Let’s take a look Redis Memory optimization .

as everyone knows ,Redis The reason for the high performance of , The reason is that its data is stored in 「 Memory 」 in , So the visit Redis The data in is extremely fast .

But in terms of resource utilization , The memory resource of the machine is compared with that of the disk , It’s more expensive .

When your business application is Redis When there is little data stored in , You may not care much about the use of memory resources . But as the business grows , Your business is stored in Redis There’s going to be more and more data in .

If the memory optimization strategy is not formulated in advance , So when the business starts to grow ,Redis The memory used will start to swell .

therefore , Formulate reasonable memory optimization strategy in advance , It is necessary to improve the utilization rate of resources .

That’s in use Redis when , How to save more memory ？ Here I summarize for you 6 Some suggestions , Let’s look at… In turn ：

1) control key The length of

The simplest and most direct memory optimization , It’s control key The length of .

In business development , You need to anticipate the whole Redis writes key The number of , If key It’s a million , that , Too long key Names also take up too much memory .

therefore , You need to guarantee key In a simple 、 On the premise of clarity , As much as possible key Short definition .

for example , The original key by user:book:123, Can be optimized to u:bk:123.

thus , Yours Redis You can save a lot of memory , This scheme is very direct and efficient for memory optimization .

2) Avoid storage bigkey

Besides controlling key Beyond the length of , You also need to focus on value Size , If there’s a lot of storage bigkey, It can also lead to Redis Memory is growing too fast .

besides , The client is reading and writing bigkey when , There are also performance problems （ I’ll go into details below ）.

therefore , You have to avoid being in Redis Storage in bigkey, My advice to you is ：

• String： The size is controlled in 10KB following
  
• List/Hash/Set/ZSet： The number of elements is controlled at 1 All of the following
  

3) Choose the right data type

Redis Provides a wealth of data types , These data types are implemented , Also optimized the use of memory . To be specific , One data type corresponds to multiple data structures ：

for example ,String、Set In the storage int Data time , Will use integer encoding storage .Hash、ZSet When the number of elements is small （ Configurable ）, Will use compressed lists （ziplist） Storage , When storing more data , Will be converted to hash table and jump table .

The reason why the author designed this way , Is to further save memory resources .

So when you store data , You can use these features to optimize Redis Of memory . Here’s my advice for you ：

• String、Set： Store as much as possible int Type data
  
• Hash、ZSet： The number of stored elements is controlled below the conversion threshold , Store… In a compressed list , To save memory
  

4) hold Redis Use as a cache

Redis Data stored in memory , It also means that its resources are limited . You’re using Redis when , Use it as a cache , Not a database .

therefore , Your app writes to  Redis Data in , Set as many as possible 「 Expiration time 」.

In the business Redis When no data is found in , And then load it from the back-end database to Redis in .

Take this approach , It can make Redis Only frequently visited 「 Thermal data 」, Memory utilization will also be relatively high .

5) Instance settings maxmemory + Elimination strategy

Although your Redis key Expiration time is set , But if you write a lot of business applications , And the expiration time is set for a long time , In such a short period of time Redis Memory will still grow rapidly .

If you don’t control Redis The maximum memory limit of , It can also lead to excessive use of memory resources .

For this scenario , You need to estimate the amount of business data ahead of time , Then set… For this instance maxmemory Control the memory limit of the instance , This can be avoided Redis Memory continues to swell .

Configured with maxmemory, At this point, you also need to set the data elimination strategy , And how to choose the elimination strategy , You need to combine your business characteristics to decide ：

• volatile-lru / allkeys-lru： Give priority to the most recently accessed data
  
• volatile-lfu / allkeys-lfu： Give priority to the most frequently accessed data （4.0+ Versioning support ）
  
• volatile-ttl ： Give priority to data that is about to expire
  
• volatile-random / allkeys-random： Random elimination data
  

6) Write data after compression Redis

The above plan basically covers Redis All aspects of memory optimization .

If you want to further optimize Redis Memory , You can also compress data in business applications first , Write it back to Redis in （ For example snappy、gzip And so on ）.

Of course , Compressing stored data , The client also needs to decompress when reading , More will be consumed in the meantime CPU resources , You need to weigh it against the actual situation .

That’s all 「 Save memory resources 」 Practice optimization in the aspect of , Are they all relatively simple ？

So let’s see 「 performance 」 Optimization of aspects .

How to keep playing Redis A high performance ？

When your system decides to introduce Redis when , If you want to see it, the key point is ： performance .

We know , A stand-alone version Redis You can achieve 10W QPS, Such high performance , It also means that if there is a delay in use , It’s not what we expected .

therefore , In the use of Redis when , How to continue to play its high performance , Avoid operation delay , It’s also our focus .

In this regard , I’ve summed it up for you 13 Piece of advice ：

1) Avoid storage bigkey

Storage bigkey In addition to using too much memory as mentioned earlier , Yes Redis Performance can also have a big impact .

because Redis Processing requests is single threaded , When your app is writing a bigkey when , More time will be spent on 「 Memory allocation 」 On , At this point, the operation delay will increase . similarly , Delete one bigkey stay 「 Free memory 」 when , It can also happen .

and , When you’re reading this bigkey when , Will also be in 「 Network data transmission 」 Spend more time on , At this point, the subsequent requests to be executed will be queued ,Redis Performance degradation .

therefore , Try not to store your business applications bigkey, Avoid operation delay .

If you do have storage bigkey The needs of , You can take bigkey Split into small key Storage .

2) Turn on lazy-free Mechanism

If you can’t avoid storage bigkey, So I suggest you turn on Redis Of lazy-free Mechanism .（4.0+ Versioning support ）

When this mechanism is turned on ,Redis In deleting a bigkey when , Time consuming operations to free memory , Will be put into the background thread to execute , This can, to the greatest extent , Avoid the impact on the main thread .

3) Don’t use complex commands

Redis It’s a single threaded model that handles requests , Besides operation bigkey It will cause the subsequent requests to be out of the queue , When executing an overly complex command , And that’s what happens .

Because it’s too complex to execute commands , It will consume more CPU resources , Other requests in the main thread can only wait , Queuing delays also occur .

therefore , You need to avoid doing things like SORT、SINTER、SINTERSTORE、ZUNIONSTORE、ZINTERSTORE And so on .

For this aggregation class operation , I suggest you put it on the client side to execute , Don’t let Redis Too much computing .

4) perform O(N) On command , Focus on N Size

Avoid using overly complex commands , You can rest easy ？

The answer is No .

When you’re doing O(N) On command , Also pay attention to N Size .

If you query too much data at once , It will also take too long in the process of network transmission , The operation delay becomes larger .

therefore , For container types （List/Hash/Set/ZSet）, When the number of elements is unknown , Don’t be mindless LRANGE key 0 -1 / HGETALL / SMEMBERS / ZRANGE key 0 -1.

When querying data , You should follow the following principles ：

1. First query the number of data elements （LLEN/HLEN/SCARD/ZCARD）
   
2. There are fewer elements , It can query all data at one time
   
3. There are a lot of elements , Batch query data （LRANGE/HASCAN/SSCAN/ZSCAN）
   

5) Focus on DEL Time complexity

You read that right , In deleting a key when , If the posture is not right , It could also affect Redis performance .

Delete one key, We usually use DEL command , Think about it , You feel DEL What is the time complexity of ？

O(1) ？ Not necessarily .

When you delete a String type key when , Time complexity is really O(1).

But when you want to delete key yes List/Hash/Set/ZSet type , Its complexity is actually O(N),N Represents the number of elements .

in other words , Delete one key, The more elements there are , perform DEL The slower it gets ！

The reason lies in , When deleting a large number of elements , You need to reclaim the memory of each element in turn , The more elements , The longer it takes ！

and , This process is executed in the main thread by default , This is bound to block the main thread , There are performance issues .

Then delete this element more key, How to deal with it ？

My advice to you is , Delete in batches ：

• List type ： Execute many times LPOP/RPOP, Until all the elements are removed
  
• Hash/Set/ZSet type ： Execute first HSCAN/SSCAN/SCAN Query element , Re execution HDEL/SREM/ZREM Delete each element in turn
  

I didn’t think! ？ A small delete operation , A little careless , It can also cause performance problems , You need to pay special attention when you operate .

6) Batch command instead of single command

When you need to operate more than one at a time key when , You should use batch commands to handle .

The advantage of batch operations over multiple single operations is , Can significantly reduce the number of clients 、 Back and forth network of the server IO frequency .

So my advice to you is ：

• String / Hash Use MGET/MSET replace GET/SET,HMGET/HMSET replace HGET/HSET
  
• Other data types use Pipeline, Package and send multiple commands to the server for execution at one time
  

7) Avoid centralized expiration key

Redis Cleanup expired key It’s timing + Lazy way to do it , And this process is executed in the main thread .

If you have a lot of business key Focus on overdue situations , that Redis After cleaning up expired key when , There is also a risk of blocking the main thread .

To avoid this , You can set the expiration time , Add a random time , Put these key The expiration date of , So as to reduce the impact of centralized expiration on the main thread .

8) Use long connection operation Redis, Reasonable configuration of connection pool

Your business should use long connection operations Redis, Avoid short connections .

When using short connection operation Redis when , Every time you go through TCP Three handshakes 、 Four waves , This process also increases the operation time .

meanwhile , Your clients should use connection pooling to access Redis, And set reasonable parameters , Do not operate for a long time Redis when , Need to release connection resources in time .

9) Use only db0

Even though Redis Provides 16 individual db, But I only recommend that you use db0.

Why? ？ I sum up the following 3 Point cause ：

1. Operate multiple… On a single connection db Data time , You need to do it first every time SELECT, This will give Redis Bring extra pressure
   
2. The use of multiple db Is the purpose of , Store data by line of business , So why not split multiple instance storage ？ Split multiple instance deployment , Multiple lines of business don’t affect each other , Can also improve Redis Access performance for
   
3. Redis Cluster Only support db0, If you want to move to Redis Cluster, Migration costs are high
   

10) Using read write separation + Fragmentation cluster

If you have a large number of business read requests , Then you can deploy multiple slave Libraries , Read and write separation , Give Way Redis Share the reading pressure from the library , To improve performance .

If your business has a large number of write requests , Single Redis The instance can’t support such a large amount of write traffic , So at this point you need to use fragmentation clustering , Share the pressure of writing .

11) Don’t open AOF or AOF Configured to flash disk per second

If the business is not sensitive to the loss of data , I suggest you don’t turn on AOF, avoid AOF Write disk slow down Redis Performance of .

If it really needs to be turned on AOF, So I suggest you configure it as appendfsync everysec, Persistent data scrubbing operation , Put it in the background thread to execute , Try to reduce Redis The impact of writing to disk on performance .

12) Using physical machine deployment Redis

Redis When doing data persistence , In the way of creating a child process .

Creating a child process calls the operating system’s fork system call , This system call takes time to execute , It’s about the system environment .

Virtual machine environment execution fork Time consuming , It’s much slower than a physical machine , So your Redis It should be deployed on physical machines as much as possible .

13) Turn off the large page mechanism of the operating system

Linux The operating system provides a large page mechanism , It is characterized by , Every time an application requests memory from the operating system , The applicant is from the previous 4KB Change into 2MB.

What’s the problem with this ？

When Redis When doing data persistence , Will first fork A subprocess , At this point, the main process and the child process share the same memory address space .

When the main process needs to modify the existing data , Copy on write （Copy On Write） The way to operate , In the process , Need to re apply for memory .

If the application memory unit becomes 2MB, It will increase the time of memory application , If the main process has a large number of write operations at this time , Need to modify the original data , So in the meantime , The delay of operation will be greater .

therefore , To avoid this kind of problem , You need to turn off the large page mechanism on the operating system .

Okay , These are Redis 「 High performance 」 Practice optimization in the aspect of . If you care very much Redis Performance problems of , We can combine these aspects to optimize .

Let’s see Redis 「 reliability 」 How to ensure .

How to ensure Redis The reliability of the ？

Here I want to remind you that , Guarantee Redis Reliability is not difficult , But the hard part is how to do it 「 Sustained stability 」.

I’ll start with 「 Resource isolation 」、「 Multiple copies 」、「 Fault recovery 」 These three dimensions , Take you to analyze the security Redis Best practices for reliability .

1) Deploy instances by line of business

The first step to improving reliability , Namely 「 Resource isolation 」.

You’d better deploy according to different lines of business Redis example , So when one of the instances fails , It won’t affect other businesses .

This kind of resource isolation scheme , The cost of implementation is the lowest , But the results are very great .

2) Deploy the master-slave cluster

If you only use the stand-alone version Redis, Then there is a risk of machine downtime and service unavailability .

therefore , You need to deploy 「 Multiple copies 」 example , That is, the master-slave cluster , So when the main library goes down , There are still slave libraries available , Avoid the risk of data loss , It also reduces the time when services are unavailable .

When deploying a master-slave cluster , You also need to pay attention , The master and slave libraries need to be distributed on different machines , Avoid cross deployment .

The reason for this is , Usually ,Redis The main library will bear all the read and write traffic , So we must give priority to the stability of the main library , Even if the slave machine is abnormal , And don’t affect the main library .

and , Sometimes we need to be right about Redis Do routine maintenance , For example, regular data backup and other operations , Then you can just do it from the library , This will only consume the resources of the slave machine , It also avoids the impact on the main library .

3) Reasonable configuration of master-slave replication parameters

When deploying a master-slave cluster , If the parameter configuration is unreasonable , It may also cause problems with master-slave replication ：

• Master slave replication interrupt
  
• Initiate full replication from the library , The performance of the main library is affected
  

In this regard, I give you the following suggestions 2 spot ：

1. Reasonable setting repl-backlog Parameters ： Too small repl-backlog In the case of large write traffic , The interruption of master-slave replication will lead to the risk of full replication of data
   
2. Reasonable setting slave client-output-buffer-limit： When there is a problem with copying from the library , Too small buffer Can cause overflow from the library buffer , This leads to replication interruption
   

4) Deploy sentry groups , Automatic fault switch is realized

Only master and slave nodes are deployed , But when a fault occurs, it can’t be switched automatically , therefore , You also need to deploy sentinel clusters , Realize the failure of 「 Automatic switch 」.

and , Multiple sentinel nodes need to be distributed on different machines , The number of instances is odd , To prevent the sentinel election from failing , Affect the switching time .

These are the guarantees Redis「 Highly reliable 」 Practice optimization , You should have found out, too , These are the optimization of deployment and operation and maintenance layer .

besides , You may also be interested in Redis Do some 「 Daily Operation 」 Work , What problems should you pay attention to ？

Daily Operation Redis What to pay attention to ？

If you are DBA Operations staff , In normal operation and maintenance Redis when , Also need to pay attention to the following 6 In terms of .

1) No use KEYS/FLUSHALL/FLUSHDB command

Execute these orders , For a long time Redis The main thread , Great harm , So you have to ban it .

If you really want to use these commands , My advice to you is ：

• SCAN Replace KEYS
  
• 4.0+ Version available FLUSHALL/FLUSHDB ASYNC, The operation of clearing data is executed in the background thread
  

2) When scanning online instances , Set sleep time

Whether you use SCAN Scan online examples , Or do it with examples bigkey Statistical analysis , I suggest you remember to set the sleep time when scanning .

Prevent during scanning , example OPS Too high, right Redis Produce performance jitter .

3) Use with caution MONITOR command

Sometimes I’m checking Redis The question is , You will use it. MONITOR see Redis The order being executed .

But if your Redis OPS Relatively high , Then in execution MONITOR It can lead to Redis The output buffer memory continues to grow , It’s going to take a lot of Redis Memory resources of , It can even cause instance memory to exceed maxmemory, Trigger data culling , You need to pay special attention to this situation .

So you’re doing MONITOR On command , Be careful , Use less as far as possible .

4) The slave library must be set to slave-read-only

Your slave library must be set to slave-read-only state , Avoid writing data from the library , This leads to inconsistency between master and slave data .

besides , From the library, if it’s right or wrong read-only state , If you’re using 4.0 Following Redis, It exists like this Bug：

Data with expiration time is written from the library , No regular cleaning and memory release .

This can cause a memory leak from the library ！ This question goes on until 4.0 Version only fixed , You need to pay special attention when configuring slave libraries .

5) The reasonable configuration timeout and tcp-keepalive Parameters

If it’s because of the Internet , Cause a lot of your client connections to Redis Unexpected interruption , It just happened that your Redis Configured maxclients The parameters are small , At this time, the client may not be able to establish a new connection with the server （ The server thinks it’s more than maxclients）.

The reason for this problem is , Every time a connection is established between the client and the server ,Redis Will assign a client fd.

When there is a problem between the client and the server , The server does not release this immediately client fd.

When will it be released ？

Redis There is a scheduled task inside , All of them will be checked regularly client Whether the idle time of is more than the configured timeout value .

If Redis It’s not turned on tcp-keepalive Words , From the server to the configured timeout After time , To clean up and release this client fd.

Before cleaning up , If there are a lot of new connections coming in , It may lead to Redis It is held inside the server client fd More than the maxclients, The new connection is rejected .

In this case , My optimization advice to you is ：

1. Don’t over configure timeout： Let the server send the invalid as soon as possible client fd Clean up
   
2. Redis Turn on tcp-keepalive： In this way, the server will send it to the client regularly TCP Heartbeat bag , Check connectivity , When the network is abnormal , We can clean up the zombies as soon as possible client fd
   

6) adjustment maxmemory when , Pay attention to the adjustment order of master and slave libraries

Redis 5.0 There is a problem with the following version ： If the slave memory exceeds maxmemory, It also triggers data culling .

In some cases , The slave library may have priority over the master library maxmemory Of （ For example, executing from the library MONITOR command , The output buffer takes up a lot of memory ）, At this point, the data will be eliminated from the library , There will be inconsistencies between master and slave libraries .

To avoid this problem , Adjusting maxmemory when , Be sure to pay attention to the modification order of the master-slave library ：

• turn up maxmemory： Modify the slave library first , Then modify the main library
  
• The small maxmemory： Modify the main library first , Then modify the slave library
  

until Redis 5.0,Redis Just added a configuration replica-ignore-maxmemory, The default slave library exceeds maxmemory It doesn’t eliminate data , To solve this problem .

Okay , These are 「 Daily Operation 」Redis Need to pay attention to , You can make up for each configuration item , See what needs to be optimized .

Next , Let’s take a look , guarantee Redis「 Security 」 What should we pay attention to .

Redis How to ensure safety ？

in any case , In the age of the Internet , Security must be something we need to be on guard at any time .

You may have heard of Redis Injected with an executable script , And get the machine root The security of permissions , It’s all about deployment Redis when , Not paying attention to security risks .

In this respect , My advice to you is ：

1. Don’t put the Redis Deployed on a server accessible to the public network
   
2. Do not use the default port for deployment 6379
   
3. Start as a normal user Redis process , prohibit root User start
   
4. Limit Redis Directory access to configuration files
   
5. Password authentication is recommended
   
6. Ban / Rename dangerous command （KEYS/FLUSHALL/FLUSHDB/CONFIG/EVAL）
   

As long as you put this in place , Basically, it can guarantee Redis The security risk of the project is within the controllable range .

thus , We analyzed Redis In the memory 、 performance 、 reliability 、 Best practice optimization in daily operation and maintenance .

In addition to the above , You also need to be ahead of time 「 The prevention of 」.

How to prevent Redis problem ？

To prevent in advance Redis problem , You need to do two things well ：

1. Rational resource planning
   
2. Perfect monitoring and early warning
   

Let’s start with resource planning .

In the deployment Redis when , If you can plan your resources ahead of time , Many problems caused by insufficient resources can be avoided . My suggestions for you are as follows 3 spot ：

1. Make sure the machine has enough CPU、 Memory 、 bandwidth 、 Disk resources
   
2. Plan capacity ahead of time , The main library machine reserves half of the memory resources , Prevent network failure of master-slave machines , Large area full synchronization , A problem that causes the main library machine to run out of memory
   
3. Single instance memory is recommended to be controlled in 10G following , Large instances in master-slave full synchronization 、RDB There is a risk of blocking when backing up
   

Let’s see how surveillance works .

Monitoring and early warning is an important part of improving stability , Perfect monitoring and early warning , You can expose problems ahead of time , So that we can react quickly , Minimize the problem .

My advice on this is ：

1. Make the machine CPU、 Memory 、 bandwidth 、 Disk monitoring , Call the police in time when resources are insufficient , Any shortage of resources will affect Redis performance
   
2. Reasonable setting slowlog threshold , And monitor it ,slowlog Too many, call the police in time
   
3. Monitoring components collect Redis INFO Information , Use long connection , Avoid frequent short connections
   
4. Do a good job of instance runtime monitoring , Focus on expired_keys、evicted_keys、latest_fork_usec indicators , A sudden increase in these indicators may lead to the risk of congestion
   

summary

Okay , To sum up , I’ll take you through this article Redis Best practice optimization path , This includes memory resources 、 High performance 、 Highly reliable 、 Daily Operation 、 Resource Planning 、 monitor 、 Security 7 Dimensions .

Here I draw a mind map , For your reference in practice .

I also Optimize these practices , according to 「 Business development 」 and 「 Operation and maintenance 」 Two dimensions , Further divided .

And take 「 mandatory 」、「 recommend 」、「 Reference resources 」3 There are three levels marked , So when you practice optimization , It will be more clear what to do , What needs to be further analyzed in combination with actual business scenarios .

These grades Other implementation rules are as follows ：

• mandatory ： Strictly observe , Otherwise, it will do great harm
  
• recommend ： It is recommended that , Can improve performance 、 Reduce memory 、 Convenient for operation and maintenance
  
• Reference resources ： Refer to the implementation according to the business characteristics
  

If you’re a business developer , You need to understand Redis Operation mechanism of , For example, the execution time complexity of each command 、 Data expiration strategy 、 Data elimination strategy, etc , Use reasonable commands , And combined with business scenarios to optimize .

If you are DBA shipment Maintenance personnel , You need to be in resource planning 、 Operation and maintenance 、 monitor 、 The security level is in place , Prepare for the future .

Postscript

If you can read this patiently , We should be concerned about how to 「 The use of good 」Redis With a new understanding .

This article is mainly about Redis Best practices , about 「 Best practices 」 This topic , I want to talk to you more .

If you’re not facing Redis, It’s other middleware , for example MySQL、Kafka, When you use these components , Will there be any optimization ideas ？

You can also use these dimensions of this article to analyze ：

• performance
  
• reliability
  
• resources
  
• Operation and maintenance
  
• monitor
  
• Security
  

in addition , In terms of learning skills , We are in the process of software development , Think and explore as much as possible 「 Best practices 」 The way .

Because that’s the only way , We will constantly urge ourselves to think , Set higher demands on yourself , Make continuous progress .

Previous recommendation

Thoroughly understand the event driven model – Reactor

How to scientifically crack slow SQL?

Last , Welcome to pay attention Kaito And iron pillar , Growing up together .