[BUG] Background saving not working with 6.2.1

[Talkabout]

Describe the bug

Background saving seems to not work after update from 6.0.18 to 6.2.1

To reproduce

Not sure about that. I updated to new version and restarted 2 servers (active-active replica). After initial sync there has been no background saving at all (since 3 days) Before that this task was executed every 5 minutes...

Expected behavior

Background saving should work :)

Additional information

Using 2 raspberry pis with active-active replication.

[Talkabout]

As an additional information, these are the settings in conf file I am using for background saving:

[VivekSainiEQ]

Hi @Talkabout,

I unfortunately cannot seem to replicate this on my end, what do your config files look like?

[Talkabout]

Hi @VivekSainiEQ,

these are my settings:

bind 192.168.XX.XX protected-mode no port 6379 timeout 0 tcp-keepalive 0 daemonize no supervised systemd pidfile /var/run/redis/redis-server.pid loglevel notice syslog-enabled yes syslog-ident keydb databases 4 always-show-logo no save 900 1 save 300 100 save 60 10000 stop-writes-on-bgsave-error no rdbcompression yes rdbchecksum no dbfilename dump.rdb dir /var/lib/redis repl-diskless-sync no replica-priority 100 maxmemory 512M maxmemory-policy allkeys-lru lazyfree-lazy-eviction no lazyfree-lazy-expire no lazyfree-lazy-server-del no replica-lazy-flush no appendonly no appendfilename "appendonly.aof" appendfsync everysec no-appendfsync-on-rewrite no auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb aof-load-truncated yes aof-use-rdb-preamble yes lua-time-limit 5000 slowlog-log-slower-than 10000 slowlog-max-len 16 latency-monitor-threshold 0 notify-keyspace-events "" hash-max-ziplist-entries 512 hash-max-ziplist-value 64 list-max-ziplist-size -2 list-compress-depth 0 set-max-intset-entries 512 zset-max-ziplist-entries 128 zset-max-ziplist-value 64 hll-sparse-max-bytes 3000 stream-node-max-bytes 4096 stream-node-max-entries 100 activerehashing yes client-output-buffer-limit normal 0 0 0 client-output-buffer-limit replica 256mb 64mb 60 client-output-buffer-limit pubsub 32mb 8mb 60 hz 10 dynamic-hz yes aof-rewrite-incremental-fsync yes rdb-save-incremental-fsync yes active-replica yes replicaof 192.168.XX.XX 6379 server-threads 2

As you can see I have migrated from redis and using the same config file with specific keydb options added.

thanks for your help!

Bye

[sts]

@VivekSainiEQ

We've got the very same issue with KeyDB. I reported it to the community forum two weeks ago: https://community.keydb.dev/t/keydb-rdb-bgsave-never-competes/189

Is there anything else we can provide to debug the issue?

[Talkabout]

Thanks @sts, I killed the running processes and immediately the background saving took place. Concurrent background savings are also working as of now, but I am assuming that at some point it will stop again (the background process will hang). Would be nice to have somebody looking into that.

Thanks!

[Talkabout]

And now one of the keydb servers has a hanging background saving process again:

the other one is still executing the background saving...

[Talkabout]

And here we have the second hanging process:

There is surely an issue somewhere with the new version of KeyDB. Any other information needed for you guys to analyze it?

[VivekSainiEQ]

Hi @Talkabout and @sts,

Do BGSAVEs always hang or can they run multiple times before hanging? And if so how many times does it take before hanging? I suspect there is a bug in how having systemd supervision enabled interacts with the BGSAVE mechanism.

[Talkabout]

Hi @VivekSainiEQ,

the first server executed 7 background savings before it got stuck, the second one executed 18 of them. I have now disabled systemd supervision in my config file and restarted both keydb instances. Let's see if it helps.
Thanks for taking a look at the issue!

[Talkabout]

At the moment background saving runs without problems:

Will keep you guys posted about status.

[Talkabout]

Unfortunately:

even after removing the supervisor systemd from the config. Any other idea?

[Talkabout]

Another update, both servers are stuck now:

[esatterwhite]

@Talkabout For sake of clarity and narrowing the problem, Does this happen on 6.2.0 ?

[Talkabout]

Hi @esatterwhite,

I have not used 6.2.0 because it was causing major memory issues on my system, I switched directly from 6.0.16 to 6.2.1...

[MalavanEQAlpha]

Hi @Talkabout @sts @esatterwhite it turns out this issue has always been present but just made incredibly more likely with 6.2.1(so 6.2.0 is as safe as 6.0.18).

The localtime_r function internally requires a lock, so any multithreaded program that forks when another thread is in the middle of a call to localtime_r will hang when the forked process calls localtime_r.

Prior to 6.2.1 that was extremely unlikely to happen, but in 6.2.1 we added a thread dedicated to checking the time. This makes repeated calls to localtime_r, massively increasing the chance that one is in flight when we fork for a background save.

The only call to localtime_r in the background save is within the syslog command, so disabling syslog(by setting the config: syslog-enabled no) should solve your issue for now while we work on a more complete fix.