[Feature] Server need backoff fetch when fetch empty record to avoid send fetch records again and again

[luoyuxia]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

When client fetch a empty table, the response will be returned quickly, and immediately, it'll send fetch request again and again..
In my case, a table with 512 buckets, 9 tablet servers, and 512 paralism to read the table.
It will bring 258MB/s in bytes and 181 MB/s out bytes for network, and ~28w fetch request for each tablet server..

Solution

We may need introduce a backoff mechanims to fetch log when fetch empty record...
A simple test with sleep 1s when fetch empty record, then about 500 KB/s in bytes and 400 KB/s for network, ~50 fetch requests for each tablet server..

Anything else?

No response

Willingness to contribute

• I'm willing to submit a PR!

[luoyuxia]

I think we can introduce fetch.min.bytes as well as fetch.max.wait.ms like kafka to avoid too many empty response...

[wuchong]

+1, we can introduce log.fetch.min-bytes, log.fetch.max-timeout in server size.