Issue 4900: Read an unflushed ledger LAC from the memtable

[kalayciburak]

Descriptions of the changes in this PR:

Fix #4900

Motivation

SortedLedgerStorage.getLastAddConfirmed delegated entirely to the index. Until the memtable is flushed, that LAC lives only in a FileInfo cache entry. Once the cache evicts the ledger, the index file still does not exist, so a later read fails with NoLedgerException / NoEntryException even though the entry is still in the memtable. The bookie turns that into ENOLEDGER.

Changes

On those misses, read the LAC from the last memtable entry (the same offset addEntry stores). If the entry was flushed between the miss and the lookup, ask the index again. A ledger that is in neither place still fails.

Tests

mvn -pl bookkeeper-server -am test -Dtest=SortedLedgerStorageTest -Dsurefire.failIfNoSpecifiedTests=false

SortedLedgerStorageTest 6 tests, 0 failures. The new case expects LAC 3 and got NoEntryException: Entry 0 not found in 0 before the change.

mvn -pl bookkeeper-server checkstyle:check — 0 violations.

[djsweet]

I have an alternative PR, #4902, that we've been using in production as a workaround for this bug.

It's worth pointing out that both of these PRs require a fix for #4895 to function correctly when ledger metadata is in use.

[djsweet]

Thinking a bit more on this, I'm not sure this is a fully correct fix. Consider the scenario posed by #4900, but where the first ledger already exists, and is committed into InterleavedLedgerStorage.

If you write to over 20,000 ledgers with small payloads below the skipListSizeLimit, then the call to interleavedLedgerStorage.getLastAddConfirmed will not be able to access the LAC from the Ledger Cache, but instead will read from the committed value in InterleavedLedgerStorage, which will be behind the LAC in the memTable. This means the LAC gets reset backwards, spuriously.

#4902 always reads from the entryMemTable before interleavedLedgerStorage, so it shouldn't be susceptible to the same problem.