[#1040] Leave the session to a total update when a configuration change asks for a restart

[vharseko]

A configuration change which restarts the session of a replication domain for what it carries restarts it while a total update into this replica is reading it. LDAPReplicationDomain.restartService() refuses the restart under ownsItsSession() - shutdown.get() || disabled - and readAssuredConfig()/readFractionalConfig() take allowReconnection from the same predicate; a total update into this replica never sets disabled: preBackendImport() takes the backend away without disabling the domain, because the import streams over the session, on the listener thread, and the domain can not stop the session it is importing over.

The issue describes the consequence as the replay road of #968 has it - the broker stops, receiveEntryBytes() returns null on shuttingDown() without recording an exception, and the import ends on the entries which had arrived, reported as finished. On the configuration road it is worse than that, and the case below shows it: the change holds the lock of the configuration (ConfigurationBackend.replaceEntry, synchronized (configLock)) for the length of applyConfigurationChange(), disableService() stops the broker and then joins the listener thread, and the listener thread - the import - ends its stream and goes to enable the backend back, closeBackendImport() → TaskUtils.enableBackend() → an internal modify of the backend entry → the same configLock. The change never returns; configLock and serviceStateLock stay held, so every configuration change of the server after it waits, disable(), the replay threads' restartSession() and shutdown() of the domain wait - stop-ds with them - the backend stays deregistered, and the import context stays acquired, so every later total update is refused as a simultaneous one.

configuration change during the import [WAITING]
    at java.lang.Thread.join(Thread.java:1990)
    at ReplicationDomain.disableService(ReplicationDomain.java:3358)
    at ReplicationDomain.restartService(ReplicationDomain.java:3497)
    at LDAPReplicationDomain.restartService(LDAPReplicationDomain.java:5581)
    at ReplicationDomain.changeConfig(ReplicationDomain.java:3477)
    at ExternalChangelogDomain.applyConfigurationChange(ExternalChangelogDomain.java:110)
    ...
    at ConfigurationBackend.replaceEntry(ConfigurationBackend.java:497)

Replica DS(1) listener for domain "dc=example,dc=com" [BLOCKED]
    at ConfigurationBackend.replaceEntry(ConfigurationBackend.java:497)
    ...
    at TaskUtils.enableBackend(TaskUtils.java:173)
    at LDAPReplicationDomain.closeBackendImport(LDAPReplicationDomain.java:5286)
    at LDAPReplicationDomain.importBackend(LDAPReplicationDomain.java:5214)
    at ReplicationDomain.initialize(ReplicationDomain.java:2590)

The roads in. The listener of the domain entry refuses a change while ieRunning(), which leaves an import which starts between "acceptable" and "apply" - inside the lock, so the listener thread is then parked on preBackendImport()'s disableBackend(), and the join is the same. The external changelog entry was not covered at all: ExternalChangelogDomain.isConfigurationChangeAcceptable() returned true unconditionally, and dsconfig set-external-changelog-domain-prop --set ecl-include:... on a domain being initialized went applyConfigurationChange() → domain.changeConfig(eclIncludes, ...) → restartService() with nothing asking whether an import was reading the session.

What this changes

• restartService() and allowReconnection take sessionHasAnOwner() - ownsItsSession() || importInProgress(), the predicate [#956] Tell a failed entryUUID search apart from an entry which is not there #968 gave the restart a failed replay asks for. The owner spans the import context, from acquireIEContext() to releaseIEContext(), and both TaskUtils.*Backend() calls of the import lie inside it, so no join under configLock is left. The restart is reported the way it is for a disabled domain - onSessionRestartSuppressed(), adminActionRequired and 327 on the domain listener - and the import starts the next session itself when it ends, from the state it loaded; the broker's stored configuration, the assured configuration and the attributes published to the external changelog are all read as that session comes up (connectAsDataServer(), StartSessionMsg).
• ExternalChangelogDomain.isConfigurationChangeAcceptable() refuses the change while domain.ieRunning(), with NOTE_ERR_CANNOT_CHANGE_CONFIG_DURING_TOTAL_UPDATE, as the listener of the domain entry does. ieRunning() rather than importInProgress() on purpose: it is the predicate the domain listener uses, an export is covered by both the same way, and importInProgress() is protected in ReplicationDomain - not visible from this class.
• 327 names the third case - a total update into this replica reading the session - and says that this one starts the session again itself when it ends; the wording for a domain left disabled is as it was.

Base

Rebased onto master after #968 merged (9300ffec88, the round-5 head 00798abf33 this branch sat on, squashed - the trees are identical). One commit, the patch unchanged (git range-diff reports it =); sessionHasAnOwner() and importInProgress() as an owner are #968's. #977 (eef0757515), the one commit of master on top of #968, touches LDAPReplicationDomain.java in the checkpointer and does not meet this change.

Tests

ConfigChangeDuringImportTest, a class of its own with the fixture of ReplayDuringImportTest: the userRoot backend, a real replication server, a broker of the test as the exporter which says when the entries arrive, the change made while the import waits for them.

• aChangeOfTheExternalChangelogEntryIsRefusedWhileATotalUpdateRuns - the domain configured through the server configuration, so that the listeners of its entries are registered and the change goes through configLock; ds-cfg-ecl-include replaced on the cn=external changelog entry from a thread of the test, the way dsconfig does it. The change is refused with UNWILLING_TO_PERFORM and the total-update reason, the import runs to its end, and the attributes are not applied. On the base the change does not return: the case waits 30 s, captures the two stacks above, interrupts the change - which gives up the join, so that the domain can be taken down - and fails on them.
• aChangeOfTheDomainEntryIsRefusedWhileATotalUpdateRuns - the twin on the domain entry (ds-cfg-assured-type), as it was before this change; pinned so that the two entries keep answering the same thing.
• aChangeOfTheDomainConfigurationLeavesTheSessionToTheImport - applyConfigurationChange() reached directly with an assured configuration which needs a restart, the way a change accepted before the total update reaches it: SUCCESS, adminActionRequired, 327, the listener thread the import runs on is the same thread afterwards, every exported entry arrives, and the assured mode is stored. On the base: adminActionRequired is false, the restart went through.
• aChangeOfTheExternalChangelogAttributesLeavesTheSessionToTheImport - changeConfig(eclIncludes, ...) reached directly: the listener thread is the same afterwards, every entry arrives, the attributes are stored. On the base: a new listener thread.

Base = the head of #968 without this commit: 1/4 (the domain-entry twin), the three others red each on its own assertion. Mutations, each run against the final class: restartService() back on ownsItsSession() - 3/4, aChangeOfTheExternalChangelogAttributesLeavesTheSessionToTheImport red on the listener thread; allowReconnection back on ownsItsSession() - 3/4, aChangeOfTheDomainConfigurationLeavesTheSessionToTheImport red on adminActionRequired. The listener's refusal is what the base run measures for the first case: with it gone the change does not return.

Neighbours on the fixed tree, one JVM per class: LDAPReplicationDomainConfigChangeTest 9/9, SessionRestartTest 2/2, ReplayDuringImportTest 3/3, InitOnLineTest 10/10, GenerationIdTest 4/4, and ConfigChangeDuringImportTest 4/4, nothing skipped.

On the head rebased onto master (c8dcf56e1f), the same classes and ServerStateFlushTest of #977: ConfigChangeDuringImportTest 4/4, LDAPReplicationDomainConfigChangeTest 9/9, SessionRestartTest 2/2, ReplayDuringImportTest 3/3, InitOnLineTest 10/10, GenerationIdTest 4/4, ServerStateFlushTest 4/4, nothing skipped.

Not in this change

• Replication: a total update whose session stops before the DoneMsg ends as a finished import, with the exporter's generationId over partial data #1039 - the silent truncation on the roads where the broker is stopped and the change returns: the replay road [#956] Tell a failed entryUUID search apart from an entry which is not there #968 closes, Replication: the owner read of a session restart and the listener's claim of a remote-initiated import share no lock #1041, the server shutdown. On the configuration road there is no truncated import, finished or not: the change never returned. Replication: a total update whose session stops before the DoneMsg ends as a finished import, with the exporter's generationId over partial data #1039 lists this road among the ones which end as a finished import; it does not.
• Replication: the owner read of a session restart and the listener's claim of a remote-initiated import share no lock #1041 - the owner read and the listener's acquireIEContext() share no lock, and this road has the same window: a change which reads importInProgress() false under serviceStateLock a few statements before the listener claims a remote-initiated InitializeTargetMsg joins a listener thread which is about to need configLock. The CAS under serviceStateLock which Replication: the owner read of a session restart and the listener's claim of a remote-initiated import share no lock #1041 asks for closes both.
• The suppressed restart on the external changelog road is not reported: ExternalChangelogDomain.applyConfigurationChange() returns plain success and reads no sessionRestartSuppressed, and the next change of the domain entry resets the flag as none of its business. Pre-existing to [#926] Restart the session of a replication domain in one place, under the lock and the generation #974 for a disabled domain; with the listener refusing the change, what is left of it is the window between "acceptable" and "apply".

Fixes #1040

[vharseko]

@maximthomas rebased onto master: #968, which this branch was stacked on, merged as 9300ffec88 - the squash of the round-5 head 00798abf33 the branch sat on, identical trees - so the six [#956] commits fell away and the conflict with them. One commit left, c8dcf56e1f; the patch is unchanged (git range-diff 00798abf33..60ad913c02 origin/master..HEAD reports =). The other commit of master since, #977, touches LDAPReplicationDomain.java in the checkpointer only and does not meet this change; no new message ordinal, 327 is reworded as before.

On the rebased head, one JVM per class: ConfigChangeDuringImportTest 4/4, LDAPReplicationDomainConfigChangeTest 9/9, SessionRestartTest 2/2, ReplayDuringImportTest 3/3, InitOnLineTest 10/10, GenerationIdTest 4/4, and ServerStateFlushTest of #977 4/4, nothing skipped. The description's "Stacked on #968" section is replaced by the base as it is now.