diff --git a/content/articles/2026/09/reordering-an-unordered-azure-service-bus-stream-with-powershell/index.md b/content/articles/2026/09/reordering-an-unordered-azure-service-bus-stream-with-powershell/index.md new file mode 100644 index 000000000..d6eae0392 --- /dev/null +++ b/content/articles/2026/09/reordering-an-unordered-azure-service-bus-stream-with-powershell/index.md @@ -0,0 +1,278 @@ +--- +title: "Reordering an Unordered Azure Service Bus Stream with PowerShell" +description: "Explore a PowerShell approach that combines Azure Service Bus message deferral, sequence numbers, and session state to turn an out-of-order stream into ordered output." +author: Andrey Vernigora +authors: + - Andrey Vernigora +date: "2026-09-14T00:00:00+00:00" +categories: + - PowerShell for Developers +tags: + - powershell + - azure-service-bus + - messaging + - message-ordering + - distributed-systems +--- + +Azure Service Bus sessions are the natural choice when a consumer must process related messages in order. But what if the messages already arrive through a non-session subscription, while a downstream consumer still expects an ordered, session-aware stream? + +This article explores one possible bridge between those two models. The idea is to use message deferral as a broker-backed buffer and session state as a small index that remembers which deferred messages can be released later. + +It is deliberately an exploration of the approach, not a production-ready implementation. The interesting part is the state machine and the failure modes it exposes. + +## The scenario + +Suppose every message carries two pieces of application-level metadata: + +- `SessionId` identifies one logical stream. +- `order` is a monotonically increasing integer within that stream. + +The input subscription does **not** require sessions. Messages for one logical stream can therefore be observed as: + +```text +1, 3, 4, 2 +``` + +The downstream subscription does require sessions and should expose: + +```text +1, 2, 3, 4 +``` + +In the sample, the flow looks like this: + +```text +NO_SESSION / NO_SESS_SUB + | + v + PowerShell reorderer + | | + | defer | forward + v v + broker storage ORDERED_TOPIC / SESS_SUB + ^ + | + session state: LastSeen + [(order, sequence number)] +``` + +The full sample is implemented in [`reorderAndForward2.ps1`](https://github.com/eosfor/pubs/blob/main/scripts/orderingTest/reorderAndForward2.ps1) in the [`pubs`](https://github.com/eosfor/pubs) repository. + +## Keep payloads in the broker + +An obvious design would keep early messages in a PowerShell collection until the missing message arrives. That creates a fragile in-memory buffer: a process restart loses it, and payloads consume memory while a gap remains open. + +Service Bus already has a better place for those payloads. A deferred message stays in the broker and can later be retrieved by its `SequenceNumber`. + +The reorderer only persists compact metadata in session state: + +```text +LastSeenOrderNum = 1 +Deferred = [ + { Order = 3; Seq = 42 }, + { Order = 4; Seq = 43 } +] +``` + +This gives the script enough information to retrieve deferred messages without copying their bodies into its own state. + +## The three decisions + +For each message, the script loads state for its logical `SessionId` and calculates: + +```powershell +$expected = $state.LastSeenOrderNum + 1 +``` + +It then makes one of three decisions: + +| Condition | Action | +| --- | --- | +| `order -eq expected` | Forward the message, complete the input, then drain any contiguous deferred messages. | +| `order -gt expected` | Defer the input and save its `order` and `SequenceNumber`. | +| `order -lt expected` | Treat it as stale and dead-letter it. | + +The heart of the approach can be reduced to this pseudocode: + +```powershell +if ($order -eq $expected) { + Send-ToOrderedTopic $message + Complete-InputMessage $message + $state.LastSeenOrderNum = $order + + while ($state contains ($state.LastSeenOrderNum + 1)) { + $next = Receive-DeferredMessage -SequenceNumber $sequenceNumber + Send-ToOrderedTopic $next + Complete-InputMessage $next + $state.LastSeenOrderNum++ + } +} +elseif ($order -gt $expected) { + Defer-InputMessage $message + $state.Deferred.Add(@{ + Order = $order + Seq = $message.SequenceNumber + }) +} +else { + DeadLetter-InputMessage $message +} + +Save-State $state +``` + +The actual script uses typed `SessionOrderingState` and `OrderSeq` objects, rather than untyped hashtables, and separates these operations into small PowerShell functions. + +## Walking through a gap + +First, the producer sends `1`, `3`, and `4`. The reorderer forwards `1`, but it cannot forward `3` or `4`: both depend on the missing `2`. Those two messages are deferred and their sequence numbers are saved. + +![Terminal output after orders 3 and 4 have been deferred](service-bus-reordering-deferred.png) + +The state is now: + +```text +LastSeen = 1 +Deferred = [3, 4] +``` + +When `2` arrives, the reorderer forwards it and advances `LastSeen` to `2`. It can now retrieve deferred `3` by sequence number. After forwarding `3`, the same check makes `4` contiguous, so the script retrieves and forwards that message too. + +![Terminal output showing the ordered result 1, 2, 3, 4](service-bus-reordering-ordered-output.png) + +Here is the complete run: + +![Terminal screencast of the reordering example](service-bus-reordering-demo.gif) + +## Running the experiment locally + +The repository contains a Docker Compose definition for the Azure Service Bus Emulator and SQL Edge. You need Docker Desktop, PowerShell 7, and the .NET 8 or 9 SDK. + +Clone the repository, create the `.env` file described in its README, and start the emulator: + +```bash +git clone https://github.com/eosfor/pubs.git +cd pubs +docker compose -f docker-compose.sbus.yml up -d +dotnet build src/SBPowerShell/SBPowerShell.csproj -c Release +``` + +Then open PowerShell and load the module and the reordering functions: + +```powershell +Import-Module ./src/SBPowerShell/bin/Release/net8.0/pubs.psd1 -Force +. ./scripts/orderingTest/reorderAndForward2.ps1 + +$conn = 'Endpoint=sb://localhost;' + + 'SharedAccessKeyName=RootManageSharedAccessKey;' + + 'SharedAccessKey=LocalEmulatorKey123!;' + + 'UseDevelopmentEmulator=true;' + +$sessionId = "ordering-demo-$([guid]::NewGuid().ToString('N'))" +``` + +Send the first three messages: + +```powershell +foreach ($order in 1, 3, 4) { + $message = New-SBMessage ` + -Body "event-$order" ` + -SessionId $sessionId ` + -CustomProperties @{ order = [int]$order } + + Send-SBMessage ` + -Topic 'NO_SESSION' ` + -Message $message ` + -ServiceBusConnectionString $conn +} + +Receive-SBMessage ` + -Topic 'NO_SESSION' ` + -Subscription 'NO_SESS_SUB' ` + -ServiceBusConnectionString $conn ` + -NoComplete ` + -MaxMessages 3 | + Process-Message -ConnStr $conn -Verbose +``` + +Now send the missing message and process it: + +```powershell +$message = New-SBMessage ` + -Body 'event-2' ` + -SessionId $sessionId ` + -CustomProperties @{ order = [int]2 } + +Send-SBMessage ` + -Topic 'NO_SESSION' ` + -Message $message ` + -ServiceBusConnectionString $conn + +Receive-SBMessage ` + -Topic 'NO_SESSION' ` + -Subscription 'NO_SESS_SUB' ` + -ServiceBusConnectionString $conn ` + -NoComplete ` + -MaxMessages 1 | + Process-Message -ConnStr $conn -Verbose +``` + +Finally, read the session-aware output: + +```powershell +Receive-SBMessage ` + -Topic 'ORDERED_TOPIC' ` + -Subscription 'SESS_SUB' ` + -ServiceBusConnectionString $conn ` + -MaxMessages 4 | + Select-Object ` + @{ Name = 'Order'; Expression = { $_.ApplicationProperties['order'] } }, + @{ Name = 'Body'; Expression = { $_.Body.ToString() } } +``` + +The expected result is: + +```text +Order Body +----- ---- + 1 event-1 + 2 event-2 + 3 event-3 + 4 event-4 +``` + +## Where the approach stops being an implementation + +The experiment makes several simplifying assumptions that matter in a real system. + +### The first message defines the starting point + +The state is initialized from the first message the reorderer receives. If `7` is first, the script accepts `7` as the beginning; messages `1` through `6` arriving later are stale. A production design needs an explicit starting-order contract if that behavior is unacceptable. + +### Forward, complete, and save are not atomic + +The sample forwards a message, completes its input copy, and then saves state. A crash between these operations can produce duplicates or state that no longer reflects the broker. Downstream processing must be idempotent, or the bridge needs a stronger transactional and recovery design. + +### An open gap needs limits + +If message `2` never arrives, the list of deferred sequence numbers continues to grow. A real worker needs limits for gap size and age, plus a policy for expiry, dead-lettering, alerting, and recovery. + +### One logical stream needs one coordinator + +Two workers updating the same logical stream can race unless ownership is coordinated. Service Bus sessions normally provide that coordination through a session lock; this example borrows session state for bookkeeping while consuming from a non-session subscription, so concurrency needs deliberate treatment. + +### Ordering does not remove the need for duplicate handling + +Retries, redelivery, and failures around settlement still exist. The `order` property helps identify stale messages, but a business-level message identifier and idempotent downstream operations are still valuable. + +## Why the pattern is useful + +Even with those limitations, this is a useful experiment because it separates three concerns: + +1. Service Bus stores deferred payloads. +2. Session state stores the minimum ordering index. +3. PowerShell expresses the state transition in a compact, inspectable form. + +That makes it practical for exploring message-ordering behavior locally, testing failure hypotheses, and deciding which guarantees a production implementation would actually need. + +The result is not “ordered messaging added to a non-session subscription.” It is a small bridge that demonstrates how deferral, sequence numbers, and session state can cooperate—and where their guarantees end. diff --git a/content/articles/2026/09/reordering-an-unordered-azure-service-bus-stream-with-powershell/service-bus-reordering-deferred.png b/content/articles/2026/09/reordering-an-unordered-azure-service-bus-stream-with-powershell/service-bus-reordering-deferred.png new file mode 100644 index 000000000..d0e639f28 Binary files /dev/null and b/content/articles/2026/09/reordering-an-unordered-azure-service-bus-stream-with-powershell/service-bus-reordering-deferred.png differ diff --git a/content/articles/2026/09/reordering-an-unordered-azure-service-bus-stream-with-powershell/service-bus-reordering-demo.gif b/content/articles/2026/09/reordering-an-unordered-azure-service-bus-stream-with-powershell/service-bus-reordering-demo.gif new file mode 100644 index 000000000..2989ce37d Binary files /dev/null and b/content/articles/2026/09/reordering-an-unordered-azure-service-bus-stream-with-powershell/service-bus-reordering-demo.gif differ diff --git a/content/articles/2026/09/reordering-an-unordered-azure-service-bus-stream-with-powershell/service-bus-reordering-ordered-output.png b/content/articles/2026/09/reordering-an-unordered-azure-service-bus-stream-with-powershell/service-bus-reordering-ordered-output.png new file mode 100644 index 000000000..b96ea4c35 Binary files /dev/null and b/content/articles/2026/09/reordering-an-unordered-azure-service-bus-stream-with-powershell/service-bus-reordering-ordered-output.png differ diff --git a/scripts/validate-article-bundles.mjs b/scripts/validate-article-bundles.mjs index 777168258..ab7df9dd9 100755 --- a/scripts/validate-article-bundles.mjs +++ b/scripts/validate-article-bundles.mjs @@ -87,13 +87,13 @@ for (const year of readdirSync(contentRoot, { withFileTypes: true }).filter((ent if (!existsSync(join(contentRoot, year.name, '_index.md'))) fail(`Year branch is missing _index.md: ${year.name}`); else checkOutput(yearRoute, `Year archive ${year.name}`); const yearHtml = archiveHtml(yearRoute); - for (const [route, source] of sourceArticles) if (source.year === year.name && !inventoryByRoute.get(route).draft && !includesRoute(yearHtml, route)) fail(`Article is missing from its year archive: ${route}`); + for (const [route, source] of sourceArticles) if (source.year === year.name && !inventoryByRoute.get(route)?.draft && !includesRoute(yearHtml, route)) fail(`Article is missing from its year archive: ${route}`); for (const month of readdirSync(join(contentRoot, year.name), { withFileTypes: true }).filter((entry) => entry.isDirectory())) { const monthRoute = `${yearRoute}${month.name}/`; if (!existsSync(join(contentRoot, year.name, month.name, '_index.md'))) fail(`Month branch is missing _index.md: ${year.name}/${month.name}`); else checkOutput(monthRoute, `Month archive ${year.name}/${month.name}`); const monthHtml = archiveHtml(monthRoute); - for (const [route, source] of sourceArticles) if (source.year === year.name && source.month === month.name && !inventoryByRoute.get(route).draft && !includesRoute(monthHtml, route)) fail(`Article is missing from its month archive: ${route}`); + for (const [route, source] of sourceArticles) if (source.year === year.name && source.month === month.name && !inventoryByRoute.get(route)?.draft && !includesRoute(monthHtml, route)) fail(`Article is missing from its month archive: ${route}`); } }