From a26cb76610927595510ce4c1f7e07a36d9854725 Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Thu, 17 Sep 2026 18:53:51 -0500 Subject: [PATCH 01/23] Added a PowerShell version of the WarmPerformnaceTier script. --- .../warm_performance_tier/README.md | 38 ++- .../WarmPerformanceTier.ps1 | 233 ++++++++++++++++++ 2 files changed, 264 insertions(+), 7 deletions(-) create mode 100755 Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 diff --git a/Management-Utilities/warm_performance_tier/README.md b/Management-Utilities/warm_performance_tier/README.md index 2605ea20..4d8fcbbe 100644 --- a/Management-Utilities/warm_performance_tier/README.md +++ b/Management-Utilities/warm_performance_tier/README.md @@ -52,19 +52,24 @@ script twice does, typically, get more data into the performance tier so if you are trying to get as much data as possible into the performance tier, it is recommended to run the script twice. +There are two versions of the script. One that designed to run on a Linux operating system name `warm_performance_tier` +and one that is meant to run on a Windows based one name `WarmPerformanceTier.ps1`. + ## Set Up The first step is to ensure the volume's tiering policy is set to something other than "all" or "snapshot-only". You should also ensure that the cloud-retrieval-policy is set to "on-read". You can make both of these changes with the following commands: ``` -set advanced -c off +set advanced volume modify -vserver -volume -tiering-policy auto -cloud-retrieval-policy on-read ``` Where `` is the name of the SVM and `` is the name of the volume. -The next step is to copy the script to a Linux based host that is able to NFS -mount the volume to be warmed. If the volume is already mounted, then +The next step is to copy the appropriate script, either the Linux based one or the Windows +based one, to the system that has access to the volume to be warmed. + +On a Linux based system, if the volume is already mounted, then any user that has read access to all the files in the volume can run it. Otherwise, the script needs to be run as 'root' so it can mount the volume before reading the files. @@ -73,7 +78,9 @@ If the 'root' user can't read the all files in the volume, then you should use t to mount the volume and then run the script from a user ID that can read the contents of all the files in the volume. -# Running The Script +## Running The Script + +### Runing the script on a Linux based operating system There are two main ways to run the script. The first is to just provide the script with a directory to start from using the -d option. The script will then read every file in that directory and all its subdirectories. The second way @@ -115,7 +122,7 @@ Where: -d directory - Is the root directory to start the process from. -t max_directory_threads - Is the maximum number of threads to use to process directories. The default is 2. -x max_read_threads - Is the maximum number of threads to use to read files. The default is 5. - -V - Enable verbose output. Displays the thread ID, date (in epoch seconds), then the directory or file being processed. + -V - Enable verbose output. Displays progress information on the files being read. -h - Prints this help information. Notes: @@ -126,11 +133,28 @@ Notes: reading files. ``` +### Runing the script on a Windows based operating system + +To run on a Windows based operating system, you will need to run the script from a +PowerShell prompt. You can run the script with the following command: +``` +.\WarmPerformanceTier.ps1 -Path \\fsx-server\myshare +``` +Where `\\fsx-server\myshare` is the UNC path or mapped drive letter to the share/volume root. + +The script supports the following options: +o Path - UNC path or mapped drive letter to the share/volume root, e.g. \\fsx-server\myshare +o ThreadCount - Number of files to read in parallel. Default 8. +o BlockSizeMB - Block size in MB used for the out-of-order reads. Default 2 +o ReadMethod - 'Reverse' (read blocks back-to-front) or 'Random' (shuffle block order). Default is 'Reverse'. +o BatchSize - How many files to queue at once before waiting for that batch to finish. + Bounds memory usage on volumes with very large file counts. Default is ThreadCount * 4. + ## Finishing Step After running the script, you should set the cloud-retrieval-policy back to "default" by running the following commands: ``` -set advanced -c off +set advanced volume modify -vserver -volume -cloud-retrieval-policy default ``` Where `` is the name of the SVM and `` is the name of the volume. @@ -149,4 +173,4 @@ Unless required by applicable law or agreed to in writing, software distributed See the License for the specific language governing permissions and limitations under the License. -© 2024 NetApp, Inc. All Rights Reserved. +© 2026 NetApp, Inc. All Rights Reserved. diff --git a/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 b/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 new file mode 100755 index 00000000..ca02e494 --- /dev/null +++ b/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 @@ -0,0 +1,233 @@ +<# +.SYNOPSIS + Reads every file in a CIFS/SMB share to force promotion of capacity-tier + (cold/tiered) data back to the performance tier on an FSx for ONTAP volume. + +.DESCRIPTION + ONTAP's tiering scanner intentionally avoids promoting data when it detects + a sequential, backup-like read pattern (to avoid backup/AV jobs needlessly + re-filling the performance tier). This script defeats that detection by + reading each file's blocks in reverse order (or random order) instead of + front-to-back, so the read pattern looks like real client access rather + than a scan. + + This is a native Windows/PowerShell port of NetApp's warm_performance_tier + bash script (NFS-only) for environments with no Linux available. + +.PARAMETER Path + UNC path or mapped drive letter to the share/volume root, e.g. \\fsx-server\myshare + +.PARAMETER ThreadCount + Number of files to read in parallel. Default 8. + +.PARAMETER BlockSizeMB + Block size in MB used for the out-of-order reads. Default 2 (matches the + original script's default of 2MB). + +.PARAMETER ReadMethod + 'Reverse' (read blocks back-to-front) or 'Random' (shuffle block order). + Default 'Reverse'. + +.PARAMETER BatchSize + How many files to queue at once before waiting for that batch to finish. + Bounds memory usage on volumes with very large file counts. Default is + ThreadCount * 4. + +.PARAMETER Help + Print invocation usage and exit. + +.NOTES + Promotion observed on FSx for ONTAP has been gradual rather than + immediate - a single pass over a volume may only promote a small + fraction of the read data back to the performance tier, with the rest + following over subsequent passes. Expect to run this multiple times + (or in a loop until volume footprint stabilizes) rather than a single + one-shot sweep for full promotion. + +.EXAMPLE + .\WarmPerformanceTier.ps1 -Path \\fsx-server\myshare -ThreadCount 12 + +.EXAMPLE + .\WarmPerformanceTier.ps1 -Path Z:\ -ReadMethod Random -BlockSizeMB 4 + +.EXAMPLE + .\WarmPerformanceTier.ps1 -Help +#> + +[CmdletBinding(DefaultParameterSetName = 'Run')] +param( + [Parameter(Mandatory = $true, ParameterSetName = 'Run', Position = 0)] + [string]$Path, + + [Parameter(ParameterSetName = 'Run')] + [int]$ThreadCount = 8, + + [Parameter(ParameterSetName = 'Run')] + [int]$BlockSizeMB = 2, + + [Parameter(ParameterSetName = 'Run')] + [ValidateSet('Reverse', 'Random')] + [string]$ReadMethod = 'Reverse', + + [Parameter(ParameterSetName = 'Run')] + [int]$BatchSize = 0, + + [Parameter(ParameterSetName = 'Run')] + [switch]$VerboseProgress, + + [Parameter(ParameterSetName = 'Help')] + [switch]$Help +) + +$ErrorActionPreference = 'Stop' + +if ($Help) { + Write-Host @" +Usage: .\WarmPerformanceTier.ps1 -Path [-ThreadCount n] [-BlockSizeMB n] [-ReadMethod Reverse|Random] [-BatchSize n] [-VerboseProgress] [-Help] + +Where: + -Path UNC path or mapped drive to the share/volume root (required). + Example: \\fsx-server\myshare or Z:\ + -ThreadCount Number of files to read in parallel. Default: 8 + -BlockSizeMB Block size in MB for out-of-order reads. Default: 2 + -ReadMethod Reverse (back-to-front) or Random. Default: Reverse + -BatchSize Files to queue before waiting. Default: ThreadCount * 4 + -VerboseProgress Print extra progress lines in addition to the progress bar + -Help Print this help and exit + +Examples: + .\WarmPerformanceTier.ps1 -Path \\fsx-server\myshare -ThreadCount 12 + .\WarmPerformanceTier.ps1 -Path Z:\ -ReadMethod Random -BlockSizeMB 4 +"@ + return +} + +if ($BatchSize -le 0) { + $BatchSize = $ThreadCount * 4 +} + +if (-not (Test-Path -LiteralPath $Path)) { + throw "Path '$Path' does not exist or is not accessible." +} + +Write-Host "Enumerating files under '$Path' ..." +$files = Get-ChildItem -LiteralPath $Path -Recurse -File -Force -ErrorAction SilentlyContinue +$total = $files.Count + +if ($total -eq 0) { + Write-Host "No files found under '$Path'. Nothing to do." + return +} + +Write-Host "Found $total files. Reading with $ThreadCount parallel workers (method: $ReadMethod, block size: ${BlockSizeMB}MB, batch size: $BatchSize)..." + +$blockSize = [int64]$BlockSizeMB * 1MB + +# Script block executed inside each runspace. Reads the file's blocks out of +# sequential order and discards the contents - the goal is purely to trigger +# ONTAP's read-driven promotion, not to use the data. +$readScriptBlock = { + param($filePath, $blockSize, $readMethod) + + try { + $fs = [System.IO.File]::Open( + $filePath, + [System.IO.FileMode]::Open, + [System.IO.FileAccess]::Read, + [System.IO.FileShare]::ReadWrite + ) + try { + $length = $fs.Length + if ($length -eq 0) { + return [PSCustomObject]@{ Path = $filePath; Success = $true; Bytes = 0 } + } + + $buffer = New-Object byte[] $blockSize + $totalBlocks = [math]::Ceiling($length / $blockSize) + + if ($readMethod -eq 'Reverse') { + $blockOrder = ($totalBlocks - 1)..0 + } + else { + $blockOrder = 0..($totalBlocks - 1) | Get-Random -Count $totalBlocks + } + + foreach ($i in $blockOrder) { + $offset = [int64]$i * $blockSize + $toRead = [Math]::Min($blockSize, $length - $offset) + if ($toRead -le 0) { continue } + [void]$fs.Seek($offset, [System.IO.SeekOrigin]::Begin) + [void]$fs.Read($buffer, 0, [int]$toRead) + } + + return [PSCustomObject]@{ Path = $filePath; Success = $true; Bytes = $length } + } + finally { + $fs.Dispose() + } + } + catch { + return [PSCustomObject]@{ Path = $filePath; Success = $false; Error = $_.Exception.Message } + } +} + +$sessionState = [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault() +$pool = [System.Management.Automation.Runspaces.RunspaceFactory]::CreateRunspacePool(1, $ThreadCount, $sessionState, $Host) +$pool.Open() + +$startTime = Get-Date +$doneCount = 0 +$errorCount = 0 +$errorList = New-Object System.Collections.Generic.List[string] + +function Wait-AndCollect($jobs) { + foreach ($job in $jobs) { + $result = $job.Pipeline.EndInvoke($job.Handle) + $job.Pipeline.Dispose() + $script:doneCount++ + if ($result.Success -eq $false) { + $script:errorCount++ + $script:errorList.Add("$($result.Path): $($result.Error)") + } + } +} + +$batch = New-Object System.Collections.Generic.List[object] + +foreach ($file in $files) { + $ps = [System.Management.Automation.PowerShell]::Create() + $ps.RunspacePool = $pool + [void]$ps.AddScript($readScriptBlock).AddArgument($file.FullName).AddArgument($blockSize).AddArgument($ReadMethod) + $handle = $ps.BeginInvoke() + $batch.Add([PSCustomObject]@{ Pipeline = $ps; Handle = $handle; File = $file.FullName }) + + if ($batch.Count -ge $BatchSize) { + Wait-AndCollect $batch + $batch.Clear() + + $elapsed = (Get-Date) - $startTime + $pct = [math]::Round(($doneCount / $total) * 100, 1) + Write-Progress -Activity "Warming performance tier" -Status "$doneCount / $total files ($errorCount errors)" -PercentComplete $pct + if ($VerboseProgress) { + Write-Host ("[{0:hh\:mm\:ss}] {1}/{2} files done ({3}%), {4} errors" -f $elapsed, $doneCount, $total, $pct, $errorCount) + } + } +} + +# Drain the final partial batch. +if ($batch.Count -gt 0) { + Wait-AndCollect $batch +} + +$pool.Close() +$pool.Dispose() + +$elapsed = (Get-Date) - $startTime +Write-Progress -Activity "Warming performance tier" -Completed + +Write-Host "" +Write-Host "Done in $($elapsed.ToString('hh\:mm\:ss')). Processed $doneCount files, $errorCount errors." +if ($errorCount -gt 0) { + Write-Host "Errors:" + $errorList | ForEach-Object { Write-Host " $_" } +} From 12f9814d6f45b6cf092bce3015a652a8dee2d879 Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Thu, 17 Sep 2026 19:15:26 -0500 Subject: [PATCH 02/23] Added a PowerShell version of the WarmPerformnaceTier script. --- .../list_fsxn_filesystems | 17 ++++++++++++----- .../fsx-ontap-aws-cli-scripts/list_fsxn_svms | 16 +++++++++++++--- .../fsx-ontap-aws-cli-scripts/list_fsxn_volumes | 17 +++++++++++++---- .../warm_performance_tier/README.md | 11 ++++++----- 4 files changed, 44 insertions(+), 17 deletions(-) diff --git a/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_filesystems b/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_filesystems index 49bb98e7..c5d9077b 100755 --- a/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_filesystems +++ b/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_filesystems @@ -44,7 +44,7 @@ usage () { cat 1>&2 < /dev/null; then + i=1 + while [ $i -lt $((maxTime * 2)) ]; do + if ! ps -p $pid > /dev/null 2>&1; then + return 0 + fi + sleep .5 + let i+=1 + done + + if ps -p $pid > /dev/null 2>&1; then printf "\nWarning, timed out while getting the list of FSxNs from the $region region.\nConsider excluding it with the -X option or raising the timeout with the -t option.\n" 1>&2 pkill -P $pid kill $pid @@ -265,7 +273,7 @@ maxTime=20 # Process command line arguments. while getopts "bhcxsaX:nr:i:f:e:t:" option; do case "$option" in - r) region="$OPTARG" + r) region="$(echo $OPTARG | sed -e 's/,/ /g')" ;; a) allRegions=true ;; @@ -322,7 +330,6 @@ if [ "$allRegions" == "true" ]; then # support FSxN allEndabledRegions=$(aws ec2 describe-regions --query "Regions[].RegionName" --output=json | jq -r '.[]') allFsxnRegions=$(curl -s https://api.regional-table.region-services.aws.a2z.com/index.json | jq -r '.prices[] | select(.attributes."aws:serviceName" == "Amazon FSx for NetApp ONTAP") .attributes."aws:region"' | egrep -v "$excludeRegions") -# allFsxnRegions=$(curl -s https://api.regional-table.region-services.aws.a2z.com/index.json | jq -r '.prices[] | select(.attributes."aws:serviceName" == "Amazon FSx for NetApp ONTAP") .attributes."aws:region"') for reg in $allEndabledRegions; do for fsxnReg in $allFsxnRegions; do if [ $reg == $fsxnReg ]; then diff --git a/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_svms b/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_svms index fc8d5104..ae25d299 100755 --- a/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_svms +++ b/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_svms @@ -30,6 +30,7 @@ usage () { cat 1>&2 < /dev/null; then + + i=1 + while [ $i -lt $((maxTime*2)) ]; do + if ! ps -p $pid > /dev/null 2>&1; then + return + fi + sleep .5 + let i+=1 + done + + if ps -p $pid > /dev/null 2>&1; then printf "\nWarning, timed out while getting the list of SVMs from the $region region.\nConsider excluding it with the -X option or raising the timeout with the -t option.\n" 1>&2 pkill -P $pid kill $pid @@ -129,7 +139,7 @@ excludeRegions="none" maxTime=20 while getopts "haX:nr:i:f:pe:t:" option; do case "$option" in - r) region="$OPTARG" + r) region="$(echo "$OPTARG" | sed -e 's/,/ /g')" ;; a) allRegions=true ;; diff --git a/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_volumes b/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_volumes index 4d61d667..840487a0 100755 --- a/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_volumes +++ b/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_volumes @@ -36,7 +36,7 @@ usage () { Usage $(basename $0) [-r region] [-a] [-X excludeRegions] [-o] [-f fileSystemName] [-i fileSystemId] [-n] [-s svmID] [-x] [-t seconds] [-e endpoint_url] [volume_pattern] Where: volume_pattern - Only volumes that match the egrep pattern will be included. An assumed '^' and '$' is added to the beginning and end of the pattern respectively. - -r region allows you to specify the region you want the list from. + -r means to only include the specified region(s) (comma seperated list). -a means all regions. -X excludeRegions - Comma separated list of regions to exclude from the search. Only valid when -a is specified. -n means include the file system and SVM names. Not compatible with the -x option. @@ -61,8 +61,17 @@ start_watchdog () { pid=$1 maxTime=$2 region=$3 - sleep $maxTime - if ps -p $pid > /dev/null; then + + i=1 + while [ $i -lt $((maxTime * 2)) ]; do + if ! ps -p $pid > /dev/null 2>&1; then + return + fi + sleep .5 + let i+=1 + done + + if ps -p $pid > /dev/null 2>&1; then printf "\nWarning, timed out while getting the list of SVMs from the $region region.\nConsider excluding it with the -X option or raising the timeout with the -t option.\n" 1>&2 pkill -P $pid kill $pid @@ -214,7 +223,7 @@ volumePattern="*" maxTime=30 while getopts "hr:aX:f:i:nos:xe:t:" option; do case "$option" in - r) region="$OPTARG" + r) region="$(echo $OPTARG | sed -e 's/,/ /g')" ;; a) allRegions=true ;; diff --git a/Management-Utilities/warm_performance_tier/README.md b/Management-Utilities/warm_performance_tier/README.md index 4d8fcbbe..2f9eec9c 100644 --- a/Management-Utilities/warm_performance_tier/README.md +++ b/Management-Utilities/warm_performance_tier/README.md @@ -143,11 +143,12 @@ PowerShell prompt. You can run the script with the following command: Where `\\fsx-server\myshare` is the UNC path or mapped drive letter to the share/volume root. The script supports the following options: -o Path - UNC path or mapped drive letter to the share/volume root, e.g. \\fsx-server\myshare -o ThreadCount - Number of files to read in parallel. Default 8. -o BlockSizeMB - Block size in MB used for the out-of-order reads. Default 2 -o ReadMethod - 'Reverse' (read blocks back-to-front) or 'Random' (shuffle block order). Default is 'Reverse'. -o BatchSize - How many files to queue at once before waiting for that batch to finish. + +* Path - UNC path or mapped drive letter to the share/volume root, e.g. \\fsx-server\myshare +* ThreadCount - Number of files to read in parallel. Default 8. +* BlockSizeMB - Block size in MB used for the out-of-order reads. Default 2 +* ReadMethod - 'Reverse' (read blocks back-to-front) or 'Random' (shuffle block order). Default is 'Reverse'. +* BatchSize - How many files to queue at once before waiting for that batch to finish. Bounds memory usage on volumes with very large file counts. Default is ThreadCount * 4. ## Finishing Step From 6bef5d1f2c16ab65f53ed9d7246d93ad30b8080d Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Tue, 22 Sep 2026 08:33:27 -0500 Subject: [PATCH 03/23] Ensure valid parameters were provided. --- .../WarmPerformanceTier.ps1 | 20 ++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 b/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 index ca02e494..3c658042 100755 --- a/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 +++ b/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 @@ -60,9 +60,11 @@ param( [string]$Path, [Parameter(ParameterSetName = 'Run')] + [ValidateRange(1, 1024)] [int]$ThreadCount = 8, [Parameter(ParameterSetName = 'Run')] + [ValidateScript({ $_ -gt 0 -and ([int64]$_ * 1MB) -le [int]::MaxValue })] [int]$BlockSizeMB = 2, [Parameter(ParameterSetName = 'Run')] @@ -111,9 +113,14 @@ if (-not (Test-Path -LiteralPath $Path)) { } Write-Host "Enumerating files under '$Path' ..." -$files = Get-ChildItem -LiteralPath $Path -Recurse -File -Force -ErrorAction SilentlyContinue +$enumerationErrors = @() +$files = Get-ChildItem -LiteralPath $Path -Recurse -File -Force -ErrorAction SilentlyContinue -ErrorVariable $enumerationErrors $total = $files.Count +if ($enumerationErrors.Count -gt 0) { + Write-Warning "Encountered $($enumerationErrors.Count) errors while enumerating files. Some files may be skipped." +} + if ($total -eq 0) { Write-Host "No files found under '$Path'. Nothing to do." return @@ -157,9 +164,15 @@ $readScriptBlock = { $toRead = [Math]::Min($blockSize, $length - $offset) if ($toRead -le 0) { continue } [void]$fs.Seek($offset, [System.IO.SeekOrigin]::Begin) - [void]$fs.Read($buffer, 0, [int]$toRead) + $remaining = [int]$toRead + while ($remaining -gt 0) { + $bytesRead = $fs.Read($buffer, 0, $remaining) + if ($bytesRead -eq 0) { + throw "Unexpected end of file while reading '$filePath'." + } + $remaining -= $bytesRead + } } - return [PSCustomObject]@{ Path = $filePath; Success = $true; Bytes = $length } } finally { @@ -230,4 +243,5 @@ Write-Host "Done in $($elapsed.ToString('hh\:mm\:ss')). Processed $doneCount fil if ($errorCount -gt 0) { Write-Host "Errors:" $errorList | ForEach-Object { Write-Host " $_" } + exit 1 } From 587cfb896c90bf8fda07a3d60ec36848b9afb36c Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Tue, 22 Sep 2026 12:43:46 -0500 Subject: [PATCH 04/23] Change it so it would stream the file objects to loop that processes the files so that it could scale to volumes with a high number of files. --- .../WarmPerformanceTier.ps1 | 68 ++++++++++--------- 1 file changed, 36 insertions(+), 32 deletions(-) diff --git a/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 b/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 index 3c658042..6cef5f5d 100755 --- a/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 +++ b/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 @@ -112,21 +112,7 @@ if (-not (Test-Path -LiteralPath $Path)) { throw "Path '$Path' does not exist or is not accessible." } -Write-Host "Enumerating files under '$Path' ..." -$enumerationErrors = @() -$files = Get-ChildItem -LiteralPath $Path -Recurse -File -Force -ErrorAction SilentlyContinue -ErrorVariable $enumerationErrors -$total = $files.Count - -if ($enumerationErrors.Count -gt 0) { - Write-Warning "Encountered $($enumerationErrors.Count) errors while enumerating files. Some files may be skipped." -} - -if ($total -eq 0) { - Write-Host "No files found under '$Path'. Nothing to do." - return -} - -Write-Host "Found $total files. Reading with $ThreadCount parallel workers (method: $ReadMethod, block size: ${BlockSizeMB}MB, batch size: $BatchSize)..." +Write-Host "Streaming files under '$Path' and reading with $ThreadCount parallel workers (method: $ReadMethod, block size: ${BlockSizeMB}MB, batch size: $BatchSize)..." $blockSize = [int64]$BlockSizeMB * 1MB @@ -206,26 +192,35 @@ function Wait-AndCollect($jobs) { } $batch = New-Object System.Collections.Generic.List[object] +$queuedCount = 0 +$enumerationErrors = @() -foreach ($file in $files) { - $ps = [System.Management.Automation.PowerShell]::Create() - $ps.RunspacePool = $pool - [void]$ps.AddScript($readScriptBlock).AddArgument($file.FullName).AddArgument($blockSize).AddArgument($ReadMethod) - $handle = $ps.BeginInvoke() - $batch.Add([PSCustomObject]@{ Pipeline = $ps; Handle = $handle; File = $file.FullName }) - - if ($batch.Count -ge $BatchSize) { - Wait-AndCollect $batch - $batch.Clear() - - $elapsed = (Get-Date) - $startTime - $pct = [math]::Round(($doneCount / $total) * 100, 1) - Write-Progress -Activity "Warming performance tier" -Status "$doneCount / $total files ($errorCount errors)" -PercentComplete $pct - if ($VerboseProgress) { - Write-Host ("[{0:hh\:mm\:ss}] {1}/{2} files done ({3}%), {4} errors" -f $elapsed, $doneCount, $total, $pct, $errorCount) +# Get-ChildItem streams FileInfo objects one at a time as it walks the tree, +# and piping straight into ForEach-Object dispatches each file to the worker +# queue as soon as it is discovered - the full listing is never held in +# memory, so this scales to directories with very large file counts. +Get-ChildItem -LiteralPath $Path -Recurse -File -Force -ErrorAction SilentlyContinue -ErrorVariable enumerationErrors | + ForEach-Object { + $file = $_ + $queuedCount++ + + $ps = [System.Management.Automation.PowerShell]::Create() + $ps.RunspacePool = $pool + [void]$ps.AddScript($readScriptBlock).AddArgument($file.FullName).AddArgument($blockSize).AddArgument($ReadMethod) + $handle = $ps.BeginInvoke() + $batch.Add([PSCustomObject]@{ Pipeline = $ps; Handle = $handle; File = $file.FullName }) + + if ($batch.Count -ge $BatchSize) { + Wait-AndCollect $batch + $batch.Clear() + + $elapsed = (Get-Date) - $startTime + Write-Progress -Activity "Warming performance tier" -Status "$doneCount processed, $queuedCount queued so far ($errorCount errors)" + if ($VerboseProgress) { + Write-Host ("[{0:hh\:mm\:ss}] {1} processed, {2} queued so far, {3} errors" -f $elapsed, $doneCount, $queuedCount, $errorCount) + } } } -} # Drain the final partial batch. if ($batch.Count -gt 0) { @@ -238,6 +233,15 @@ $pool.Dispose() $elapsed = (Get-Date) - $startTime Write-Progress -Activity "Warming performance tier" -Completed +if ($enumerationErrors.Count -gt 0) { + Write-Warning "Encountered $($enumerationErrors.Count) errors while enumerating files. Some files may be skipped." +} + +if ($queuedCount -eq 0) { + Write-Host "No files found under '$Path'. Nothing to do." + return +} + Write-Host "" Write-Host "Done in $($elapsed.ToString('hh\:mm\:ss')). Processed $doneCount files, $errorCount errors." if ($errorCount -gt 0) { From b4bfb51d818f0f9daab8d1d75a595c96059779f2 Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Tue, 22 Sep 2026 12:44:36 -0500 Subject: [PATCH 05/23] Made it clearer. --- Management-Utilities/warm_performance_tier/README.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/Management-Utilities/warm_performance_tier/README.md b/Management-Utilities/warm_performance_tier/README.md index 2f9eec9c..a66549fc 100644 --- a/Management-Utilities/warm_performance_tier/README.md +++ b/Management-Utilities/warm_performance_tier/README.md @@ -52,8 +52,9 @@ script twice does, typically, get more data into the performance tier so if you are trying to get as much data as possible into the performance tier, it is recommended to run the script twice. -There are two versions of the script. One that designed to run on a Linux operating system name `warm_performance_tier` -and one that is meant to run on a Windows based one name `WarmPerformanceTier.ps1`. +There are two versions of the script: +o `warm_performance_tier` - Disigned for Linux. +o `WarmPerformanceTier.ps1` - Designed for Windows. ## Set Up The first step is to ensure the volume's tiering policy is set @@ -80,7 +81,7 @@ of all the files in the volume. ## Running The Script -### Runing the script on a Linux based operating system +### Running the script on a Linux-based operating system There are two main ways to run the script. The first is to just provide the script with a directory to start from using the -d option. The script will then read every file in that directory and all its subdirectories. The second way From 10f7641a698d833f1ed6eb15bd62524de0c8563e Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Tue, 22 Sep 2026 12:49:07 -0500 Subject: [PATCH 06/23] Added multi-threading for the -a option. --- .../fsx-ontap-aws-cli-scripts/list_fsxn_filesystems | 2 +- .../fsx-ontap-aws-cli-scripts/list_fsxn_svms | 10 +++++----- .../fsx-ontap-aws-cli-scripts/list_fsxn_volumes | 8 ++++---- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_filesystems b/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_filesystems index c5d9077b..a314c97e 100755 --- a/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_filesystems +++ b/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_filesystems @@ -72,7 +72,7 @@ start_watchdog () { region=$3 i=1 - while [ $i -lt $((maxTime * 2)) ]; do + while [ $i -le $((maxTime * 2)) ]; do if ! ps -p $pid > /dev/null 2>&1; then return 0 fi diff --git a/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_svms b/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_svms index ae25d299..3583f259 100755 --- a/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_svms +++ b/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_svms @@ -18,7 +18,7 @@ # o Region # o File system ID # o File System Name - optional -# o The managment and NAS IP address of the SVM - optional +# o The management and NAS IP address of the SVM - optional # o SVM ID # o SVM Name ################################################################################ @@ -30,7 +30,7 @@ usage () { cat 1>&2 < /dev/null 2>&1; then return fi diff --git a/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_volumes b/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_volumes index 840487a0..b5af05ee 100755 --- a/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_volumes +++ b/Management-Utilities/fsx-ontap-aws-cli-scripts/list_fsxn_volumes @@ -36,7 +36,7 @@ usage () { Usage $(basename $0) [-r region] [-a] [-X excludeRegions] [-o] [-f fileSystemName] [-i fileSystemId] [-n] [-s svmID] [-x] [-t seconds] [-e endpoint_url] [volume_pattern] Where: volume_pattern - Only volumes that match the egrep pattern will be included. An assumed '^' and '$' is added to the beginning and end of the pattern respectively. - -r means to only include the specified region(s) (comma seperated list). + -r means to only include the specified region(s) (comma separated list). -a means all regions. -X excludeRegions - Comma separated list of regions to exclude from the search. Only valid when -a is specified. -n means include the file system and SVM names. Not compatible with the -x option. @@ -53,8 +53,8 @@ EOF ################################################################################ # This function is used to kill a process, who's ID is passed as the first -# argument, that is running longger than the time specified as the second -# argurment. +# argument, that is running longer than the time specified as the second +# argument. ################################################################################ start_watchdog () { @@ -63,7 +63,7 @@ start_watchdog () { region=$3 i=1 - while [ $i -lt $((maxTime * 2)) ]; do + while [ $i -le $((maxTime * 2)) ]; do if ! ps -p $pid > /dev/null 2>&1; then return fi From eb44b89cb208c4ae65dd13dcad23ce9799d1f7a5 Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Tue, 22 Sep 2026 13:16:38 -0500 Subject: [PATCH 07/23] Adding a PowerShell version of the warm_performance_tier script. --- .../warm_performance_tier/WarmPerformanceTier.ps1 | 3 +++ 1 file changed, 3 insertions(+) diff --git a/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 b/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 index 6cef5f5d..a8f624ea 100755 --- a/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 +++ b/Management-Utilities/warm_performance_tier/WarmPerformanceTier.ps1 @@ -54,6 +54,9 @@ .\WarmPerformanceTier.ps1 -Help #> +[Diagnostics.CodeAnalysis.SuppressMessageAttribute("PSAvoidUsingWriteHost", "")] +[Diagnostics.CodeAnalysis.SuppressMessageAttribute("PSReviewUnusedParameter", "")] # Since the GitHub PS Lint thinks $VerboseProgress is unused. + [CmdletBinding(DefaultParameterSetName = 'Run')] param( [Parameter(Mandatory = $true, ParameterSetName = 'Run', Position = 0)] From 1206f985f72f1be202c4035b5c048c3de35d0fa0 Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Tue, 29 Sep 2026 15:49:55 -0500 Subject: [PATCH 08/23] Added the "set_fsxn_password" script. --- .../fsx-ontap-aws-cli-scripts/README.md | 9 +- .../set_fsxn_password | 150 ++++++++++++++++++ 2 files changed, 155 insertions(+), 4 deletions(-) create mode 100755 Management-Utilities/fsx-ontap-aws-cli-scripts/set_fsxn_password diff --git a/Management-Utilities/fsx-ontap-aws-cli-scripts/README.md b/Management-Utilities/fsx-ontap-aws-cli-scripts/README.md index 056f1393..31d1095e 100644 --- a/Management-Utilities/fsx-ontap-aws-cli-scripts/README.md +++ b/Management-Utilities/fsx-ontap-aws-cli-scripts/README.md @@ -8,7 +8,7 @@ Windows Subsystem for Linux (WSL) based Linux distribution installed. Before running the UNIX based scripts, make sure the following package is installed: * jq - lightweight and flexible command-line JSON processor -* aws-cli - Command Line Interface for AWS +* aws - Command Line Interface for AWS ## Summary of the convenience scripts @@ -17,15 +17,16 @@ Before running the UNIX based scripts, make sure the following package is instal |create_fsxn_filesystem | Creates a new FSx for NetApp ONTAP file system. | |create_fsxn_svm | Creates a new Storage Virtual Server (SVM) in a specified FSx for ONTAP file system. | |create_fsxn_volume | Creates a new volume under a specified SVM. | +|delete_fsxn_filesystem | Deletes a specified FSx for ONTAP file system. Including all the SVMs and volumes on it. | +|delete_fsxn_svm | Deletes a specified SVM. Including all the volumes assigned to it. | +|delete_fsxn_volume | Deletes a specified volume. | |list_fsx_filesystems | List all the FSx for NetApp ONTAP file systems that the user has access to. | |list_fsx_filesystems.ps1 | List all the FSx for NetApp ONTAP file systems that the user has access to, written in PowerShell. | |list_fsxn_volumes | List all the FSx for NetApp ONTAP volumes that the user has access to. | |list_fsxn_svms | List all the storage virtual machines that the user access to. | |list_aws_subnets | List all the AWS subnets. | |list_aws_vpcs | List all the AWS VPCs. | -|delete_fsxn_filesystem | Deletes a specified FSx for ONTAP file system. Including all the SVMs and volumes on it. | -|delete_fsxn_svm | Deletes a specified SVM. Including all the volumes assigned to it. | -|delete_fsxn_volume | Deletes a specified volume. | +|set_fsxn_password | Sets the password for the fsxadmin user of a specified FSxN file system. | ## Author Information diff --git a/Management-Utilities/fsx-ontap-aws-cli-scripts/set_fsxn_password b/Management-Utilities/fsx-ontap-aws-cli-scripts/set_fsxn_password new file mode 100755 index 00000000..f9709217 --- /dev/null +++ b/Management-Utilities/fsx-ontap-aws-cli-scripts/set_fsxn_password @@ -0,0 +1,150 @@ +#!/bin/bash +################################################################################ +# +# This script is used to set the fsxadmin password for a FSxN file system. +################################################################################ + +################################################################################ +# This function just outputs the usage information and exits. +################################################################################ +usage () { +cat 1>&2 < /dev/null 2>&1; then + : +else + echo "Error, both the 'aws' and 'jq' commands are required to run this script." 1>&2 + exit 1 +fi +# +# Set some defaults. +region=$(aws configure get region) +waitForCompletion=false +maxIterations=24 +sleepTime=5 +# +# Process command line arguments. +while getopts "hwp:r:f:i:e:" option; do + case $option in + p) password=$OPTARG + ;; + r) region=$OPTARG + ;; + f) fileSystemName=$OPTARG + ;; + i) fsid=$OPTARG + ;; + w) waitForCompletion=true + ;; + e) endpoint_url="--endpoint-url $OPTARG" + ;; + *) usage + ;; + esac +done + +if [ ! -z "$fileSystemName" -a ! -z "$fsid" ]; then + echo "Error, you can only specify the -i OR the -f option. Not both." 1>&2 + usage +fi + +if [ -z "$password" ]; then + read -sp "Enter the password to set for the fsxadmin user: " password1 + echo "" + read -sp "Confirm the password: " password2 + echo "" + if [ "$password1" != "$password2" ]; then + echo "Error, the passwords do not match." 1>&2 + exit 1 + else + password=$password1 + fi +fi +# +# Ensure all the required parameters have been provided. +if [ -z "$password" -o -z "$fileSystemName" -a -z "$fsid" ]; then + echo "Error, missing required arguments." 1>&2 + usage +fi +# +# Get the file system id from the file system name. +if [ -z "$fsid" ]; then + fsid=$(aws fsx describe-file-systems $endpoint_url --region $region --output=json 2> /dev/null | jq -r ".FileSystems[] | if((.Tags[] | select(.Key == \"Name\") .Value) == \"${fileSystemName}\") then .FileSystemId else empty end" 2> /dev/null) +fi + +if [ -z "$fsid" ]; then + echo "Error, could not find the file system with name '${fileSystemName}' in region $region." 1>&2 + exit 1 +fi +# +# Set the password +aws fsx update-file-system $endpoint_url --ontap-configuration '{"FsxAdminPassword": "'$password'"}' --region=$region --file-system-id "$fsid" --output=json > $tmpout 2>&1 +if [ $? != "0" ]; then + echo "Failed to set the password." 1>&2 + cat $tmpout 1>&2 + exit 1 +else + status=$(jq -r .FileSystem.AdministrativeActions[0].Status $tmpout 2> /dev/null) + if [ "$status" == "PENDING" ]; then + printf "Password is being set." + # + # Wait for the svm to be deleted. + if [ $waitForCompletion == true ]; then + i=0 + while [ $i -lt $maxIterations ]; do + aws fsx describe-file-systems $endpoint_url --file-system-id $fsid --output=json --region=$region > $tmpout 2>&1 + if [ $? -eq 0 ]; then + # + # Assume the first administrative action is the one we just created. + status=$(jq -r .FileSystems[0].AdministrativeActions[0].Status $tmpout 2> /dev/null) + if [ "$status" == "COMPLETED" ]; then + printf "\nPassword has been set.\n" + break + fi + if [ "$status" != "PENDING" -a "$status" != "IN_PROGRESS" ]; then + printf "\nError, failed to set the password on '$fsid'. Status = $status\n" 1>&2 + if [ "$status" == "FAILED" ]; then + jq -r .FileSystems[0].AdministrativeActions[0].FailureDetails.Message $tmpout 1>&2 + fi + exit 1 + fi + else + printf "\nError, failed to get status of filesystem with ID '$fsid'.\n" 1>&2 + cat $tmpout 1>&2 + exit 1 + fi + printf "." + sleep $sleepTime + let i+=1 + done + if [ $i -ge $maxIterations ]; then + printf "\nFailed to set the password on file system id '$fsid'. Taking too long.\n" 1>&2 + exit 1 + fi + else + printf "\n" + fi + else + echo "Unknown status '$status'. Complete output returned from the AWS api:" 1>&2 + cat $tmpout 1>&2 + exit 1 + fi +fi +exit 0 From 17138c455094d3e06bd2adff1dd5fb10b26cffb2 Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Sat, 3 Oct 2026 09:24:21 -0500 Subject: [PATCH 09/23] Updated it to check the destinatio file system to see if the relationship exist. Added support for an S3 bucket to contain the partners and secrets information. Added a CloudFormation template. --- .../auto_create_sm_relationships/README.md | 259 +++- .../auto_create_sm_relationships.py | 827 ++++++++---- .../cloudformation.yaml | 1155 +++++++++++++++++ .../updateCFTemplate | 55 + 4 files changed, 1992 insertions(+), 304 deletions(-) create mode 100644 Management-Utilities/auto_create_sm_relationships/cloudformation.yaml create mode 100755 Management-Utilities/auto_create_sm_relationships/updateCFTemplate diff --git a/Management-Utilities/auto_create_sm_relationships/README.md b/Management-Utilities/auto_create_sm_relationships/README.md index 27e80f71..8d6cd611 100644 --- a/Management-Utilities/auto_create_sm_relationships/README.md +++ b/Management-Utilities/auto_create_sm_relationships/README.md @@ -1,75 +1,206 @@ # Automatically create SnapMirror relationships for FSxN file systems ## Introduction -This script is used to ensure that all your volumes, in all your FSxN file systems, are SnapMirror'ed to a remote FSxN file system. It does this by: +This program is used to ensure that all the volumes, in all of your FSxN file systems, +are SnapMirror'ed to a remote FSxN file system. It does this by: - Looping on all the AWS regions that support FSx. - For each region, it loops on all the FSxN file systems. -- For each FSxN file system, it obtains the list of the volumes it has using the ONTAP API. -- For each of the volumes it checks if the "snapmirror.destinations.is_ontap" field is set to false. Meaning it doesn't have an existing SnapMirror relationship. -- If it is set to false, then it checks from the AWS side to see if there is a "protect_volume" tag set to "skip". -- If there isn't one, then it uses the ONTAP SnapMirror API to create the SnapMirror relationship. This API will create a destination volume if it does not already exist. - -**NOTE:** This script only creates SnapMirror relationships for FlexVol volumes. It does not create SnapMirror relationships for FlexGroup volumes. - -## Set Up -There are a few things you need to do in order to get this script to run properly: - -- Set up a secret for each of the FSxN file systems in the AWS Secrets Manager. Each secret should have two "keys" (they can be named anything, since you set the name of the key the scretsTable defined below): - - username - set to the username you want the script to use when issuing API to the ONTAP system. - - password - set to the password of the username specified with the username key. -- Edit the top of the script and fill in a few variables: - - partnerTable - This table provides the association with a source FSxN file system to its partner cluster (i.e. where its volumes should be SnapMirror'ed to.) There should be five fields for each entry: - - fsxId - Set to the AWS ID of the FSx file system. - - svmName - Set to the SVM name on the FSxN file system. - - partnerFsxnIP - Set to the IP address of the management port of the partner FSxN file system. - - partnerSvmName - The name of the SVM where you want the SnapMirror destination volume to reside. - - partnerSvmSourceName - Is the "peered name" of the source SVM. Usually, it is the same as the source SVM, but can be different if that same name already exists on the partner file system. When you peer the SVM it will require you to create an alias for the source SVM so all the SVM names are unique. - - dynamodbPartnersTableName and dynamodbRegion - Instead of filling out the partnerTable in the source code, you can populate a DynamoDB table with the information and define these two variables to point to that table. Note that if both partnerTable and these two variables are defined, the partnerTable will be used. Also note that the required fields in the DynamoDB are a little different than the partnerTable. Here are the fields that should be defined in the DynamoDB table: - - soureceId - Which is the concatenation of the source file system ID followed by a ":" followed by the SVM name. It is done this way because the id has to be unique in the table. It is split up into its two components in the script when it is read in. - - partnerFsxnIp - Set to the IP address of the management port of the partner FSxN file system. - - partnerSvmName - The name of the SVM where you want the SnapMirror destination volume to reside. - - partnerSvmSourceName - Is the "peered name" of the source SVM. Usually, it is the same as the source SVM, but can be different if that same name already exists on the partner file system. When you peer the SVM it will require you to create an alias for the source SVM so all the SVM names are unique. - - secretsTable - This table provides the secret name, and username and password keys to use for each of the file systems. It should have 4 fields: - - fsxId - Set to the AWS File System ID. - - secretName - Set to the name of the secret created in step one. - - usernameKey - Set to the name of the key that holds the username. - - passwordKey - Set to the name of the key that holds the password. - - dynamodbSecretsTableName and dynamodbRegion - Instead of defining the secretsTable in the source code, you can populate a DynamoDB table with the information and define these variables to point to that table. The table should have the same fields as the secretsTable defined above - - secretsManagerRegion - Set to the region where the Secrets Manager has been set up. - - destinationVolumeSuffix - Set to the string you want appended to the source volume name to create the destination volume name. - - snapMirrorPolicy - Set to the Data ONTAP SnapMirror policy you want the assigned to the SnapMirror relationship. - - maxSnapMirrorRelationships - Set to the maximum number of SnapMirror relationship initializations you want this script to create in a single run. - - dryRun - If set to 'True' (case sensitive) the script will just show what it would have done, instead of actually creating the SnapMirror relationships. - - protectAll - If set to 'True' (case sensitive) the script will protect all volumes that don't have a "protect_volume" tag set to "skip". If set to 'False' it will only protect volumes that have a "protect_volume" tag set to "protect". - -- If you want to run this script as a Lambda program, then you'll need to: - - Create a role that has the following permissions: - - secretsmanager:GetSecretValue - - ec2:DescribeRegions - - fsx:DescribeFileSystems - - fsx:DescribeVolumes - - fsx:ListTagsForResources - - dynamodb:GetItem - Optional, only needed if you are using a DynamoDB table to access the secretsTable or partnerTable. - - - Create AWS endpoints for any services that it uses. Currently that is: - - ec2 - - fsx - - SecretsManager - - dynamodb - Optional, only needed if you are using a DynamoDB table to access the secretsTable or partnersTable. - -# Running The Script -To run the script on a Linux host, you just need to change the UNIX permissions on the file to be executable, then run it as a command: +- For each FSxN file system, obtain the list of the volumes it has. +- For each volumes it checks to see if it is: + - A RW type volume + - Not a clone volume + - Not a FlexCache volume + - Not already in an existing SnapMirror relationship + - Doesn't have an AWS tag with a key of `protect_volume` with a value of `skip`. +- If it passes all those check then it uses the ONTAP SnapMirror API to create the SnapMirror relationship. + +**NOTES** +- The ONTAP API creates the destination volume and will error out if the destination volume already exists. +- The program assumes that the cluster and vservers have already been peered. + +## Prerequisites +There are a couple of things you need to do in order to get this script to run properly: + +1. Set up an AWS SecretsManager secret for each of the FSxN file systems, both sources and destinations, +that the program is going to work with. Each secret should have two "keys" (they can be named anything, since you set the +name of the key in the `secretsTable` defined below): + + |Key|Description| + |-|-| + |username | Set to the username you want the script to use.| + |password | Set to the password of the username.| + +1. If you are planning on deploying as a Lambda function, or just want an easier way of maintaining the +associations of the source and destination SVMs and the file system to Secrets Manager secrets then you can +either create CSV (Comma Separated Values) files with the information and store them in an S3 bucket or +you can put the information into a DynamoDB table. The specific formats of the file and tables are listed +in the [Appendix](#appendix) section below. + +## Deployment Options + +There are two way to deploy the program: + +- As a Lambda function. +- As a standalone program running on a system that has Python installed + +### Deploying as a Lambda function +Note that since Lambda function has to communicate with the FSxN file systems it has to run +with a subnet of a VPC. Because of that it is requirement to pick a subnet that has network +connectivity to all the file systems, as well as the AWS service endpoints. Since accessing +AWS service endpoints can be problematic if the Lambda function is deployed +in a "Public subnet" (i.e. one with an Internet Gateway in it) it is highly recommended to +deploy it in private subnet that has access to the Internet, typically via a NAT gateway. + +To make it easy to deploy the program as a Lambda function a CloudFormation template has +been created. To use it just download the template file [found here](cloudformation.yaml) +and create a CloudFormation stack from the AWS console. + +Once you have loaded the file, it will prompt you for the following items: + +|Parameter|Required|Purpose| +|--------|--------|------| +|Stack Name| Yes | Provides a unique name for the stack. Note that this value is used as a suffix to a lot of the resources so it is recommended to keep to short.| +|Subnet IDs| Yes | The subnets that the Lambda function run from. It is recommended to select at least two. They must have connectivity to the FSxNs and be in private subnets (i.e. subnets without an Internet Gateway attached).| +|Security Group IDs | Yes | The AWS Security Group IDs to assign to the Lambda function. The only requirement is for outbound TCP port 443 to the FSxNs and to the AWS endpoints.| +|Check Interval | Yes | The number of hours between checks for new volumes to protect.| +|Secrets ARN pattern | No | An ARN pattern that will match all the secrets that the program will get values from to obtain the credentials for the FSxN file systems. This value is ignored if a role ARN is provided below.| +|Secrets Manager Region | Yes | This specifies the AWS region where all the secrets reside.| +|Role ARN | No | An ARN to a role to assign to the Lambda function. See the [Required Permissions](#required-permissions) section in the appendix for the required permissions.| +|Create Watchdog Alarm | No | If set to True a CloudWatch alarm will be created that will alert you if the Lambda function fails for any reason.| +|SNS Topic ARN | No | The ARN to an SNS topic that the CloudWatch Alarm will send alerts to.| +|Max Run Time | Yes | The maximum number of seconds to allow the program to run. Recommended to be at least 90 seconds if you want it to scan all regions. Must to be less than 900 (15 minutes).| +|Regions | No | Provide a comma separated list of regions to scan for FSxN file systems. If left blank it will scan all AWS regions.| +|FSx IDs | No | Provide a comma separated list of FSxN file systems for the program to protect. If left blank will attempt to protect all FSxNs in the regions that it is allowed to scan.| +|Destination Volume Suffix | No | The string to append to the volume name at the destination file system. Can be left blank to keep the names the same.| +|SnapMirror Policy | Yes | The SnapMirror Policy to assign to the SnapMirror relationship.| +|Schedule Name | No | The name of the update schedule to assign to the SnapMirror relationship.| +|Tiering Policy | Yes | The tiering policy to assign to the destination volume.| +|Maximum SnapMirror Relationships | Yes | The maximum number of SnapMirror relationships you want to allow the program to create on any give invocation. This is a safe guard to ensure it doesn't create too many at a time. To disable the safe guard, just set to a high number.| +|Protect All | Yes | Allows you to change the default behavior of creating SnapMirror relationships. If this parameter is set to `True` it will create SnapMirror relationships for all volumes that don't have a `protect_volume` tag set to `skip`. If set to `False` it will only creating SnapMirror relationships for volumes that have a `protect_volume` tag set to `protect`.| +|S3 Bucket Name | No | The S3 bucket that contains the files that have the list of secrets ARNs and svm to svm relationships.| +|S3 Bucket Region | No | The region where the S3 bucket resides.| +|Partners File | No | The file within the S3 bucket that contains the SVM to SVM relationship information. The format of this file is specified in the [S3 Bucket File formats](#s3-bucket-file-formats) section below.| +|Secrets File | No | The file within the S3 bucket that contains the secrets to file system associations. The format of this file is specified in the [S3 Bucket File formats](#s3-bucket-file-formats) section below.| +|DynamoDB Region | No | The region of the DynamoDB that contains the partners and secret tables. | +|Partners Table | No | The DynamoDB table that contains the SVM to SVM relationship information. The format of this table is specified in the [DynamoDB Table Structures](#dynamodb-table-structures) section below.| +|Secrets Table | No | The DynamoDB table that contains the secrets to file system associations. The format of this table is specified in the [DynamoDB Table Structures](#dynamodb-table-structures) section below.| + +Once you have filled in the values click on the `Next` button. From that page, just scroll to the bottom and click on the check box next to +the disclaimer acknowledging that the template might create a role. If you have provided a role ARN it will not create a role. Next, click +on the `Next` button. This last page just summarizes all the selections that have been made. Scroll to the bottom and click on the `Submit` button. + +Within a few minutes it should deploy the solution. Once it is done, click on the `Resources` tab and click on the Lambda function URL. That +should bring you to the Lambda function within the AWS console. On that page click on the `Test` tab and click on the `Test` button. This forces the program to run. +Once it is done it will display the output that came from the program, including any error messages. If there are error messages +try to resolve them. If you can't open an issue on this GitHub repository and someone will help you. + +### Deploying as a standalone program + +To use this program in a standalone program you'll need to ensure Python (version 3.12 or higher) is installed as +well as the 'boto3' Python package. Boto3 is the AWS SDK that allows the program to issue AWS APIs. The program +will need some AWS permissions to run. The list of them are in the [Required Permissions](#required-permissions) section below. You can either create a role with at least +these permissions and assign it to the EC2 instance that is used to run the program. Or, if not running from an +EC2 instance, download and install the aws cli and configure it with a access key and secret key. While a session +key would work, it would not be practical to have to update it everyday. + +Once you have given the program the permissions it needs you'll need to configure it. There are two ways to do so, +either by editing the top of the script itself, or by creating environment variables with the same name as the +global variables. The one exception to the global variable option is for the partners and secrets tables. For those +you'll need to either create a Python dictionary within the script that contains the information, or create a partners and secrets file +and put them into an S3 bucket. The format of the files are listed [Appendix](#appendix) below. Another option is to create tables +in a DynamoDB that contains the information. The structure of those tables are also listed in the [Appendix](#appendix) below. + +Here are the list of variable that you can set: + +|Parameter|Required|Purpose| +|--------|--------|------| +|s3BucketName | No | The S3 bucket that contains the files that have the list of secrets and svm to svm relationships.| +|s3BucketRegion | No | The region where the S3 bucket resides.| +|partnersConfigFile | No | The file within the S3 bucket that contains the SVM to SVM relationship information. The format of this file is specified in the [S3 Bucket File formats](#s3-bucket-file-formats) section below.| +|secretsConfigFile | No | The file within the S3 bucket that contains the secrets to file system associations. The format of this file is specified in the [S3 Bucket File formats](#s3-bucket-file-formats) section below.| +|dynamodbRegion | No | The region of the DynamoDB that contains the partners and secret tables. | +|dynamodbPartnersTableName | No | The DynamoDB table that contains the SVM to SVM relationship information. The format of this table is specified in the [DynamoDB Table Structures](#dynamodb-table-structures) section below.| +|dynamodbSecretsTableName | No | The DynamoDB table that contains the secrets to file system associations. The format of this table is specified in the [DynamoDB Table Structures](#dynamodb-table-structures) section below.| +|secretsManagerRegion | Yes | This specifies the AWS region where all the secrets reside.| +|regions | No | Provide a comma separated list of regions to scan for FSxN file systems. If left blank it will scan all AWS regions.| +|FsxIds | No | Provide a comma separated list of FSxN file systems for the program to protect. If left blank will attempt to protect all FSxNs in the regions that it is allowed to scan.| +|destinationVolumeSuffix | No | The string to append to the volume name at the destination file system. Can be left blank to keep the names the same.| +|snapMirrorPolicy | Yes | The SnapMirror Policy to assign to the SnapMirror relationship.| +|scheduleName | No | The name of the update schedule to assign to the SnapMirror relationship.| +|tieringPolicy | Yes | The tieting policy to assign to the destination volume.| +|maxSnapMirrorRelationships | Yes | The maximum number of SnapMirror relationships you want to allow the program to create on any give invocation. This is a safe guard to ensure it doesn't create too many at a time. To disable, just set it to a high number.| +|protectAll | Yes | Allows you to change the default behavior of creating SnapMirror relationships. If this parameter is set to `True` it will create SnapMirror relationships for all volumes that don't have a `protect_volume` tag set to `skip`. If set to `False` it will only creating SnapMirror relationships for volumes that have a `protect_volume` tag set to `protect`.| +|dryRun | No | If set to True it will not create any SnapMirror relationships, but instead just output a message saying what it would have done. Note this variable can only be set within the program, not via an environment variable.| + +Once you have filled in the variables you can run the script by just passing it as an argument to the python program: ``` -chmod +x auto_creaate_sm_relationships - ./auto_create_sm_relationships +python3 auto_create_sm_relationships.py ``` -To run it as a Lambda function you will need to: -- Create the Lambda function with a Python runtime, from scratch, and paste the program into code box and save it. -- Associate the role created above with the Lambda function. -- Create the AWS service endpoints mentioned above. -- Adjust the default timeout from 4 seconds to at least 60 seconds. -- Once you have tested that it run successfully, creating an eventBridge that will trigger it to run on a regular basis (e.g. once or twice a day). + +## Appendix + +### DynamoDB Table Structures +#### Partners Table +This table provides the association with a source FSxN file system to its partner cluster (i.e. where its volumes should be SnapMirror'ed to.) There should be five fields for each entry: +- soureceId - Which is the concatenation of the source file system ID followed by a ":" followed by the SVM name. It is done this way because the id has to be unique in the table. It is split up into its two components in the script when it is read in. +- partnerFsxnId - Set to the AWS ID the partner FSxN file system. +- partnerFsxnIp - Set to the IP address of the management port of the partner FSxN file system. +- partnerSvmName - The name of the SVM where you want the SnapMirror destination volume to reside. +- partnerSvmSourceName - Is the "local name" of the source SVM. Usually, it is the same as the source SVM, but can be different if that same name already exists on the partner file system. When you peer the SVM it will require you to create an alias for the source SVM so all the SVM names are unique. + +#### Secrets Table +This table provides the secret name, and username and password keys to use for each of the file systems. It should have 4 fields: +- fsxId - Set to the AWS File System ID. +- secretName - Set to the name of the secret created in step one. +- usernameKey - Set to the name of the key that holds the username. +- passwordKey - Set to the name of the key that holds the password. + +### S3 Bucket File formats +#### Partners file +The partners file is a comma separated values file with the following fields: +- FSxN Id - Set to the AWS file system ID. +- SVM Name - Set to the SVM name. +- Partner FSxN ID - Set to the AWS ID the partner FSxN file system. +- Partner FSxN IP - Set to the IP address of the management port of the partner FSxN file system. +- Partner SVM Name - The name of the SVM where you want the SnapMirror destination volume to reside. +- Partner SVM Source Name - Is the "local name" of the source SVM. Usually, it is the same as the source SVM, but can be different if that same name already exists on the partner file system. When you peer the SVM it will require you to create an alias for the source SVM so all the SVM names are unique. + +Example File: +``` +fs-04f0ebeeXXXXXXXXX,fsx,fs-07bcb7adXXXXXXXXX,198.19.254.83,fsx,prod-fsx +fs-04f0ebeeXXXXXXXXX,smb_svm,fs-07bcb7adXXXXXXXXX,198.19.254.83,fsx,smb_svm +``` + +#### Secrets file +The secret file is a comma separated values file with the following fields: +- fsxId - Set to the AWS File System ID. +- Secret Name - Set to the name of the secret created in step one. +- Username Key - Set to the name of the key that holds the username. +- Password Key - Set to the name of the key that holds the password. + +Example File: +``` +fs-04f0ebeeXXXXXXXXX,FSxSecret-Default,username,password +fs-07bcb7adXXXXXXXXX,FSxSecret-Default,username,password +``` + +### Required permissions +Here are the permissions that are required for the program to run: + +|Action|Resource|Description| +|-|-| +|fsx:DescribeFileSystems| `*` | Allows the program to discovery the file systems.| +|fsx:DescribeVolumes| `*` | Allows the program to discovery the volumes within the file systems.| +|tag:GetResources|`*` | Allows the program to get tag information.| +|ec2:DescribeRegions| `*` | Allow it to discovery all the regions.| +|secretsmanager:GetSecretValue| pattern that matches the required secrets | Allows the program to obtain the credentials for the file systems.| +|s3:GetObject| ARN of the bucket to get the partners and secrets files from | Not required if you aren't using an S3 bucket to store the partners and secrets ARN information.| +|s3:ListBucket| ARN of the bucket to get the partners and secrets files from | Not required if you aren't using an S3 bucket to store the partners and secrets ARN information.| +|dynamodb:DescribeTable | ARN of the DynamoDB tables that holds the partner and secrets ARN information.| Allows the program to access the table. | +|dynamodb:Scan | ARN of the DynamoDB tables that holds the partner and secrets ARN information.| Allows the program to get the contents from the table. | + +If running as a Lambda function you will also need to attached an AWS Managed Policy named `AWSLambdaVPCAccessExecutionRole`. This managed policy is required for a function that is to run within a VPC. ## Author Information diff --git a/Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py b/Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py index 48287107..4d801287 100755 --- a/Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py +++ b/Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py @@ -1,113 +1,179 @@ -#!/usr/bin/python3 -################################################################################ -# THIS SOFTWARE IS PROVIDED BY NETAPP "AS IS" AND ANY EXPRESS OR IMPLIED -# WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF -# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO -# EVENT SHALL NETAPP BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, -# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, -# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; -# OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, -# WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR' -# OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF -# ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -################################################################################ +#!/bin/python3 # ################################################################################ -# This program is used to create SnapMirror relationships for all RW volumes -# it finds on any FSxN File System that the user running the program has -# access to. It does this by: +# This program is used to create SnapMirror relationships for all RW, +# non-clone, non-FlexCache volumes it finds on any FSxN File System that the +# user (or Lambda function) running the program has access to. +# +# It does this by: # o Looping on all the regions. Skipping on any that don't support FSx. # o Looping on all the FSx file systems. Skipping any non-FSxN file systems. # o Obtaining information on all the volumes from the FSxN file system -# using the ONTAP API. This data includes whether the volumes has an -# existing SnapMirror relationship. This only is reliable for flexvols and -# not flexgroups. +# using the ONTAP API. # o If a SnapMirror relationship doesn't already exist, and the volume on the # AWS side doesn't have a "protect_volume" tag set to "skip", then it # creates a SnapMirror relationship using the partner information # provided below. +# +# Version: %%VERSION%% +# Date: %%DATE%% +# ################################################################################ # -# Create a table of source FSxN IDs and SVMs and its partner (destination) -# cluster and SVM. How to fill in the table should be pretty obvious except -# for the "partnerSvmSourceName". It is the label for the source SVM at the -# destination cluster. Usually, the SVM name is the same as the source SVM name -# unless there is a name conflict with SVM name at the destination, in which -# case an "alias" is has to be created when you peer the SVMs (a.k.a. vservers). -# In this case partnerSvmSourceName should be set to that alias, otherwise -# it should just be set the same as the svmName. It must be set though. -# -# You can either define an array named "partnersTable", like the one defined -# below, or define dynamodbPartnersTableName that specifies a DynamoDB table -# to scan to get the information. The DynamoDB table should have the -# following attributes: -# sourceId - Which is the concatentation of the source file system ID -# followed by a ":" followed by the SVM name. It is done this -# way because the id has to be unique in the table. -# partnerFsxnIp - The IP address of the destination cluster. -# partnerSvmName - The name of the SVM at the destination cluster. -# partnerSvmSourceName - The name of the SVM at the destination cluster that -# will be the source of the SnapMirror relationship. -partnersTable = [ - { - 'fsxId': 'fs-0e8d9172XXXXXXXXX', - 'svmName': 'fsx', - 'partnerFsxnIp': '198.19.253.1', - 'partnerSvmName': 'fsx', - 'partnerSvmSourceName': 'fsx_source' - }, - { - 'fsxId': 'fs-0e8d9172XXXXXXXXX', - 'svmName': 'fsx_smb', - 'partnerFsxnIp': '198.19.253.1', - 'partnerSvmName': 'fsx', - 'partnerSvmSourceName': 'fsx_smb' - }, - { - 'fsxId': 'fs-020de268XXXXXXXXX', - 'svmName': 'fsx', - 'partnerFsxnIp': '198.19.255.1', - 'partnerSvmName': 'fsx', - 'partnerSvmSourceName': 'fsx_dest' - }, - ] +# The program needs a list of source SVMs to partner associations to know where +# to create the SnapMirror relationships. That information is conveyed via +# the 'partnersTable' variable. The values of each of the fields in the table +# There are three ways to provide the partner information to the program. +# You can either: +# 1) Define an array named "partnersTable" in this script +# 2) Define a DynamoDB table to scan for the information +# 3) Define an S3 bucket and file to fetch the information from. +# +# If you want to define a variable in the program, then name needs to be +# partnersTable. It should be an array of dictionaries, where each dictionary +# has these fields: +# fsxId - The AWS ID of the source cluster. +# svmName - The name of the SVM at the source cluster. +# partnerFsxId - The AWS ID of the destination cluster. +# partnerFsxnIp - The IP address of the destination cluster. +# partnerSvmName - The name of the SVM at the destination cluster. +# partnerSvmSourceName - The name of the source SVM at the destination cluster. +# +# All of those should be pretty straight forward except for the "partnerSvmSourceName". +# That is the "local name" for the source SVM at the destination cluster. Usually the +# local name is the same as the source SVM name unless the destination cluster +# already has an SVM with the same name. In that case a "local name" has to be +# assigned when you peer the SVMs (a.k.a. vservers) so ONTAP can distinguish them +# apart. To be able to handle that situation the partnerSvmSourceName should +# be set to that local name even if it is the same as the source SVM name. +# +# Here is an example of the partnersTable array: +# partnersTable = [ +# { +# "fsxId": "fs-04f0ebeXXXXXXXXXX", +# "svmName": "fsx", +# "partnerFsxId": "fs-07bcb7aXXXXXXXXXX", +# "partnerFsxnIp": "198.19.254.83", +# "partnerSvmName": "fsx", +# "partnerSvmSourceName": "prod-fsx" +# }, +# { +# "fsxId": "fs-04f0ebeXXXXXXXXXX", +# "svmName": "smb_svm", +# "partnerFsxId": "fs-07bcb7aXXXXXXXXXX", +# "partnerFsxnIp": "198.19.254.83", +# "partnerSvmName": "fsx", +# "partnerSvmSourceName": "smb_svm" +# } +# ] +# +# -------- +# +# If you want to use a DynamoDB table, the program expect the following +# attributes for each entry: +# sourceId - Which is the concatenation of the source file system ID +# followed by a ":" followed by the SVM name. It is done this +# way because the id has to be unique in the table. +# partnerFsxId - The AWS ID of the destination cluster. +# partnerFsxnIp - The IP address of the destination cluster. +# partnerSvmName - The name of the SVM at the destination cluster. +# partnerSvmSourceName - The name of the source SVM at the destination cluster. +# +# You should set the following variables to instruct the program to use +# the DynamoDB table for the partner information: +# dynamodbRegion="us-west-2" +dynamodbRegion="" +dynamodbPartnersTableName="" +# +# Of course adjust for your region and table name. +# +# -------- +# +# If you want to use a file in an S3 bucket, the format of file should be a +# comma separated list of the following fields: +# +# fsxId,svmName,partnerFsxId,partnerFsxnIp,partnerSvmName,partnerSvmSourceName # -# If you don't want to define the partnersTable in this script, you can -# define the following variables to use a DynamoDB table to get the -# partner information. +# Here are the variable to set to get the program to read a file in an S3 +# bucket for the partner information. +s3BucketName="" +s3BucketRegion="" +partnersConfigFile="" # -# NOTE: If both the partnersTable and dynamodbPartnersTableName are defined, -# the partnersTable will be used. -#dynamodbRegion="us-west-2" -#dynamodbPartnersTableName="fsx_partners" +# Of course adjust for your bucket name, region and file name. # -# Create a table of secret names and keys for the username and password for each of the FSxIds. -# You can either define an array named "secretsTable", like below or define -# dynamodbSecretsTableName, that will specify a DynamoDB table to use. It should have the -# following attributes: +# NOTE: If both the DynamoDB table and S3 bucket variables are defined, the +# DynamoDB table will be used and if either are defined, the will be used +# instead of the global variable "partnersTable" defined in this script. +# +################################################################################ +# +# The program needs credentials to use to authenticate with the FSxNs with. +# There are three ways you can conveyed this information to the program. You can +# either: +# 1) Define an array named "secretsTable" in this script +# 2) Define a DynamoDB table to scan for the information +# 3) Define an S3 bucket and file to fetch the information from. +# +# If you want to define a variable in the program, then name needs to be +# secretsTable. It should be an array of dictionaries, where each dictionary +# has these fields: +# fsxId - The AWS ID of the source cluster. +# secretName - The name of the Amazon SecretsManager secret that holds the username and password keys. +# usernameKey - The name of the key that holds the username to use. +# passwordKey - The name of the key that holds the password to use. +# +# Here is an example of the secretsTable array: +# secretsTable = [ +# {"fsxId": "fs-04f0ebeXXXXXXXXXX", "secretName": "FSxSecret-Default", "usernameKey": "username", "passwordKey": "password"}, +# {"fsxId": "fs-07bcb7aXXXXXXXXXX", "secretName": "FSxSecret-Default", "usernameKey": "username", "passwordKey": "password"} +# ] +# +# If you want to use a DynamoDB table, the program expect the following +# attributes for each entry: # fsxId - The file system ID -# SecretName - The name of the Amazon SecretManger secret that holds the username and password keys. +# SecretName - The name of the Amazon SecretsManager secret that holds the username and password keys. # usernameKey - The name of the key that holds the username to use. # passwordKey - The name of the key that holds the password to use. # -secretsTable = [ - {"fsxId": "fs-0e8d9172XXXXXXXXX", "secretName": "fsxn-credentials", "usernameKey": "fsxn-username", "passwordKey": "fsxn-password"}, - {"fsxId": "fs-020de268XXXXXXXXX", "secretName": "fsxn-credentials", "usernameKey": "fsxn-username", "passwordKey": "fsxn-password"}, - {"fsxId": "fs-07bcb7adXXXXXXXXX", "secretName": "fsxn-credentials", "usernameKey": "fsxn-username", "passwordKey": "fsxn-password"}, - {"fsxId": "fs-077b5ff4XXXXXXXXX", "secretName": "fsxn-credentials", "usernameKey": "fsxn-username", "passwordKey": "fsxn-password"} - ] +# You should set the following variables to instruct the program to use +# the DynamoDB table for the secrets information. +# dynamodbRegion="" # Should be defined above for the partners information. +dynamodbSecretsTableName="" +# +# ------ +# +# If you want to use a file in an S3 bucket, the format of file should be a +# comma separated list of the following fields: +# fsxId,secretName,usernameKey,passwordKey # -# If you don't want to define the secretsTable in this script, you can -# define the following variables to use a DynamoDB table to get the -# secret information. +# You'll need to set the following variables to get the program to read a file in an S3 +# bucket for the secrets information. +# s3BucketName="" # Should be defined above for the partners information. +# s3Region="" # Should be defined above for the partners information. +secretsConfigFile="" # -# NOTE: If both the secretsTable, and dynamodbSecretsTableName are defined, -# the secretsTable will be used. -#dynamodbRegion="us-west-2" -#dynamodbSecretsTableName="fsx_secrets" +# Of course adjust for your bucket name, region and file name. # +# NOTE: If both the DynamoDB table and S3 bucket variables are defined, the +# DynamoDB table will be used and if either are defined, they will be used +# instead of the global variable "secretsTable" defined in this script. +# +# ------ +# +# If you want to limit the regions that are scanned, you can define them here. +# The expected format is a comma separated list of region names. +# If you don't define it then all regions will be scanned. +# regions="us-west-2,us-east-1" +regions="" +# +# If you want to limit the FSxN IDs to look for source volumes from, you +# an define them here. Expected format is a comma separated list of FSxN IDs. +# fsxIds="fs-04f0ebeeXXXXXXXXX,fs-07bcbXXXXXXXXXXXX" +fsxIds="" +# # Provide the region the secrets manager resides in: -secretsManagerRegion='us-west-2' +secretsManagerRegion="us-west-2" # # Set the suffix string to append to the destination volume. destinationVolumeSuffix="_dp" @@ -120,21 +186,22 @@ scheduleName="hourly" # # Set the Tiering policy for the destination volume if it is created. +# Supported values are: "all", "none", "auto", "snapshot-only". tieringPolicy="all" # # Set to the maximum number of SnapMirror relationships to create during -# a signle run. -maxSnapMirrorRelationships=10 -# -# Set the following to 'True' (case sensitive) to have the program just -# show what it would have done and not really perform any actions. -dryRun=True +# a single run. +maxSnapMirrorRelationships=20 # # Set the following to 'True' (case sensitive) to have the program protect # all volumes that don't have a "protect_volume" tag set to "skip". Or, set # it to 'False' to only protect volumes that have a "protect_volume" tag # set to "protect". -protectAll=False +protectAll=True +# +# Set the following to 'True' (case sensitive) to have the program just +# show what it would have done instead of really doing it. +dryRun=False ################################################################################ # !!!!!!!! You shouldn't have to modify anything below here. !!!!!!!!!!!!!!!!!!! @@ -146,10 +213,14 @@ import urllib3 from urllib3.util import Retry import logging +import botocore +from botocore.config import Config import boto3 # +################################################################################ # Define a custom exception so we can gracefully exit the program if too many # snapmirror relationships have been created. +################################################################################ class TooManySMs(Exception): # Constructor or Initializer @@ -160,19 +231,19 @@ def __init__(self, value): def __str__(self): return(repr(self.value)) - ################################################################################ # This function returns the value assigned to the "protect_volume" tag # associated with the ARN passed in. If none is found, it returns an empty # string. ################################################################################ -def getVolumeProtectTagValue(fsxClient, arn): +def getVolumeProtectTagValue(tags, arn): + + for resource in tags: + if resource['ResourceARN'] == arn: + for tag in resource['Tags']: + if tag['Key'].lower() == "protect_volume": + return tag['Value'].lower() - if not arn == "": - tags = fsxClient.list_tags_for_resource(ResourceARN=arn) - for tag in tags['Tags']: - if(tag['Key'].lower() == "protect_volume"): - return(tag['Value'].lower()) return("") ################################################################################ @@ -184,7 +255,7 @@ def getVolumeARN(awsVolumes, volumeUUID): global logger for awsVolume in awsVolumes: - if awsVolume['OntapConfiguration']['UUID'] == volumeUUID: + if awsVolume.get('OntapConfiguration') is not None and awsVolume['OntapConfiguration'].get('UUID') == volumeUUID: return(awsVolume['ResourceARN']) logger.warning(f'Failed to get ARN for volume with UUID={volumeUUID}.') return("") @@ -192,32 +263,108 @@ def getVolumeARN(awsVolumes, volumeUUID): ################################################################################ # This function is used to obtain the username and password from AWS's Secrets # Manager for the fsxnId passed in. It returns empty strings if it can't -# find the credentials. +# find the credentials. It creates a cache of the responses so that it doesn't +# have to call Secrets Manager for the same fsxnId multiple times. ################################################################################ +credentialsCache = {} def getCredentials(fsxnId): global secretsManagerClient, secretsTable + if credentialsCache.get(fsxnId) is not None: + return (credentialsCache[fsxnId]['username'], credentialsCache[fsxnId]['password']) + for secretItem in secretsTable: if secretItem['fsxId'] == fsxnId: secretsInfo = secretsManagerClient.get_secret_value(SecretId=secretItem['secretName']) secrets = json.loads(secretsInfo['SecretString']) username = secrets[secretItem['usernameKey']] password = secrets[secretItem['passwordKey']] + credentialsCache[fsxnId] = {'username': username, 'password': password} return (username, password) return ("", "") ################################################################################ -# Since the ONTAP SnapMirror API does not work with FlexGroups, this function -# was created as a place holder for coming up with an alternative way to -# protect them. +# This function returns the partner information for the sources FSxN and +# SVM name passed in. ################################################################################ -def protectFlexGroup(fsxId, svmName, volumeName): +def getPartnerInfo(fsxId, svmName): - global logger, http, numSnapMirrorRelationships + partnerId = "" + partnerIp = "" + partnerSvmName = "" + partnerSvmSourceName = "" - logger.warning('Unfortunately, creating snapmirror relationships for FlexGroups is currently not supported.') - return + for fsx in partnersTable: + if fsx['fsxId'] == fsxId and fsx['svmName'] == svmName: + partnerId = fsx['partnerFsxId'] + partnerIp = fsx['partnerFsxnIp'] + partnerSvmName = fsx['partnerSvmName'] + partnerSvmSourceName = fsx['partnerSvmSourceName'] + break + + return (partnerId, partnerIp, partnerSvmName, partnerSvmSourceName) + +################################################################################ +# The following function will check to see if a SnapMirror relationship already +# exists for the source and destination volumes passed in. It returns True if +# it exists, False otherwise. +# +# It caches the snapmirror relationships for each destination cluster so it +# doesn't have to call the API for each volume. +################################################################################ +snapMirrorRelationshipsCache = {} +def checkSnapMirrorRelationshipExists(sourceSVM, sourceVolume, destinationFsId, destinationFsIp, destinationSVM, destinationVolume): + + if snapMirrorRelationshipsCache.get(destinationFsId) is None: + snapMirrorRelationshipsCache[destinationFsId] = getSnapMirrorRelationships(destinationFsId, destinationFsIp) + + for relationship in snapMirrorRelationshipsCache[destinationFsId]: + if (relationship['source']['path'] == f'{sourceSVM}:{sourceVolume}' and + relationship['destination']['path'] == f'{destinationSVM}:{destinationVolume}'): + return True + + return False + +################################################################################ +# This function gets all the snapmirror relationships for the fsxnIp passed in. +################################################################################ +def getSnapMirrorRelationships(fsxnId, fsxnIp): + + global logger, http + + relationships = [] + + try: + (username, password) = getCredentials(fsxnId) + if username == "" or password == "": + logger.error(f'No credentials for FSxN ID: {fsxnId}.') + return relationships + + auth = urllib3.make_headers(basic_auth=f'{username}:{password}') + headers = { **auth } + + endpoint = f'https://{fsxnIp}/api/snapmirror/relationships?fields=source,destination' + logger.debug(f'Trying {endpoint}.') + response = http.request('GET', endpoint, headers=headers) + if response.status < 200 or response.status > 299: + logger.error(f'API call to {endpoint} failed. HTTP status code: {response.status}.') + return relationships + body = json.loads(response.data.decode('utf-8')) + relationships += body['records'] + while body.get('next'): + endpoint = f'https://{fsxnIp}{body["next"]}' + logger.debug(f'Trying {endpoint}.') + response = http.request('GET', endpoint, headers=headers) + if response.status < 200 or response.status > 299: + logger.error(f'API call to {endpoint} failed. HTTP status code: {response.status}.') + return relationships + body = json.loads(response.data.decode('utf-8')) + relationships += body['records'] + except Exception as err: + logger.critical(f'API against {fsxnIp} failed. The error returned: "{err}".') + + return relationships ################################################################################ # This function is used to setup a snapmirror relationship for the source @@ -225,62 +372,74 @@ def protectFlexGroup(fsxId, svmName, volumeName): # capabilities of the snapmirror API which will create the destination volume # with the same name as the source volume with a suffix appended to it. # The suffix is defined above. -# -# NOTE: This program does not check that the Snapmirror relationship is -# successfully created mostly because all the API does is queue up a job -# that should create the relationship. ################################################################################ -def protectVolume(fsxId, svmName, volumeName): +def protectVolume(fsxId, svmName, volumeName, partnerId, partnerIp, partnerSvmName, partnerSvmSourceName): - global logger, http, numSnapMirrorRelationships, scheduleName, partnersTable + global logger, http, numSnapMirrorRelationships, dryRun # - # find the partner cluster management IP and svm for the source fsxId and svm. - partnerIp = "" - for fsx in partnersTable: - if fsx['fsxId'] == fsxId and fsx['svmName'] == svmName: - partnerIp = fsx['partnerFsxnIp'] - partnerSvmName = fsx['partnerSvmName'] - partnerSvmSourceName = fsx['partnerSvmSourceName'] - break - - if partnerIp == "": - logger.error(f'No partner found for fsxId: {fsxId} and svmName: {svmName}.') + # First check to see if the relationship already exists. + if checkSnapMirrorRelationshipExists(partnerSvmSourceName, volumeName, partnerId, partnerIp, partnerSvmName, f'{volumeName}{config["destinationVolumeSuffix"]}'): + logger.debug(f'SnapMirror relationship for {fsxId}::{svmName}:{volumeName} to {partnerIp}::{partnerSvmName}:{volumeName}{config["destinationVolumeSuffix"]} already exists.') return - - (username, password) = getCredentials(fsxId) + # + # Create the relationship. + (username, password) = getCredentials(partnerId) if username == "" or password == "": - logger.error(f'No credentials for FSxN ID: {fsxId}.') + logger.error(f'No credentials for FSxN ID: {partnerId}.') return - auth = urllib3.make_headers(basic_auth=f'{username}:{password}') headers = { **auth } - try: - endpoint = f'https://{partnerIp}/api/snapmirror/relationships/' - data = {"source": {"path": f"{partnerSvmSourceName}:{volumeName}"}, - "destination": {"path": f"{partnerSvmName}:{volumeName}{destinationVolumeSuffix}"}, - "create_destination": {"enabled" : True, "tiering": {"supported": True, "policy": tieringPolicy}}, - "state": "snapmirrored", - "policy": snapMirrorPolicy} - # - # To be safe, check that the variable exist, since it might have been - # commented out above. - if not 'scheduleName' in globals(): - scheduleName = "" + data = {"source": {"path": f"{partnerSvmSourceName}:{volumeName}"}, + "destination": {"path": f"{partnerSvmName}:{volumeName}{config['destinationVolumeSuffix']}"}, + "create_destination": {"enabled" : True, "tiering": {"supported": True, "policy": config['tieringPolicy']}}, + "state": "snapmirrored", + "policy": config['snapMirrorPolicy']} + if config['scheduleName'] is not None: + data["transfer_schedule"] = {"name": config['scheduleName']} - if scheduleName != "": - data["transfer_schedule"] = {"name": scheduleName} - - logger.debug(f'Trying {endpoint} with {data}.') + try: if not dryRun: + endpoint = f'https://{partnerIp}/api/snapmirror/relationships' + logger.debug(f'Trying {endpoint} with {data}.') response = http.request('POST', endpoint, headers=headers, body=json.dumps(data)) if response.status < 200 or response.status > 299: - logger.error(f'API call to {endpoint} failed. HTTP status code: {response.status}.') + errMessage = "N/A" + if response.data is not None: + errMessage = response.data.decode("utf-8") + logger.error(f'API call to {endpoint} failed. HTTP status code: {response.status}. Response from the API: {errMessage}') return - logger.info(f'Path {fsxId}::{svmName}:{volumeName} is being SnapMirrored to {partnerIp}::{partnerSvmName}:{volumeName}{destinationVolumeSuffix}.') + logger.info(f'Path {fsxId}::{svmName}:{volumeName} is being SnapMirrored to {partnerId}::{partnerSvmName}:{volumeName}{config["destinationVolumeSuffix"]}.') + # + # Now check that it worked. + body = json.loads(response.data.decode('utf-8')) + jobId = body['job']['uuid'] + state = "unknown" + failsafeCounter = 0 + while state != "success" and state != "failure" and failsafeCounter < 60: + time.sleep(1) + endpoint = f'https://{partnerIp}/api/cluster/jobs/{jobId}' + logger.debug(f'Trying {endpoint}.') + response = http.request('GET', endpoint, headers=headers) + if response.status < 200 or response.status > 299: + logger.error(f'API call to {endpoint} failed. HTTP status code: {response.status}.') + return + body = json.loads(response.data.decode('utf-8')) + state = body['state'] + failsafeCounter += 1 + + if state == "success": + logger.info(f'SnapMirror relationship for {fsxId}::{svmName}:{volumeName} to {partnerIp}::{partnerSvmName}:{volumeName}{config["destinationVolumeSuffix"]} was successfully created.') + numSnapMirrorRelationships += 1 + else: + if body.get("error") is not None and body["error"].get("message") is not None: + errMessage = body["error"]["message"] + else: + errMessage = "N/A" + logger.error(f'SnapMirror relationship for {fsxId}::{svmName}:{volumeName} to {partnerIp}::{partnerSvmName}:{volumeName}{config["destinationVolumeSuffix"]} failed to be created. Last state was: {state}. Erorr message returned by the API:\n{errMessage}') else: - logger.info(f'Path {fsxId}::{svmName}:{volumeName} would have been SnapMirrored to {partnerIp}::{partnerSvmName}:{volumeName}{destinationVolumeSuffix}.') - numSnapMirrorRelationships += 1 # pylint: disable=E0602 + logger.info(f'Path {fsxId}::{svmName}:{volumeName} would have been SnapMirrored to {partnerId}::{partnerSvmName}:{volumeName}{config["destinationVolumeSuffix"]}.') + numSnapMirrorRelationships += 1 except Exception as err: logger.critical(f'API against {partnerIp} failed. Volume not protected. The error returned: "{err}".') return @@ -299,89 +458,232 @@ def getOntapVolumes(fsxId, fsxnIp): return([]) auth = urllib3.make_headers(basic_auth=f'{username}:{password}') headers = { **auth } - + + volumes = [] try: - endpoint = f'https://{fsxnIp}/api/storage/volumes?fields=*' + endpoint = f'https://{fsxnIp}/api/storage/volumes?fields=name,svm,type,clone,flexcache_endpoint_type' logger.debug(f'Trying {endpoint}.') response = http.request('GET', endpoint, headers=headers, timeout=5.0) if response.status == 200: data = json.loads(response.data) - return(data['records']) + volumes = data['records'] else: logger.error(f'API call to {endpoint} failed. HTTP status code: {response.status}.') + return(volumes) + + while data.get('next'): + endpoint = f'https://{fsxnIp}{data["next"]}' + logger.debug(f'Trying {endpoint}.') + response = http.request('GET', endpoint, headers=headers, timeout=5.0) + if response.status == 200: + data = json.loads(response.data) + volumes += data['records'] + else: + logger.error(f'API call to {endpoint} failed. HTTP status code: {response.status}.') + break + except Exception as err: logger.critical(f'Failed to issue API against {fsxnIp}. Cluster could be down. The error messages received: "{err}".') - return([]) + return(volumes) + +################################################################################ +# This function is used to consolidate all the configuration values into a +# single dictionary. +################################################################################ +def readinConfig(): + + global config, boto3Config, logger, config, secretsTable, partnersTable + + config = { + "regions": None, + "fsxIds": None, + "partnersConfigFile": None, + "secretsConfigFile": None, + "secretsManagerRegion": None, + "s3BucketName": None, + "s3BucketRegion": None, + "dynamodbSecretsTableName": None, + "dynamodbPartnersTableName": None, + "dynamodbRegion": None, + "destinationVolumeSuffix": "_dp", + "snapMirrorPolicy": "MirrorAllSnapshots", + "scheduleName": None, + "tieringPolicy": "all", + "maxSnapMirrorRelationships": 10, + "protectAll": True + } + # + # Get the values for the variables giving the environment variables the + # highest priority, then the global variables, and finally the default values. + for var in config: + if os.environ.get(var) is not None: + config[var] = os.environ.get(var) + elif var in globals() and globals()[var] is not None: + config[var] = globals()[var] + # + # Since the CloudFormation template will set the environment variables + # to an empty string if someone doesn't provide a value, reset the + # values back to None. + for var in config: + if config[var] == "": + config[var] = None + # + # Convert the logical string to a logic. + if config.get('protectAll') is not None and type(config['protectAll']) == str and config['protectAll'].lower() == "false": + config['protectAll'] = False + else: + config['protectAll'] = True + # + # If the s3BucketName is set, then open a client to the s3 service and read + # in the partners and secrets config files. + if config['s3BucketName'] is not None: + # + # Open a client to the s3 service. + s3Client = boto3.client('s3', region_name=config['s3BucketRegion'], config=boto3Config) + if config['partnersConfigFile'] is not None: + partnersTable = [] + try: + response = s3Client.get_object(Bucket=config['s3BucketName'], Key=config['partnersConfigFile']) + partnerContents = response['Body'].read().decode('utf-8') + for partnerLine in partnerContents.splitlines(): + if partnerLine.strip() == "": + continue + if partnerLine.startswith("#"): + continue + (fsxId, svmName, partnerFsxId, partnerFsxnIp, partnerSvnName, partnerSvmSourceName) = partnerLine.split(",") + partnersTable.append({ + "fsxId": fsxId, + "svmName": svmName, + "partnerFsxId": partnerFsxId, + "partnerFsxnIp": partnerFsxnIp, + "partnerSvmName": partnerSvnName, + "partnerSvmSourceName": partnerSvmSourceName + }) + except Exception as err: + message = f'Unable to read the partners config file {config["partnersConfigFile"]} from bucket {config["s3BucketName"]}. Error message: "{err}".' + logger.critical(message) + raise Exception(message) + + if config['secretsConfigFile'] is not None: + secretsTable = [] + try: + response = s3Client.get_object(Bucket=config['s3BucketName'], Key=config['secretsConfigFile']) + secretsContents = response['Body'].read().decode('utf-8') + for secretsLine in secretsContents.splitlines(): + if secretsLine.strip() == "": + continue + if secretsLine.startswith("#"): + continue + (fsxId, secretName, usernameKey, passwordKey) = secretsLine.split(",") + secretsTable.append({ + "fsxId": fsxId, + "secretName": secretName, + "usernameKey": usernameKey, + "passwordKey": passwordKey + }) + except Exception as err: + message = f'Unable to read the secrets config file {config["secretsConfigFile"]} from bucket {config["s3BucketName"]}. Error message: "{err}".' + logger.critical(message) + raise Exception(message) + s3Client.close() + # + # If the dynamodbRegion is set, then open a client to the dynamodb service and read + # in the partners and secrets config tables. + if config['dynamodbRegion'] is not None: + dynamodbClient = boto3.resource("dynamodb", region_name=config['dynamodbRegion']) + + if config['dynamodbSecretsTableName'] is not None: + table = dynamodbClient.Table(config['dynamodbSecretsTableName']) + response = table.scan() + secretsTable = response["Items"] + if config['dynamodbPartnersTableName'] is not None: + table = dynamodbClient.Table(config['dynamodbPartnersTableName']) + response = table.scan() + items = response["Items"] + partnersTable = [] + for item in items: + partnersTable.append({ + 'fsxId': item['sourceId'].split(":")[0], + 'svmName': item['sourceId'].split(":")[1], + 'partnerFsxId': item['partnerFsxId'], + 'partnerFsxnIp': item['partnerFsxnIp'], + 'partnerSvmName': item['partnerSvmName'], + 'partnerSvmSourceName': item['partnerSvmSourceName'] + }) + + if 'partnersTable' not in globals() or len(partnersTable) == 0: + message = "Error, No partners were have been configured." + logger.critical(message) + raise Exception(message) + + if 'secretsTable' not in globals() or len(secretsTable) == 0: + message = "Error, No secrets were have been configured." + logger.critical(message) + raise Exception(message) ################################################################################ # This is the main logic of the program. It loops on all the regions, then all -# the fsx volumes within each region, checking to see if there are any volumes +# the fsx FSxN within each region, checking to see if there are any volumes # that don't have a snapmirror relationship. -# -# NOTE: It depends on the "snapmiror.destination.ontap" field from the source -# ontap volume structure to determine of a SM relationship already exist. -# Turns out that field is not maintained for FlexGroup volumes. ################################################################################ def lambda_handler(event, context): # # Define some globals so we don't have to pass them around. - global logger, http, secretsManagerClient, numSnapMirrorRelationships, secretsTable, partnersTable + global logger, http, secretsManagerClient, numSnapMirrorRelationships, secretsTable, partnersTable, snapMirrorRelationshipsCache, credentialsCache, boto3Config, config # - # Get a list of all the regions. - ec2Client = boto3.client('ec2') - regions = ec2Client.describe_regions() + # Clear out the caches to prevent them from being re-used for previous Lambda invocations. + snapMirrorRelationshipsCache = {} + credentialsCache = {} + # + # Configure boto3 to use the more advanced "adaptive" retry method and + # so we aren't waiting if a region is down, which, at the time of this + # writing, the Middle East regions are down. + boto3Config = Config( + connect_timeout = 2, + read_timeout = 30, + retries = { + 'total_max_attempts': 2, + 'mode': 'adaptive' + }, + s3={'us_east_1_regional_endpoint':'regional'} + ) # # Set up "logging" to appropriately display messages. It can be set it up # to send messages to a syslog server. logging.basicConfig(datefmt='%Y-%m-%d_%H:%M:%S', format='%(asctime)s:%(name)s:%(levelname)s:%(message)s', encoding='utf-8') logger = logging.getLogger("auto_create_sm_relationships") -# logger.setLevel(logging.DEBUG) - logger.setLevel(logging.INFO) + logger.setLevel(logging.DEBUG) +# logger.setLevel(logging.INFO) # - # Ensure the logging level on higher for these noisy modules to mute thier messages. + # Set the logging level higher for these noisy modules to mute thier messages + # when debugging things. logging.getLogger("boto3").setLevel(logging.WARNING) logging.getLogger("urllib3").setLevel(logging.WARNING) # - if dryRun: - logger.info('Running in Dry Run mode.') + # Read in the configuration from the environment variables, global variables or an s3 bucket. + readinConfig() # - # Create a Secrets Manager client. - session = boto3.session.Session() - secretsManagerClient = session.client(service_name='secretsmanager', region_name=secretsManagerRegion) + # Get the list of regions to check. It can be set in the environment + # variable "regions" or in the global variable "regions". If neither + # is set, then it will check all regions. + regionsEnv = config['regions'] + if regionsEnv is not None: + limitRegions = [item.strip() for item in regionsEnv.split(',')] + else: + limitRegions = [] + ec2Client = boto3.client('ec2', config=boto3Config) + ec2Regions = ec2Client.describe_regions()['Regions'] + for region in ec2Regions: + limitRegions += [region['RegionName']] + ec2Client.close() # - # Read in the secretTable if it is not already defined. - if 'dynamodbRegion' in globals(): - dynamodbClient = boto3.resource("dynamodb", region_name=dynamodbRegion) # pylint: disable=E0602 - - if 'secretsTable' not in globals(): - if 'dynamodbRegion' not in globals() or 'dynamodbSecretsTableName' not in globals(): - raise Exception('Error, you must either define the secretsTable array at the top of this script, or define dynamodbRegion and dynamodbSecretsTableName') - - table = dynamodbClient.Table(dynamodbSecretsTableName) # pylint: disable=E0602 - - response = table.scan() - secretsTable = response["Items"] # - # Read in the partnersTable if it is not already defined. - if 'partnersTable' not in globals(): - if 'dynamodbRegion' not in globals() or 'dynamodbPartnersTableName' not in globals(): - raise Exception('Error, you must either define the partnersTable array at the top of this script, or define dynamodbRegion and dynamodbPartnersTableName') - - table = dynamodbClient.Table(dynamodbPartnersTableName) # pylint: disable=E0602 - - response = table.scan() - items = response["Items"] - partnersTable = [] - for item in items: - partner = { - 'fsxId': item['sourceId'].split(":")[0], - 'svmName': item['sourceId'].split(":")[1], - 'partnerFsxnIp': item['partnerFsxnIp'], - 'partnerSvmName': item['partnerSvmName'], - 'partnerSvmSourceName': item['partnerSvmSourceName'] - } - partnersTable.append(partner) + if dryRun: + logger.info('Running in Dry Run mode.') + # + # Open a client to the SecretsManager service. + secretsManagerClient = boto3.client('secretsmanager', region_name=config['secretsManagerRegion'], config=boto3Config) # # Disable warning about connecting to servers with self-signed SSL certificates. urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) @@ -395,59 +697,104 @@ def lambda_handler(event, context): # # Get the list of regions that support fsx. fsxRegions = boto3.Session().get_available_regions('fsx') + # + # Create an array of the FSxN IDs to scan. + fsxIdsEnv = config['fsxIds'] + if fsxIdsEnv is not None: + limitFsxIds = [item.strip() for item in fsxIdsEnv.split(',')] + else: + limitFsxIds = [] try: - for region in regions['Regions']: - regionName=region['RegionName'] + for regionName in limitRegions: # # Skip regions that don't support fsx. if regionName in fsxRegions: logger.debug(f'Scanning region {regionName}.') - fsxClient = boto3.client('fsx', region_name=regionName) - # - # Create an array of all the AWS FSx file systems. - data = fsxClient.describe_file_systems() - fsxs = data['FileSystems'] - nextToken = data.get('NextToken') - while nextToken is not None: + fsxClient = boto3.client('fsx', region_name=regionName, config=boto3Config) + tagsClient = boto3.client('resourcegroupstaggingapi', region_name=regionName, config=boto3Config) + try: + # + # Create an array of all the AWS FSx file systems. data = fsxClient.describe_file_systems() - fsxs += data['FileSystems'] + fsxs = data['FileSystems'] nextToken = data.get('NextToken') - # - # Create an array of all the AWS volumes. - data = fsxClient.describe_volumes() - awsVolumes = data['Volumes'] - nextToken = data.get('NextToken') - while nextToken is not None: - data = fsxClient.describe_volumes(NextToken=nextToken) - awsVolumes += data['Volumes'] + while nextToken is not None: + data = fsxClient.describe_file_systems(NextToken=nextToken) + fsxs += data['FileSystems'] + nextToken = data.get('NextToken') + # + # Create an array of all the AWS volumes. + data = fsxClient.describe_volumes() + awsVolumes = data['Volumes'] nextToken = data.get('NextToken') + while nextToken is not None: + data = fsxClient.describe_volumes(NextToken=nextToken) + awsVolumes += data['Volumes'] + nextToken = data.get('NextToken') + # + # Create an array of all the tags associated with the fsx resource type. + data = tagsClient.get_resources(ResourceTypeFilters=['fsx'], ResourcesPerPage=100) + tags = data['ResourceTagMappingList'] + nextToken = data.get('PaginationToken', "") + + while nextToken != "": + response = tagsClient.get_resources(ResourceTypeFilters=['fsx'], ResourcesPerPage=100, PaginationToken=nextToken) + tags += response['ResourceTagMappingList'] + nextToken = response.get('PaginationToken', "") + # + # Handle the case where one of those AWS API calls fails due to a timeout. + # Chances are it means the region is down. + except (botocore.exceptions.ReadTimeoutError, botocore.exceptions.ConnectTimeoutError, botocore.exceptions.ConnectionClosedError) as e: + logger.warning(f"boto3 client error while getting the list of FSxNs, volumes or resource tags from region {regionName}. Exception: {e}. Skipping the region.") + fsxClient.close() + tagsClient.close() + continue + + fsxClient.close() + tagsClient.close() # # Loop on all the file systems in the region. for fsxn in fsxs: # - # Skip file systems that are not ONTAP. - if fsxn['FileSystemType'] == "ONTAP": + # Skip file systems that are not ONTAP or are not listed in the limitFsxIds array of defined.. + if fsxn['FileSystemType'] == "ONTAP" and (limitFsxIds == [] or fsxn['FileSystemId'] in limitFsxIds): fsxnId = fsxn['FileSystemId'] - fsxnIp = fsxn['OntapConfiguration']['Endpoints']['Management']['IpAddresses'][0] - logger.debug(f'Scanning fsxn with IP {fsxnIp}.') - ontapVolumes = getOntapVolumes(fsxnId, fsxnIp) # Get all the volumes in the file system. + if (fsxn.get('OntapConfiguration') is not None and + fsxn['OntapConfiguration'].get('Endpoints') is not None and + fsxn['OntapConfiguration']['Endpoints'].get('Management') is not None and + fsxn['OntapConfiguration']['Endpoints']['Management'].get('IpAddresses') is not None): + fsxnIp = fsxn['OntapConfiguration']['Endpoints']['Management']['IpAddresses'][0] + else: + logger.warning(f'No management IP address found for fsxId: {fsxnId}. Probably because it is being created, or deleted.') + continue + logger.debug(f'Getting all the volumes from fsxn with IP {fsxnIp}.') + ontapVolumes = getOntapVolumes(fsxnId, fsxnIp) for ontapVolume in ontapVolumes: - if ontapVolume['type'].lower() == "rw" and not ontapVolume['snapmirror']['destinations']['is_ontap']: + # + # Only create SnapMirror relationships for RW volumes that arne't clones or FlexCaches. + if ontapVolume['type'].lower() == "rw" and ontapVolume.get('clone') is not None and not ontapVolume['clone']['is_flexclone'] and ontapVolume.get('flexcache_endpoint_type') != "cache": volumeUUID = ontapVolume['uuid'] volumeARN = getVolumeARN(awsVolumes, volumeUUID) - protectTag = getVolumeProtectTagValue(fsxClient, volumeARN) - - if protectAll and protectTag != "skip" or not protectAll and protectTag == "protect": - volumeName = ontapVolume['name'] - svmName = ontapVolume['svm']['name'] - if ontapVolume['style'] == "flexgroup": - protectFlexGroup(fsxnId, svmName, volumeName) - else: - protectVolume(fsxnId, svmName, volumeName) - if numSnapMirrorRelationships >= maxSnapMirrorRelationships: - raise TooManySMs("Too Many SnapMirror relationships being created.") - + if volumeARN != "": + protectTag = getVolumeProtectTagValue(tags, volumeARN) + + if config['protectAll'] and protectTag != "skip" or not config['protectAll'] and protectTag == "protect": + volumeName = ontapVolume['name'] + svmName = ontapVolume['svm']['name'] + (partnerId, partnerIp, partnerSvmName, partnerSvmSourceName) = getPartnerInfo(fsxnId, svmName) + if partnerId == "": + logger.warning(f'No partner found for fsxId: {fsxnId} and svmName: {svmName}.') + continue + + if getCredentials(partnerId) == ("", ""): + logger.warning(f'No credentials found for partner fsxId: {partnerId}.') + continue + + protectVolume(fsxnId, svmName, volumeName, partnerId, partnerIp, partnerSvmName, partnerSvmSourceName) + + if numSnapMirrorRelationships >= config['maxSnapMirrorRelationships']: + raise TooManySMs("Too Many SnapMirror relationships being created.") except TooManySMs: logger.warning(f'Hit the maximum number of SnapMirorr relationships ({numSnapMirrorRelationships}) created in one run. Exiting.') diff --git a/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml b/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml new file mode 100644 index 00000000..9d49634e --- /dev/null +++ b/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml @@ -0,0 +1,1155 @@ +Description: "Deploy auto_create_sm_relationships." +# +# This section just formats the page that prompts for the parameters when using the AWS Console to deploy your stack. +Metadata: + AWS::CloudFormation::Interface: + ParameterGroups: + - Label: + default: "Deployment Parameters" + Parameters: + - subNetIds + - securityGroupIds + - checkInterval + - secretArnPattern + - secretsManagerRegion + - roleArn + - createWatchdogAlarm + - snsTopicArn + - maxRunTime + - Label: + default: "Configuration Parmeters" + Parameters: + - regions + - fsxIds + - destinationVolumeSuffix + - snapMirrorPolicy + - scheduleName + - tieringPolicy + - maxSnapMirrorRelationships + - protectAll + - Label: + default: "AWS S3 Configuration - Fill in this in OR the DynamoDB section." + Parameters: + - s3BucketName + - s3BucketRegion + - partnersConfigFile + - secretsConfigFile + - Label: + default: "DynamoDB Configuration - Fill in this OR the S3 section" + Parameters: + - dynamodbRegion + - dynamodbPartnersTableName + - dynamodbSecretsTableName + + ParameterLabels: + regions: + default: "The AWS region names to scan for FSxNs for. Comma separated list" + fsxIds: + default: "The specific FSxNs to look for volumes from. Comma separated list" + s3BucketName: + default: "S3 Bucket Name that will contain the partners and secrets configuration files" + s3BucketRegion: + default: "The region the S3 bucket resides" + partnersConfigFile: + default: "The file that contains the list of source to partner relationships" + secretsConfigFile: + default: "The file that contains the list of secrets for the FSxNs" + secretsManagerRegion: + default: "The region the SecretsManager secrets reside" + dynamodbPartnersTableName: + default: "The name of the DynamoDB table that contains the partner relationships" + dynamodbSecretsTableName: + default: "The name of the DynamoDB table that contains the secrets ARNs for the FSxNs" + dynamodbRegion: + default: "The region the DynamoDB tables reside" + secretArnPattern: + default: "SecretsManager secret ARN pattern for FSxN credentials" + destinationVolumeSuffix: + default: "The suffix to append to the destination volume name when creating a new SnapMirror relationship" + snapMirrorPolicy: + default: "The SnapMirror policy to use when creating a new SnapMirror relationship" + scheduleName: + default: "The schedule name to use when creating a new SnapMirror relationship" + tieringPolicy: + default: "The tiering policy to assign to the destination volume when creating a new SnapMirror relationship" + maxSnapMirrorRelationships: + default: "The maximum number of SnapMirror relationships to create in a single run" + protectAll: + default: "Protect all volumes (true) or just ones with a protect_volume tag set to 'protect' (false)" + subNetIds: + default: "Subnet IDs" + securityGroupIds: + default: "Security Group IDs" + checkInterval: + default: "Check Interval (hours)" + createWatchdogAlarm: + default: "Create Watchdog Alarms" + snsTopicArn: + default: "SNS Topic ARN to send alarms to" + roleArn: + default: "Role ARN to assign to the Lambda function." + maxRunTime: + default: "Maximum time the Lambda is allowed to run (seconds)" + +Parameters: + regions: + Description: "The AWS region names to scan for FSxNs for. Comma separated list. If empty, all regions will be scanned." + Type: String + Default: "" + + fsxIds: + Description: "The specific FSxNs to look for volumes from. Comma separated list. If empty, all FSxNs will be scanned." + Type: String + Default: "" + + s3BucketName: + Description: "The name of the S3 bucket to read the configuration files from." + Type: String + Default: "" + + s3BucketRegion: + Description: "The region the S3 bucket resides in." + Type: String + Default: "" + + partnersConfigFile: + Description: "The file in the S3 bucket that contains the list of partner relationships. The format of the file is specified in the README file on the GitHub repo where this program is maintained." + Type: String + Default: "" + + secretsConfigFile: + Description: "The file in the S3 bucket that contains the list of secrets for the FSxN file systems. The format of the file is specified in the README file on the GitHub repo where this program is maintained." + Type: String + Default: "" + + secretsManagerRegion: + Description: "The region the SecretsManager secrets reside in." + Type: String + Default: "" + + dynamodbPartnersTableName: + Description: "The name of the DynamoDB table that contains the partner relationships." + Type: String + Default: "" + + dynamodbSecretsTableName: + Description: "The name of the DynamoDB table that contains the secrets ARNs for the FSxNs." + Type: String + Default: "" + + dynamodbRegion: + Description: "The region the DynamoDB tables reside in." + Type: String + Default: "" + + secretArnPattern: + Description: "The ARN pattern of the SecretsManager secrets that holds the FSxN credentials." + Type: String + Default: "arn:aws:secretsmanager:*:*:secret:FSxSecret*" + + destinationVolumeSuffix: + Description: "The suffix to append to the destination volume name when creating a new SnapMirror relationship." + Type: String + Default: "_dp" + + snapMirrorPolicy: + Description: "The SnapMirror policy to use when creating a new SnapMirror relationship." + Type: String + Default: "MirrorAllSnapshots" + + scheduleName: + Description: "The schedule name to use when creating a new SnapMirror relationship." + Type: String + Default: "daily" + + tieringPolicy: + Description: "The tiering policy to assign to the destination volume when creating a new SnapMirror relationship." + Type: String + Default: "all" + AllowedValues: ["all", "auto", "none", "snapshot-only"] + + maxSnapMirrorRelationships: + Description: "The maximum number of SnapMirror relationships to create in a single run." + Type: Number + Default: 10 + MinValue: 1 + + protectAll: + Description: "Protect all volumes by default, or just ones with a protect_volume tag set to protect." + Type: String + Default: "true" + AllowedValues: ["true", "false"] + + subNetIds: + Description: "The subnet IDs where you want to deploy the Lambda function. Must have connectivity to all the source and destination FSxN file systems. Recommended to have at least two. Also recommended to be in a private subnet." + Type: "List" + + securityGroupIds: + Description: "The security group IDs to associate with the Lambda function. Must allow outbound traffic over TCP port 443 to the FSxN file system, and the AWS service endpoints." + Type: "List" + + createWatchdogAlarm: + Description: "Create CloudWatch alarms that monitors the Lambda function. It will alert you if Lambda function fails to run successfully." + Type: String + Default: "true" + AllowedValues: ["true", "false"] + + snsTopicArn: + Description: "The ARN of the SNS topic to send alarms to. This is only needed if createWatchdogAlarm is set to 'true'." + Type: String + Default: "" + + roleArn: + Description: "The ARN of the role to use for the Lambda function. This is only needed if you want to provide an existing role, otherwise an appropriate one will be created for you." + Type: String + Default: "" + + maxRunTime: + Description: "The maximum time, in seconds, the Lambda function is allowed to run before being terminated. Must be between 60 and 900 seconds." + Type: Number + Default: 60 + MinValue: 60 + MaxValue: 900 + + checkInterval: + Description: "The interval, in hours, between checks." + Type: Number + Default: 12 + MinValue: 1 + +Conditions: + CreateWatchdogAlarm: !Equals [!Ref createWatchdogAlarm, "true"] + CreateLambdaRole: !Equals [!Ref roleArn, ""] + IncludeDynamodbPartners: !Not [!Equals [!Ref dynamodbPartnersTableName, ""]] + IncludeDynamodbSecrets: !Not [!Equals [!Ref dynamodbPartnersTableName, ""]] + IncludeDynamodb: !Or [!Condition IncludeDynamodbPartners, !Condition IncludeDynamodbSecrets] + IncludeS3: !Not [!Equals [!Ref s3BucketName, ""]] + +Resources: + # + # This is the CloudWatch alarm that will trigger when the Lambda function fails + # to run successfully. It will send an SNS message to the SNS topic. + watchdogAlarm: + Type: "AWS::CloudWatch::Alarm" + Condition: CreateWatchdogAlarm + Properties: + AlarmName: !Sub "auto-create-sm-relationships-watchdog-${AWS::StackName}" + AlarmDescription: !Sub "Watchdog alarm for the auto-create-sm-relationships-${AWS::StackName} Lambda function." + Namespace: "AWS/Lambda" + MetricName: "Errors" + Dimensions: + - Name: "FunctionName" + Value: !Sub "auto-create-sm-relationships-${AWS::StackName}" + Statistic: "Maximum" + Period: 900 + EvaluationPeriods: 1 + DatapointsToAlarm: 1 + TreatMissingData: "ignore" + Threshold: 0.5 + ComparisonOperator: "GreaterThanThreshold" + AlarmActions: + - !Ref snsTopicArn + # + # This is the role that will be assigned to the Lambda function. + # It will only be created if the user hasn't provided a roleARN. + lambdaRole: + Type: "AWS::IAM::Role" + Condition: CreateLambdaRole + Properties: + RoleName: !Sub "auto-create-sm-relationships-${AWS::StackName}" + AssumeRolePolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: "Allow" + Principal: + Service: "lambda.amazonaws.com" + Action: "sts:AssumeRole" + + ManagedPolicyArns: + - "arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole" + + Policies: + - PolicyName: "LambdaPolicy" + PolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: "Allow" + Action: + - "fsx:DescribeFileSystems" + - "fsx:DescribeVolumes" + - "tags:ListTagsForResource" + - "ec2:DescribeRegions" + Resource: + - "*" + - Effect: "Allow" + Action: + - "secretsmanager:GetSecretValue" + - !If [IncludeS3, "s3:GetObject", !Ref "AWS::NoValue"] + - !If [IncludeS3, "s3:ListBucket", !Ref "AWS::NoValue"] + - !If [IncludeDynamodb, "dynamodb:DescribeTable", !Ref "AWS::NoValue"] + - !If [IncludeDynamodb, "dynamodb:Scan", !Ref "AWS::NoValue"] + Resource: + - !Ref secretArnPattern + - !If [IncludeS3, !Sub "arn:aws:s3:::${s3BucketName}", !Ref "AWS::NoValue"] + - !If [IncludeS3, !Sub "arn:aws:s3:::${s3BucketName}/*", !Ref "AWS::NoValue"] + - !If [IncludeDynamodbPartners, !Sub "arn:aws:dynamodb:${dynamodbRegion}:${AWS::AccountId}:table/${dynamodbPartnersTableName}", !Ref "AWS::NoValue"] + - !If [IncludeDynamodbSecrets, !Sub "arn:aws:dynamodb:${dynamodbRegion}:${AWS::AccountId}:table/${dynamodbSecretsTableName}", !Ref "AWS::NoValue"] + # + # Instead of creating a EventBridge schedule, create an EventBridge rule triggered by a schedule. + # This way a separate role doesn't have to be created, just a Lambda permission to allow + # EventBridge to invoke the Lambda function. + LambdaEventRule: + Type: "AWS::Events::Rule" + Properties: + Description: "Event rule to trigger the Lambda function." + Name: !Sub "auto-create-sm-relationships-event-rule-${AWS::StackName}" + ScheduleExpression: !Sub "rate(${checkInterval} hours)" + State: "ENABLED" + Targets: + - Arn: !GetAtt LambdaFunction.Arn + Id: "AutoCreateSMRelationshipsTarget" + + LambdaPermissionEventRule: + Type: "AWS::Lambda::Permission" + Properties: + Action: "lambda:InvokeFunction" + FunctionName: !GetAtt LambdaFunction.Arn + Principal: "events.amazonaws.com" + SourceArn: !GetAtt LambdaEventRule.Arn + + LambdaFunction: + Type: "AWS::Lambda::Function" + Properties: + FunctionName: !Sub "auto-create-sm-relationships-${AWS::StackName}" + Role: !If [CreateLambdaRole, !GetAtt lambdaRole.Arn, !Ref roleArn] + VpcConfig: + SecurityGroupIds: !Ref securityGroupIds + SubnetIds: !Ref subNetIds + PackageType: "Zip" + Runtime: "python3.12" + Handler: "index.lambda_handler" + Timeout: !Ref maxRunTime + Environment: + Variables: + regions: !Ref regions + fsxIds: !Ref fsxIds + secretsManagerRegion: !Ref secretsManagerRegion + destinationVolumeSuffix: !Ref destinationVolumeSuffix + snapMirrorPolicy: !Ref snapMirrorPolicy + scheduleName: !Ref scheduleName + tieringPolicy: !Ref tieringPolicy + maxSnapmirrorRelationships: !Ref maxSnapMirrorRelationships + protectAll: !Ref protectAll + s3BucketName: !Ref s3BucketName + s3BucketRegion: !Ref s3BucketRegion + partnersConfigFile: !Ref partnersConfigFile + secretsConfigFile: !Ref secretsConfigFile + dynamodbRegion: !Ref dynamodbRegion + dynamodbPartnersTableName: !Ref dynamodbPartnersTableName + dynamodbSecretsTableName: !Ref dynamodbSecretsTableName + Code: + ZipFile: | + #!/bin/python3 + # + ################################################################################ + # This program is used to create SnapMirror relationships for all RW, + # non-clone, non-FlexCache volumes it finds on any FSxN File System that the + # user (or Lambda function) running the program has access to. + # + # It does this by: + # o Looping on all the regions. Skipping on any that don't support FSx. + # o Looping on all the FSx file systems. Skipping any non-FSxN file systems. + # o Obtaining information on all the volumes from the FSxN file system + # using the ONTAP API. + # o If a SnapMirror relationship doesn't already exist, and the volume on the + # AWS side doesn't have a "protect_volume" tag set to "skip", then it + # creates a SnapMirror relationship using the partner information + # provided below. + # + # Version: v1.5 + # Date: 2026-10-03-09:21:59 + # + ################################################################################ + # + # The program needs a list of source SVMs to partner associations to know where + # to create the SnapMirror relationships. That information is conveyed via + # the 'partnersTable' variable. The values of each of the fields in the table + # There are three ways to provide the partner information to the program. + # You can either: + # 1) Define an array named "partnersTable" in this script + # 2) Define a DynamoDB table to scan for the information + # 3) Define an S3 bucket and file to fetch the information from. + # + # If you want to define a variable in the program, then name needs to be + # partnersTable. It should be an array of dictionaries, where each dictionary + # has these fields: + # fsxId - The AWS ID of the source cluster. + # svmName - The name of the SVM at the source cluster. + # partnerFsxId - The AWS ID of the destination cluster. + # partnerFsxnIp - The IP address of the destination cluster. + # partnerSvmName - The name of the SVM at the destination cluster. + # partnerSvmSourceName - The name of the source SVM at the destination cluster. + # + # All of those should be pretty straight forward except for the "partnerSvmSourceName". + # That is the "local name" for the source SVM at the destination cluster. Usually the + # local name is the same as the source SVM name unless the destination cluster + # already has an SVM with the same name. In that case a "local name" has to be + # assigned when you peer the SVMs (a.k.a. vservers) so ONTAP can distinguish them + # apart. To be able to handle that situation the partnerSvmSourceName should + # be set to that local name even if it is the same as the source SVM name. + # + # Here is an example of the partnersTable array: + # partnersTable = [ + # { + # "fsxId": "fs-04f0ebeXXXXXXXXXX", + # "svmName": "fsx", + # "partnerFsxId": "fs-07bcb7aXXXXXXXXXX", + # "partnerFsxnIp": "198.19.254.83", + # "partnerSvmName": "fsx", + # "partnerSvmSourceName": "prod-fsx" + # }, + # { + # "fsxId": "fs-04f0ebeXXXXXXXXXX", + # "svmName": "smb_svm", + # "partnerFsxId": "fs-07bcb7aXXXXXXXXXX", + # "partnerFsxnIp": "198.19.254.83", + # "partnerSvmName": "fsx", + # "partnerSvmSourceName": "smb_svm" + # } + # ] + # + # -------- + # + # If you want to use a DynamoDB table, the program expect the following + # attributes for each entry: + # sourceId - Which is the concatenation of the source file system ID + # followed by a ":" followed by the SVM name. It is done this + # way because the id has to be unique in the table. + # partnerFsxId - The AWS ID of the destination cluster. + # partnerFsxnIp - The IP address of the destination cluster. + # partnerSvmName - The name of the SVM at the destination cluster. + # partnerSvmSourceName - The name of the source SVM at the destination cluster. + # + # You should set the following variables to instruct the program to use + # the DynamoDB table for the partner information: + # dynamodbRegion="us-west-2" + dynamodbRegion="" + dynamodbPartnersTableName="" + # + # Of course adjust for your region and table name. + # + # -------- + # + # If you want to use a file in an S3 bucket, the format of file should be a + # comma separated list of the following fields: + # + # fsxId,svmName,partnerFsxId,partnerFsxnIp,partnerSvmName,partnerSvmSourceName + # + # Here are the variable to set to get the program to read a file in an S3 + # bucket for the partner information. + s3BucketName="" + s3BucketRegion="" + partnersConfigFile="" + # + # Of course adjust for your bucket name, region and file name. + # + # NOTE: If both the DynamoDB table and S3 bucket variables are defined, the + # DynamoDB table will be used and if either are defined, the will be used + # instead of the global variable "partnersTable" defined in this script. + # + ################################################################################ + # + # The program needs credentials to use to authenticate with the FSxNs with. + # There are three ways you can conveyed this information to the program. You can + # either: + # 1) Define an array named "secretsTable" in this script + # 2) Define a DynamoDB table to scan for the information + # 3) Define an S3 bucket and file to fetch the information from. + # + # If you want to define a variable in the program, then name needs to be + # secretsTable. It should be an array of dictionaries, where each dictionary + # has these fields: + # fsxId - The AWS ID of the source cluster. + # secretName - The name of the Amazon SecretsManager secret that holds the username and password keys. + # usernameKey - The name of the key that holds the username to use. + # passwordKey - The name of the key that holds the password to use. + # + # Here is an example of the secretsTable array: + # secretsTable = [ + # {"fsxId": "fs-04f0ebeXXXXXXXXXX", "secretName": "FSxSecret-Default", "usernameKey": "username", "passwordKey": "password"}, + # {"fsxId": "fs-07bcb7aXXXXXXXXXX", "secretName": "FSxSecret-Default", "usernameKey": "username", "passwordKey": "password"} + # ] + # + # If you want to use a DynamoDB table, the program expect the following + # attributes for each entry: + # fsxId - The file system ID + # SecretName - The name of the Amazon SecretsManager secret that holds the username and password keys. + # usernameKey - The name of the key that holds the username to use. + # passwordKey - The name of the key that holds the password to use. + # + # You should set the following variables to instruct the program to use + # the DynamoDB table for the secrets information. + # dynamodbRegion="" # Should be defined above for the partners information. + dynamodbSecretsTableName="" + # + # ------ + # + # If you want to use a file in an S3 bucket, the format of file should be a + # comma separated list of the following fields: + # fsxId,secretName,usernameKey,passwordKey + # + # You'll need to set the following variables to get the program to read a file in an S3 + # bucket for the secrets information. + # s3BucketName="" # Should be defined above for the partners information. + # s3Region="" # Should be defined above for the partners information. + secretsConfigFile="" + # + # Of course adjust for your bucket name, region and file name. + # + # NOTE: If both the DynamoDB table and S3 bucket variables are defined, the + # DynamoDB table will be used and if either are defined, they will be used + # instead of the global variable "secretsTable" defined in this script. + # + # ------ + # + # If you want to limit the regions that are scanned, you can define them here. + # The expected format is a comma separated list of region names. + # If you don't define it then all regions will be scanned. + # regions="us-west-2,us-east-1" + regions="" + # + # If you want to limit the FSxN IDs to look for source volumes from, you + # an define them here. Expected format is a comma separated list of FSxN IDs. + # fsxIds="fs-04f0ebeeXXXXXXXXX,fs-07bcbXXXXXXXXXXXX" + fsxIds="" + # + # Provide the region the secrets manager resides in: + secretsManagerRegion="us-west-2" + # + # Set the suffix string to append to the destination volume. + destinationVolumeSuffix="_dp" + # + # Set the SnapMirror policy to use. + snapMirrorPolicy="MirrorAllSnapshots" + # + # If your policy doesn't have a schedule associated with it, you + # can specify a schedule name here. Set to an empty string otherwise. + scheduleName="hourly" + # + # Set the Tiering policy for the destination volume if it is created. + # Supported values are: "all", "none", "auto", "snapshot-only". + tieringPolicy="all" + # + # Set to the maximum number of SnapMirror relationships to create during + # a single run. + maxSnapMirrorRelationships=20 + # + # Set the following to 'True' (case sensitive) to have the program protect + # all volumes that don't have a "protect_volume" tag set to "skip". Or, set + # it to 'False' to only protect volumes that have a "protect_volume" tag + # set to "protect". + protectAll=True + # + # Set the following to 'True' (case sensitive) to have the program just + # show what it would have done instead of really doing it. + dryRun=False + + ################################################################################ + # !!!!!!!! You shouldn't have to modify anything below here. !!!!!!!!!!!!!!!!!!! + ################################################################################ + + import json + import os + import time + import urllib3 + from urllib3.util import Retry + import logging + import botocore + from botocore.config import Config + import boto3 + # + ################################################################################ + # Define a custom exception so we can gracefully exit the program if too many + # snapmirror relationships have been created. + ################################################################################ + class TooManySMs(Exception): + + # Constructor or Initializer + def __init__(self, value): + self.value = value + + # __str__ is to print() the value + def __str__(self): + return(repr(self.value)) + + ################################################################################ + # This function returns the value assigned to the "protect_volume" tag + # associated with the ARN passed in. If none is found, it returns an empty + # string. + ################################################################################ + def getVolumeProtectTagValue(tags, arn): + + for resource in tags: + if resource['ResourceARN'] == arn: + for tag in resource['Tags']: + if tag['Key'].lower() == "protect_volume": + return tag['Value'].lower() + + return("") + + ################################################################################ + # This function returns the ARN of the volume that has the UUID passed in. It + # returns an empty string if the UUID is not found. + ################################################################################ + def getVolumeARN(awsVolumes, volumeUUID): + + global logger + + for awsVolume in awsVolumes: + if awsVolume.get('OntapConfiguration') is not None and awsVolume['OntapConfiguration'].get('UUID') == volumeUUID: + return(awsVolume['ResourceARN']) + logger.warning(f'Failed to get ARN for volume with UUID={volumeUUID}.') + return("") + + ################################################################################ + # This function is used to obtain the username and password from AWS's Secrets + # Manager for the fsxnId passed in. It returns empty strings if it can't + # find the credentials. It creates a cache of the responses so that it doesn't + # have to call Secrets Manager for the same fsxnId multiple times. + ################################################################################ + credentialsCache = {} + def getCredentials(fsxnId): + + global secretsManagerClient, secretsTable + + if credentialsCache.get(fsxnId) is not None: + return (credentialsCache[fsxnId]['username'], credentialsCache[fsxnId]['password']) + + for secretItem in secretsTable: + if secretItem['fsxId'] == fsxnId: + secretsInfo = secretsManagerClient.get_secret_value(SecretId=secretItem['secretName']) + secrets = json.loads(secretsInfo['SecretString']) + username = secrets[secretItem['usernameKey']] + password = secrets[secretItem['passwordKey']] + credentialsCache[fsxnId] = {'username': username, 'password': password} + return (username, password) + return ("", "") + + ################################################################################ + # This function returns the partner information for the sources FSxN and + # SVM name passed in. + ################################################################################ + def getPartnerInfo(fsxId, svmName): + + partnerId = "" + partnerIp = "" + partnerSvmName = "" + partnerSvmSourceName = "" + + for fsx in partnersTable: + if fsx['fsxId'] == fsxId and fsx['svmName'] == svmName: + partnerId = fsx['partnerFsxId'] + partnerIp = fsx['partnerFsxnIp'] + partnerSvmName = fsx['partnerSvmName'] + partnerSvmSourceName = fsx['partnerSvmSourceName'] + break + + return (partnerId, partnerIp, partnerSvmName, partnerSvmSourceName) + + ################################################################################ + # The following function will check to see if a SnapMirror relationship already + # exists for the source and destination volumes passed in. It returns True if + # it exists, False otherwise. + # + # It caches the snapmirror relationships for each destination cluster so it + # doesn't have to call the API for each volume. + ################################################################################ + snapMirrorRelationshipsCache = {} + def checkSnapMirrorRelationshipExists(sourceSVM, sourceVolume, destinationFsId, destinationFsIp, destinationSVM, destinationVolume): + + if snapMirrorRelationshipsCache.get(destinationFsId) is None: + snapMirrorRelationshipsCache[destinationFsId] = getSnapMirrorRelationships(destinationFsId, destinationFsIp) + + for relationship in snapMirrorRelationshipsCache[destinationFsId]: + if (relationship['source']['path'] == f'{sourceSVM}:{sourceVolume}' and + relationship['destination']['path'] == f'{destinationSVM}:{destinationVolume}'): + return True + + return False + + ################################################################################ + # This function gets all the snapmirror relationships for the fsxnIp passed in. + ################################################################################ + def getSnapMirrorRelationships(fsxnId, fsxnIp): + + global logger, http + + relationships = [] + + try: + (username, password) = getCredentials(fsxnId) + if username == "" or password == "": + logger.error(f'No credentials for FSxN ID: {fsxnId}.') + return relationships + + auth = urllib3.make_headers(basic_auth=f'{username}:{password}') + headers = { **auth } + + endpoint = f'https://{fsxnIp}/api/snapmirror/relationships?fields=source,destination' + logger.debug(f'Trying {endpoint}.') + response = http.request('GET', endpoint, headers=headers) + if response.status < 200 or response.status > 299: + logger.error(f'API call to {endpoint} failed. HTTP status code: {response.status}.') + return relationships + body = json.loads(response.data.decode('utf-8')) + relationships += body['records'] + while body.get('next'): + endpoint = f'https://{fsxnIp}{body["next"]}' + logger.debug(f'Trying {endpoint}.') + response = http.request('GET', endpoint, headers=headers) + if response.status < 200 or response.status > 299: + logger.error(f'API call to {endpoint} failed. HTTP status code: {response.status}.') + return relationships + body = json.loads(response.data.decode('utf-8')) + relationships += body['records'] + except Exception as err: + logger.critical(f'API against {fsxnIp} failed. The error returned: "{err}".') + + return relationships + + ################################################################################ + # This function is used to setup a snapmirror relationship for the source + # volume passed in. It leverages the "create destination endpoint" + # capabilities of the snapmirror API which will create the destination volume + # with the same name as the source volume with a suffix appended to it. + # The suffix is defined above. + ################################################################################ + def protectVolume(fsxId, svmName, volumeName, partnerId, partnerIp, partnerSvmName, partnerSvmSourceName): + + global logger, http, numSnapMirrorRelationships, dryRun + # + # First check to see if the relationship already exists. + if checkSnapMirrorRelationshipExists(partnerSvmSourceName, volumeName, partnerId, partnerIp, partnerSvmName, f'{volumeName}{config["destinationVolumeSuffix"]}'): + logger.debug(f'SnapMirror relationship for {fsxId}::{svmName}:{volumeName} to {partnerIp}::{partnerSvmName}:{volumeName}{config["destinationVolumeSuffix"]} already exists.') + return + # + # Create the relationship. + (username, password) = getCredentials(partnerId) + if username == "" or password == "": + logger.error(f'No credentials for FSxN ID: {partnerId}.') + return + auth = urllib3.make_headers(basic_auth=f'{username}:{password}') + headers = { **auth } + + data = {"source": {"path": f"{partnerSvmSourceName}:{volumeName}"}, + "destination": {"path": f"{partnerSvmName}:{volumeName}{config['destinationVolumeSuffix']}"}, + "create_destination": {"enabled" : True, "tiering": {"supported": True, "policy": config['tieringPolicy']}}, + "state": "snapmirrored", + "policy": config['snapMirrorPolicy']} + if config['scheduleName'] is not None: + data["transfer_schedule"] = {"name": config['scheduleName']} + + try: + if not dryRun: + endpoint = f'https://{partnerIp}/api/snapmirror/relationships' + logger.debug(f'Trying {endpoint} with {data}.') + response = http.request('POST', endpoint, headers=headers, body=json.dumps(data)) + if response.status < 200 or response.status > 299: + errMessage = "N/A" + if response.data is not None: + errMessage = response.data.decode("utf-8") + logger.error(f'API call to {endpoint} failed. HTTP status code: {response.status}. Response from the API: {errMessage}') + return + logger.info(f'Path {fsxId}::{svmName}:{volumeName} is being SnapMirrored to {partnerId}::{partnerSvmName}:{volumeName}{config["destinationVolumeSuffix"]}.') + # + # Now check that it worked. + body = json.loads(response.data.decode('utf-8')) + jobId = body['job']['uuid'] + state = "unknown" + failsafeCounter = 0 + while state != "success" and state != "failure" and failsafeCounter < 60: + time.sleep(1) + endpoint = f'https://{partnerIp}/api/cluster/jobs/{jobId}' + logger.debug(f'Trying {endpoint}.') + response = http.request('GET', endpoint, headers=headers) + if response.status < 200 or response.status > 299: + logger.error(f'API call to {endpoint} failed. HTTP status code: {response.status}.') + return + body = json.loads(response.data.decode('utf-8')) + state = body['state'] + failsafeCounter += 1 + + if state == "success": + logger.info(f'SnapMirror relationship for {fsxId}::{svmName}:{volumeName} to {partnerIp}::{partnerSvmName}:{volumeName}{config["destinationVolumeSuffix"]} was successfully created.') + numSnapMirrorRelationships += 1 + else: + if body.get("error") is not None and body["error"].get("message") is not None: + errMessage = body["error"]["message"] + else: + errMessage = "N/A" + logger.error(f'SnapMirror relationship for {fsxId}::{svmName}:{volumeName} to {partnerIp}::{partnerSvmName}:{volumeName}{config["destinationVolumeSuffix"]} failed to be created. Last state was: {state}. Erorr message returned by the API:\n{errMessage}') + else: + logger.info(f'Path {fsxId}::{svmName}:{volumeName} would have been SnapMirrored to {partnerId}::{partnerSvmName}:{volumeName}{config["destinationVolumeSuffix"]}.') + numSnapMirrorRelationships += 1 + except Exception as err: + logger.critical(f'API against {partnerIp} failed. Volume not protected. The error returned: "{err}".') + return + + ################################################################################ + # This function is used to return all the volumes that are in the FSxN cluster. + # It returns an empty list if there are no volumes or if there was an error. + ################################################################################ + def getOntapVolumes(fsxId, fsxnIp): + + global logger, http + + (username, password) = getCredentials(fsxId) + if username == "" or password == "": + logger.error(f'No credentials for FSxN ID: {fsxId}.') + return([]) + auth = urllib3.make_headers(basic_auth=f'{username}:{password}') + headers = { **auth } + + volumes = [] + try: + endpoint = f'https://{fsxnIp}/api/storage/volumes?fields=name,svm,type,clone,flexcache_endpoint_type' + logger.debug(f'Trying {endpoint}.') + response = http.request('GET', endpoint, headers=headers, timeout=5.0) + if response.status == 200: + data = json.loads(response.data) + volumes = data['records'] + else: + logger.error(f'API call to {endpoint} failed. HTTP status code: {response.status}.') + return(volumes) + + while data.get('next'): + endpoint = f'https://{fsxnIp}{data["next"]}' + logger.debug(f'Trying {endpoint}.') + response = http.request('GET', endpoint, headers=headers, timeout=5.0) + if response.status == 200: + data = json.loads(response.data) + volumes += data['records'] + else: + logger.error(f'API call to {endpoint} failed. HTTP status code: {response.status}.') + break + + except Exception as err: + logger.critical(f'Failed to issue API against {fsxnIp}. Cluster could be down. The error messages received: "{err}".') + + return(volumes) + + ################################################################################ + # This function is used to consolidate all the configuration values into a + # single dictionary. + ################################################################################ + def readinConfig(): + + global config, boto3Config, logger, config, secretsTable, partnersTable + + config = { + "regions": None, + "fsxIds": None, + "partnersConfigFile": None, + "secretsConfigFile": None, + "secretsManagerRegion": None, + "s3BucketName": None, + "s3BucketRegion": None, + "dynamodbSecretsTableName": None, + "dynamodbPartnersTableName": None, + "dynamodbRegion": None, + "destinationVolumeSuffix": "_dp", + "snapMirrorPolicy": "MirrorAllSnapshots", + "scheduleName": None, + "tieringPolicy": "all", + "maxSnapMirrorRelationships": 10, + "protectAll": True + } + # + # Get the values for the variables giving the environment variables the + # highest priority, then the global variables, and finally the default values. + for var in config: + if os.environ.get(var) is not None: + config[var] = os.environ.get(var) + elif var in globals() and globals()[var] is not None: + config[var] = globals()[var] + # + # Since the CloudFormation template will set the environment variables + # to an empty string if someone doesn't provide a value, reset the + # values back to None. + for var in config: + if config[var] == "": + config[var] = None + # + # Convert the logical string to a logic. + if config.get('protectAll') is not None and type(config['protectAll']) == str and config['protectAll'].lower() == "false": + config['protectAll'] = False + else: + config['protectAll'] = True + # + # If the s3BucketName is set, then open a client to the s3 service and read + # in the partners and secrets config files. + if config['s3BucketName'] is not None: + # + # Open a client to the s3 service. + s3Client = boto3.client('s3', region_name=config['s3BucketRegion'], config=boto3Config) + if config['partnersConfigFile'] is not None: + partnersTable = [] + try: + response = s3Client.get_object(Bucket=config['s3BucketName'], Key=config['partnersConfigFile']) + partnerContents = response['Body'].read().decode('utf-8') + for partnerLine in partnerContents.splitlines(): + if partnerLine.strip() == "": + continue + if partnerLine.startswith("#"): + continue + (fsxId, svmName, partnerFsxId, partnerFsxnIp, partnerSvnName, partnerSvmSourceName) = partnerLine.split(",") + partnersTable.append({ + "fsxId": fsxId, + "svmName": svmName, + "partnerFsxId": partnerFsxId, + "partnerFsxnIp": partnerFsxnIp, + "partnerSvmName": partnerSvnName, + "partnerSvmSourceName": partnerSvmSourceName + }) + except Exception as err: + message = f'Unable to read the partners config file {config["partnersConfigFile"]} from bucket {config["s3BucketName"]}. Error message: "{err}".' + logger.critical(message) + raise Exception(message) + + if config['secretsConfigFile'] is not None: + secretsTable = [] + try: + response = s3Client.get_object(Bucket=config['s3BucketName'], Key=config['secretsConfigFile']) + secretsContents = response['Body'].read().decode('utf-8') + for secretsLine in secretsContents.splitlines(): + if secretsLine.strip() == "": + continue + if secretsLine.startswith("#"): + continue + (fsxId, secretName, usernameKey, passwordKey) = secretsLine.split(",") + secretsTable.append({ + "fsxId": fsxId, + "secretName": secretName, + "usernameKey": usernameKey, + "passwordKey": passwordKey + }) + except Exception as err: + message = f'Unable to read the secrets config file {config["secretsConfigFile"]} from bucket {config["s3BucketName"]}. Error message: "{err}".' + logger.critical(message) + raise Exception(message) + s3Client.close() + # + # If the dynamodbRegion is set, then open a client to the dynamodb service and read + # in the partners and secrets config tables. + if config['dynamodbRegion'] is not None: + dynamodbClient = boto3.resource("dynamodb", region_name=config['dynamodbRegion']) + + if config['dynamodbSecretsTableName'] is not None: + table = dynamodbClient.Table(config['dynamodbSecretsTableName']) + response = table.scan() + secretsTable = response["Items"] + if config['dynamodbPartnersTableName'] is not None: + table = dynamodbClient.Table(config['dynamodbPartnersTableName']) + response = table.scan() + items = response["Items"] + partnersTable = [] + for item in items: + partnersTable.append({ + 'fsxId': item['sourceId'].split(":")[0], + 'svmName': item['sourceId'].split(":")[1], + 'partnerFsxId': item['partnerFsxId'], + 'partnerFsxnIp': item['partnerFsxnIp'], + 'partnerSvmName': item['partnerSvmName'], + 'partnerSvmSourceName': item['partnerSvmSourceName'] + }) + + if 'partnersTable' not in globals() or len(partnersTable) == 0: + message = "Error, No partners were have been configured." + logger.critical(message) + raise Exception(message) + + if 'secretsTable' not in globals() or len(secretsTable) == 0: + message = "Error, No secrets were have been configured." + logger.critical(message) + raise Exception(message) + + ################################################################################ + # This is the main logic of the program. It loops on all the regions, then all + # the fsx FSxN within each region, checking to see if there are any volumes + # that don't have a snapmirror relationship. + ################################################################################ + def lambda_handler(event, context): + # + # Define some globals so we don't have to pass them around. + global logger, http, secretsManagerClient, numSnapMirrorRelationships, secretsTable, partnersTable, snapMirrorRelationshipsCache, credentialsCache, boto3Config, config + # + # Clear out the caches to prevent them from being re-used for previous Lambda invocations. + snapMirrorRelationshipsCache = {} + credentialsCache = {} + # + # Configure boto3 to use the more advanced "adaptive" retry method and + # so we aren't waiting if a region is down, which, at the time of this + # writing, the Middle East regions are down. + boto3Config = Config( + connect_timeout = 2, + read_timeout = 30, + retries = { + 'total_max_attempts': 2, + 'mode': 'adaptive' + }, + s3={'us_east_1_regional_endpoint':'regional'} + ) + # + # Set up "logging" to appropriately display messages. It can be set it up + # to send messages to a syslog server. + logging.basicConfig(datefmt='%Y-%m-%d_%H:%M:%S', format='%(asctime)s:%(name)s:%(levelname)s:%(message)s', encoding='utf-8') + logger = logging.getLogger("auto_create_sm_relationships") + logger.setLevel(logging.DEBUG) + # logger.setLevel(logging.INFO) + # + # Set the logging level higher for these noisy modules to mute thier messages + # when debugging things. + logging.getLogger("boto3").setLevel(logging.WARNING) + logging.getLogger("urllib3").setLevel(logging.WARNING) + # + # Read in the configuration from the environment variables, global variables or an s3 bucket. + readinConfig() + # + # Get the list of regions to check. It can be set in the environment + # variable "regions" or in the global variable "regions". If neither + # is set, then it will check all regions. + regionsEnv = config['regions'] + if regionsEnv is not None: + limitRegions = [item.strip() for item in regionsEnv.split(',')] + else: + limitRegions = [] + ec2Client = boto3.client('ec2', config=boto3Config) + ec2Regions = ec2Client.describe_regions()['Regions'] + for region in ec2Regions: + limitRegions += [region['RegionName']] + ec2Client.close() + # + # + if dryRun: + logger.info('Running in Dry Run mode.') + # + # Open a client to the SecretsManager service. + secretsManagerClient = boto3.client('secretsmanager', region_name=config['secretsManagerRegion'], config=boto3Config) + # + # Disable warning about connecting to servers with self-signed SSL certificates. + urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) + # + # Set the https retries to 1. + retries = Retry(total=None, connect=1, read=1, redirect=10, status=0, other=0) # pylint: disable=E1123 + http = urllib3.PoolManager(cert_reqs='CERT_NONE', retries=retries) + # + # Create a counter of the number of SM relationships created. + numSnapMirrorRelationships = 0 + # + # Get the list of regions that support fsx. + fsxRegions = boto3.Session().get_available_regions('fsx') + # + # Create an array of the FSxN IDs to scan. + fsxIdsEnv = config['fsxIds'] + if fsxIdsEnv is not None: + limitFsxIds = [item.strip() for item in fsxIdsEnv.split(',')] + else: + limitFsxIds = [] + + try: + for regionName in limitRegions: + # + # Skip regions that don't support fsx. + if regionName in fsxRegions: + logger.debug(f'Scanning region {regionName}.') + fsxClient = boto3.client('fsx', region_name=regionName, config=boto3Config) + tagsClient = boto3.client('resourcegroupstaggingapi', region_name=regionName, config=boto3Config) + try: + # + # Create an array of all the AWS FSx file systems. + data = fsxClient.describe_file_systems() + fsxs = data['FileSystems'] + nextToken = data.get('NextToken') + while nextToken is not None: + data = fsxClient.describe_file_systems(NextToken=nextToken) + fsxs += data['FileSystems'] + nextToken = data.get('NextToken') + # + # Create an array of all the AWS volumes. + data = fsxClient.describe_volumes() + awsVolumes = data['Volumes'] + nextToken = data.get('NextToken') + while nextToken is not None: + data = fsxClient.describe_volumes(NextToken=nextToken) + awsVolumes += data['Volumes'] + nextToken = data.get('NextToken') + # + # Create an array of all the tags associated with the fsx resource type. + data = tagsClient.get_resources(ResourceTypeFilters=['fsx'], ResourcesPerPage=100) + tags = data['ResourceTagMappingList'] + nextToken = data.get('PaginationToken', "") + + while nextToken != "": + response = tagsClient.get_resources(ResourceTypeFilters=['fsx'], ResourcesPerPage=100, PaginationToken=nextToken) + tags += response['ResourceTagMappingList'] + nextToken = response.get('PaginationToken', "") + # + # Handle the case where one of those AWS API calls fails due to a timeout. + # Chances are it means the region is down. + except (botocore.exceptions.ReadTimeoutError, botocore.exceptions.ConnectTimeoutError, botocore.exceptions.ConnectionClosedError) as e: + logger.warning(f"boto3 client error while getting the list of FSxNs, volumes or resource tags from region {regionName}. Exception: {e}. Skipping the region.") + fsxClient.close() + tagsClient.close() + continue + + fsxClient.close() + tagsClient.close() + # + # Loop on all the file systems in the region. + for fsxn in fsxs: + # + # Skip file systems that are not ONTAP or are not listed in the limitFsxIds array of defined.. + if fsxn['FileSystemType'] == "ONTAP" and (limitFsxIds == [] or fsxn['FileSystemId'] in limitFsxIds): + fsxnId = fsxn['FileSystemId'] + if (fsxn.get('OntapConfiguration') is not None and + fsxn['OntapConfiguration'].get('Endpoints') is not None and + fsxn['OntapConfiguration']['Endpoints'].get('Management') is not None and + fsxn['OntapConfiguration']['Endpoints']['Management'].get('IpAddresses') is not None): + fsxnIp = fsxn['OntapConfiguration']['Endpoints']['Management']['IpAddresses'][0] + else: + logger.warning(f'No management IP address found for fsxId: {fsxnId}. Probably because it is being created, or deleted.') + continue + logger.debug(f'Getting all the volumes from fsxn with IP {fsxnIp}.') + ontapVolumes = getOntapVolumes(fsxnId, fsxnIp) + for ontapVolume in ontapVolumes: + # + # Only create SnapMirror relationships for RW volumes that arne't clones or FlexCaches. + if ontapVolume['type'].lower() == "rw" and ontapVolume.get('clone') is not None and not ontapVolume['clone']['is_flexclone'] and ontapVolume.get('flexcache_endpoint_type') != "cache": + volumeUUID = ontapVolume['uuid'] + volumeARN = getVolumeARN(awsVolumes, volumeUUID) + if volumeARN != "": + protectTag = getVolumeProtectTagValue(tags, volumeARN) + + if config['protectAll'] and protectTag != "skip" or not config['protectAll'] and protectTag == "protect": + volumeName = ontapVolume['name'] + svmName = ontapVolume['svm']['name'] + (partnerId, partnerIp, partnerSvmName, partnerSvmSourceName) = getPartnerInfo(fsxnId, svmName) + if partnerId == "": + logger.warning(f'No partner found for fsxId: {fsxnId} and svmName: {svmName}.') + continue + + if getCredentials(partnerId) == ("", ""): + logger.warning(f'No credentials found for partner fsxId: {partnerId}.') + continue + + protectVolume(fsxnId, svmName, volumeName, partnerId, partnerIp, partnerSvmName, partnerSvmSourceName) + + if numSnapMirrorRelationships >= config['maxSnapMirrorRelationships']: + raise TooManySMs("Too Many SnapMirror relationships being created.") + except TooManySMs: + logger.warning(f'Hit the maximum number of SnapMirorr relationships ({numSnapMirrorRelationships}) created in one run. Exiting.') + + return + + if os.environ.get('AWS_LAMBDA_FUNCTION_NAME') == None: + lambda_handler(None, None) diff --git a/Management-Utilities/auto_create_sm_relationships/updateCFTemplate b/Management-Utilities/auto_create_sm_relationships/updateCFTemplate new file mode 100755 index 00000000..3e345972 --- /dev/null +++ b/Management-Utilities/auto_create_sm_relationships/updateCFTemplate @@ -0,0 +1,55 @@ +#!/bin/bash +# +# This script is used to update the CloudFormation template with the latest +# version of the Lambda function. It will also update the version number in +# the template. +################################################################################# + +majorVersionNum=1 +file="auto_create_sm_relationships.py" +codeChanged=1 + +tmpfile1=$(mktemp /tmp/tmpfile1.XXXXXX) +tmpfile2=$(mktemp /tmp/tmpfile2.XXXXXX) +trap "rm -f $tmpfile1 $tmpfile2" exit +# +# First get the code out of the CF template so we can compare it. +# Assume it is the last thing in the file. +sed -e '1,\(#!/bin/python3(d' cloudformation.yaml > cloudformation.yaml.tmp +# +# Now get the Date and Version lines out of both files so that's not included in the diff. +egrep -v '^ # Date:|^ # Version' cloudformation.yaml.tmp > $tmpfile1 +sed -e 1d $file | egrep -v '^# Date:|^# Version:' > $tmpfile2 +# +# Now compare the two ignoring whitespace. +if diff -w $tmpfile1 $tmpfile2 > /dev/null; then + echo "No changes to the code." + codeChanged=0 +fi + +if [ $codeChanged -eq 0 ]; then + echo "No changes to CloudFormation template required." + rm -f cloudformation.yaml.tmp + rm -f $tmpfile1 $tmpfile2 + exit 0 +fi + +if [ $codeChanged -eq 1 ]; then + echo "Updating monitor code in CloudFormation template." + # + # Get the number of commits in the git history for the file to calculate the minor version number. + minorVersionNum="$(git log "$file" | egrep '^commit' | wc -l)" + if [ -z "$minorVersionNum" ]; then + echo "Failed to calculate version number." 1>&2 + exit 1 + fi + + version="v${majorVersionNum}.${minorVersionNum}" + # + # Strip out the monitoring code. + sed -e '\(#!/bin/python3(,$d' cloudformation.yaml > cloudformation.yaml.tmp + # + # Add the monitoring code to the CF template while updating the version and date. + cat "$file" | sed -e 's/^/ /' -e "s/%%VERSION%%/${version}/" -e "s/%%DATE%%/$(date +%Y-%m-%d-%H:%M:%S)/" >> cloudformation.yaml.tmp + mv cloudformation.yaml.tmp cloudformation.yaml +fi From f3dfe6e66ad42d082a93329ab2b125395b977d60 Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Sat, 3 Oct 2026 09:31:24 -0500 Subject: [PATCH 10/23] Updated it to check the destinatio file system to see if the relationship exist. Added support for an S3 bucket to contain the partners and secrets information. Added a CloudFormation template. --- Management-Utilities/auto_create_sm_relationships/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Management-Utilities/auto_create_sm_relationships/README.md b/Management-Utilities/auto_create_sm_relationships/README.md index 8d6cd611..df837b80 100644 --- a/Management-Utilities/auto_create_sm_relationships/README.md +++ b/Management-Utilities/auto_create_sm_relationships/README.md @@ -189,7 +189,7 @@ fs-07bcb7adXXXXXXXXX,FSxSecret-Default,username,password Here are the permissions that are required for the program to run: |Action|Resource|Description| -|-|-| +|-|-|-| |fsx:DescribeFileSystems| `*` | Allows the program to discovery the file systems.| |fsx:DescribeVolumes| `*` | Allows the program to discovery the volumes within the file systems.| |tag:GetResources|`*` | Allows the program to get tag information.| From 6e1e8a97b7e2272a1a408a80c647cef1a4a4ea32 Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Sat, 3 Oct 2026 09:56:45 -0500 Subject: [PATCH 11/23] Updated it to check the destinatio file system to see if the relationship exist. Added support for an S3 bucket to contain the partners and secrets information. Added a CloudFormation template. --- .../auto_create_sm_relationships/README.md | 2 +- .../auto_create_sm_relationships.py | 16 ++++++++-------- .../cloudformation.yaml | 2 +- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/Management-Utilities/auto_create_sm_relationships/README.md b/Management-Utilities/auto_create_sm_relationships/README.md index df837b80..83948484 100644 --- a/Management-Utilities/auto_create_sm_relationships/README.md +++ b/Management-Utilities/auto_create_sm_relationships/README.md @@ -189,7 +189,7 @@ fs-07bcb7adXXXXXXXXX,FSxSecret-Default,username,password Here are the permissions that are required for the program to run: |Action|Resource|Description| -|-|-|-| +|-|:-:|-| |fsx:DescribeFileSystems| `*` | Allows the program to discovery the file systems.| |fsx:DescribeVolumes| `*` | Allows the program to discovery the volumes within the file systems.| |tag:GetResources|`*` | Allows the program to get tag information.| diff --git a/Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py b/Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py index 4d801287..3e330ef5 100755 --- a/Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py +++ b/Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py @@ -222,6 +222,7 @@ # snapmirror relationships have been created. ################################################################################ class TooManySMs(Exception): + """ Custom exception to raise when too many SnapMirror relationships have been created. """ # Constructor or Initializer def __init__(self, value): @@ -430,7 +431,7 @@ def protectVolume(fsxId, svmName, volumeName, partnerId, partnerIp, partnerSvmNa if state == "success": logger.info(f'SnapMirror relationship for {fsxId}::{svmName}:{volumeName} to {partnerIp}::{partnerSvmName}:{volumeName}{config["destinationVolumeSuffix"]} was successfully created.') - numSnapMirrorRelationships += 1 + numSnapMirrorRelationships += 1 # pylint: disable=E0602 else: if body.get("error") is not None and body["error"].get("message") is not None: errMessage = body["error"]["message"] @@ -530,7 +531,7 @@ def readinConfig(): config[var] = None # # Convert the logical string to a logic. - if config.get('protectAll') is not None and type(config['protectAll']) == str and config['protectAll'].lower() == "false": + if config.get('protectAll') is not None and type(config['protectAll']) == str and config['protectAll'].lower() == "false": # pylint: disable=E1101 config['protectAll'] = False else: config['protectAll'] = True @@ -653,8 +654,8 @@ def lambda_handler(event, context): # to send messages to a syslog server. logging.basicConfig(datefmt='%Y-%m-%d_%H:%M:%S', format='%(asctime)s:%(name)s:%(levelname)s:%(message)s', encoding='utf-8') logger = logging.getLogger("auto_create_sm_relationships") - logger.setLevel(logging.DEBUG) -# logger.setLevel(logging.INFO) + # logger.setLevel(logging.DEBUG) + logger.setLevel(logging.INFO) # # Set the logging level higher for these noisy modules to mute thier messages # when debugging things. @@ -677,8 +678,7 @@ def lambda_handler(event, context): for region in ec2Regions: limitRegions += [region['RegionName']] ec2Client.close() - # - # + if dryRun: logger.info('Running in Dry Run mode.') # @@ -710,7 +710,7 @@ def lambda_handler(event, context): # # Skip regions that don't support fsx. if regionName in fsxRegions: - logger.debug(f'Scanning region {regionName}.') + logger.info(f'Scanning region {regionName}.') fsxClient = boto3.client('fsx', region_name=regionName, config=boto3Config) tagsClient = boto3.client('resourcegroupstaggingapi', region_name=regionName, config=boto3Config) try: @@ -768,7 +768,7 @@ def lambda_handler(event, context): else: logger.warning(f'No management IP address found for fsxId: {fsxnId}. Probably because it is being created, or deleted.') continue - logger.debug(f'Getting all the volumes from fsxn with IP {fsxnIp}.') + logger.info(f'Getting all the volumes from {fsxnId}.') ontapVolumes = getOntapVolumes(fsxnId, fsxnIp) for ontapVolume in ontapVolumes: # diff --git a/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml b/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml index 9d49634e..9611d2f6 100644 --- a/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml +++ b/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml @@ -207,7 +207,7 @@ Parameters: maxRunTime: Description: "The maximum time, in seconds, the Lambda function is allowed to run before being terminated. Must be between 60 and 900 seconds." Type: Number - Default: 60 + Default: 90 MinValue: 60 MaxValue: 900 From 10332fd4e1cb852b49fed692993427bffd794a9c Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Sat, 3 Oct 2026 09:56:58 -0500 Subject: [PATCH 12/23] Updated it to check the destinatio file system to see if the relationship exist. Added support for an S3 bucket to contain the partners and secrets information. Added a CloudFormation template. --- .../cloudformation.yaml | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml b/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml index 9611d2f6..aaa41872 100644 --- a/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml +++ b/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml @@ -366,8 +366,8 @@ Resources: # creates a SnapMirror relationship using the partner information # provided below. # - # Version: v1.5 - # Date: 2026-10-03-09:21:59 + # Version: v1.7 + # Date: 2026-10-03-09:56:51 # ################################################################################ # @@ -573,6 +573,7 @@ Resources: # snapmirror relationships have been created. ################################################################################ class TooManySMs(Exception): + """ Custom exception to raise when too many SnapMirror relationships have been created. """ # Constructor or Initializer def __init__(self, value): @@ -781,7 +782,7 @@ Resources: if state == "success": logger.info(f'SnapMirror relationship for {fsxId}::{svmName}:{volumeName} to {partnerIp}::{partnerSvmName}:{volumeName}{config["destinationVolumeSuffix"]} was successfully created.') - numSnapMirrorRelationships += 1 + numSnapMirrorRelationships += 1 # pylint: disable=E0602 else: if body.get("error") is not None and body["error"].get("message") is not None: errMessage = body["error"]["message"] @@ -881,7 +882,7 @@ Resources: config[var] = None # # Convert the logical string to a logic. - if config.get('protectAll') is not None and type(config['protectAll']) == str and config['protectAll'].lower() == "false": + if config.get('protectAll') is not None and type(config['protectAll']) == str and config['protectAll'].lower() == "false": # pylint: disable=E1101 config['protectAll'] = False else: config['protectAll'] = True @@ -1004,8 +1005,8 @@ Resources: # to send messages to a syslog server. logging.basicConfig(datefmt='%Y-%m-%d_%H:%M:%S', format='%(asctime)s:%(name)s:%(levelname)s:%(message)s', encoding='utf-8') logger = logging.getLogger("auto_create_sm_relationships") - logger.setLevel(logging.DEBUG) - # logger.setLevel(logging.INFO) + # logger.setLevel(logging.DEBUG) + logger.setLevel(logging.INFO) # # Set the logging level higher for these noisy modules to mute thier messages # when debugging things. @@ -1028,8 +1029,7 @@ Resources: for region in ec2Regions: limitRegions += [region['RegionName']] ec2Client.close() - # - # + if dryRun: logger.info('Running in Dry Run mode.') # @@ -1061,7 +1061,7 @@ Resources: # # Skip regions that don't support fsx. if regionName in fsxRegions: - logger.debug(f'Scanning region {regionName}.') + logger.info(f'Scanning region {regionName}.') fsxClient = boto3.client('fsx', region_name=regionName, config=boto3Config) tagsClient = boto3.client('resourcegroupstaggingapi', region_name=regionName, config=boto3Config) try: @@ -1119,7 +1119,7 @@ Resources: else: logger.warning(f'No management IP address found for fsxId: {fsxnId}. Probably because it is being created, or deleted.') continue - logger.debug(f'Getting all the volumes from fsxn with IP {fsxnIp}.') + logger.info(f'Getting all the volumes from {fsxnId}.') ontapVolumes = getOntapVolumes(fsxnId, fsxnIp) for ontapVolume in ontapVolumes: # From 44c481d1bf53fd2a6ee763fc9b36e96222875b8d Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Sat, 3 Oct 2026 10:00:40 -0500 Subject: [PATCH 13/23] Updated it to check the destinatio file system to see if the relationship exist. Added support for an S3 bucket to contain the partners and secrets information. Added a CloudFormation template. --- .../auto_create_sm_relationships/cloudformation.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml b/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml index aaa41872..67fffc30 100644 --- a/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml +++ b/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml @@ -277,7 +277,7 @@ Resources: Action: - "fsx:DescribeFileSystems" - "fsx:DescribeVolumes" - - "tags:ListTagsForResource" + - "tag:GetResources" - "ec2:DescribeRegions" Resource: - "*" From 883b54b77b1c46f74de6da99fe9bef38aac77b3c Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Tue, 6 Oct 2026 07:54:00 -0500 Subject: [PATCH 14/23] Added support for a space in the CIFS share name or path. --- .../Workload-Factory-API-Samples/volume_cifs_share_create | 8 ++++---- .../Workload-Factory-API-Samples/volume_cifs_share_delete | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/Management-Utilities/Workload-Factory-API-Samples/volume_cifs_share_create b/Management-Utilities/Workload-Factory-API-Samples/volume_cifs_share_create index a285f137..f82af7e9 100755 --- a/Management-Utilities/Workload-Factory-API-Samples/volume_cifs_share_create +++ b/Management-Utilities/Workload-Factory-API-Samples/volume_cifs_share_create @@ -112,8 +112,8 @@ required_options["CIFS_SHARE_NAME"]='Error: The name of the cifs share is requir check_required_options # -# Check that the CIFS_SHARE_NAME only contains allowed characters (alphanumeric, underscores and dashes). -if [[ ! "$CIFS_SHARE_NAME" =~ ^[a-zA-Z0-9_-]+$ ]]; then +# Check that the CIFS_SHARE_NAME only contains allowed characters (alphanumeric, underscores, spaces and dashes). +if [[ ! "$CIFS_SHARE_NAME" =~ ^[[:alnum:]_[:space:]-]+$ ]]; then echo "Error: The CIFS share name can only contain alphanumeric characters, underscores and dashes." >&2 exit 1 fi @@ -135,7 +135,7 @@ fi # Get the existing shares and the junction path run_curl GET "$token" "https://api.workloads.netapp.com/accounts/${BLUEXP_ACCOUNT_ID}/fsx/v2/credentials/${CREDENTIALS_ID}/regions/${AWS_REGION}/file-systems/${FILESYSTEM_ID}/volumes/${VOLUME_ID}?include=cifsShares" $tmpout $tmperr cifsShares=$(jq -r '.cifsShares' $tmpout) -alreadExist=$(echo "$cifsShares" | jq -r 'map(select(.name == "'$CIFS_SHARE_NAME'")) | length') +alreadExist=$(echo "$cifsShares" | jq -r --arg shareName "$CIFS_SHARE_NAME" 'map(select(.name == $shareName)) | length') if [ "$alreadExist" -gt 0 ]; then echo "A share with the name '$CIFS_SHARE_NAME' already exists. No changes will be made." >&2 exit 1 @@ -159,6 +159,6 @@ newShare='{"name":"'$CIFS_SHARE_NAME'","path":"'$SHARE_PATH'","acls":[{"userOrGr echo "Creating share named '$CIFS_SHARE_NAME' with path '$SHARE_PATH'." # # Add the new share to the existing ones. -cifsShares=$(echo "$cifsShares" | jq -r '. += ['$newShare']') +cifsShares=$(echo "$cifsShares" | jq -r --argjson newShare "$newShare" '. += [$newShare]') cifsShares="{\"cifsShares\": $cifsShares}" run_curl PATCH "$token" "https://api.workloads.netapp.com/accounts/${BLUEXP_ACCOUNT_ID}/fsx/v2/credentials/${CREDENTIALS_ID}/regions/${AWS_REGION}/file-systems/${FILESYSTEM_ID}/volumes/${VOLUME_ID}" $tmpout $tmperr "$cifsShares" diff --git a/Management-Utilities/Workload-Factory-API-Samples/volume_cifs_share_delete b/Management-Utilities/Workload-Factory-API-Samples/volume_cifs_share_delete index f45df635..12f97f9d 100755 --- a/Management-Utilities/Workload-Factory-API-Samples/volume_cifs_share_delete +++ b/Management-Utilities/Workload-Factory-API-Samples/volume_cifs_share_delete @@ -118,6 +118,6 @@ run_curl GET "$token" "https://api.workloads.netapp.com/accounts/${BLUEXP_ACCOUN cifsShares=$(jq -r '.cifsShares' $tmpout) # # Remove the specified share from the existing ones. -cifsShares=$(echo "$cifsShares" | jq -r 'map(select(.name != "'$CIFS_SHARE_NAME'"))') +cifsShares=$(echo "$cifsShares" | jq -r --arg shareName "$CIFS_SHARE_NAME" 'map(select(.name != $shareName))') cifsShares="{\"cifsShares\": $cifsShares}" run_curl PATCH "$token" "https://api.workloads.netapp.com/accounts/${BLUEXP_ACCOUNT_ID}/fsx/v2/credentials/${CREDENTIALS_ID}/regions/${AWS_REGION}/file-systems/${FILESYSTEM_ID}/volumes/${VOLUME_ID}" $tmpout $tmperr "$cifsShares" From 9e881de32c662a352df32c0feec01a5e043ff93d Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Tue, 6 Oct 2026 13:16:49 -0500 Subject: [PATCH 15/23] Updated it to check the destinatio file system to see if the relationship exist. Added support for an S3 bucket to contain the partners and secrets information. Added a CloudFormation template. --- .../auto_create_sm_relationships.py | 12 +++++++----- .../auto_create_sm_relationships/cloudformation.yaml | 10 ++++++---- 2 files changed, 13 insertions(+), 9 deletions(-) diff --git a/Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py b/Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py index 3e330ef5..ee9d1241 100755 --- a/Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py +++ b/Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py @@ -462,7 +462,7 @@ def getOntapVolumes(fsxId, fsxnIp): volumes = [] try: - endpoint = f'https://{fsxnIp}/api/storage/volumes?fields=name,svm,type,clone,flexcache_endpoint_type' + endpoint = f'https://{fsxnIp}/api/storage/volumes?fields=name,svm,type,clone,flexcache_endpoint_type,is_svm_root' logger.debug(f'Trying {endpoint}.') response = http.request('GET', endpoint, headers=headers, timeout=5.0) if response.status == 200: @@ -772,8 +772,10 @@ def lambda_handler(event, context): ontapVolumes = getOntapVolumes(fsxnId, fsxnIp) for ontapVolume in ontapVolumes: # - # Only create SnapMirror relationships for RW volumes that arne't clones or FlexCaches. - if ontapVolume['type'].lower() == "rw" and ontapVolume.get('clone') is not None and not ontapVolume['clone']['is_flexclone'] and ontapVolume.get('flexcache_endpoint_type') != "cache": + # Only create SnapMirror relationships for RW volumes that arne't clones, vserver_roots or FlexCaches. + if (ontapVolume['type'].lower() == "rw" and ontapVolume.get('clone') is not None + and not ontapVolume['clone']['is_flexclone'] and ontapVolume.get('flexcache_endpoint_type') != "cache" + and ontapVolume.get('is_svm_root') is not True): volumeUUID = ontapVolume['uuid'] volumeARN = getVolumeARN(awsVolumes, volumeUUID) if volumeARN != "": @@ -784,11 +786,11 @@ def lambda_handler(event, context): svmName = ontapVolume['svm']['name'] (partnerId, partnerIp, partnerSvmName, partnerSvmSourceName) = getPartnerInfo(fsxnId, svmName) if partnerId == "": - logger.warning(f'No partner found for fsxId: {fsxnId} and svmName: {svmName}.') + logger.warning(f'No partner found for fsxId: {fsxnId} and svmName: {svmName} while trying to protect {volumeName}.') continue if getCredentials(partnerId) == ("", ""): - logger.warning(f'No credentials found for partner fsxId: {partnerId}.') + logger.warning(f'No credentials found for partner fsxId: {partnerId} while trying to protect {volumeName}.') continue protectVolume(fsxnId, svmName, volumeName, partnerId, partnerIp, partnerSvmName, partnerSvmSourceName) diff --git a/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml b/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml index 67fffc30..e553d783 100644 --- a/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml +++ b/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml @@ -367,7 +367,7 @@ Resources: # provided below. # # Version: v1.7 - # Date: 2026-10-03-09:56:51 + # Date: 2026-10-06-10:44:45 # ################################################################################ # @@ -813,7 +813,7 @@ Resources: volumes = [] try: - endpoint = f'https://{fsxnIp}/api/storage/volumes?fields=name,svm,type,clone,flexcache_endpoint_type' + endpoint = f'https://{fsxnIp}/api/storage/volumes?fields=name,svm,type,clone,flexcache_endpoint_type,is_svm_root' logger.debug(f'Trying {endpoint}.') response = http.request('GET', endpoint, headers=headers, timeout=5.0) if response.status == 200: @@ -1123,8 +1123,10 @@ Resources: ontapVolumes = getOntapVolumes(fsxnId, fsxnIp) for ontapVolume in ontapVolumes: # - # Only create SnapMirror relationships for RW volumes that arne't clones or FlexCaches. - if ontapVolume['type'].lower() == "rw" and ontapVolume.get('clone') is not None and not ontapVolume['clone']['is_flexclone'] and ontapVolume.get('flexcache_endpoint_type') != "cache": + # Only create SnapMirror relationships for RW volumes that arne't clones, vserver_roots or FlexCaches. + if (ontapVolume['type'].lower() == "rw" and ontapVolume.get('clone') is not None + and not ontapVolume['clone']['is_flexclone'] and ontapVolume.get('flexcache_endpoint_type') != "cache" + and ontapVolume.get('is_svm_root') is not True): volumeUUID = ontapVolume['uuid'] volumeARN = getVolumeARN(awsVolumes, volumeUUID) if volumeARN != "": From 9189fdd0ba942390ebc384085a31987cd0ef9fe3 Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Tue, 6 Oct 2026 16:51:15 -0500 Subject: [PATCH 16/23] Updated it to check the destinatio file system to see if the relationship exist. Added support for an S3 bucket to contain the partners and secrets information. Added a CloudFormation template. --- ...nTemplate-auto_create_sm_relationships.xml | 42 +++++++++++++++++++ .../auto_create_sm_relationships/README.md | 5 ++- .../auto_create_sm_relationships.py | 2 +- .../cloudformation.yaml | 10 ++--- 4 files changed, 51 insertions(+), 8 deletions(-) create mode 100644 .github/workflows/update-CloudformationTemplate-auto_create_sm_relationships.xml diff --git a/.github/workflows/update-CloudformationTemplate-auto_create_sm_relationships.xml b/.github/workflows/update-CloudformationTemplate-auto_create_sm_relationships.xml new file mode 100644 index 00000000..3943873a --- /dev/null +++ b/.github/workflows/update-CloudformationTemplate-auto_create_sm_relationships.xml @@ -0,0 +1,42 @@ +--- +# Copyright (c) NetApp, Inc. +# SPDX-License-Identifier: Apache-2.0 + +name: "Update Cloudformation Template" + +on: + pull_request: + paths: + - 'Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py' + push: + paths: + - 'Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py' + branches: + - main + +jobs: + update-Cloudformation-Template: + runs-on: ubuntu-latest + permissions: + # Give the default GITHUB_TOKEN write permission to commit and push the + # added or changed files to the repository. + contents: write + + steps: + - name: Harden the runner (Audit all outbound calls) + uses: step-security/harden-runner@fa2e9d605c4eeb9fcad4c99c224cee0c6c7f3594 # v2.16.0 + with: + egress-policy: audit + + - name: Checkout pull request + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + with: + ref: ${{ github.event.pull_request.head.ref }} + + - name: Update the Cloudformation Template + shell: bash + working-directory: Management-Utilities/auto_create_sm_relationships + run: ./update-CFTemplate + + - name: Commit the changes + uses: step-security/git-auto-commit-action@905c3cd6e9ed2b67b4d46ff401fdb6d745d0ff9d # v7.1.0 diff --git a/Management-Utilities/auto_create_sm_relationships/README.md b/Management-Utilities/auto_create_sm_relationships/README.md index 83948484..202e7d3b 100644 --- a/Management-Utilities/auto_create_sm_relationships/README.md +++ b/Management-Utilities/auto_create_sm_relationships/README.md @@ -11,9 +11,10 @@ are SnapMirror'ed to a remote FSxN file system. It does this by: - A RW type volume - Not a clone volume - Not a FlexCache volume - - Not already in an existing SnapMirror relationship + - Not a vserver root volume + - Not already in an existing SnapMirror relationship to the specified destination - Doesn't have an AWS tag with a key of `protect_volume` with a value of `skip`. -- If it passes all those check then it uses the ONTAP SnapMirror API to create the SnapMirror relationship. +- If it passes all those checks then it uses the ONTAP SnapMirror API to create the SnapMirror relationship. **NOTES** - The ONTAP API creates the destination volume and will error out if the destination volume already exists. diff --git a/Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py b/Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py index ee9d1241..664c1406 100755 --- a/Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py +++ b/Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py @@ -527,7 +527,7 @@ def readinConfig(): # to an empty string if someone doesn't provide a value, reset the # values back to None. for var in config: - if config[var] == "": + if config[var] == "" and var != "destinationVolumeSuffix": config[var] = None # # Convert the logical string to a logic. diff --git a/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml b/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml index e553d783..cf28bf8d 100644 --- a/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml +++ b/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml @@ -366,8 +366,8 @@ Resources: # creates a SnapMirror relationship using the partner information # provided below. # - # Version: v1.7 - # Date: 2026-10-06-10:44:45 + # Version: v1.8 + # Date: 2026-10-06-16:42:22 # ################################################################################ # @@ -878,7 +878,7 @@ Resources: # to an empty string if someone doesn't provide a value, reset the # values back to None. for var in config: - if config[var] == "": + if config[var] == "" and var != "destinationVolumeSuffix": config[var] = None # # Convert the logical string to a logic. @@ -1137,11 +1137,11 @@ Resources: svmName = ontapVolume['svm']['name'] (partnerId, partnerIp, partnerSvmName, partnerSvmSourceName) = getPartnerInfo(fsxnId, svmName) if partnerId == "": - logger.warning(f'No partner found for fsxId: {fsxnId} and svmName: {svmName}.') + logger.warning(f'No partner found for fsxId: {fsxnId} and svmName: {svmName} while trying to protect {volumeName}.') continue if getCredentials(partnerId) == ("", ""): - logger.warning(f'No credentials found for partner fsxId: {partnerId}.') + logger.warning(f'No credentials found for partner fsxId: {partnerId} while trying to protect {volumeName}.') continue protectVolume(fsxnId, svmName, volumeName, partnerId, partnerIp, partnerSvmName, partnerSvmSourceName) From c6bbf5f8a0c88166ca39cb4daaf75b6b08c0bc5a Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Tue, 6 Oct 2026 20:10:47 -0500 Subject: [PATCH 17/23] Fixed a typo --- ...ate-CloudformationTemplate-auto_create_sm_relationships.yml} | 0 Management-Utilities/warm_performance_tier/README.md | 2 +- 2 files changed, 1 insertion(+), 1 deletion(-) rename .github/workflows/{update-CloudformationTemplate-auto_create_sm_relationships.xml => update-CloudformationTemplate-auto_create_sm_relationships.yml} (100%) diff --git a/.github/workflows/update-CloudformationTemplate-auto_create_sm_relationships.xml b/.github/workflows/update-CloudformationTemplate-auto_create_sm_relationships.yml similarity index 100% rename from .github/workflows/update-CloudformationTemplate-auto_create_sm_relationships.xml rename to .github/workflows/update-CloudformationTemplate-auto_create_sm_relationships.yml diff --git a/Management-Utilities/warm_performance_tier/README.md b/Management-Utilities/warm_performance_tier/README.md index a66549fc..5bb1f21a 100644 --- a/Management-Utilities/warm_performance_tier/README.md +++ b/Management-Utilities/warm_performance_tier/README.md @@ -53,7 +53,7 @@ if you are trying to get as much data as possible into the performance tier, it is recommended to run the script twice. There are two versions of the script: -o `warm_performance_tier` - Disigned for Linux. +o `warm_performance_tier` - Designed for Linux. o `WarmPerformanceTier.ps1` - Designed for Windows. ## Set Up From e58582183713fede6fc0c7f87e57639e6f17446e Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Tue, 6 Oct 2026 20:11:57 -0500 Subject: [PATCH 18/23] Added a set fsxn password script. --- .../set_fsxn_password | 24 ++++++++++++------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/Management-Utilities/fsx-ontap-aws-cli-scripts/set_fsxn_password b/Management-Utilities/fsx-ontap-aws-cli-scripts/set_fsxn_password index f9709217..6b32a511 100755 --- a/Management-Utilities/fsx-ontap-aws-cli-scripts/set_fsxn_password +++ b/Management-Utilities/fsx-ontap-aws-cli-scripts/set_fsxn_password @@ -9,14 +9,16 @@ ################################################################################ usage () { cat 1>&2 <&2 usage fi +# +# Ensure all the required parameters have been provided. +if [ -z "$fileSystemName" -a -z "$fsid" ]; then + echo "Error, missing required arguments." 1>&2 + usage +fi if [ -z "$password" ]; then read -sp "Enter the password to set for the fsxadmin user: " password1 @@ -79,8 +87,8 @@ if [ -z "$password" ]; then fi # # Ensure all the required parameters have been provided. -if [ -z "$password" -o -z "$fileSystemName" -a -z "$fsid" ]; then - echo "Error, missing required arguments." 1>&2 +if [ -z "$password" ]; then + echo "Error, the password can't be blank." usage fi # @@ -95,7 +103,7 @@ if [ -z "$fsid" ]; then fi # # Set the password -aws fsx update-file-system $endpoint_url --ontap-configuration '{"FsxAdminPassword": "'$password'"}' --region=$region --file-system-id "$fsid" --output=json > $tmpout 2>&1 +aws fsx update-file-system $endpoint_url --ontap-configuration '{"FsxAdminPassword": "'"$password"'"}' --region=$region --file-system-id "$fsid" --output=json > $tmpout 2>&1 if [ $? != "0" ]; then echo "Failed to set the password." 1>&2 cat $tmpout 1>&2 From 823cab49cbfb73b6e3e8f2ebbe40fbc412033409 Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Wed, 7 Oct 2026 08:59:36 -0500 Subject: [PATCH 19/23] Updated it to check the destinatio file system to see if the relationship exist. Added support for an S3 bucket to contain the partners and secrets information. Added a CloudFormation template. --- .../auto_create_sm_relationships/README.md | 2 +- .../auto_create_sm_relationships.py | 131 ++++++++-------- .../cloudformation.yaml | 143 +++++++++--------- 3 files changed, 144 insertions(+), 132 deletions(-) diff --git a/Management-Utilities/auto_create_sm_relationships/README.md b/Management-Utilities/auto_create_sm_relationships/README.md index 202e7d3b..23802d32 100644 --- a/Management-Utilities/auto_create_sm_relationships/README.md +++ b/Management-Utilities/auto_create_sm_relationships/README.md @@ -145,7 +145,7 @@ python3 auto_create_sm_relationships.py #### Partners Table This table provides the association with a source FSxN file system to its partner cluster (i.e. where its volumes should be SnapMirror'ed to.) There should be five fields for each entry: - soureceId - Which is the concatenation of the source file system ID followed by a ":" followed by the SVM name. It is done this way because the id has to be unique in the table. It is split up into its two components in the script when it is read in. -- partnerFsxnId - Set to the AWS ID the partner FSxN file system. +- partnerFsxId - Set to the AWS ID the partner FSxN file system. - partnerFsxnIp - Set to the IP address of the management port of the partner FSxN file system. - partnerSvmName - The name of the SVM where you want the SnapMirror destination volume to reside. - partnerSvmSourceName - Is the "local name" of the source SVM. Usually, it is the same as the source SVM, but can be different if that same name already exists on the partner file system. When you peer the SVM it will require you to create an alias for the source SVM so all the SVM names are unique. diff --git a/Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py b/Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py index 664c1406..69c1fe16 100755 --- a/Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py +++ b/Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py @@ -334,34 +334,31 @@ def getSnapMirrorRelationships(fsxnId, fsxnIp): global logger, http - relationships = [] + (username, password) = getCredentials(fsxnId) + if username == "" or password == "": + logger.error(f'No credentials for FSxN ID: {fsxnId}.') + return relationships + + auth = urllib3.make_headers(basic_auth=f'{username}:{password}') + headers = { **auth } + + relationships = [] try: - (username, password) = getCredentials(fsxnId) - if username == "" or password == "": - logger.error(f'No credentials for FSxN ID: {fsxnId}.') - return relationships - - auth = urllib3.make_headers(basic_auth=f'{username}:{password}') - headers = { **auth } - - endpoint = f'https://{fsxnIp}/api/snapmirror/relationships?fields=source,destination' - logger.debug(f'Trying {endpoint}.') - response = http.request('GET', endpoint, headers=headers) - if response.status < 200 or response.status > 299: - logger.error(f'API call to {endpoint} failed. HTTP status code: {response.status}.') - return relationships - body = json.loads(response.data.decode('utf-8')) - relationships += body['records'] - while body.get('next'): - endpoint = f'https://{fsxnIp}{body["next"]}' + url = '/api/snapmirror/relationships?fields=source,destination' + while url is not None: + endpoint = f'https://{fsxnIp}{url}' logger.debug(f'Trying {endpoint}.') - response = http.request('GET', endpoint, headers=headers) + response = http.request('GET', endpoint, headers=headers, timeout=16) if response.status < 200 or response.status > 299: logger.error(f'API call to {endpoint} failed. HTTP status code: {response.status}.') return relationships - body = json.loads(response.data.decode('utf-8')) - relationships += body['records'] + data = json.loads(response.data.decode('utf-8')) + relationships.extend(data['records']) + if data.get("_links") is not None and data["_links"].get("next") is not None and data["_links"]["next"].get("href") is not None: + url = data["_links"]["next"]["href"] + else: + url = None except Exception as err: logger.critical(f'API against {fsxnIp} failed. The error returned: "{err}".') @@ -421,7 +418,7 @@ def protectVolume(fsxId, svmName, volumeName, partnerId, partnerIp, partnerSvmNa time.sleep(1) endpoint = f'https://{partnerIp}/api/cluster/jobs/{jobId}' logger.debug(f'Trying {endpoint}.') - response = http.request('GET', endpoint, headers=headers) + response = http.request('GET', endpoint, headers=headers, timeout=16) if response.status < 200 or response.status > 299: logger.error(f'API call to {endpoint} failed. HTTP status code: {response.status}.') return @@ -462,26 +459,20 @@ def getOntapVolumes(fsxId, fsxnIp): volumes = [] try: - endpoint = f'https://{fsxnIp}/api/storage/volumes?fields=name,svm,type,clone,flexcache_endpoint_type,is_svm_root' - logger.debug(f'Trying {endpoint}.') - response = http.request('GET', endpoint, headers=headers, timeout=5.0) - if response.status == 200: - data = json.loads(response.data) - volumes = data['records'] - else: - logger.error(f'API call to {endpoint} failed. HTTP status code: {response.status}.') - return(volumes) - - while data.get('next'): - endpoint = f'https://{fsxnIp}{data["next"]}' + url = '/api/storage/volumes?fields=name,svm,type,clone,flexcache_endpoint_type,is_svm_root' + while url is not None: + endpoint = f'https://{fsxnIp}{url}' logger.debug(f'Trying {endpoint}.') - response = http.request('GET', endpoint, headers=headers, timeout=5.0) - if response.status == 200: - data = json.loads(response.data) - volumes += data['records'] - else: + response = http.request('GET', endpoint, headers=headers, timeout=16) + if response.status < 200 or response.status > 299: logger.error(f'API call to {endpoint} failed. HTTP status code: {response.status}.') - break + return(volumes) + data = json.loads(response.data) + volumes.extend(data['records']) + if data.get("_links") is not None and data["_links"].get("next") is not None and data["_links"]["next"].get("href") is not None: + url = data["_links"]["next"]["href"] + else: + url = None except Exception as err: logger.critical(f'Failed to issue API against {fsxnIp}. Cluster could be down. The error messages received: "{err}".') @@ -536,6 +527,10 @@ def readinConfig(): else: config['protectAll'] = True # + # Convert string to an int. + if config.get('maxSnapMirrorRelationships') is not None: + config['maxSnapMirrorRelationships'] = int(config['maxSnapMirrorRelationships']) + # # If the s3BucketName is set, then open a client to the s3 service and read # in the partners and secrets config files. if config['s3BucketName'] is not None: @@ -592,36 +587,46 @@ def readinConfig(): # If the dynamodbRegion is set, then open a client to the dynamodb service and read # in the partners and secrets config tables. if config['dynamodbRegion'] is not None: - dynamodbClient = boto3.resource("dynamodb", region_name=config['dynamodbRegion']) - - if config['dynamodbSecretsTableName'] is not None: - table = dynamodbClient.Table(config['dynamodbSecretsTableName']) - response = table.scan() - secretsTable = response["Items"] - if config['dynamodbPartnersTableName'] is not None: - table = dynamodbClient.Table(config['dynamodbPartnersTableName']) - response = table.scan() - items = response["Items"] - partnersTable = [] - for item in items: - partnersTable.append({ - 'fsxId': item['sourceId'].split(":")[0], - 'svmName': item['sourceId'].split(":")[1], - 'partnerFsxId': item['partnerFsxId'], - 'partnerFsxnIp': item['partnerFsxnIp'], - 'partnerSvmName': item['partnerSvmName'], - 'partnerSvmSourceName': item['partnerSvmSourceName'] - }) + try: + dynamodbClient = boto3.resource("dynamodb", region_name=config['dynamodbRegion']) + + if config['dynamodbSecretsTableName'] is not None: + table = dynamodbClient.Table(config['dynamodbSecretsTableName']) + response = table.scan() + secretsTable = response["Items"] + while 'LastEvaluatedKey' in response: + response = table.scan(ExclusiveStartKey=response['LastEvaluatedKey']) + secretsTable.extend(response['Items']) + if config['dynamodbPartnersTableName'] is not None: + table = dynamodbClient.Table(config['dynamodbPartnersTableName']) + response = table.scan() + items = response["Items"] + while 'LastEvaluatedKey' in response: + response = table.scan(ExclusiveStartKey=response['LastEvaluatedKey']) + items.extend(response['Items']) + partnersTable = [] + for item in items: + partnersTable.append({ + 'fsxId': item['sourceId'].split(":")[0], + 'svmName': item['sourceId'].split(":")[1], + 'partnerFsxId': item['partnerFsxId'], + 'partnerFsxnIp': item['partnerFsxnIp'], + 'partnerSvmName': item['partnerSvmName'], + 'partnerSvmSourceName': item['partnerSvmSourceName'] + }) + except Exception as err: + message = f'Error, unable to read the secrets and/or partners config tables "{config["dynamodbSecretsTableName"]}", "{config["dynamodbPartnersTableName"]}" from Dynamodb region {config["dynamodbRegion"]}. Error message: "{err}".' + logger.critical(message) if 'partnersTable' not in globals() or len(partnersTable) == 0: message = "Error, No partners were have been configured." logger.critical(message) - raise Exception(message) + raise Exception(message) # Send up a flare. Hopefully they enabled the watchdog alarm and will get a notification. if 'secretsTable' not in globals() or len(secretsTable) == 0: message = "Error, No secrets were have been configured." logger.critical(message) - raise Exception(message) + raise Exception(message) # Send up a flare. Hopefully they enabled the watchdog alarm and will get a notification. ################################################################################ # This is the main logic of the program. It loops on all the regions, then all diff --git a/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml b/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml index cf28bf8d..0d4492cb 100644 --- a/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml +++ b/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml @@ -221,9 +221,11 @@ Conditions: CreateWatchdogAlarm: !Equals [!Ref createWatchdogAlarm, "true"] CreateLambdaRole: !Equals [!Ref roleArn, ""] IncludeDynamodbPartners: !Not [!Equals [!Ref dynamodbPartnersTableName, ""]] - IncludeDynamodbSecrets: !Not [!Equals [!Ref dynamodbPartnersTableName, ""]] + IncludeDynamodbSecrets: !Not [!Equals [!Ref dynamodbSecretsTableName, ""]] IncludeDynamodb: !Or [!Condition IncludeDynamodbPartners, !Condition IncludeDynamodbSecrets] IncludeS3: !Not [!Equals [!Ref s3BucketName, ""]] + IncludeSNS: !Not [!Equals [!Ref snsTopicArn, ""]] + IntervalRateUnitSingular: !Equals [!Ref checkInterval, 1] Resources: # @@ -248,7 +250,7 @@ Resources: Threshold: 0.5 ComparisonOperator: "GreaterThanThreshold" AlarmActions: - - !Ref snsTopicArn + - !If [IncludeSNS, !Ref snsTopicArn, !Ref "AWS::NoValue"] # # This is the role that will be assigned to the Lambda function. # It will only be created if the user hasn't provided a roleARN. @@ -303,7 +305,7 @@ Resources: Properties: Description: "Event rule to trigger the Lambda function." Name: !Sub "auto-create-sm-relationships-event-rule-${AWS::StackName}" - ScheduleExpression: !Sub "rate(${checkInterval} hours)" + ScheduleExpression: !If [IntervalRateUnitSingular, "rate(1 hour)", !Sub "rate(${checkInterval} hours)"] State: "ENABLED" Targets: - Arn: !GetAtt LambdaFunction.Arn @@ -338,7 +340,7 @@ Resources: snapMirrorPolicy: !Ref snapMirrorPolicy scheduleName: !Ref scheduleName tieringPolicy: !Ref tieringPolicy - maxSnapmirrorRelationships: !Ref maxSnapMirrorRelationships + maxSnapMirrorRelationships: !Ref maxSnapMirrorRelationships protectAll: !Ref protectAll s3BucketName: !Ref s3BucketName s3BucketRegion: !Ref s3BucketRegion @@ -366,8 +368,8 @@ Resources: # creates a SnapMirror relationship using the partner information # provided below. # - # Version: v1.8 - # Date: 2026-10-06-16:42:22 + # Version: v1.9 + # Date: 2026-10-07-08:58:52 # ################################################################################ # @@ -685,34 +687,31 @@ Resources: global logger, http - relationships = [] - try: - (username, password) = getCredentials(fsxnId) - if username == "" or password == "": - logger.error(f'No credentials for FSxN ID: {fsxnId}.') - return relationships + (username, password) = getCredentials(fsxnId) + if username == "" or password == "": + logger.error(f'No credentials for FSxN ID: {fsxnId}.') + return relationships - auth = urllib3.make_headers(basic_auth=f'{username}:{password}') - headers = { **auth } + auth = urllib3.make_headers(basic_auth=f'{username}:{password}') + headers = { **auth } - endpoint = f'https://{fsxnIp}/api/snapmirror/relationships?fields=source,destination' - logger.debug(f'Trying {endpoint}.') - response = http.request('GET', endpoint, headers=headers) - if response.status < 200 or response.status > 299: - logger.error(f'API call to {endpoint} failed. HTTP status code: {response.status}.') - return relationships - body = json.loads(response.data.decode('utf-8')) - relationships += body['records'] - while body.get('next'): - endpoint = f'https://{fsxnIp}{body["next"]}' + relationships = [] + try: + url = '/api/snapmirror/relationships?fields=source,destination' + while url is not None: + endpoint = f'https://{fsxnIp}{url}' logger.debug(f'Trying {endpoint}.') - response = http.request('GET', endpoint, headers=headers) + response = http.request('GET', endpoint, headers=headers, timeout=16) if response.status < 200 or response.status > 299: logger.error(f'API call to {endpoint} failed. HTTP status code: {response.status}.') return relationships - body = json.loads(response.data.decode('utf-8')) - relationships += body['records'] + data = json.loads(response.data.decode('utf-8')) + relationships.extend(data['records']) + if data.get("_links") is not None and data["_links"].get("next") is not None and data["_links"]["next"].get("href") is not None: + url = data["_links"]["next"]["href"] + else: + url = None except Exception as err: logger.critical(f'API against {fsxnIp} failed. The error returned: "{err}".') @@ -772,7 +771,7 @@ Resources: time.sleep(1) endpoint = f'https://{partnerIp}/api/cluster/jobs/{jobId}' logger.debug(f'Trying {endpoint}.') - response = http.request('GET', endpoint, headers=headers) + response = http.request('GET', endpoint, headers=headers, timeout=16) if response.status < 200 or response.status > 299: logger.error(f'API call to {endpoint} failed. HTTP status code: {response.status}.') return @@ -813,26 +812,20 @@ Resources: volumes = [] try: - endpoint = f'https://{fsxnIp}/api/storage/volumes?fields=name,svm,type,clone,flexcache_endpoint_type,is_svm_root' - logger.debug(f'Trying {endpoint}.') - response = http.request('GET', endpoint, headers=headers, timeout=5.0) - if response.status == 200: - data = json.loads(response.data) - volumes = data['records'] - else: - logger.error(f'API call to {endpoint} failed. HTTP status code: {response.status}.') - return(volumes) - - while data.get('next'): - endpoint = f'https://{fsxnIp}{data["next"]}' + url = '/api/storage/volumes?fields=name,svm,type,clone,flexcache_endpoint_type,is_svm_root' + while url is not None: + endpoint = f'https://{fsxnIp}{url}' logger.debug(f'Trying {endpoint}.') - response = http.request('GET', endpoint, headers=headers, timeout=5.0) - if response.status == 200: - data = json.loads(response.data) - volumes += data['records'] - else: + response = http.request('GET', endpoint, headers=headers, timeout=16) + if response.status < 200 or response.status > 299: logger.error(f'API call to {endpoint} failed. HTTP status code: {response.status}.') - break + return(volumes) + data = json.loads(response.data) + volumes.extend(data['records']) + if data.get("_links") is not None and data["_links"].get("next") is not None and data["_links"]["next"].get("href") is not None: + url = data["_links"]["next"]["href"] + else: + url = None except Exception as err: logger.critical(f'Failed to issue API against {fsxnIp}. Cluster could be down. The error messages received: "{err}".') @@ -887,6 +880,10 @@ Resources: else: config['protectAll'] = True # + # Convert string to an int. + if config.get('maxSnapMirrorRelationships') is not None: + config['maxSnapMirrorRelationships'] = int(config['maxSnapMirrorRelationships']) + # # If the s3BucketName is set, then open a client to the s3 service and read # in the partners and secrets config files. if config['s3BucketName'] is not None: @@ -943,36 +940,46 @@ Resources: # If the dynamodbRegion is set, then open a client to the dynamodb service and read # in the partners and secrets config tables. if config['dynamodbRegion'] is not None: - dynamodbClient = boto3.resource("dynamodb", region_name=config['dynamodbRegion']) - - if config['dynamodbSecretsTableName'] is not None: - table = dynamodbClient.Table(config['dynamodbSecretsTableName']) - response = table.scan() - secretsTable = response["Items"] - if config['dynamodbPartnersTableName'] is not None: - table = dynamodbClient.Table(config['dynamodbPartnersTableName']) - response = table.scan() - items = response["Items"] - partnersTable = [] - for item in items: - partnersTable.append({ - 'fsxId': item['sourceId'].split(":")[0], - 'svmName': item['sourceId'].split(":")[1], - 'partnerFsxId': item['partnerFsxId'], - 'partnerFsxnIp': item['partnerFsxnIp'], - 'partnerSvmName': item['partnerSvmName'], - 'partnerSvmSourceName': item['partnerSvmSourceName'] - }) + try: + dynamodbClient = boto3.resource("dynamodb", region_name=config['dynamodbRegion']) + + if config['dynamodbSecretsTableName'] is not None: + table = dynamodbClient.Table(config['dynamodbSecretsTableName']) + response = table.scan() + secretsTable = response["Items"] + while 'LastEvaluatedKey' in response: + response = table.scan(ExclusiveStartKey=response['LastEvaluatedKey']) + secretsTable.extend(response['Items']) + if config['dynamodbPartnersTableName'] is not None: + table = dynamodbClient.Table(config['dynamodbPartnersTableName']) + response = table.scan() + items = response["Items"] + while 'LastEvaluatedKey' in response: + response = table.scan(ExclusiveStartKey=response['LastEvaluatedKey']) + items.extend(response['Items']) + partnersTable = [] + for item in items: + partnersTable.append({ + 'fsxId': item['sourceId'].split(":")[0], + 'svmName': item['sourceId'].split(":")[1], + 'partnerFsxId': item['partnerFsxId'], + 'partnerFsxnIp': item['partnerFsxnIp'], + 'partnerSvmName': item['partnerSvmName'], + 'partnerSvmSourceName': item['partnerSvmSourceName'] + }) + except Exception as err: + message = f'Error, unable to read the secrets and/or partners config tables "{config["dynamodbSecretsTableName"]}", "{config["dynamodbPartnersTableName"]}" from Dynamodb region {config["dynamodbRegion"]}. Error message: "{err}".' + logger.critical(message) if 'partnersTable' not in globals() or len(partnersTable) == 0: message = "Error, No partners were have been configured." logger.critical(message) - raise Exception(message) + raise Exception(message) # Send up a flare. Hopefully they enabled the watchdog alarm and will get a notification. if 'secretsTable' not in globals() or len(secretsTable) == 0: message = "Error, No secrets were have been configured." logger.critical(message) - raise Exception(message) + raise Exception(message) # Send up a flare. Hopefully they enabled the watchdog alarm and will get a notification. ################################################################################ # This is the main logic of the program. It loops on all the regions, then all From 5ea1475f23562f2b6cc9ebeba8efdc0d9c2ee1ff Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Wed, 7 Oct 2026 10:09:12 -0500 Subject: [PATCH 20/23] Updated it to check the destinatio file system to see if the relationship exist. Added support for an S3 bucket to contain the partners and secrets information. Added a CloudFormation template. --- .../auto_create_sm_relationships.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py b/Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py index 69c1fe16..6febc80c 100755 --- a/Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py +++ b/Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py @@ -335,6 +335,7 @@ def getSnapMirrorRelationships(fsxnId, fsxnIp): global logger, http + relationships = [] (username, password) = getCredentials(fsxnId) if username == "" or password == "": logger.error(f'No credentials for FSxN ID: {fsxnId}.') @@ -343,7 +344,6 @@ def getSnapMirrorRelationships(fsxnId, fsxnIp): auth = urllib3.make_headers(basic_auth=f'{username}:{password}') headers = { **auth } - relationships = [] try: url = '/api/snapmirror/relationships?fields=source,destination' while url is not None: From 97bbb7a97a3179bee894246763335b38d11ace44 Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Wed, 7 Oct 2026 10:14:12 -0500 Subject: [PATCH 21/23] Updated it to check the destinatio file system to see if the relationship exist. Added support for an S3 bucket to contain the partners and secrets information. Added a CloudFormation template. --- ...date-CloudformationTemplate-auto_create_sm_relationships.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/update-CloudformationTemplate-auto_create_sm_relationships.yml b/.github/workflows/update-CloudformationTemplate-auto_create_sm_relationships.yml index 3943873a..90d2b916 100644 --- a/.github/workflows/update-CloudformationTemplate-auto_create_sm_relationships.yml +++ b/.github/workflows/update-CloudformationTemplate-auto_create_sm_relationships.yml @@ -36,7 +36,7 @@ jobs: - name: Update the Cloudformation Template shell: bash working-directory: Management-Utilities/auto_create_sm_relationships - run: ./update-CFTemplate + run: ./updateCFTemplate - name: Commit the changes uses: step-security/git-auto-commit-action@905c3cd6e9ed2b67b4d46ff401fdb6d745d0ff9d # v7.1.0 From 09f6e6c47255a0e7f76ed323c228a0a1ed630ff6 Mon Sep 17 00:00:00 2001 From: kcantrel <22017779+kcantrel@users.noreply.github.com> Date: Wed, 7 Oct 2026 15:21:22 +0000 Subject: [PATCH 22/23] Apply automatic changes --- .../auto_create_sm_relationships/cloudformation.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml b/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml index 0d4492cb..84befc78 100644 --- a/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml +++ b/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml @@ -368,8 +368,8 @@ Resources: # creates a SnapMirror relationship using the partner information # provided below. # - # Version: v1.9 - # Date: 2026-10-07-08:58:52 + # Version: v1.1 + # Date: 2026-10-07-15:21:19 # ################################################################################ # @@ -688,6 +688,7 @@ Resources: global logger, http + relationships = [] (username, password) = getCredentials(fsxnId) if username == "" or password == "": logger.error(f'No credentials for FSxN ID: {fsxnId}.') @@ -696,7 +697,6 @@ Resources: auth = urllib3.make_headers(basic_auth=f'{username}:{password}') headers = { **auth } - relationships = [] try: url = '/api/snapmirror/relationships?fields=source,destination' while url is not None: From 9ba1175ec40350b4a23d646c852f77d68af4e873 Mon Sep 17 00:00:00 2001 From: Keith Cantrell Date: Wed, 7 Oct 2026 10:30:07 -0500 Subject: [PATCH 23/23] Updated it to check the destinatio file system to see if the relationship exist. Added support for an S3 bucket to contain the partners and secrets information. Added a CloudFormation template. --- .../auto_create_sm_relationships.py | 1 - .../auto_create_sm_relationships/cloudformation.yaml | 5 ++--- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py b/Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py index 6febc80c..b2b1a162 100755 --- a/Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py +++ b/Management-Utilities/auto_create_sm_relationships/auto_create_sm_relationships.py @@ -334,7 +334,6 @@ def getSnapMirrorRelationships(fsxnId, fsxnIp): global logger, http - relationships = [] (username, password) = getCredentials(fsxnId) if username == "" or password == "": diff --git a/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml b/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml index 84befc78..b8a52a38 100644 --- a/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml +++ b/Management-Utilities/auto_create_sm_relationships/cloudformation.yaml @@ -368,8 +368,8 @@ Resources: # creates a SnapMirror relationship using the partner information # provided below. # - # Version: v1.1 - # Date: 2026-10-07-15:21:19 + # Version: v1.11 + # Date: 2026-10-07-10:29:35 # ################################################################################ # @@ -687,7 +687,6 @@ Resources: global logger, http - relationships = [] (username, password) = getCredentials(fsxnId) if username == "" or password == "":