diff --git a/.gitignore b/.gitignore index 3bdf40f..1066029 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,4 @@ __pycache__ .coverage .vscode +.superpowers/ diff --git a/README.md b/README.md index 742922f..ec69a23 100644 --- a/README.md +++ b/README.md @@ -12,9 +12,32 @@ Notes: - The `relink.py` script was previously used for step 3 above, but that functionality is now built into `rimport`. It's still there if you want to use it by itself. - A relative filename passed to `rimport` directly (via `--file` or as a positional argument) is always resolved against your current directory — never against the inputdata root, and it doesn't matter whether you're running from inside or outside the inputdata tree. Pass an absolute path if you want to name a file without regard to your current directory. - A relative entry in a `--list` file is always resolved against that list file's own directory — again never against the inputdata root, wherever the list file itself lives. Pass absolute entries in the list if you want them independent of the list file's location. -- Before staging anything, `rimport` validates every file it's about to process (all `--file`/`--list`/positional entries together). If any of them fail — missing, a directory, a broken symlink, outside the inputdata root, etc. — none of them are touched, and every failing path is reported at once so you can fix them all in one pass. This is a promise about rejected input, not about success: pre-flight passing doesn't guarantee the whole batch will finish, since a file can still fail later for a reason pre-flight can't see (e.g. a runtime/relink failure partway through). -- `--check` is gated by the same pre-flight validation as a real run: if any file in the batch fails validation, `rimport` reports the failures and exits without checking (or reporting on) any of the other files. This is deliberate, not a bug — fix the bad entries and re-run to see the rest. -- Exit codes: `0` means everything succeeded (or, under `--check`, everything checked cleanly); `2` means the run was rejected before touching anything (bad arguments, a missing/empty list file, or a pre-flight validation failure); `1` means pre-flight passed but something failed for real while actually being staged or relinked. +- Before staging anything, `rimport` validates every path you named (all `--file`/`--list`/positional entries together). If any of them fail — missing, a broken symlink, outside the inputdata root, etc. — none of them are touched, and every failing path is reported at once so you can fix them all in one pass. Anything found by expanding a directory you named — an invalid file, or a subdirectory that cannot be read — is not covered by this promise; it is skipped on its own, and the rest of the run continues (see "Directory arguments" below). This is a promise about rejected input, not about success: pre-flight passing doesn't guarantee the whole batch will finish, since a file can still fail later for a reason pre-flight can't see (e.g. a runtime/relink failure partway through). +- `--check` is gated by the same pre-flight validation as a real run: if any path you named fails validation, `rimport` reports the failures and exits without checking (or reporting on) any of the other files. This is deliberate, not a bug — fix the bad entries and re-run to see the rest. + +### Directory arguments + +Any name you give `rimport` — positional, `--file`, or a `--list` entry — may be a directory inside the inputdata tree. Every file beneath it is enumerated recursively and acted on. A directory outside the tree is rejected without being enumerated, exactly as a file outside it is. The directory itself is never copied to staging or replaced with a symlink. A symlink to a directory is the one carve-out: it is not expanded, but treated as a single entry — reported as already published if it points into the staging directory, and an error otherwise. + +Why symlinks to directories are left alone, rather than enumerated: + +- A symlink into staging is what `rimport` itself creates, so a symlink here is usually a published file rather than a detour to follow. Treating it as a single entry is what lets you re-run `rimport` over a tree it has already published. +- Enumeration never follows a directory symlink either, so it cannot loop on a link that points at its own ancestor, and cannot wander outside the directory you named and publish files you did not ask for. The rule for a name you give matches the rule used while walking, so a path behaves the same whichever way `rimport` reaches it. + +Anything found by enumeration that cannot be staged does not abort the run — an unstageable file, or a subdirectory that cannot be read. It is reported, skipped, and repeated in a summary at the end so it does not scroll away. A bad name you gave directly is still fatal, and nothing is published. + +## Exit codes + +| Code | Meaning | +|------|---------| +| 0 | Everything staged or checked, nothing skipped | +| 1 | A file could not be staged | +| 2 | A name you gave failed validation; nothing was published | +| 3 | Finished, but one or more items were skipped (listed at the end) | + +When more than one code applies, the precedence is 2 > 1 > 3 > 0. + +A name you gave failing is fatal (2). Anything found by expanding a directory you named is skipped instead, and the run continues (3). ## Filenames and metadata: diff --git a/rimport b/rimport index 408287e..29fb9e4 100755 --- a/rimport +++ b/rimport @@ -15,7 +15,7 @@ import pwd import shutil import sys from pathlib import Path -from typing import Iterable, List +from typing import Iterable, List, NamedTuple from urllib.request import Request, urlopen from urllib.error import HTTPError @@ -33,6 +33,20 @@ INPUTDATA_URL = "https://osdf-data.gdex.ucar.edu/ncar/gdex/d651077/cesmdata/inpu logger = shared.logger +class _EpilogRawFormatter(argparse.HelpFormatter): + """Wrap the description as argparse normally would, but leave the epilog verbatim. + + RawDescriptionHelpFormatter would preserve the epilog's exit-code list, but it also + stops re-wrapping the description, which then renders as one long line on a narrow + terminal. Only the epilog needs its line breaks kept. + """ + + def _fill_text(self, text, width, indent): + if text.lstrip().startswith("exit codes:"): + return "".join(indent + line for line in text.splitlines(keepends=True)) + return super()._fill_text(text, width, indent) + + def build_parser() -> argparse.ArgumentParser: """Build and configure the argument parser for rimport. @@ -44,6 +58,20 @@ def build_parser() -> argparse.ArgumentParser: f"Copy files from CESM inputdata directory ({DEFAULT_INPUTDATA_ROOT}) to a publishing" " directory, then replace the original with a symlink to the copy." ), + epilog=( + "exit codes:\n" + " 0: everything staged or checked, nothing skipped\n" + " 1: a file could not be staged\n" + " 2: nothing was published -- a bad command line, a missing or empty\n" + " --list file, or a name you gave failing validation\n" + " 3: finished, but one or more items were skipped (listed at the end)\n" + "\n" + "a name you gave failing is fatal (2). anything found by expanding a\n" + "directory you named -- a bad file, or a subdirectory that cannot be\n" + "read -- is skipped instead, and the run continues (3).\n" + "when several apply, the precedence is 2 > 1 > 3 > 0.\n" + ), + formatter_class=_EpilogRawFormatter, add_help=False, # Disable automatic help to add custom -help flag ) @@ -55,7 +83,8 @@ def build_parser() -> argparse.ArgumentParser: help=( "Provide a file to import. Must be in the CESM inputdata directory. A relative name" " is resolved against the current directory; there is no fallback to the" - " inputdata root." + " inputdata root. If the name is a directory, every file beneath it is" + " enumerated recursively and acted on; a symlink to a directory is not expanded." ), ) @@ -67,7 +96,9 @@ def build_parser() -> argparse.ArgumentParser: help=( "Provide a file that contains a list of filenames to import. All filenames in the list" " must be in the CESM inputdata directory. A relative entry is resolved against the" - " list file's own directory, wherever that directory is." + " list file's own directory, wherever that directory is. A list entry naming a" + " directory is enumerated recursively, the same as a name given on the command" + " line." ), ) @@ -76,8 +107,10 @@ def build_parser() -> argparse.ArgumentParser: nargs="*", help=( "One or more files to process. (Optional; can use --file instead to process just one.)" - " A relative name is resolved against the current directory; there is no fallback to" - " the inputdata root." + " Must be in the CESM inputdata directory. A relative name is resolved against the" + " current directory; there is no fallback to the inputdata root. If the name is a" + " directory, every file beneath it is enumerated recursively and acted on; a symlink" + " to a directory is not expanded." ), ) @@ -90,9 +123,10 @@ def build_parser() -> argparse.ArgumentParser: "-c", action="store_true", help=( - "Check whether file(s) is/are already published, without staging anything. A bad" - " path anywhere in the batch aborts before any file is checked, reporting all bad" - " paths at once." + "Check whether item(s) is/are already published, without staging anything. A bad" + " name that you gave aborts before any file is checked, reporting all bad names" + " at once; anything found by enumerating a directory is instead reported and" + " skipped individually." ), ) @@ -163,6 +197,216 @@ def normalize_paths(root: Path, relnames: Iterable[str]) -> List[Path]: return paths +class Entry(NamedTuple): + """One path to consider staging, and how it got into the batch. + + `named` is True when the user typed this path (positional, `--file`, or a `--list` + entry) and False when a directory walk discovered it. The distinction is what lets a + bad file inside a large tree warn-and-skip while a bad path the user asked for by name + still aborts the whole batch. + """ + + path: Path + named: bool + + +class Skip(NamedTuple): + """One path that will not be staged, the reason why, and whether the user named it. + + Both validation failures on discovered files and directories that could not be read + become Skips, so `report_skips` can present them in one block. + + `named` defaults to False so a walker that has no idea what the user typed can build a + Skip with two arguments. `expand_directories` is the one place that knows the whole + batch, so it is the one place that fills this in; `main` then trusts it rather than + re-deriving it, which previously meant two copies of the same set having to agree. + """ + + path: Path + reason: Exception + named: bool = False + + +def walk_files(root: Path) -> tuple[List[Path], List[Skip]]: + """Recursively list the non-directory entries under `root`. + + Descends only into real directories (`follow_symlinks=False`), so the walk cannot + leave the tree the user named and cannot loop on a cyclic link. A symlink is therefore + always a leaf: it is returned as an entry to act on, including when its target happens + to be a directory, and `validate_source_path` decides what that means. + + Every non-directory entry is returned -- regular files, symlinks, dotfiles alike. There + is no owner filter (unlike relink's walker): rimport runs as the staging owner and the + point is to publish what is there. + + Entries are sorted at each level so output and tests are deterministic; `os.scandir` + order is otherwise arbitrary. + + Args: + root: Directory to walk. + + Returns: + (files, skips). `files` are absolute paths in depth-first, per-level sorted order. + `skips` are directories that could not be read; an unreadable directory does not + raise and does not abort the rest of the walk. + """ + files: List[Path] = [] + skips: List[Skip] = [] + + try: + with os.scandir(root) as scan: + children = sorted(scan, key=lambda entry: entry.name) + except OSError as exc: + return [], [Skip(Path(root), exc)] + + for child in children: + child_path = Path(child.path) + if child.is_dir(follow_symlinks=False): + sub_files, sub_skips = walk_files(child_path) + files.extend(sub_files) + skips.extend(sub_skips) + else: + files.append(child_path) + + return files, skips + + +def expand_directories( + paths: Iterable[Path], inputdata_root: Path +) -> tuple[List[Entry], List[Skip]]: + """Replace each directory in `paths` with the files beneath it, tagging provenance. + + A path is expanded only if it is a real directory INSIDE `inputdata_root`. A symlink to + a directory is left alone as a single named entry, matching `walk_files`' refusal to + descend through one. Nothing here validates: a nonexistent path, or a directory outside + the tree, passes straight through as named, so `main`'s pre-flight gate can reject it + with its usual message. Declining to expand is not a verdict -- it just leaves the path + for the gate that already knows how to judge it. + + Scoping expansion to the tree matters for more than tidiness. Walking recurses, so + expanding a directory this tool has no business in -- a mistyped `rimport ~` -- would + stat an arbitrarily large tree before rejecting every file in it, and would downgrade + the user's own bad argument from a fatal named failure to a heap of discovered skips. + + Duplicates collapse in first-seen order. If the same path is both named and discovered + -- the user typed a file that also lives inside a directory they named -- `named` wins, + so the path the user asked for by name keeps its fatal-on-failure treatment. + + Args: + paths: Absolute paths from `normalize_paths`. + inputdata_root: Root of the inputdata tree; only directories under it are expanded. + + Returns: + (entries, skips), each skip carrying its own provenance. `skips` holds two things: + directories that could not be read during a walk, and named paths whose is_dir() + probe itself raised (a file under an unreadable parent, say). A walk skip is usually + discovered, and a discovered skip warns and continues with exit 3 rather than + aborting; only a skip the user named is fatal. Skips are collapsed by path, so a + directory named twice yields one. + """ + # Consumed twice -- once to walk, once to test provenance -- so a generator will not do. + # Collapsing here means a directory named twice is walked once, so it counts once toward + # the expansion total and reports anything unreadable beneath it once. + paths = list(dict.fromkeys(paths)) + named_paths = set(paths) + named_by_path: dict[Path, bool] = {} + skips: List[Skip] = [] + n_dirs = 0 + expanded_files: set[Path] = set() + + for path in paths: + try: + is_expandable_dir = path.is_dir() and not path.is_symlink() + except OSError as exc: + # Path.is_dir() ignores only ENOENT, ENOTDIR, EBADF and ELOOP. It propagates + # everything else -- EACCES included, on every supported version -- so a path + # under an unreadable parent would abort the run with a traceback. Record it and + # let main decide: every path here was NAMED by the user, so main routes it to + # the fatal pre-flight block and the run exits 2 having published nothing. + skips.append(Skip(path, exc)) + continue + if is_expandable_dir and not path.resolve().is_relative_to(inputdata_root.resolve()): + # Outside the tree: hand it to the pre-flight gate unexpanded. Resolve both + # sides, as validate_source_path does, so the two agree about what "outside" + # means for a path reached through a symlinked parent. + is_expandable_dir = False + if is_expandable_dir: + n_dirs += 1 + found, walk_skips = walk_files(path) + skips.extend(walk_skips) + if not found and not walk_skips: + logger.warning("rimport: no files found under %s", path) + for found_path in found: + expanded_files.add(found_path) + named_by_path.setdefault(found_path, False) + else: + # A named path always wins over the same path discovered by a walk. + named_by_path[path] = True + + # Collapse by path and stamp provenance before anything counts or prints these. Two + # named directories can overlap -- `` and `/sub` -- and then the same + # unreadable grandchild is found by both walks. + deduped: dict[Path, Skip] = {} + for skip in skips: + deduped.setdefault(skip.path, Skip(skip.path, skip.reason, skip.path in named_paths)) + skips = list(deduped.values()) + + if n_dirs: + logger.info( + "rimport: expanded %d director(ies) to %d file(s)", n_dirs, len(expanded_files) + ) + + for skip in skips: + # Report a DISCOVERED skip here, during expansion, where the run reached it; the + # end-of-run summary repeats it. A skip the user NAMED is not reported here at all: + # main treats it as a fatal failure, and calling it "skipping" would contradict the + # "nothing was published" that follows. Note this asks about the whole batch, not + # about the directory being walked: a directory can be named AND discovered beneath + # another named one, and then both are true of it. + if not skip.named: + logger.warning( + "%srimport: skipping '%s': %s", INDENT, skip.path, reason_text(skip.reason) + ) + + entries = [Entry(path, named) for path, named in named_by_path.items()] + return entries, skips + + +def report_skips(skips: List[Skip]) -> None: + """Re-list every skipped path at the very end of the run. + + Each skip was already reported inline, at WARNING level, where it happened. This block + repeats them at ERROR level -- so stderr, and so surviving `-q` -- because in a run over + a large tree the inline warnings scroll away and the whole point is that a file which + went unpublished cannot be missed. + + Args: + skips: Every skipped path with its reason. An empty list prints nothing. + """ + if not skips: + return + + logger.error("rimport: %d item(s) skipped (not stageable):", len(skips)) + for skip in skips: + logger.error("%s%s: %s", INDENT, skip.path, reason_text(skip.reason)) + + +def reason_text(reason: Exception) -> str: + """Render a skip or failure reason for a message that already names the path. + + `OSError`'s str() appends the filename, so a line built as "'': " prints + the path twice. Its `strerror` is the same message without that, and errno is dropped + deliberately: "Permission denied" is what the user can act on. This is safe only because + every OSError reaching here was raised on the same path the message names. + + It changes nothing for the RuntimeErrors `validate_source_path` builds: those carry no + `strerror`, and several write the path into their own text, which this cannot undo. + """ + if isinstance(reason, OSError) and reason.strerror: + return reason.strerror + return str(reason) + + def check_relink_worked(src: Path, dst: Path) -> None: """Check whether relink worked @@ -182,11 +426,11 @@ def validate_source_path( ) -> Exception | None: """Run stage_data's read-only guardrails against `src` without raising or logging. - This is the pre-flight half of `stage_data`'s checks: broken symlink, live symlink whose - target is outside staging, missing file, directory, outside the inputdata root, and already - under the staging directory. It performs no I/O beyond stat-ing `src` and its resolved - target — in particular it never makes a network call — so it is cheap to run over an entire - batch before anything is staged. + This is the pre-flight half of `stage_data`'s checks, in the order they run: broken + symlink, live symlink whose target is outside staging, missing file, outside the inputdata + root, already under the staging directory, and directory. It performs no I/O beyond + stat-ing `src` and its resolved target — in particular it never makes a network call — so + it is cheap to run over an entire batch before anything is staged. Critically, a *live* symlink whose target resolves under `staging_root` is NOT a failure: that is the normal state of a file that has already been published and linked by a previous @@ -221,9 +465,9 @@ def validate_source_path( if not src.exists(): return FileNotFoundError(f"source not found: {src}") - if src.is_dir(): - return RuntimeError(f"source is a directory, not a file: {src}") - + # Containment is checked before the is-a-directory backstop so that a DIRECTORY outside + # the tree names the reason the user can act on. Told "source is a directory, not a + # file", they would reasonably reply that directories are supported now. try: src.resolve().relative_to(inputdata_root.resolve()) except ValueError: @@ -235,6 +479,9 @@ def validate_source_path( f"source not under inputdata root: {src} not in {inputdata_root}" ) + if src.is_dir(): + return RuntimeError(f"source is a directory, not a file: {src}") + return None @@ -435,6 +682,10 @@ def get_files_to_process(file: str, filelist: str, items_to_process: list): entries, and all `--list` entries (which never anchor to cwd), are unaffected by an undeterminable cwd. + An empty or whitespace-only `file`/`items_to_process` entry is rejected with `(None, 2)` + before any anchoring happens, because an empty name would otherwise resolve to the cwd + and be expanded. + Args: file (str): Single file to process. filelist (str): File containing list of files to process. @@ -445,6 +696,17 @@ def get_files_to_process(file: str, filelist: str, items_to_process: list): int: Result code """ cli_args = ([file] if file is not None else []) + list(items_to_process or []) + # An empty name anchors to cwd and resolves to the cwd itself, which then expands: an + # unset shell variable (`rimport "$maybe_unset"`) would recursively publish the whole + # subtree the user is standing in. Refuse it. + empty_cli_args = [name for name in cli_args if not str(name).strip()] + if empty_cli_args: + logger.error( + "rimport: %d empty filename argument(s) given; refusing to resolve an empty " + "name to the current directory. Did a shell variable expand to nothing?", + len(empty_cli_args), + ) + return None, 2 relative_cli_args = [name for name in cli_args if not Path(name).is_absolute()] cwd = None @@ -507,14 +769,17 @@ def main(argv: List[str] | None = None) -> int: argv: Command-line arguments to parse. If None, uses sys.argv. Returns: - int: Exit code (0 for success, 1 if any files had errors, 2 for fatal errors). + int: Exit code (0 for success with nothing skipped, 1 if any files had errors, + 2 for fatal errors, or 3 if the run completed but one or more paths were + skipped). Environment Variables: RIMPORT_SKIP_USER_CHECK: Set to "1" to skip automatic user switching. RIMPORT_STAGING: Override the default staging root directory. Exit Codes: - 0: All files staged successfully (or, under --check, checked without error). + 0: All files staged successfully (or, under --check, checked without error), and + nothing was skipped. 1: Pre-flight validation passed, but one or more files failed while actually being staged or relinked -- a genuine runtime failure, not a rejected input (errors printed to stderr for each). @@ -534,6 +799,14 @@ def main(argv: List[str] | None = None) -> int: inputdata root, etc.). Pre-flight checks every resolved path before staging anything and reports every failure at once, so a bad path never leaves the batch half-published. This gate applies to --check runs too. + 3: The run finished -- nothing was rejected outright and no file failed while being + staged or relinked -- but at least one discovered path was skipped rather than + staged. Two things cause that, and either alone is enough: a file found by + expanding a directory argument failed pre-flight validation, or a directory + beneath a named one could not be read. Each skip is warned about inline during + expansion or pre-flight and the full list is repeated on stderr at the very end, + so a skipped path cannot be missed. Note that only DISCOVERED paths are skipped; + the same failure on a path the user named is fatal, and exits 2. """ parser = build_parser() args = parser.parse_args(argv) @@ -564,27 +837,52 @@ def main(argv: List[str] | None = None) -> int: paths = normalize_paths(root, files_to_process) staging_root = get_staging_root() - # Pre-flight: validate every path before staging anything, so a batch containing a bad - # path is reported (all bad paths at once) instead of half-completing. Runs for --check - # too: a bad entry aborts the whole batch rather than being reported per-file. - failures = [] - for p in paths: - error = validate_source_path(p, root, staging_root) - if error is not None: - failures.append((p, error)) - if failures: + # Expand any directory argument into the files beneath it, tagging each path as named + # (the user asked for it) or discovered (a walk found it). + entries, skips = expand_directories(paths, root) + + # Pre-flight: validate everything before staging anything. A NAMED bad path is fatal and + # aborts the whole batch, so a batch containing one is never half-completed. A DISCOVERED + # bad path is only a skip: one unstageable file inside a large tree must not block the + # thousands of good ones around it. + # A directory the user NAMED that could not be read is a named failure, not a skip. + # `expand_directories` stamped that on each Skip; deriving it a second time here would + # mean two sets that have to agree, and nothing enforcing it. An unreadable directory + # found BENEATH a named one stays a skip: the user did not name it, so it is discovered, + # and discovered failures warn and skip. + named_failures: List[tuple[Path, Exception]] = [ + (skip.path, skip.reason) for skip in skips if skip.named + ] + skips = [skip for skip in skips if not skip.named] + # An unreadable named directory never became an Entry, so it has to be added to the + # count of paths considered; otherwise a lone one reports "1 of 0". + n_considered = len(entries) + len(named_failures) + to_stage: List[Path] = [] + for entry in entries: + error = validate_source_path(entry.path, root, staging_root) + if error is None: + to_stage.append(entry.path) + elif entry.named: + named_failures.append((entry.path, error)) + else: + logger.warning( + "%srimport: skipping '%s': %s", INDENT, entry.path, reason_text(error) + ) + skips.append(Skip(entry.path, error)) + + if named_failures: logger.error( - "rimport: %d of %d file(s) failed pre-flight validation; nothing was published:", - len(failures), - len(paths), + "rimport: %d of %d item(s) failed pre-flight validation; nothing was published:", + len(named_failures), + n_considered, ) - for p, error in failures: - logger.error("%srimport: '%s': %s", INDENT, p, error) + for path, error in named_failures: + logger.error("%srimport: '%s': %s", INDENT, path, reason_text(error)) return 2 # Execute the new action per file errors = 0 - for p in paths: + for p in to_stage: logger.info("'%s':", p) try: stage_data(p, root, staging_root, args.check) @@ -593,10 +891,18 @@ def main(argv: List[str] | None = None) -> int: errors += 1 logger.error("%srimport: error processing %s: %s", INDENT, p, e) + if not errors and not args.check: + logger.info("\nNo need to run relink.py") + + # Last, so it cannot be scrolled past. + report_skips(skips) + + # Precedence 2 > 1 > 3 > 0; 2 already returned above. A real staging failure must never + # be masked by "completed with skips". if errors: return 1 - if not args.check: - logger.info("\nNo need to run relink.py") + if skips: + return 3 return 0 diff --git a/tests/rimport/test_build_parser.py b/tests/rimport/test_build_parser.py index b54d710..1c93dd3 100644 --- a/tests/rimport/test_build_parser.py +++ b/tests/rimport/test_build_parser.py @@ -199,3 +199,29 @@ def test_quiet_and_verbose_mutually_exclusive(self, capsys): captured = capsys.readouterr() stderr_lines = captured.err.strip().split("\n") assert "not allowed with argument" in stderr_lines[-1] + + def test_help_documents_directory_expansion(self): + """A user reading --help must learn that a directory argument is enumerated.""" + help_text = rimport.build_parser().format_help() + # argparse WRAPS argument help across lines, so match on whitespace-normalized text + normalized = " ".join(help_text.split()).lower() + + assert "directory" in normalized + assert "enumerated recursively" in normalized + + def test_help_documents_that_directory_symlinks_are_not_expanded(self): + """The one surprising carve-out belongs in the help, not just the source.""" + help_text = rimport.build_parser().format_help() + + assert "symlink to a directory is not expanded" in help_text + + def test_help_documents_all_four_exit_codes(self): + """Exit 3 is new and scriptable; all four codes must be discoverable.""" + help_text = rimport.build_parser().format_help() + help_lower = help_text.lower() + + assert "exit codes:" in help_lower + exit_section = help_lower.split("exit codes:", 1)[1] + for code in ["0:", "1:", "2:", "3:"]: + assert code in exit_section + assert "skipped" in exit_section diff --git a/tests/rimport/test_cmdline.py b/tests/rimport/test_cmdline.py index d08d89e..8b3bac9 100644 --- a/tests/rimport/test_cmdline.py +++ b/tests/rimport/test_cmdline.py @@ -322,7 +322,7 @@ def test_error_for_nonexistent_file(self, rimport_script, test_env, rimport_env) ) # Verify error. Assert the actual reason, not the substring "error": pre-flight - # reports "N of M file(s) failed pre-flight validation", which contains no such + # reports "N of M item(s) failed pre-flight validation", which contains no such # word, so a bare "error" check is satisfied only by tmp_path echoing this test's # own name back in the offending path. assert result.returncode != 0 @@ -928,13 +928,13 @@ def test_check_doesnt_relink_published(self, rimport_script, test_env, rimport_e assert "Created symbolic link".lower() not in result.stdout.lower() assert "Error creating symlink".lower() not in result.stdout.lower() - def test_directory_argument_from_subdir_errors_and_leaves_tree_intact( + def test_directory_argument_from_subdir_stages_contents_and_leaves_tree_intact( self, rimport_script, test_env, rimport_env ): """Test that pointing rimport at a directory (e.g. via the cwd-anchored positional from - inside an inputdata subdir) errors cleanly instead of falling into the destructive - replace-with-symlink path, which would rename the directory to '.tmp', symlink it - away, and then fail to roll back.""" + inside an inputdata subdir) enumerates the files beneath it and stages them, instead of + falling into the destructive replace-with-symlink path, which would rename the directory + itself to '.tmp', symlink it away, and then fail to roll back.""" inputdata_root = test_env["inputdata_root"] staging_root = test_env["staging_root"] @@ -967,28 +967,28 @@ def test_directory_argument_from_subdir_errors_and_leaves_tree_intact( cwd=subdir.parent, ) - # Verify failure - assert result.returncode != 0, f"Command unexpectedly passed: {result.stdout}" - assert "directory" in result.stderr - assert "not a file" in result.stderr + assert result.returncode == 0, f"Command unexpectedly failed: {result.stderr}" - # Verify the tree is intact: clm2 is still a real directory, not a symlink; no - # '.tmp' path was left anywhere under inputdata_root; and its contents are - # untouched. + # The file inside was published and relinked. + assert (staging_mirror / "data.nc").read_text() == "clm2 data" + assert inner_file.is_symlink() + + # The directory itself must survive untouched: still a real directory, never + # symlinked away, and no '.tmp' left behind by a half-finished replacement. assert subdir.is_dir() and not subdir.is_symlink(), ( - f"clm2 should still be a plain, non-symlink directory after the error; " + f"clm2 should still be a plain, non-symlink directory; " f"is_dir={subdir.is_dir()} is_symlink={subdir.is_symlink()}" ) tmp_paths = list(inputdata_root.rglob("*.tmp")) assert not tmp_paths, f"Found unexpected '.tmp' path(s) left behind: {tmp_paths}" - assert inner_file.read_text() == "clm2 data" def test_empty_string_argument_errors_and_leaves_tree_intact( self, rimport_script, test_env, rimport_env ): """Test that an empty-string positional (as from an unset shell variable, e.g. `rimport "$maybe_unset"`) errors cleanly instead of anchoring to the inputdata root - itself and running that root through the destructive replace-with-symlink path.""" + itself and, now that directories are enumerated, recursively publishing the whole + subtree.""" inputdata_root = test_env["inputdata_root"] # staging_root itself need not be assigned here — the fixture already created it, and # its mere existence is what makes dst.exists() true for rel="." — the same thing that @@ -1018,26 +1018,19 @@ def test_empty_string_argument_errors_and_leaves_tree_intact( ) # Verify failure - assert result.returncode != 0, f"Command unexpectedly passed: {result.stdout}" - assert "directory" in result.stderr - assert "not a file" in result.stderr - - # Verify the inputdata root itself is untouched: still a real directory, not renamed, - # not replaced with a symlink, no '.tmp' sibling. - assert inputdata_root.is_dir() and not inputdata_root.is_symlink(), ( - f"inputdata root should still be a plain, non-symlink directory after the error; " - f"is_dir={inputdata_root.is_dir()} is_symlink={inputdata_root.is_symlink()}" - ) - tmp_siblings = list(inputdata_root.parent.glob(f"{inputdata_root.name}.tmp")) - assert not tmp_siblings, f"Found unexpected '.tmp' sibling(s): {tmp_siblings}" - assert marker_file.read_text() == "root marker" + assert result.returncode == 2, f"Command unexpectedly passed: {result.stdout}" + assert "empty filename" in result.stderr - def test_check_directory_argument_reports_error_not_publishable( + # The inputdata root was NOT expanded and published wholesale. + assert not list(test_env["staging_root"].rglob("*")) + assert not marker_file.is_symlink() + + def test_check_directory_argument_reports_each_file_inside( self, rimport_script, test_env, rimport_env ): - """Test that --check on a directory argument reports it as an error, rather than - misreporting the directory as already published but not linked and available for - download.""" + """Test that --check on a directory argument enumerates the files beneath it and + reports on each one individually, rather than describing the directory itself as + already published or available for download.""" inputdata_root = test_env["inputdata_root"] staging_root = test_env["staging_root"] @@ -1068,24 +1061,29 @@ def test_check_directory_argument_reports_error_not_publishable( cwd=subdir.parent, ) - # Verify failure - assert result.returncode != 0, f"Command unexpectedly passed: {result.stdout}" - assert "directory" in result.stderr - assert "not a file" in result.stderr + assert result.returncode == 0, f"Command unexpectedly failed: {result.stderr}" - # Verify --check does NOT claim the directory is already published / downloadable - assert "already published" not in result.stdout.lower() - assert "available for download" not in result.stdout.lower() + # --check reports on the file inside, and stages nothing. + assert "data.nc" in result.stdout + assert not (staging_mirror / "data.nc").exists() + assert not inner_file.is_symlink() - # Verify the tree is intact + # The file is reported on its own terms, and the directory itself is never + # described as published or downloadable. rimport heads each reported item with + # "'':", so the directory's own header is what must be absent -- its name + # appears anyway inside the path of the file beneath it. + assert f"'{inner_file}':" in result.stdout + assert f"'{subdir}':" not in result.stdout + assert "not already published" in result.stdout + assert "available for download" not in result.stdout.lower() assert subdir.is_dir() and not subdir.is_symlink() assert not list(inputdata_root.rglob("*.tmp")) assert inner_file.read_text() == "clm2 data" def _run_mixed_validity_list(self, rimport_script, test_env, rimport_env, *, check): """Set up a --list with one valid entry (good.nc) and two entries that are invalid - in DIFFERENT ways (missing.nc, and a directory named adir), all as absolute paths in - a list file OUTSIDE the inputdata tree, then run rimport against it -- with + in DIFFERENT ways (missing.nc, and a broken symlink named broken.nc), all as absolute + paths in a list file OUTSIDE the inputdata tree, then run rimport against it -- with --check when `check` is True (which also requires deleting RIMPORT_SKIP_USER_CHECK, since --check needs ensure_running_as() to actually run), without it otherwise. @@ -1110,12 +1108,12 @@ def _run_mixed_validity_list(self, rimport_script, test_env, rimport_env, *, che missing_file = inputdata_root / "missing.nc" - bad_dir = inputdata_root / "adir" - bad_dir.mkdir() + broken_link = inputdata_root / "broken.nc" + broken_link.symlink_to(inputdata_root / "nonexistent_target.nc") # List file OUTSIDE the tree, with absolute entries. filelist = tmp_path / "filelist.txt" - filelist.write_text(f"{valid_file}\n{missing_file}\n{bad_dir}\n") + filelist.write_text(f"{valid_file}\n{missing_file}\n{broken_link}\n") command = [ sys.executable, @@ -1141,9 +1139,9 @@ def _run_mixed_validity_list(self, rimport_script, test_env, rimport_env, *, che # Verify failure: rc 2, all reasons present, correct "N of M" count. Identical for # both callers; not what either test discriminates on. assert result.returncode == 2, f"Command unexpectedly passed: {result.stdout}" - assert "2 of 3 file(s) failed pre-flight validation" in result.stderr + assert "2 of 3 item(s) failed pre-flight validation" in result.stderr assert f"source not found: {missing_file}" in result.stderr - assert f"source is a directory, not a file: {bad_dir}" in result.stderr + assert f"Source is a broken symlink: {broken_link}" in result.stderr return result, valid_file, staging_root @@ -1151,7 +1149,7 @@ def test_mixed_validity_list_aborts_and_stages_nothing( self, rimport_script, test_env, rimport_env ): """Test the pre-flight gate end to end: a --list with one valid entry and two entries - that are invalid in DIFFERENT ways (missing, and a directory) aborts the whole batch + that are invalid in DIFFERENT ways (missing, and a broken symlink) aborts the whole batch with rc 2, reports every failure reason, gets the "N of M" count right, and — the assertion that matters most — never stages or relinks the valid entry. @@ -1193,3 +1191,197 @@ def test_check_mode_is_gated_too_and_reports_nothing_for_valid_entry( # Verify nothing was staged assert not any(staging_root.rglob("*")) assert not valid_file.is_symlink() + + def test_directory_argument_recurses_into_subdirectories( + self, rimport_script, test_env, rimport_env + ): + """Enumeration is recursive, and the mirrored staging structure is preserved.""" + inputdata_root = test_env["inputdata_root"] + staging_root = test_env["staging_root"] + + deep = inputdata_root / "lnd" / "clm2" / "paramdata" + deep.mkdir(parents=True) + (inputdata_root / "lnd" / "top.nc").write_text("top") + (deep / "deep.nc").write_text("deep") + + result = subprocess.run( + [ + sys.executable, + rimport_script, + str(inputdata_root / "lnd"), + "-inputdata", + str(inputdata_root), + ], + capture_output=True, + text=True, + check=False, + env=rimport_env, + ) + + assert result.returncode == 0, f"Command unexpectedly failed: {result.stderr}" + assert (staging_root / "lnd" / "top.nc").read_text() == "top" + assert (staging_root / "lnd" / "clm2" / "paramdata" / "deep.nc").read_text() == "deep" + + def test_file_option_expands_a_directory(self, rimport_script, test_env, rimport_env): + """--help promises expansion on all three input channels. The positional channel is + covered above; this is --file, which nothing else pins.""" + inputdata_root = test_env["inputdata_root"] + staging_root = test_env["staging_root"] + + d = inputdata_root / "lnd" / "clm2" + d.mkdir(parents=True) + (d / "a.nc").write_text("a") + (d / "sub").mkdir() + (d / "sub" / "b.nc").write_text("b") + + result = subprocess.run( + [ + sys.executable, + rimport_script, + "--file", + str(d), + "-inputdata", + str(inputdata_root), + ], + capture_output=True, + text=True, + check=False, + env=rimport_env, + ) + + assert result.returncode == 0, f"Command unexpectedly failed: {result.stderr}" + assert (staging_root / "lnd" / "clm2" / "a.nc").read_text() == "a" + assert (staging_root / "lnd" / "clm2" / "sub" / "b.nc").read_text() == "b" + # The directory itself must survive untouched: still a real directory, never + # replaced by a symlink to a staged copy of itself. + assert d.is_dir() and not d.is_symlink() + + def test_list_entry_expands_a_directory(self, rimport_script, test_env, rimport_env): + """The third channel. A --list entry naming a directory is enumerated the same way, + including when the entry is relative and anchors to the list file's own directory.""" + inputdata_root = test_env["inputdata_root"] + staging_root = test_env["staging_root"] + + d = inputdata_root / "lnd" / "clm2" + d.mkdir(parents=True) + (d / "a.nc").write_text("a") + (d / "sub").mkdir() + (d / "sub" / "b.nc").write_text("b") + + # Relative, so this also pins that directory entries anchor like file entries do. + list_file = inputdata_root / "lnd" / "todo.txt" + list_file.write_text("clm2\n") + + result = subprocess.run( + [ + sys.executable, + rimport_script, + "--list", + str(list_file), + "-inputdata", + str(inputdata_root), + ], + capture_output=True, + text=True, + check=False, + env=rimport_env, + ) + + assert result.returncode == 0, f"Command unexpectedly failed: {result.stderr}" + assert (staging_root / "lnd" / "clm2" / "a.nc").read_text() == "a" + assert (staging_root / "lnd" / "clm2" / "sub" / "b.nc").read_text() == "b" + assert d.is_dir() and not d.is_symlink() + + def test_skip_summary_repeats_skipped_files_on_stderr_at_the_end( + self, rimport_script, test_env, rimport_env + ): + """A skipped file must survive a long scroll: reported inline on stdout, then + repeated in a summary block on stderr after everything else.""" + inputdata_root = test_env["inputdata_root"] + + subdir = inputdata_root / "lnd" + subdir.mkdir() + (subdir / "good.nc").write_text("good") + (subdir / "broken.nc").symlink_to(inputdata_root / "nonexistent.nc") + + result = subprocess.run( + [ + sys.executable, + rimport_script, + str(subdir), + "-inputdata", + str(inputdata_root), + ], + capture_output=True, + text=True, + check=False, + env=rimport_env, + ) + + assert result.returncode == 3 + assert "1 item(s) skipped (not stageable)" in result.stderr + assert "broken.nc" in result.stderr + # The summary is the LAST thing on stderr. The broken-symlink message names the + # link itself, not its target, so the final line ends with broken.nc. + assert result.stderr.rstrip().endswith("broken.nc") + + def test_skip_summary_survives_quiet_mode( + self, rimport_script, test_env, rimport_env + ): + """-q hides progress but must not hide what went unpublished.""" + inputdata_root = test_env["inputdata_root"] + + subdir = inputdata_root / "lnd" + subdir.mkdir() + (subdir / "good.nc").write_text("good") + (subdir / "broken.nc").symlink_to(inputdata_root / "nonexistent.nc") + + result = subprocess.run( + [ + sys.executable, + rimport_script, + str(subdir), + "-inputdata", + str(inputdata_root), + "-q", + ], + capture_output=True, + text=True, + check=False, + env=rimport_env, + ) + + assert result.returncode == 3 + assert "1 item(s) skipped (not stageable)" in result.stderr + + def test_expansion_count_is_logged_before_staging( + self, rimport_script, test_env, rimport_env + ): + """The blast radius reaches the user before the first file is written, so a run over + an unexpectedly large tree can still be interrupted.""" + inputdata_root = test_env["inputdata_root"] + + subdir = inputdata_root / "lnd" + subdir.mkdir() + (subdir / "a.nc").write_text("a") + (subdir / "b.nc").write_text("b") + + result = subprocess.run( + [ + sys.executable, + rimport_script, + str(subdir), + "-inputdata", + str(inputdata_root), + ], + capture_output=True, + text=True, + check=False, + env=rimport_env, + ) + + assert result.returncode == 0 + assert ( + result.stdout.index("expanded 1 director(ies) to 2 file(s)") + < result.stdout.index("[rimport] staged") + ) diff --git a/tests/rimport/test_expand_directories.py b/tests/rimport/test_expand_directories.py new file mode 100644 index 0000000..7bb0029 --- /dev/null +++ b/tests/rimport/test_expand_directories.py @@ -0,0 +1,388 @@ +""" +Tests for expand_directories() function in rimport script. +""" + +import os +import logging +import importlib.util +from importlib.machinery import SourceFileLoader + + +# Import rimport module from file without .py extension +rimport_path = os.path.join( + os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))), + "rimport", +) +loader = SourceFileLoader("rimport", rimport_path) +spec = importlib.util.spec_from_loader("rimport", loader) +if spec is None: + raise ImportError(f"Could not create spec for rimport from {rimport_path}") +rimport = importlib.util.module_from_spec(spec) +# Don't add to sys.modules to avoid conflict with other test files +loader.exec_module(rimport) + + +def test_directory_expands_to_discovered_files(tmp_path): + """Files found by walking a named directory are tagged discovered, not named.""" + d = tmp_path / "d" + d.mkdir() + (d / "a.nc").write_text("a") + (d / "b.nc").write_text("b") + + entries, skips = rimport.expand_directories([d], tmp_path) + + assert skips == [] + assert entries == [ + rimport.Entry(d / "a.nc", False), + rimport.Entry(d / "b.nc", False), + ] + + +def test_plain_file_passes_through_as_named(tmp_path): + """A file the user typed stays named, so its failures stay fatal.""" + f = tmp_path / "f.nc" + f.write_text("data") + + entries, _skips = rimport.expand_directories([f], tmp_path) + + assert entries == [rimport.Entry(f, True)] + + +def test_nonexistent_path_passes_through_as_named(tmp_path): + """Expansion does not validate. A missing path stays named so pre-flight can reject it.""" + missing = tmp_path / "missing.nc" + + entries, _skips = rimport.expand_directories([missing], tmp_path) + + assert entries == [rimport.Entry(missing, True)] + + +def test_symlink_to_directory_is_not_expanded(tmp_path): + """A named symlink-to-directory stays ONE named entry under the per-file rules. + + It is not walked, matching walk_files' refusal to descend through a directory symlink. + """ + real_dir = tmp_path / "real_dir" + real_dir.mkdir() + (real_dir / "inside.nc").write_text("data") + link = tmp_path / "dirlink" + link.symlink_to(real_dir) + + entries, _skips = rimport.expand_directories([link], tmp_path) + + assert entries == [rimport.Entry(link, True)] + + +def test_duplicates_collapse_and_named_wins(tmp_path): + """Naming a file that also lives inside a named directory keeps it named. + + Otherwise the user's own explicitly typed path would be demoted to a discovered + entry, and its validation failure would warn-and-skip instead of aborting. + """ + d = tmp_path / "d" + d.mkdir() + inner = d / "inner.nc" + inner.write_text("data") + + entries, _skips = rimport.expand_directories([d, inner], tmp_path) + + assert entries == [rimport.Entry(inner, True)] + + +def test_named_wins_when_the_file_is_named_before_its_directory(tmp_path): + """Naming the file before the directory that contains it reaches the same verdict as + naming it after (the test above). The two orders take different routes: this one sets + `named` first and the walk must not clear it.""" + d = tmp_path / "d" + d.mkdir() + inner = d / "inner.nc" + inner.write_text("data") + + entries, _skips = rimport.expand_directories([inner, d], tmp_path) + + assert entries == [rimport.Entry(inner, True)] + + +def test_duplicate_order_is_first_seen(tmp_path): + """De-duplication preserves the order a path was first encountered, across arguments.""" + d1 = tmp_path / "d1" + d1.mkdir() + (d1 / "b.nc").write_text("b") + d2 = tmp_path / "d2" + d2.mkdir() + (d2 / "a.nc").write_text("a") + + # d1 first, so its file leads, even though "a.nc" sorts before "b.nc" within a walk. + entries, _skips = rimport.expand_directories([d1, d2, d1], tmp_path) + + assert [e.path.name for e in entries] == ["b.nc", "a.nc"] + + +def test_walk_skips_are_passed_through(tmp_path): + """An unreadable directory found during expansion surfaces as a Skip.""" + d = tmp_path / "d" + d.mkdir() + locked = d / "locked" + locked.mkdir() + os.chmod(locked, 0o000) + + try: + _entries, skips = rimport.expand_directories([d], tmp_path) + finally: + os.chmod(locked, 0o700) + + assert len(skips) == 1 + assert skips[0].path == locked + + +def test_empty_directory_warns_but_is_not_a_skip(tmp_path, caplog): + """A named directory containing nothing is worth saying out loud, but is not an error + and must not affect the exit code.""" + d = tmp_path / "empty" + d.mkdir() + + with caplog.at_level(logging.WARNING, logger="rimport_relink"): + entries, skips = rimport.expand_directories([d], tmp_path) + + assert entries == [] + assert skips == [] + assert f"no files found under {d}" in caplog.text + + +def test_logs_expansion_counts(tmp_path, caplog): + """The blast radius is printed by `expand_directories()` -- i.e., before anything is staged.""" + d1 = tmp_path / "d1" + d1.mkdir() + (d1 / "a.nc").write_text("a") + d2 = tmp_path / "d2" + d2.mkdir() + (d2 / "b.nc").write_text("b") + (d2 / "c.nc").write_text("c") + + with caplog.at_level(logging.INFO, logger="rimport_relink"): + rimport.expand_directories([d1, d2], tmp_path) + + assert "expanded 2 director(ies) to 3 file(s)" in caplog.text + + +def test_no_expansion_logs_no_count_line(tmp_path, caplog): + """A batch of plain files should not emit a confusing 'expanded 0' line.""" + f = tmp_path / "f.nc" + f.write_text("data") + + with caplog.at_level(logging.INFO, logger="rimport_relink"): + rimport.expand_directories([f], tmp_path) + + assert "expanded" not in caplog.text + + +def test_unreadable_directory_does_not_warn_that_it_is_empty(tmp_path, caplog): + """A directory that could not be read is not an empty directory. Saying "no files found" + for it contradicts the Permission denied reported for the same path.""" + locked = tmp_path / "locked" + locked.mkdir() + (locked / "hidden.nc").write_text("data") + os.chmod(locked, 0o000) + + try: + with caplog.at_level(logging.WARNING, logger="rimport_relink"): + entries, skips = rimport.expand_directories([locked], tmp_path) + finally: + os.chmod(locked, 0o700) + + assert entries == [] + assert len(skips) == 1 + assert "no files found" not in caplog.text + + +def test_discovered_walk_skip_is_warned_where_it_happened(tmp_path, caplog): + """Every skip is reported twice: here, as the run reaches it, and again in the end-of-run + summary. Without this first report a skip during a fatal abort is reported zero times.""" + d = tmp_path / "d" + d.mkdir() + (d / "a.nc").write_text("a") + locked = d / "locked" + locked.mkdir() + os.chmod(locked, 0o000) + + try: + with caplog.at_level(logging.WARNING, logger="rimport_relink"): + rimport.expand_directories([d], tmp_path) + finally: + os.chmod(locked, 0o700) + + assert f"skipping '{locked}'" in caplog.text + # The line already names the path, so the reason must not repeat it. OSError's str() + # appends the filename and prefixes the errno; strerror is the part worth reading. + assert "Permission denied" in caplog.text + assert "[Errno" not in caplog.text + assert caplog.text.count(str(locked)) == 1 + + +def test_named_unreadable_directory_is_not_warned_as_skipped(tmp_path, caplog): + """A NAMED unreadable directory is fatal, and main reports it as such. Warning + it as a failure; warning "skipping" here too would contradict "nothing was published".""" + locked = tmp_path / "locked" + locked.mkdir() + os.chmod(locked, 0o000) + + try: + with caplog.at_level(logging.WARNING, logger="rimport_relink"): + rimport.expand_directories([locked], tmp_path) + finally: + os.chmod(locked, 0o700) + + assert "skipping" not in caplog.text + + +def test_named_unreadable_directory_is_not_warned_even_when_also_discovered(tmp_path, caplog): + """A path can be named AND discovered at once: name a tree and an unreadable directory + inside it, and the walk of the tree finds what the user also typed. It is still named, + so main reports it as a fatal failure and nothing here may call it "skipping" -- the two + messages contradict each other. Provenance is whole-batch membership, not "is this the + directory being walked".""" + d = tmp_path / "d" + d.mkdir() + (d / "a.nc").write_text("a") + locked = d / "locked" + locked.mkdir() + os.chmod(locked, 0o000) + + try: + with caplog.at_level(logging.WARNING, logger="rimport_relink"): + rimport.expand_directories([d, locked], tmp_path) + finally: + os.chmod(locked, 0o700) + + assert "skipping" not in caplog.text + + +def test_expansion_count_is_logged_before_any_skip_warning(tmp_path, caplog): + """The count line is the blast radius, and it belongs at the top where it cannot be + pushed down the screen by one warning per unreadable directory.""" + d = tmp_path / "d" + d.mkdir() + (d / "a.nc").write_text("a") + locked = d / "locked" + locked.mkdir() + os.chmod(locked, 0o000) + + try: + with caplog.at_level(logging.INFO, logger="rimport_relink"): + rimport.expand_directories([d], tmp_path) + finally: + os.chmod(locked, 0o700) + + assert caplog.text.index("expanded 1 director(ies)") < caplog.text.index("skipping") + + +def test_directory_named_twice_is_reported_once(tmp_path, caplog): + """Naming the same directory twice is one directory, not two, so the blast-radius line + counts it once.""" + d = tmp_path / "d" + d.mkdir() + (d / "a.nc").write_text("a") + + with caplog.at_level(logging.INFO, logger="rimport_relink"): + rimport.expand_directories([d, d], tmp_path) + + assert "expanded 1 director(ies) to 1 file(s)" in caplog.text + + +def test_duplicate_arguments_do_not_duplicate_a_skip(tmp_path, caplog): + """One unreadable directory is one skip, however many of the named arguments reach it. + Otherwise it is warned about twice, listed twice in the end-of-run summary, and counted + twice in the total.""" + d = tmp_path / "d" + d.mkdir() + locked = d / "locked" + locked.mkdir() + os.chmod(locked, 0o000) + + try: + with caplog.at_level(logging.WARNING, logger="rimport_relink"): + _entries, skips = rimport.expand_directories([d, d], tmp_path) + finally: + os.chmod(locked, 0o700) + + assert len(skips) == 1 + assert caplog.text.count("skipping") == 1 + + +def test_overlapping_named_directories_do_not_duplicate_a_skip(tmp_path): + """Same defect by another route: a subdirectory named alongside its parent is walked + twice, so anything unreadable beneath it is recorded twice.""" + d = tmp_path / "d" + d.mkdir() + sub = d / "sub" + sub.mkdir() + locked = sub / "locked" + locked.mkdir() + os.chmod(locked, 0o000) + + try: + _entries, skips = rimport.expand_directories([d, sub], tmp_path) + finally: + os.chmod(locked, 0o700) + + assert len(skips) == 1 + + +def test_directory_outside_the_root_is_not_expanded(tmp_path): + """Expansion is scoped to the inputdata tree. A directory outside it passes through as a + named entry for the pre-flight gate to reject, exactly as a nonexistent path does.""" + root = tmp_path / "inputdata" + root.mkdir() + outside = tmp_path / "elsewhere" + outside.mkdir() + (outside / "a.nc").write_text("a") + + entries, skips = rimport.expand_directories([outside], root) + + assert entries == [rimport.Entry(outside, True)] + assert skips == [] + + +def test_directory_outside_the_root_is_not_even_walked(tmp_path, monkeypatch): + """Declining to expand must happen BEFORE the walk, not by discarding its results: + walking recurses, and the whole point is not to stat a tree we have no business in.""" + root = tmp_path / "inputdata" + root.mkdir() + outside = tmp_path / "elsewhere" + outside.mkdir() + + def _fail(_path): + raise AssertionError("walk_files must not be called for a path outside the root") + + monkeypatch.setattr(rimport, "walk_files", _fail) + + rimport.expand_directories([outside], root) + + +def test_directory_at_the_root_itself_is_expanded(tmp_path): + """The boundary case: the root is not outside itself.""" + root = tmp_path / "inputdata" + root.mkdir() + (root / "a.nc").write_text("a") + + entries, _skips = rimport.expand_directories([root], root) + + assert entries == [rimport.Entry(root / "a.nc", False)] + + +def test_scope_is_decided_after_resolving_symlinks(tmp_path): + """The scope test resolves both sides, matching validate_source_path, so a directory + reached through a symlinked parent is judged by where it really is rather than by how it + was spelled. A lexical test would call this one outside the root and decline to expand + it; the two checks would then disagree about the same path.""" + root = tmp_path / "inputdata" + (root / "lnd").mkdir(parents=True) + (root / "lnd" / "a.nc").write_text("a") + # A door into the tree from outside it. Spelled through here, the path is lexically + # outside `root` but resolves inside. + door = tmp_path / "door" + door.symlink_to(root) + + entries, _skips = rimport.expand_directories([door / "lnd"], root) + + assert entries == [rimport.Entry(door / "lnd" / "a.nc", False)] diff --git a/tests/rimport/test_get_files_to_process.py b/tests/rimport/test_get_files_to_process.py index e709e5e..7d0c6ce 100644 --- a/tests/rimport/test_get_files_to_process.py +++ b/tests/rimport/test_get_files_to_process.py @@ -646,3 +646,41 @@ def test_deleted_cwd_with_multiple_relative_names_reports_all( assert filename in caplog.text for item_name in item_names: assert item_name in caplog.text + + def test_empty_positional_is_rejected(self, caplog): + """An empty positional must not anchor to cwd and become a whole-tree expansion.""" + with caplog.at_level(logging.ERROR, logger="rimport_relink"): + files, status = rimport.get_files_to_process(None, None, [""]) + + assert files is None + assert status == 2 + assert "empty filename" in caplog.text + + def test_empty_file_option_is_rejected(self, caplog): + """Same guard on --file.""" + with caplog.at_level(logging.ERROR, logger="rimport_relink"): + files, status = rimport.get_files_to_process("", None, []) + + assert files is None + assert status == 2 + assert "empty filename" in caplog.text + + def test_whitespace_only_argument_is_rejected(self, caplog): + """A name that is empty after stripping is just as dangerous as ''.""" + with caplog.at_level(logging.ERROR, logger="rimport_relink"): + files, status = rimport.get_files_to_process(None, None, [" "]) + + assert files is None + assert status == 2 + assert "empty filename" in caplog.text + + def test_empty_argument_rejected_even_alongside_valid_ones(self, tmp_path, caplog): + """One empty name poisons the batch; nothing is resolved.""" + good = tmp_path / "good.nc" + good.write_text("data") + + with caplog.at_level(logging.ERROR, logger="rimport_relink"): + files, status = rimport.get_files_to_process(None, None, [str(good), ""]) + + assert files is None + assert status == 2 diff --git a/tests/rimport/test_main.py b/tests/rimport/test_main.py index 04ba444..16178e6 100644 --- a/tests/rimport/test_main.py +++ b/tests/rimport/test_main.py @@ -534,8 +534,8 @@ def test_preflight_gate_rejects_whole_batch_and_never_calls_stage_data( valid = inputdata_root / "good.nc" valid.write_text("data") missing = inputdata_root / "missing.nc" - bad_dir = inputdata_root / "adir" - bad_dir.mkdir() + broken = inputdata_root / "broken.nc" + broken.symlink_to(inputdata_root / "nonexistent_target.nc") result = rimport.main( [ @@ -543,7 +543,7 @@ def test_preflight_gate_rejects_whole_batch_and_never_calls_stage_data( str(inputdata_root), str(valid), str(missing), - str(bad_dir), + str(broken), ] ) @@ -551,10 +551,419 @@ def test_preflight_gate_rejects_whole_batch_and_never_calls_stage_data( mock_stage_data.assert_not_called() captured = capsys.readouterr() - assert "2 of 3 file(s) failed pre-flight validation" in captured.err + assert "2 of 3 item(s) failed pre-flight validation" in captured.err assert f"source not found: {missing}" in captured.err - assert f"source is a directory, not a file: {bad_dir}" in captured.err + assert f"Source is a broken symlink: {broken}" in captured.err # The valid file was never staged or turned into a symlink. assert not (staging_root / "good.nc").exists() assert not valid.is_symlink() + + @patch.object(rimport, "get_staging_root") + @patch.object(rimport, "ensure_running_as") + def test_directory_argument_stages_the_files_inside_it( + self, _mock_ensure_running_as, mock_get_staging_root, tmp_path, capsys + ): + """A directory argument publishes the files beneath it, and is never itself staged.""" + inputdata_root = tmp_path / "inputdata" + subdir = inputdata_root / "lnd" + subdir.mkdir(parents=True) + staging_root = tmp_path / "staging" + staging_root.mkdir() + mock_get_staging_root.return_value = staging_root + (subdir / "a.nc").write_text("a") + (subdir / "b.nc").write_text("b") + + result = rimport.main(["-inputdata", str(inputdata_root), str(subdir)]) + + assert result == 0 + assert (staging_root / "lnd" / "a.nc").read_text() == "a" + assert (staging_root / "lnd" / "b.nc").read_text() == "b" + assert (subdir / "a.nc").is_symlink() + assert subdir.is_dir() and not subdir.is_symlink() + assert not (staging_root / "lnd").is_symlink() + + @patch.object(rimport, "get_staging_root") + @patch.object(rimport, "ensure_running_as") + def test_check_directory_argument_reports_each_file_and_stages_nothing( + self, _mock_ensure_running_as, mock_get_staging_root, tmp_path, capsys + ): + """--check reaches the same verdict on the same input without writing anything.""" + inputdata_root = tmp_path / "inputdata" + subdir = inputdata_root / "lnd" + subdir.mkdir(parents=True) + staging_root = tmp_path / "staging" + staging_root.mkdir() + mock_get_staging_root.return_value = staging_root + (subdir / "a.nc").write_text("a") + (subdir / "b.nc").write_text("b") + + result = rimport.main( + ["-inputdata", str(inputdata_root), str(subdir), "--check"] + ) + + assert result == 0 + captured = capsys.readouterr() + assert "a.nc" in captured.out and "b.nc" in captured.out + assert not any(staging_root.rglob("*")) + assert not (subdir / "a.nc").is_symlink() + + @patch.object(rimport, "get_staging_root") + @patch.object(rimport, "ensure_running_as") + def test_discovered_failure_warns_skips_and_returns_3( + self, _mock_ensure_running_as, mock_get_staging_root, tmp_path, capsys + ): + """A bad file FOUND BY A WALK must not abort the batch: the good files still + publish, the bad one is skipped, and the run reports 3.""" + inputdata_root = tmp_path / "inputdata" + subdir = inputdata_root / "lnd" + subdir.mkdir(parents=True) + staging_root = tmp_path / "staging" + staging_root.mkdir() + mock_get_staging_root.return_value = staging_root + (subdir / "good.nc").write_text("good") + (subdir / "broken.nc").symlink_to(inputdata_root / "nonexistent.nc") + + result = rimport.main(["-inputdata", str(inputdata_root), str(subdir)]) + + assert result == 3 + assert (staging_root / "lnd" / "good.nc").read_text() == "good" + assert (subdir / "good.nc").is_symlink() + captured = capsys.readouterr() + # Reported inline at WARNING (stdout) as the run reaches it... + assert "skipping" in captured.out + # ...and repeated at ERROR (stderr) at the very end, where it cannot be scrolled past. + assert "1 item(s) skipped (not stageable)" in captured.err + assert "broken.nc" in captured.err + + @patch.object(rimport, "get_staging_root") + @patch.object(rimport, "ensure_running_as") + def test_check_discovered_failure_also_returns_3( + self, _mock_ensure_running_as, mock_get_staging_root, tmp_path, capsys + ): + """--check reports the same skip and the same exit code, staging nothing.""" + inputdata_root = tmp_path / "inputdata" + subdir = inputdata_root / "lnd" + subdir.mkdir(parents=True) + staging_root = tmp_path / "staging" + staging_root.mkdir() + mock_get_staging_root.return_value = staging_root + (subdir / "good.nc").write_text("good") + (subdir / "broken.nc").symlink_to(inputdata_root / "nonexistent.nc") + + result = rimport.main( + ["-inputdata", str(inputdata_root), str(subdir), "--check"] + ) + + assert result == 3 + captured = capsys.readouterr() + assert "1 item(s) skipped (not stageable)" in captured.err + assert not any(staging_root.rglob("*")) + assert not (subdir / "good.nc").is_symlink() + + @patch.object(rimport, "get_staging_root") + @patch.object(rimport, "ensure_running_as") + def test_named_failure_still_aborts_everything_including_discovered_files( + self, _mock_ensure_running_as, mock_get_staging_root, tmp_path, capsys + ): + """Provenance asymmetry: a path the USER typed is still fatal, and it takes the + whole batch down with it -- including good files discovered under a good + directory.""" + inputdata_root = tmp_path / "inputdata" + subdir = inputdata_root / "lnd" + subdir.mkdir(parents=True) + staging_root = tmp_path / "staging" + staging_root.mkdir() + mock_get_staging_root.return_value = staging_root + (subdir / "good.nc").write_text("good") + missing = inputdata_root / "missing.nc" + + result = rimport.main( + ["-inputdata", str(inputdata_root), str(subdir), str(missing)] + ) + + assert result == 2 + assert not any(staging_root.rglob("*")) + assert not (subdir / "good.nc").is_symlink() + captured = capsys.readouterr() + assert "nothing was published" in captured.err + assert "1 of 2 item(s) failed pre-flight validation" in captured.err + + @patch.object(rimport, "get_staging_root") + @patch.object(rimport, "ensure_running_as") + def test_check_named_failure_also_aborts_before_checking_anything( + self, _mock_ensure_running_as, mock_get_staging_root, tmp_path, capsys + ): + """--check is gated by the same pre-flight, so a named failure aborts it too and the + good file is never reported on -- fix the bad name and re-run to see the rest.""" + inputdata_root = tmp_path / "inputdata" + subdir = inputdata_root / "lnd" + subdir.mkdir(parents=True) + staging_root = tmp_path / "staging" + staging_root.mkdir() + mock_get_staging_root.return_value = staging_root + (subdir / "good.nc").write_text("good") + missing = inputdata_root / "missing.nc" + + result = rimport.main( + ["-inputdata", str(inputdata_root), str(subdir), str(missing), "--check"] + ) + + assert result == 2 + captured = capsys.readouterr() + assert "nothing was published" in captured.err + assert "good.nc" not in captured.out + assert not any(staging_root.rglob("*")) + + @patch.object(rimport, "get_staging_root") + @patch.object(rimport, "ensure_running_as") + def test_staging_error_outranks_skip_in_exit_code( + self, _mock_ensure_running_as, mock_get_staging_root, tmp_path + ): + """Precedence 1 > 3: a real staging failure must not be masked by 'completed with + skips'.""" + inputdata_root = tmp_path / "inputdata" + subdir = inputdata_root / "lnd" + subdir.mkdir(parents=True) + staging_root = tmp_path / "staging" + staging_root.mkdir() + mock_get_staging_root.return_value = staging_root + (subdir / "good.nc").write_text("good") + (subdir / "broken.nc").symlink_to(inputdata_root / "nonexistent.nc") + + with patch.object(rimport, "stage_data", side_effect=RuntimeError("boom")): + result = rimport.main(["-inputdata", str(inputdata_root), str(subdir)]) + + assert result == 1 + + @patch.object(rimport, "get_staging_root") + @patch.object(rimport, "ensure_running_as") + def test_check_error_also_outranks_skip_in_exit_code( + self, _mock_ensure_running_as, mock_get_staging_root, tmp_path + ): + """Precedence 1 > 3 holds under --check: an item that fails while being checked is a + real failure, not a skip, even though nothing was being written. Also pins that the + flag reaches stage_data, which is the only place --check changes what happens.""" + inputdata_root = tmp_path / "inputdata" + subdir = inputdata_root / "lnd" + subdir.mkdir(parents=True) + staging_root = tmp_path / "staging" + staging_root.mkdir() + mock_get_staging_root.return_value = staging_root + (subdir / "good.nc").write_text("good") + (subdir / "broken.nc").symlink_to(inputdata_root / "nonexistent.nc") + + with patch.object( + rimport, "stage_data", side_effect=RuntimeError("boom") + ) as mock_stage_data: + result = rimport.main( + ["-inputdata", str(inputdata_root), str(subdir), "--check"] + ) + + assert result == 1 + mock_stage_data.assert_called_once_with( + subdir / "good.nc", inputdata_root, staging_root, True + ) + + @patch.object(rimport, "get_staging_root") + @patch.object(rimport, "ensure_running_as") + def test_named_unreadable_directory_is_fatal_not_a_skip( + self, _mock_ensure_running_as, mock_get_staging_root, tmp_path, capsys + ): + """A directory the USER NAMED that cannot be read is a named failure, so it aborts + the batch -- it must not be demoted to a skip that lets other named arguments + publish anyway. Demoting it would turn a hard stop into a partial publish. + """ + inputdata_root = tmp_path / "inputdata" + locked = inputdata_root / "locked" + locked.mkdir(parents=True) + (locked / "unreachable.nc").write_text("data") + other = inputdata_root / "ok" + other.mkdir() + (other / "good.nc").write_text("good") + (other / "also-good.nc").write_text("good") + staging_root = tmp_path / "staging" + staging_root.mkdir() + mock_get_staging_root.return_value = staging_root + os.chmod(locked, 0o000) + + try: + result = rimport.main( + ["-inputdata", str(inputdata_root), str(locked), str(other)] + ) + finally: + os.chmod(locked, 0o700) + + assert result == 2 + captured = capsys.readouterr() + assert "nothing was published" in captured.err + # `locked` never became an Entry, so a denominator taken from entries alone would + # report "1 of 2". Three items were considered: `locked` and the two files + # discovered under `ok`. Two arguments were given, so a denominator that counted + # those instead would also read "1 of 2". + assert "1 of 3 item(s) failed pre-flight validation" in captured.err + + # No file may have published -- not just the one that failed. + assert not any(staging_root.rglob("*")) + assert not (other / "good.nc").is_symlink() + + @patch.object(rimport, "get_staging_root") + @patch.object(rimport, "ensure_running_as") + def test_named_directory_outside_the_inputdata_root_is_rejected_not_walked( + self, _mock_ensure_running_as, mock_get_staging_root, tmp_path, capsys + ): + """A named path outside the inputdata root is fatal whether it is a file or a + directory. Expanding the directory instead would demote the user's own bad argument + to a pile of discovered skips and a "finished" exit 3. + + It must also not be walked. Expansion recurses, so a mistyped `rimport ~` would + otherwise stat an arbitrarily large tree before rejecting every file in it. + """ + inputdata_root = tmp_path / "inputdata" + inputdata_root.mkdir() + outside = tmp_path / "elsewhere" + (outside / "deep").mkdir(parents=True) + (outside / "a.nc").write_text("a") + (outside / "deep" / "b.nc").write_text("b") + staging_root = tmp_path / "staging" + staging_root.mkdir() + mock_get_staging_root.return_value = staging_root + + result = rimport.main(["-inputdata", str(inputdata_root), str(outside)]) + + assert result == 2 + captured = capsys.readouterr() + assert "nothing was published" in captured.err + # The directory itself is the failure, named once. Not its contents. + assert "1 of 1 item(s) failed pre-flight validation" in captured.err + # The reason must be the actionable one, not the is-a-directory backstop. + assert "source not under inputdata root" in captured.err + assert str(outside / "a.nc") not in captured.err + # Nothing beneath it may have been enumerated. + assert "expanded" not in captured.out + assert not any(staging_root.rglob("*")) + + @patch.object(rimport, "get_staging_root") + @patch.object(rimport, "ensure_running_as") + def test_naming_an_unreadable_directory_twice_reports_it_once( + self, _mock_ensure_running_as, mock_get_staging_root, tmp_path, capsys + ): + """The same path given twice is one bad path, not two: it is listed once, and + counted once in both halves of the "N of M" total.""" + inputdata_root = tmp_path / "inputdata" + locked = inputdata_root / "locked" + locked.mkdir(parents=True) + other = inputdata_root / "ok" + other.mkdir() + (other / "good.nc").write_text("good") + staging_root = tmp_path / "staging" + staging_root.mkdir() + mock_get_staging_root.return_value = staging_root + os.chmod(locked, 0o000) + + try: + result = rimport.main( + ["-inputdata", str(inputdata_root), str(locked), str(locked), str(other)] + ) + finally: + os.chmod(locked, 0o700) + + assert result == 2 + captured = capsys.readouterr() + # Two distinct paths were considered: `locked` and the good.nc discovered under `ok`. + assert "1 of 2 item(s) failed pre-flight validation" in captured.err + assert captured.err.count(f"rimport: '{locked}'") == 1 + + @patch.object(rimport, "get_staging_root") + @patch.object(rimport, "ensure_running_as") + def test_unreadable_subdirectory_stays_a_skip( + self, _mock_ensure_running_as, mock_get_staging_root, tmp_path, capsys + ): + """An unreadable directory found BENEATH a named one was not named by the user, so + it is not a named failure: it is warned about, skipped, and its readable siblings + still publish. Only the path the user typed is allowed to abort the batch.""" + inputdata_root = tmp_path / "inputdata" + tree = inputdata_root / "tree" + tree.mkdir(parents=True) + (tree / "good.nc").write_text("good") + locked = tree / "locked" + locked.mkdir() + (locked / "unreachable.nc").write_text("data") + staging_root = tmp_path / "staging" + staging_root.mkdir() + mock_get_staging_root.return_value = staging_root + os.chmod(locked, 0o000) + + try: + result = rimport.main(["-inputdata", str(inputdata_root), str(tree)]) + finally: + os.chmod(locked, 0o700) + + assert result == 3 + assert (staging_root / "tree" / "good.nc").read_text() == "good" + captured = capsys.readouterr() + assert "skipped (not stageable)" in captured.err + + @patch.object(rimport, "get_staging_root") + @patch.object(rimport, "ensure_running_as") + def test_walk_skip_is_still_reported_when_the_run_aborts( + self, _mock_ensure_running_as, mock_get_staging_root, tmp_path, capsys + ): + """A skipped path must be reported even when the run goes on to abort. A named + failure returns before the end-of-run summary, so if the skip were not also + reported inline it would appear nowhere at all.""" + inputdata_root = tmp_path / "inputdata" + tree = inputdata_root / "tree" + tree.mkdir(parents=True) + (tree / "good.nc").write_text("good") + locked = tree / "locked" + locked.mkdir() + (locked / "hidden.nc").write_text("data") + staging_root = tmp_path / "staging" + staging_root.mkdir() + mock_get_staging_root.return_value = staging_root + missing = inputdata_root / "missing.nc" + os.chmod(locked, 0o000) + + try: + result = rimport.main( + ["-inputdata", str(inputdata_root), str(tree), str(missing)] + ) + finally: + os.chmod(locked, 0o700) + + assert result == 2 + assert not any(staging_root.rglob("*")) + captured = capsys.readouterr() + assert str(locked) in captured.out + captured.err + + @patch.object(rimport, "get_staging_root") + @patch.object(rimport, "ensure_running_as") + def test_unreadable_parent_directory_is_an_error_not_a_traceback( + self, _mock_ensure_running_as, mock_get_staging_root, tmp_path + ): + """Naming a file inside a directory that cannot be read is a user error, and the help + text promises exit 2 for one. Path.is_dir() propagates EACCES rather than returning + False -- it ignores only ENOENT, ENOTDIR, EBADF and ELOOP -- so an unguarded probe + turns that into a stack trace on every supported version. + + An escaping exception errors this test rather than failing it, so reaching the + assertions below at all is half of what is being checked.""" + inputdata_root = tmp_path / "inputdata" + locked = inputdata_root / "locked" + locked.mkdir(parents=True) + (locked / "hidden.nc").write_text("data") + staging_root = tmp_path / "staging" + staging_root.mkdir() + mock_get_staging_root.return_value = staging_root + os.chmod(locked, 0o000) + + try: + result = rimport.main( + ["-inputdata", str(inputdata_root), str(locked / "hidden.nc")] + ) + finally: + os.chmod(locked, 0o700) + + assert result == 2 + assert not any(staging_root.rglob("*")) diff --git a/tests/rimport/test_validate_source_path.py b/tests/rimport/test_validate_source_path.py index 7f136b3..a96869d 100644 --- a/tests/rimport/test_validate_source_path.py +++ b/tests/rimport/test_validate_source_path.py @@ -114,6 +114,28 @@ def test_error_directory(tmp_path): assert "source is a directory, not a file" in str(result) +def test_error_directory_outside_root_names_containment_not_directoryness(tmp_path): + """Containment is diagnosed before the is-a-directory backstop, so the message names the + reason the user can act on. Told "source is a directory, not a file" they would + reasonably answer that directories are supported now. + + Directories inside the root never reach this in a normal run -- they are expanded -- so + the backstop above and this case are the two orderings that have to stay distinguished. + """ + inputdata_root = tmp_path / "inputdata" + inputdata_root.mkdir() + staging_root = tmp_path / "staging" + staging_root.mkdir() + + src = tmp_path / "elsewhere" + src.mkdir() + + result = rimport.validate_source_path(src, inputdata_root, staging_root) + assert isinstance(result, RuntimeError) + assert "source not under inputdata root" in str(result) + assert "is a directory" not in str(result) + + def test_error_file_outside_inputdata_root(tmp_path): """A regular file outside the inputdata root returns a RuntimeError, unraised.""" inputdata_root = tmp_path / "inputdata" diff --git a/tests/rimport/test_walk_files.py b/tests/rimport/test_walk_files.py new file mode 100644 index 0000000..92269a3 --- /dev/null +++ b/tests/rimport/test_walk_files.py @@ -0,0 +1,119 @@ +""" +Tests for walk_files() function in rimport script. +""" + +import os +import importlib.util +from importlib.machinery import SourceFileLoader + + +# Import rimport module from file without .py extension +rimport_path = os.path.join( + os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))), + "rimport", +) +loader = SourceFileLoader("rimport", rimport_path) +spec = importlib.util.spec_from_loader("rimport", loader) +if spec is None: + raise ImportError(f"Could not create spec for rimport from {rimport_path}") +rimport = importlib.util.module_from_spec(spec) +# Don't add to sys.modules to avoid conflict with other test files +loader.exec_module(rimport) + + +def test_finds_files_recursively(tmp_path): + """Files at every depth are returned, and directories themselves are not.""" + (tmp_path / "a").mkdir() + (tmp_path / "a" / "b").mkdir() + (tmp_path / "top.nc").write_text("top") + (tmp_path / "a" / "mid.nc").write_text("mid") + (tmp_path / "a" / "b" / "deep.nc").write_text("deep") + + files, skips = rimport.walk_files(tmp_path) + + assert skips == [] + assert files == [ + tmp_path / "a" / "b" / "deep.nc", + tmp_path / "a" / "mid.nc", + tmp_path / "top.nc", + ] + + +def test_entries_are_sorted_at_each_level(tmp_path): + """Order is deterministic, so output and downstream assertions are stable.""" + for name in ["zebra.nc", "apple.nc", "mango.nc"]: + (tmp_path / name).write_text(name) + + files, _skips = rimport.walk_files(tmp_path) + + assert [f.name for f in files] == ["apple.nc", "mango.nc", "zebra.nc"] + + +def test_symlinks_to_files_are_yielded(tmp_path): + """A symlink is an entry to act on, not something to pass over.""" + real = tmp_path / "real.nc" + real.write_text("data") + link = tmp_path / "link.nc" + link.symlink_to(real) + + files, _skips = rimport.walk_files(tmp_path) + + assert link in files + + +def test_does_not_descend_through_directory_symlink(tmp_path): + """A symlink to a directory is yielded as ONE entry; the walk must not follow it. + + Following it would let the walk leave the directory the user named, and would let a + cyclic link loop forever. + """ + tree = tmp_path / "tree" + tree.mkdir() + outside = tmp_path / "outside" + outside.mkdir() + (outside / "hidden_away.nc").write_text("must not be found") + link = tree / "dirlink" + link.symlink_to(outside) + + files, _skips = rimport.walk_files(tree) + + assert files == [link] + assert not any(f.name == "hidden_away.nc" for f in files) + + +def test_dotfiles_are_included(tmp_path): + """No hidden-file filter: rimport publishes what is there.""" + (tmp_path / ".hidden.nc").write_text("data") + + files, _skips = rimport.walk_files(tmp_path) + + assert [f.name for f in files] == [".hidden.nc"] + + +def test_empty_directory_yields_nothing(tmp_path): + """An empty tree is not an error.""" + files, skips = rimport.walk_files(tmp_path) + + assert files == [] + assert skips == [] + + +def test_unreadable_directory_is_returned_as_a_skip(tmp_path): + """A directory that cannot be read is reported, not raised, and does not abort the walk.""" + readable = tmp_path / "readable" + readable.mkdir() + (readable / "good.nc").write_text("data") + locked = tmp_path / "locked" + locked.mkdir() + (locked / "unreachable.nc").write_text("data") + os.chmod(locked, 0o000) + + try: + files, skips = rimport.walk_files(tmp_path) + finally: + os.chmod(locked, 0o700) + + assert files == [readable / "good.nc"] + assert len(skips) == 1 + assert skips[0].path == locked + assert isinstance(skips[0].reason, OSError)