Long node_modules paths cannot be found on Windows when LongPathsEnabled enabled

@bzoz

Version

v20.9.0

Platform

Microsoft Windows NT 10.0.20348.0 x64

Subsystem

No response

What steps will reproduce the bug?

I created a reproduction repo at https://github.com/karlhorky/node-js-max_path-windows-bug . The steps in the GitHub Actions workflow file reproduce the bug:

# Test whether Windows support for long paths is enabled
(Get-ItemProperty "HKLM:System\CurrentControlSet\Control\FileSystem").LongPathsEnabled
# 1

# Clone, setup Node.js

# Install dependencies with npm
npm install

# Run program in short path (✅ works)
node eslint.config.js

# Make directory with long path and copy all files there
mkdir long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\
cp eslint.config.js long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\
cp index.js long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\
cp package.json long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\

# Change to directory with long path
cd long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\

# Install dependencies with npm
npm install

# Run program in long path (💥 crashes)
node eslint.config.js

The error that shows up is this:

node:internal/process/esm_loader:[4](https://github.com/karlhorky/node-js-max_path-windows-bug/actions/runs/6890011394/job/18742165029#step:11:5)0
      internalBinding('errors').triggerUncaughtException(
                                ^

Error: Cannot find package 'D:\a\node-js-max_path-windows-bug\node-js-max_path-windows-bug\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\node_modules\eslint-plugin-import\package.json' imported from D:\a\node-js-max_path-windows-bug\node-js-max_path-windows-bug\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\eslint.config.js
Did you mean to import eslint-plugin-import/lib/index.js?
    at legacyMainResolve (node:internal/modules/esm/resolve:189:26)
    at packageResolve (node:internal/modules/esm/resolve:776:14)
    at moduleResolve (node:internal/modules/esm/resolve:838:20)
    at defaultResolve (node:internal/modules/esm/resolve:1043:11)
    at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:383:12)
    at ModuleLoader.resolve (node:internal/modules/esm/loader:3[5](https://github.com/karlhorky/node-js-max_path-windows-bug/actions/runs/6890011394/job/18742165029#step:11:6)2:25)
    at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:228:38)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:85:39)
    at link (node:internal/modules/esm/module_job:84:3[6](https://github.com/karlhorky/node-js-max_path-windows-bug/actions/runs/6890011394/job/18742165029#step:11:7)) {
  code: 'ERR_MODULE_NOT_FOUND'
}

Node.js v20.[9](https://github.com/karlhorky/node-js-max_path-windows-bug/actions/runs/6890011394/job/18742165029#step:11:10).0
Error: Process completed with exit code 1.

How often does it reproduce? Is there a required condition?

Always

What is the expected behavior? Why is that the expected behavior?

If Windows support for long paths is turned on (see LongPathsEnabled check above showing that it's enabled) I would expect Node.js to also support long paths (longer than the Windows ~260 characters MAX_PATH limit)

What do you see instead?

The crash and error above

Additional information

More information about LongPathsEnabled :

MAX_PATH lengths on Windows environment actions/runner-images#1052 (comment)

Context:

I'm here after originally reporting the bug in ESLint (I thought that this comment by @bzoz meant that Node.js is not affected by the 260 character path limit on Windows :

Bug: Long node_modules plugin paths cannot be found on Windows eslint/eslint#17763

cc: @nodejs/platform-windows

@karlhorky

I took a look at it, but found another issue instead. @karlhorky I forked your repo and the failing run can be found here.

I made long path longer, because the original one was shorter then 260 characters. After I did it, Even running npm install (or anything else, example here) started failing. Since I'm seeing same error regardless if I'm using CMD or PowerShell it seems safe to assume this is something Windows specific.

Back to the issue you reported, importer uses file URLs internally, thus cannot use \\?\ prefix required for handling long paths. I'll investigate this further, and add another comment based on my findings.

@al-cheb

Thanks for the response, and for the investigation! 🙌

I made long path longer, because the original one was shorter then 260 characters. After I did it, Even running npm install (or anything else, example here) started failing

I guess you mean "anything else Node.js-related"? Because the cp command on the previous step doesn't fail:

cp eslint.config.js long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\

But maybe the way that GitHub Actions is launching PowerShell + CMD has a problem - maybe a separate GitHub Actions-related problem cc @al-cheb @MaksimZhukov @chrispat:

An error occurred trying to start process 'C:\Program Files\PowerShell\7\pwsh.EXE' with working directory 'D:\a\node-js-max_path-windows-bug\node-js-max_path-windows-bug\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\'. The directory name is invalid.

I'll try to run various commands from a directory with a long path on my Windows machine locally and see what I find.

@JeremyKuhne

Ok, I ran some commands on a Windows 11 Pro x64 machine (both node and also cmd) I can confirm that running commands from within deep directories fails with The directory name is invalid:

PS C:\> node -v
v20.5.1

PS C:\> mkdir long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\


    Directory: C:\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\l
    ong-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-p
    ath\long-path\long-path\long-path\long-path


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----        17.11.2023     16:27                long-path


PS C:\> cd long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\
PS C:\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path> node -v
Program 'node.exe' failed to run: The directory name is invalidAt line:1 char:1
+ node -v
+ ~~~~~~~.
At line:1 char:1
+ node -v
+ ~~~~~~~
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed

Commands like cd still work:

PS C:\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path> cd .

cmd.exe also doesn't run:

PS C:\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path> cmd
Program 'cmd.exe' failed to run: The directory name is invalidAt line:1 char:1
+ cmd
+ ~~~.
At line:1 char:1
+ cmd
+ ~~~
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed

Also when running C:\Windows\System32\cmd.exe with the full path - still has a problem with the current directory being invalid:

PS C:\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path> C:\Windows\System32\cmd.exe
Program 'cmd.exe' failed to run: The directory name is invalidAt line:1 char:1
+ C:\Windows\System32\cmd.exe
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~.
At line:1 char:1
+ C:\Windows\System32\cmd.exe
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed

Wonder if someone from the Windows team involved in interop / this MAX_PATH limitation could confirm that running - I don't know of contacts who work on the Windows team, but maybe @JeremyKuhne (because of this blog post) or @iSazonov (because of involvement in PowerShell/PowerShell#17197 and PowerShell/PowerShell#13955) or @SteveL-MSFT (because of PowerShell/PowerShell involvement) ?

Anyway, back to the import / long path file reading issue, maybe at least this part could be supported by Node.js...

Ok, I ran some commands on a Windows 11 Pro x64 machine (both node and also cmd) I can confirm that running commands from within deep directories fails with The directory name is invalid:

That is correct. cp is a command, not an executable, so I assume only applications are facing that issue.

Back to the import with long paths problem. Investigating a bit longer got me to one of my first PRs in this repo, especially to changes in the manifest file. Applying that change solved an issue for me, but I'd like to investigate a bit more to find the root cause. The original PR was dropped in favor of this one as that was enough to fix the issue and something similar should probably be possible here. I'll get back to this later this week to continue my work.

Regarding the "PowerShell unable to launch programs from long path" problem:

I've commented in an existing MAX_PATH issue in the PowerShell/PowerShell repo:

Incorrect behavior running executable if current location path is longer than 256 characters PowerShell/PowerShell#13955 (comment)

I may need to open a new issue for this as well.

After further investigation, I saw that some parts of the code responsible for module loading, do not add the required \\?\ prefix for Windows long path support. I was able to make a PoC fixing it without the changes to the manifest file, but that's just a starting point for further work.

@karlhorky

@karlhorky we can do it both ways. I do not have time to work on this now, but I've talked to my colleague @huseyinacacak-janea and he'll take a look.

In my PR that addressed this issue, I added a new test test/es-module/test-GH-50753.js, which passes without any problems. Do you see some way to change that test and make it experience same issue you are facing with GitHub Action in your repo?

Ok great, thanks for the info!

I see the current version of the test at test/es-module/test-GH-50753.js here:

node/test/es-module/test-GH-50753.js

Lines 1 to 42 in 2aaeaa8

    
           'use strict'; 
        
           // Flags: --expose-internals 
        
           const common = require('../common'); 
        
           if (!common.isWindows) { 
        
             common.skip('this test is Windows-specific.'); 
        
           } 
        
           const fs = require('fs'); 
        
           const path = require('path'); 
        
           const tmpdir = require('../common/tmpdir'); 
        
           // https://github.com/nodejs/node/issues/50753 
        
           // Make a path that is more than 260 chars long. 
        
           // Module layout will be the following: 
        
           //  package.json 
        
           //  main 
        
           //    main.js 
        
           const packageDirNameLen = Math.max(260 - tmpdir.path.length, 1); 
        
           const packageDirName = tmpdir.resolve('x'.repeat(packageDirNameLen)); 
        
           const packageDirPath = path.resolve(packageDirName); 
        
           const packageJsonFilePath = path.join(packageDirPath, 'package.json'); 
        
           const mainDirName = 'main'; 
        
           const mainDirPath = path.resolve(packageDirPath, mainDirName); 
        
           const mainJsFile = 'main.js'; 
        
           const mainJsFilePath = path.resolve(mainDirPath, mainJsFile); 
        
           tmpdir.refresh(); 
        
           fs.mkdirSync(packageDirPath); 
        
           fs.writeFileSync( 
        
             packageJsonFilePath, 
        
             `{"main":"${mainDirName}/${mainJsFile}"}` 
        
           ); 
        
           fs.mkdirSync(mainDirPath); 
        
           fs.writeFileSync(mainJsFilePath, 'console.log("hello world");'); 
        
           require('internal/modules/run_main').executeUserEntryPoint(packageDirPath); 
        
           tmpdir.refresh();

Current version looks good, but low-level (whereas my reproduction was more on the "end to end" / "integration" end of the spectrum) - maybe there is something else in Node.js that is also failing... 🤔

@huseyinacacak-janea

@huseyinacacak-janea could you reopen this issue as a tracking issue for this work?

@huseyinacacak-janea

Oh, apparently @huseyinacacak-janea doesn't have access to reopen the issue

@StefanStojanovic could you do it instead?

Can it be merged into other TLC versions?

module,win: fix long path resolve

@huseyinacacak-janea

@huseyinacacak-janea thanks for the PR fixing the problem in #53294, and thanks @RafaelGSS and other Node.js contributors for the Node.js 22.7.0 release PR #54452 where the fix landed

I just upgraded to Node.js 22.7.0 in my reproduction repo in the PR below:

Upgrade to Node.js 22.7.0 karlhorky/node-js-max_path-windows-bug#2

I can confirm that the workflow run of the GitHub Actions workflow was successful 🎉

So for my reproduction case, this is now resolved 🙌

If there are any other further long path issues with Node.js on Windows, I'll be sure to open a different issue and mention it here (if the issue is not locked over time).

@karlhorky

@karlhorky

I tried to follow this fix, and it seemed to work. However, trying longer paths still causes this to fail. In your example, the path length is still below 260. If you try creating a path that is longer than 260 and npm init in there, what happens then?

@ibrahimqasim-abtrace

@ibrahimqasim-abtrace I have not noticed any failures, but it's possible that I'm not testing the same way you mean.

Can you fork my repo and create a reproduction that fails on GitHub Actions?

https://github.com/karlhorky/node-js-max_path-windows-bug

Then it will be a clear, reproducible failure that Node.js can look at.

Remove .npmrc workaround for Windows long path

Navigation Menu

Repository navigation

Long node_modules paths cannot be found on Windows when `LongPathsEnabled` enabled #50753

Version

Platform

Subsystem

What steps will reproduce the bug?

How often does it reproduce? Is there a required condition?

What is the expected behavior? Why is that the expected behavior?

What do you see instead?

Additional information

11 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	'use strict';

	// Flags: --expose-internals

	const common = require('../common');
	if (!common.isWindows) {
	common.skip('this test is Windows-specific.');
	}

	const fs = require('fs');
	const path = require('path');
	const tmpdir = require('../common/tmpdir');

	// https://github.com/nodejs/node/issues/50753
	// Make a path that is more than 260 chars long.
	// Module layout will be the following:
	// package.json
	// main
	// main.js

	const packageDirNameLen = Math.max(260 - tmpdir.path.length, 1);
	const packageDirName = tmpdir.resolve('x'.repeat(packageDirNameLen));
	const packageDirPath = path.resolve(packageDirName);
	const packageJsonFilePath = path.join(packageDirPath, 'package.json');
	const mainDirName = 'main';
	const mainDirPath = path.resolve(packageDirPath, mainDirName);
	const mainJsFile = 'main.js';
	const mainJsFilePath = path.resolve(mainDirPath, mainJsFile);

	tmpdir.refresh();

	fs.mkdirSync(packageDirPath);
	fs.writeFileSync(
	packageJsonFilePath,
	`{"main":"${mainDirName}/${mainJsFile}"}`
	);
	fs.mkdirSync(mainDirPath);
	fs.writeFileSync(mainJsFilePath, 'console.log("hello world");');

	require('internal/modules/run_main').executeUserEntryPoint(packageDirPath);

	tmpdir.refresh();

Long node_modules paths cannot be found on Windows when LongPathsEnabled enabled #50753

Description

Version

Platform

Subsystem

What steps will reproduce the bug?

How often does it reproduce? Is there a required condition?

What is the expected behavior? Why is that the expected behavior?

What do you see instead?

Additional information

Activity

MrJithil commented on Nov 17, 2023

StefanStojanovic commented on Nov 17, 2023

karlhorky commented on Nov 17, 2023

karlhorky commented on Nov 17, 2023

StefanStojanovic commented on Nov 20, 2023

karlhorky commented on Nov 21, 2023

StefanStojanovic commented on Dec 5, 2023

11 remaining items

StefanStojanovic commented on May 21, 2024

karlhorky commented on May 21, 2024

karlhorky commented on May 21, 2024

karlhorky commented on May 22, 2024

WangJincheng4869 commented on May 23, 2024

karlhorky commented on Sep 9, 2024

ibrahimqasim-abtrace commented on Oct 17, 2024

karlhorky commented on Oct 18, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Long node_modules paths cannot be found on Windows when `LongPathsEnabled` enabled #50753