How to migrate to 0.3 for APIs that expect PathLike

[Zan-L]

As an example, Polars used to work perfectly with UPath < 0.3:

path = UPath('s3a://...')
df = pl.read_parquet(path)

But after 0.3 it stops working. The problem is, none of the 3 solutions works from the migration guide. Any suggestions?

[ap--]

Hi @Zan-L

cross-referencing: pola-rs/polars#24921

In the example you provided, they converted remote UPaths to a string via indirectly calling os.fspath() via os.path.expanduser() see here: https://github.com/pola-rs/polars/blob/77254fce44d7b848aa9aa8bb3a0f01f5d2e63d1c/py-polars/src/polars/_utils/various.py#L235
It's better to be explicit about how you're calling this API, instead of relying on the UPath being cast to string (and losing its storage options).

Your example code is effectively equivalent to:

path = UPath('s3a://...')
df = pl.read_parquet(path.as_uri())

Warning

But be aware !! you're dropping all storage_options in the general case (in your example there are none).

I would recommend to use it like this, if you want to use fsspec and s3fs as the read backend (probably slower than polars default):

import upath
import polars

p = upath.UPath("s3://bucket/file", anon=True)

polars.scan_parquet(p.open('rb'))

Or if you want to use the polars rust backends

import upath
import polars

p = upath.UPath("s3://bucket/file", anon=True)

object_store_storage_options = ...  # convert your fsspec p.storage_options to the object store style

polars.scan_parquet(p.as_uri(), storage_options=object_store_storage_options)

Warning

Because polars uses object-store instead of fsspec and s3fs, the storage options have different names and you have to convert: https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html

I'll add this to the migration guide when I find some time.

Cheers,
Andreas

[cas--]

Ah this explains why I am encountering new errors upgrading to latest version of UPath passed to pandas.read_parquet and pyarrow.

Including the error here to help anyone else encountering it:

  File "pyarrow/_dataset_parquet.pyx", line 271, in pyarrow._dataset_parquet.ParquetFileFormat.make_fragment
  File "pyarrow/_dataset.pyx", line 1379, in pyarrow._dataset.FileFormat.make_fragment
  File "pyarrow/_dataset.pyx", line 137, in pyarrow._dataset._make_file_source
TypeError: cannot construct a FileSource from s3://bucket/test.parquet

Considering that pyarrow is fsspec compatible I assume this would be the recommended approach:

df = pd.read_parquet(path=path.as_uri(), filesystem=path.fs)

Unless it's a bug and pyarrow should handle UPaths?

Incidentally I thought path.as_uri() was same as str(path) but looking at the code there is a guard for relative paths which is probably worth documenting too

universal_pathlib/upath/core.py

Lines 1955 to 1961 in 63b6f1c

	def as_uri(self) -> str:
	"""Return the string representation of the path as a URI."""
	if self._relative_base is not None:
	raise ValueError(
	f"relative path can't be expressed as a {self.protocol} URI"
	)
	return str(self)