File tree Expand file tree Collapse file tree
Expand file tree Collapse file tree Original file line number Diff line number Diff line change @@ -100,11 +100,16 @@ Supported Object Stores are
100100- {py: class }` ~datafusion.object_store.MicrosoftAzure `
101101
102102``` python
103+ import os
104+
105+ from datafusion import SessionContext
103106from datafusion.object_store import AmazonS3
104107
105108region = " us-east-1"
106109bucket_name = " yellow-trips"
107110
111+ ctx = SessionContext()
112+
108113s3 = AmazonS3(
109114 bucket_name = bucket_name,
110115 region = region,
@@ -120,6 +125,28 @@ ctx.register_parquet("trips", path)
120125ctx.table(" trips" ).show()
121126```
122127
128+ ### Use S3 in SQL
129+
130+ Configure S3 access on an {py: class }` ~datafusion.object_store.AmazonS3 ` object and
131+ register it on the context before issuing SQL that uses an ` s3:// ` location. AWS
132+ credentials are not SQL ` OPTIONS ` : ` aws.* ` is not a recognized SQL configuration
133+ namespace.
134+
135+ After registering the object store above, a SQL external table can use the same
136+ S3 path:
137+
138+ ``` python
139+ ctx.sql(
140+ f """
141+ CREATE EXTERNAL TABLE trips_sql
142+ STORED AS PARQUET
143+ LOCATION ' { path} '
144+ """
145+ ).collect()
146+
147+ ctx.sql(" SELECT count(passenger_count) FROM trips_sql" ).show()
148+ ```
149+
123150## Other DataFrame Libraries
124151
125152DataFusion can import DataFrames directly from other libraries, such as
You can’t perform that action at this time.
0 commit comments