Context
EDITO Storage is S3-compatible, which means you can access your files programmatically from a wide range of languages, libraries and command-line tools.
This article provides ready-to-use code and configuration snippets for some of the most common clients, including R, Python, Polars, MinIO Client, s3cmd and rclone.
All these snippets are available in My Account > Connect to storage > To access your storage outside of datalab services section:
⚠️ Keep your credentials secure. Your access key, secret access key and session token provide access to your storage. Do not commit them to Git, include them in public notebooks or expose them in application logs.
The examples below use the default EDITO Storage endpoint, minio.dive.edito.eu, and region, waw3-1. To use these examples, retrieve your current storage credentials from My Account > Connect to storage, then replace the placeholder values in the relevant snippet.
Access EDITO storage from R
R (paws)
# Install the paws package
install.packages("paws", repos = "https://cloud.R-project.org")
# Configure your EDITO Storage credentials
Sys.setenv(
"AWS_ACCESS_KEY_ID" = "<YOUR_ACCESS_KEY_ID>",
"AWS_SECRET_ACCESS_KEY" = "<YOUR_SECRET_ACCESS_KEY>",
"AWS_DEFAULT_REGION" = "waw3-1",
"AWS_SESSION_TOKEN" = "<YOUR_SESSION_TOKEN>",
"AWS_S3_ENDPOINT" = "minio.dive.edito.eu"
)
# Load paws
library("paws")
# Create an S3 client
minio <- paws::s3(
config = list(
credentials = list(
creds = list(
access_key_id = Sys.getenv("AWS_ACCESS_KEY_ID"),
secret_access_key = Sys.getenv("AWS_SECRET_ACCESS_KEY"),
session_token = Sys.getenv("AWS_SESSION_TOKEN")
)
),
endpoint = paste0(
"https://",
Sys.getenv("AWS_S3_ENDPOINT")
),
region = Sys.getenv("AWS_DEFAULT_REGION")
)
)
# List the buckets available to your account
minio$list_buckets()
R (aws.S3)
# Install the aws.s3 package
install.packages("aws.s3", repos = "https://cloud.R-project.org")
# Configure your EDITO Storage credentials
Sys.setenv(
"AWS_ACCESS_KEY_ID" = "<YOUR_ACCESS_KEY_ID>",
"AWS_SECRET_ACCESS_KEY" = "<YOUR_SECRET_ACCESS_KEY>",
"AWS_DEFAULT_REGION" = "waw3-1",
"AWS_SESSION_TOKEN" = "<YOUR_SESSION_TOKEN>",
"AWS_S3_ENDPOINT" = "minio.dive.edito.eu"
)
# Load aws.s3
library("aws.s3")
# List the buckets available to your account
bucketlist(region = "")
Access EDITO storage from Python
Python (s3fs)
Use s3fs to access EDITO Storage from Python using S3-compatible credentials.
import os
import s3fs
# Configure your EDITO Storage credentials
os.environ["AWS_ACCESS_KEY_ID"] = "<YOUR_ACCESS_KEY_ID>"
os.environ["AWS_SECRET_ACCESS_KEY"] = "<YOUR_SECRET_ACCESS_KEY>"
os.environ["AWS_SESSION_TOKEN"] = "<YOUR_SESSION_TOKEN>"
os.environ["AWS_DEFAULT_REGION"] = "waw3-1"
# Create an S3 filesystem client
fs = s3fs.S3FileSystem(
client_kwargs={
"endpoint_url": "https://minio.dive.edito.eu"
},
key=os.environ["AWS_ACCESS_KEY_ID"],
secret=os.environ["AWS_SECRET_ACCESS_KEY"],
token=os.environ["AWS_SESSION_TOKEN"]
)
Python (boto3)
Use boto3 to connect directly to EDITO Storage with an S3-compatible client.
import boto3
# Create an S3 client for EDITO Storage
s3 = boto3.client(
"s3",
endpoint_url="https://minio.dive.edito.eu",
aws_access_key_id="<YOUR_ACCESS_KEY_ID>",
aws_secret_access_key="<YOUR_SECRET_ACCESS_KEY>",
aws_session_token="<YOUR_SESSION_TOKEN>",
region_name="waw3-1"
)
Python (polars)
Use Polars to read Parquet files directly from EDITO Storage without downloading them first.
import os
import polars as pl
# Option 1: Use credentials from environment variables
storage_options = {
"aws_endpoint": "https://minio.dive.edito.eu",
"aws_access_key_id": os.environ["AWS_ACCESS_KEY_ID"],
"aws_secret_access_key": os.environ["AWS_SECRET_ACCESS_KEY"],
"aws_region": os.environ["AWS_DEFAULT_REGION"],
"aws_token": os.environ["AWS_SESSION_TOKEN"],
}
# Option 2: Provide credentials directly
storage_options = {
"aws_endpoint": "https://minio.dive.edito.eu",
"aws_access_key_id": "<YOUR_ACCESS_KEY_ID>",
"aws_secret_access_key": "<YOUR_SECRET_ACCESS_KEY>",
"aws_region": "waw3-1",
"aws_token": "<YOUR_SESSION_TOKEN>",
}
# Read Parquet files from EDITO Storage
df = pl.scan_parquet(
source="s3://<YOUR_BUCKET>/*.parquet",
storage_options=storage_options,
)
print(df)
Configure your shell environment
Use these environment variables to configure your EDITO Storage credentials in a shell session.
export AWS_ACCESS_KEY_ID="<YOUR_ACCESS_KEY_ID>"
export AWS_SECRET_ACCESS_KEY="<YOUR_SECRET_ACCESS_KEY>"
export AWS_DEFAULT_REGION="waw3-1"
export AWS_SESSION_TOKEN="<YOUR_SESSION_TOKEN>"
export AWS_S3_ENDPOINT="minio.dive.edito.eu"
Access EDITO storage with command-line tools
s5cmd
Fast S3 command-line operations and bulk transfers.
export AWS_ACCESS_KEY_ID="<YOUR_ACCESS_KEY_ID>"
export AWS_SECRET_ACCESS_KEY="<YOUR_SECRET_ACCESS_KEY>"
export AWS_SESSION_TOKEN="<YOUR_SESSION_TOKEN>"
export AWS_REGION="waw3-1"
export S3_ENDPOINT_URL="https://minio.dive.edito.eu"
s5cmd ls
Example:
s5cmd cp "s3://<YOUR_BUCKET>/*.parquet" ./data/
MC client
Manage buckets and objects with the MinIO command-line client.
export MC_HOST_s3="https://<YOUR_ACCESS_KEY_ID>:<YOUR_SECRET_ACCESS_KEY>:<YOUR_SESSION_TOKEN>@minio.dive.edito.eu"
💡 To interact with your EDITO storage from outside EDITO, we recommend that you do not use the MC client, as it is no longer supported outside of EDITO. Unless you have already installed it, please use another method.
s3cmd
Access S3-compatible storage using the s3cmd CLI.
[default]
access_key = <YOUR_ACCESS_KEY_ID>
access_token = <YOUR_SESSION_TOKEN>
add_encoding_exts =
add_headers =
bucket_location = us-east-1
ca_certs_file =
cache_file =
check_ssl_certificate = False
check_ssl_hostname = False
cloudfront_host = cloudfront.amazonaws.com
default_mime_type = binary/octet-stream
delay_updates = False
delete_after = False
delete_after_fetch = False
delete_removed = False
dry_run = False
enable_multipart = True
encoding = UTF-8
encrypt = False
expiry_date =
expiry_days =
expiry_prefix =
follow_symlinks = False
force = False
get_continue = False
gpg_command = /usr/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase =
guess_mime_type = True
host_base = minio.dive.edito.eu
host_bucket = minio.dive.edito.eu
human_readable_sizes = False
invalidate_default_index_on_cf = False
invalidate_default_index_root_on_cf = True
invalidate_on_cf = False
kms_key =
limitrate = 0
list_md5 = False
log_target_prefix =
long_listing = False
max_delete = -1
mime_type =
multipart_chunk_size_mb = 15
multipart_max_chunks = 10000
preserve_attrs = True
progress_meter = True
proxy_host =
proxy_port = 0
put_continue = False
recursive = False
recv_chunk = 65536
reduced_redundancy = False
requester_pays = False
restore_days = 1
secret_key = <YOUR_SECRET_ACCESS_KEY>
send_chunk = 65536
server_side_encryption = False
signature_v2 = False
simpledb_host = sdb.amazonaws.com
skip_existing = False
socket_timeout = 300
stats = False
stop_on_error = False
storage_class =
urlencoding_mode = normal
use_https = True
use_mime_magic = True
verbosity = WARNING
website_endpoint = http://%(bucket)s.s3-website-%(location)s.amazonaws.com/
website_error =
website_index = index.html
rclone
Copy and synchronize files between EDITO Storage and other locations.
[minio]
type = s3
provider = Minio
env_auth = false
upload_concurrency = 5
acl = private
bucket_acl = private
endpoint = minio.dive.edito.eu
access_key_id = <YOUR_ACCESS_KEY_ID>
secret_access_key = <YOUR_SECRET_ACCESS_KEY>
session_token = <YOUR_SESSION_TOKEN>
Going further
The snippets above provide a quick way to connect your preferred tool or library to EDITO Storage. Once connected, you can use the standard S3 features supported by that client to list buckets, read and write objects, or work directly with datasets stored in EDITO.
Keep in mind that EDITO temporary storage credentials are short-lived. If your connection stops working after your credentials expire, retrieve or refresh your credentials before reconnecting.
For more information about how storage credentials are provided to EDITO services and processes, how to renew them, and how to work with private storage programmatically, see Access EDITO Storage from code, services and processes.
What's next?
If you have any questions, problems, or suggestions, please feel free to contact us via chat using the widget available at the bottom right of the page.