Skip to main content

Access EDITO Storage from code, services and processes

Learn how EDITO provides S3 credentials to services and processes, how to renew them, and how to read or write private storage.

Context

S3 credentials provide authenticated access to the bucket selected in the current project configuration. Depending on the permissions associated with the credentials, your code may be able to read, write or delete objects.
​

The S3 section of a compatible service or process launch form is prefilled from the S3 configurations section.

You can keep the project defaults or override the configuration for a specific launch.

⚠️ Treat access keys, secret keys and session tokens as passwords. Do not commit them to Git, include them in a public notebook or print them in application logs.

EDITO services and processes can receive an S3 configuration automatically. Your code can then list, read, write and delete objects without hard-coding storage credentials.

This article is the canonical reference for programmatic access from a running EDITO environment.

Environment variables available in a service

In a correctly configured EDITO service, the storage settings are injected as environment variables. Open a terminal and run:

env | grep -E 'AWS|S3|MC_HOST'

The variables commonly include:

Variable

Meaning

AWS_S3_ENDPOINT

S3 host, normally without the https:// prefix

AWS_DEFAULT_REGION

S3 region, normally waw3-1 for the default EDITO storage

MC_HOST_s3

Temporary MinIO Client alias used by mc

S3_ENDPOINT

Complete endpoint URL, normally including https://

AWS_SECRET_ACCESS_KEY

S3 secret access key

AWS_ACCESS_KEY_ID

S3 access-key identifier

AWS_SESSION_TOKEN

Temporary-session token; present for short-lived EDITO credentials

📌 Note: not every provider uses every variable. In particular, static access keys normally do not require AWS_SESSION_TOKEN.

Credential types

Temporary EDITO credentials

The platform normally generates temporary S3 credentials for a service. They include:

  • An access key;

  • A secret key;

  • A session token.

They are valid for 24 hours. After they expire, renew them or restart the service.

💡 Temporary credentials are recommended for interactive services because their limited lifetime reduces the impact of accidental disclosure.

Static or long-lived access keys

We do not recommend using this type of credential for security reasons. However, you can automate the refresh of your credentials for services or processes that run for more than 24 hours (see below).

External-provider credentials

An external S3 provider such as CloudFerro, AWS or OVH supplies its own endpoint, region and credentials. These credentials may be static or temporary depending on the provider.

Renew temporary credentials in a running service

When the current 24-hour session expires, you can either:

  • Stop and restart the service; or

  • Renew the credentials in the current shell.

Use the built-in refresh script

In an EDITO built-in service (such as a Jupyter), run:

source /opt/refreshS3Credentials.sh

The script asks for the EDITO username and password used to connect to the Datalab. It requests a new S3 session and updates the environment variables in the current shell.

Because the script is sourced rather than executed as a child process, the exported variables remain available after the script finishes.


Provide the username and password non-interactively

EDITO_USERNAME=<USERNAME> EDITO_PASSWORD=<PASSWORD> source /opt/refreshS3Credentials.sh

⚠️ Passing a password directly on a command line can expose it through shell history or process inspection. Prefer the interactive command, a protected secret mechanism or a short-lived environment setup that is immediately unset.

📌 The built-in script is directly available only in EDITO-provided environments.

Advanced: implement the refresh of your credentials

The following Bash implementation reproduces the behavior of the refreshS3Credentials.sh built-in script: it obtains an EDITO identity token, exchanges it for a 24-hour S3 session and exports the resulting variables:

#!/usr/bin/env bash

# Prompt for EDITO username and password if not set
if [ -z "$EDITO_USERNAME" ]; then
    read -p "EDITO username: " -i ${KUBERNETES_NAMESPACE#"user-"} -e EDITO_USERNAME
fi
if [ -z "$EDITO_PASSWORD" ]; then
    read -s -p "EDITO password: " -e EDITO_PASSWORD
fi
echo

read_dom () {
    local IFS=\>
    read -d \< ENTITY CONTENT
}

# Get access token for minio
curlKeycloakCommand="curl --silent -X POST https://auth.dive.edito.eu/auth/realms/datalab/protocol/openid-connect/token -H 'Content-Type: application/x-www-form-urlencoded' -d 'client_id=onyxia-minio' -d 'username=${EDITO_USERNAME}' --data-urlencode 'password=${EDITO_PASSWORD}' -d 'grant_type=password' -d 'scope=openid+email+profile'"
curlKeycloakResult=$(eval $curlKeycloakCommand)
keycloakToken=$(echo "$curlKeycloakResult" | tr ',' '\n' | grep -o '"access_token":"[^"]*' | sed 's/"access_token":"//')

unset EDITO_USERNAME
unset EDITO_PASSWORD

AWS_S3_ENDPOINT=${AWS_S3_ENDPOINT="minio.dive.edito.eu"}
S3_ENDPOINT=${S3_ENDPOINT="https://$AWS_S3_ENDPOINT"}

export AWS_S3_ENDPOINT
echo "export AWS_S3_ENDPOINT=$AWS_S3_ENDPOINT"
export S3_ENDPOINT
echo "export S3_ENDPOINT=$S3_ENDPOINT"

# Export AWS environment variables
curlMinioCommand="curl --silent -X POST '$S3_ENDPOINT?Action=AssumeRoleWithWebIdentity&WebIdentityToken=$keycloakToken&DurationSeconds=86400&Version=2011-06-15'"
while read_dom; do
    if [[ $ENTITY = "AccessKeyId" ]]; then
        export AWS_ACCESS_KEY_ID=$CONTENT
        echo "export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID"
    elif [[ $ENTITY = "SecretAccessKey" ]]; then
        export AWS_SECRET_ACCESS_KEY=$CONTENT
        echo "export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY"
    elif [[ $ENTITY = "SessionToken" ]]; then
        export AWS_SESSION_TOKEN=$CONTENT
        echo "export AWS_SESSION_TOKEN=$AWS_SESSION_TOKEN"
    fi
done < <(eval $curlMinioCommand)

MC_HOST_s3=https://$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY:$AWS_SESSION_TOKEN@$AWS_S3_ENDPOINT
export MC_HOST_s3
echo "export MC_HOST_s3=$MC_HOST_s3"

This exemple requires Bash, curl and Python 3.

In Python, you can use this snippet of code instead:

import requests 
import os 
from xml.etree import ElementTree  

DATALAB_USERNAME = "<USERNAME>"  # To change with your username
DATALAB_PASSWORD = "<PASSWORD>"  # To change with your password  

url = "https://auth.dive.edito.eu/auth/realms/datalab/protocol/openid-connect/token" 
headers = {
     'Content-Type': 'application/x-www-form-urlencoded'
} 
data = {
     'client_id': 'onyxia-minio',
     'username': DATALAB_USERNAME,
     'password': DATALAB_PASSWORD,
     'grant_type': 'password',
     'scope': 'openid email profile'
}  

response = requests.post(url, headers=headers, data=data) 
json_response = response.json() 
access_token = json_response["access_token"]  

params = {
     "Action": "AssumeRoleWithWebIdentity",
     "WebIdentityToken": access_token,
     "DurationSeconds": "86400",
     "Version": "2011-06-15"
}  

response = requests.post(os.environ["S3_ENDPOINT"], params=params)  

root = ElementTree.fromstring(response.content) 
namespace_as_text = root.tag[root.tag.find("{")+1:root.tag.find("}")] namespace = {'ns': namespace_as_text} 
access_key_id = root.find('.//ns:AccessKeyId', namespace).text secret_access_key = root.find('.//ns:SecretAccessKey', namespace).text session_token = root.find('.//ns:SessionToken', namespace).text  

os.environ["AWS_ACCESS_KEY_ID"] = access_key_id
os.environ["AWS_SECRET_ACCESS_KEY"] = secret_access_key
os.environ["AWS_SESSION_TOKEN"] = session_token

⚠️ Do not keep a clear-text password in a committed notebook or script. Retrieve it from an appropriate secret store (such as Vault) or request it interactively.

Configure a service or process with static credentials

For a service or process whose launch form supports S3 configuration:
​

1. Open its S3 configuration section;

2. Keep the values inherited from Project Settings or replace them for this launch;

3. Set the endpoint to minio.dive.edito.eu for the default EDITO MinIO storage;

4. Set the region to waw3-1;

5. Enter the static Access Key ID and Secret Access Key;

6. Leave SessionToken empty for a static key.

You can also update the shared configuration in Project Settings → S3 Configurations. Remember that project settings are shared with all members of a group project.

📌 Note: for a custom key, review its S3 policy and restrict it to the minimum required permissions.

Create an S3 client with boto3

The helper below works with both temporary credentials and static credentials. It uses AWS_SESSION_TOKEN only when that variable is present.

import os
from typing import Any

import boto3
from botocore.client import BaseClient


def get_s3_endpoint_url() -> str:
    endpoint_url = os.getenv("S3_ENDPOINT")
    if endpoint_url:
        return endpoint_url

    endpoint_host = os.environ["AWS_S3_ENDPOINT"]
    if endpoint_host.startswith(("http://", "https://")):
        return endpoint_host
    return f"https://{endpoint_host}"


def get_s3_client() -> BaseClient:
    client_options: dict[str, Any] = {
        "service_name": "s3",
        "endpoint_url": get_s3_endpoint_url(),
        "aws_access_key_id": os.environ["AWS_ACCESS_KEY_ID"],
        "aws_secret_access_key": os.environ["AWS_SECRET_ACCESS_KEY"],
        "region_name": os.getenv("AWS_DEFAULT_REGION", "waw3-1"),
    }

    session_token = os.getenv("AWS_SESSION_TOKEN")
    if session_token:
        client_options["aws_session_token"] = session_token

    return boto3.client(**client_options)

Upload bytes without using the local filesystem

Use put_object when your application already holds the content in memory.

def save_bytes_to_s3(
    bucket_name: str,
    object_bytes: bytes | bytearray | memoryview,
    object_key: str,
) -> None:
    response = get_s3_client().put_object(
        Bucket=bucket_name,
        Body=object_bytes,
        Key=object_key,
    )

    status_code = response["ResponseMetadata"]["HTTPStatusCode"]
    if status_code != 200:
        raise RuntimeError(
            f"Upload failed for s3://{bucket_name}/{object_key}: {response}"
        )

    print(f"Uploaded s3://{bucket_name}/{object_key}")

How bytes are produced depends on the object and library. Images, text, video and serialized scientific objects can all be uploaded this way.

Write a NetCDF dataset directly from memory

xarray.Dataset.to_netcdf() returns NetCDF bytes when called without a path.

import xarray as xr

dataset: xr.Dataset = ...

save_bytes_to_s3(
    bucket_name="my_bucket",
    object_bytes=dataset.to_netcdf(),
    object_key="path/to/my_file.nc",
)

This avoids writing an intermediate NetCDF file to the service filesystem.

Upload an existing local file

s3 = get_s3_client()
s3.upload_file(
    Filename="/local/path/my_file.nc",
    Bucket="my_bucket",
    Key="target_folder/my_file.nc",
)

Download an object to the service filesystem

import  boto3 

s3  = boto3.client(
    "s3",
    endpoint_url="https://" +  "s3.waw3-1.cloudferro.com/",
    aws_access_key_id="ACCESSKEY",
    aws_secret_access_key="SECRETKEY",
)
 
file_path = "path_to_file/file.nc"

s3.download_file("bucket_name", file_path,  'out.nc')

Or a shorter version:

s3 = get_s3_client()
s3.download_file(
    Bucket="my_bucket",
    Key="path/to/file.nc",
    Filename="/local/path/out.nc",
)

The same pattern works with an EDITO personal bucket, a project bucket or an external S3 configuration, provided that the environment variables point to the correct endpoint and credentials.

Open a private dataset with xarray and s3fs

Use s3fs to expose a private S3 bucket through a file-like interface that xarray can open. Then, you can point to your file:

import s3fs
from os import environ
import xarray

fs = s3fs.S3FileSystem(
    client_kwargs={'endpoint_url': 'https://'+environ["AWS_S3_ENDPOINT"]},
    key = environ["AWS_ACCESS_KEY_ID"],
    secret = environ["AWS_SECRET_ACCESS_KEY"],
    token = environ["AWS_SESSION_TOKEN"]
)

ds = xarray.open_dataset(fs.open("<BUCKET_NAME>/<DATASET_NAME>"))
print(ds)

For cloud-optimized formats such as Zarr, use an appropriate xarray/Zarr store rather than treating the dataset as one monolithic file.

📌 Note: direct S3 access can avoid downloading an entire large dataset when the format and library support partial reads.

Write a Zarr dataset directly to private storage

The following example writes an xarray.Dataset to an S3-backed Zarr store. It is also used in the sharing workflow described in Share files and datasets from EDITO storage.
​

import os

import s3fs
import xarray as xr

endpoint_host = os.environ["AWS_S3_ENDPOINT"]
endpoint_url = (
    endpoint_host
    if endpoint_host.startswith(("http://", "https://"))
    else f"https://{endpoint_host}"
)

s3fs_options = {
    "client_kwargs": {"endpoint_url": endpoint_url},
    "key": os.environ["AWS_ACCESS_KEY_ID"],
    "secret": os.environ["AWS_SECRET_ACCESS_KEY"],
}
if os.getenv("AWS_SESSION_TOKEN"):
    s3fs_options["token"] = os.environ["AWS_SESSION_TOKEN"]

fs = s3fs.S3FileSystem(**s3fs_options)

dataset: xr.Dataset = ...
x_chunk: int = ...
y_chunk: int = ...

encoding = {
    "my_var": {
        "chunks": (x_chunk, y_chunk),
    }
}

store = s3fs.S3Map(
    root="oidc-[YOUR_USERNAME]/foobar.zarr",
    s3=fs,
    create=True,
)

dataset.to_zarr(
    store=store,
    consolidated=True,
    mode="w",
    encoding=encoding,
)

Adapt the variables, dimensions and chunk sizes to your dataset. Chunking strongly affects access performance and should be chosen according to the expected read patterns.

Persist outputs from an EDITO process

A process runs in temporary computing storage. The process template can copy a designated output path to the user's permanent storage after the computation finishes.


Use EDITO_INFRA_OUTPUT

Write the files that must persist under the path stored in: EDITO_INFRA_OUTPUT


The template process includes an additional container named copy-output in job.yaml. This step copies the content of EDITO_INFRA_OUTPUT to the user's personal storage.

Your process code should therefore:

  1. Read the value of EDITO_INFRA_OUTPUT from the environment;

  2. Create the directory if required;

  3. Write final outputs there;

  4. Finish successfully so that the copy step can run.

Example in Python:

from pathlib import Path
import os

output_directory = Path(os.environ["EDITO_INFRA_OUTPUT"])
output_directory.mkdir(parents=True, exist_ok=True)

result_path = output_directory / "result.nc"
# dataset.to_netcdf(result_path)

Inject the S3 secret into the copy container

The process template exposes the generated S3 secret with envFrom:

envFrom:
  {{- if .Values.s3.enabled }}
  - secretRef:
      name: {{ include "library-chart.secretNameS3" . }}
  {{- end }}

The copy container requires:

  • AWS_ACCESS_KEY_ID;

  • AWS_SECRET_ACCESS_KEY;

  • AWS_SESSION_TOKEN for temporary credentials;

  • AWS_S3_ENDPOINT;

  • AWS_DEFAULT_REGION.

These variables are generated from:

  • The s3 section of values.schema.json; and

  • The secret-s3.yaml template.

The process chart must keep these elements aligned with the S3 configuration exposed in the launch form.
​

The implementation relies on the process Helm chart configuration, in particular job.yaml, the s3 section of values.schema.json, and the secret-s3.yaml template.


​

Security in shared group-project services

All members of a group project can access the project storage, and project S3 settings are shared.

Do not configure a shared service with an unrestricted credential that also grants access to your personal bucket. Instead:

1. Create a dedicated access key;

2. Restrict its policy to the required project bucket and prefixes;

3. Grant only the necessary actions, such as read-only or write to one output prefix;

4. Revoke or rotate the key when the shared service no longer needs it.

The same principle applies to external-provider credentials.


​

Troubleshooting

AccessDenied or 403

Check that:

  • The credentials belong to the expected personal or project context;

  • The policy allows the requested operation;

  • The bucket and object key are correct;

  • The configured endpoint belongs to that bucket;

  • A group-project service is not accidentally using a personal-only configuration.

ExpiredToken, InvalidToken or authentication failure after several hours

Temporary EDITO credentials expire after 24 hours. Run:

source /opt/refreshS3Credentials.sh

Then restart the Python kernel or recreate any S3 client that cached the previous credentials.

Endpoint errors

For the default EDITO storage, use:

  • Endpoint host: minio.dive.edito.eu

  • Endpoint URL: https://minio.dive.edito.eu

  • Region: waw3-1

Some configuration forms expect the host without https://, while Python clients normally expect the complete URL.


Static credentials fail when a session token is present

Remove or unset an obsolete AWS_SESSION_TOKEN when switching from temporary credentials to static access keys:

unset AWS_SESSION_TOKEN

Then recreate the S3 client.


​

What's next?

If you have any questions, problems, or suggestions, please feel free to contact us via chat using the widget available at the bottom right of the page.


​

Did this answer your question?

Related Articles

EDITO Help Center
This page is currently being created and is constantly evolving to cover as many topics as possible.

Funded by the European Union