Context
On EDITO, we define a process this way: a process is a remote function that generates data, such as data transformation, pre/post-processing, reanalysis, forecasts, detections, What-If scenarios, quality controls. It can be piped, scheduled, or triggered on-demand. Its inputs and outputs locations can be configured at runtime. In opposite to a service, it is not interactive during execution (it does not host a web server or UI).
In order to have a broad compatibility, we chose to follow standards, and in the case of computation, the OGC API - process correspond to our needs. Their definition of a process is the following:
"The OGC API - Processes standard supports the wrapping of computational tasks into executable processes that can be offered by a server through a Web API and be invoked by a client application. The standard specifies a processing interface to communicate over a RESTful protocol using JavaScript Object Notation (JSON) encodings. Typically, these processes execute well-defined algorithms that ingest vector and/or coverage data to produce new datasets.”
⚠️ WARNING ⚠️
Even if the process is running, it may be deleted during platform maintenance. We strongly recommend saving the process configuration at the time of creation using the “Save” button to facilitate restart in case of an outage, and saving your code to a remote repository or in your “My Files” space. The process content will use resources from your temporary storage, not the permanent storage corresponding to the content in “My Files”.
Getting started
First thing first, you will need to have a container image hosted on a public repository. This image should run a container exposing environment variables to determine inputs and outputs location, if they are needed. We strongly recommand to follow the 12-factor app methodology.
In this tutorial, we will take the Coral bleaching detection process as an example. At the time being, it only has one input parameter determining if the process is launched as a small demonstration (small resources needed to run).
And as you can see, this project satisfies the minimal requirement for hosting a process on the datalab, which is having a public container image available with environment variables if an input is needed.
Deploy your process using add-your-process
⚠️ The tool add-your-process is an early feature. Please, contact the user support to be authorized to access the contribution tools.
add-your-process is a tool available in the Contribution tab of the Datalab Process Catalogue. It will help you create and register a new EDITO process from a Docker image, a Dockerfile, or a Python configuration file. It automates your Helm Chart creation (metadata setup, code retrieval) and publication into the Datalab Playground Process Catalogue.
💡 EDITO Pro Tips :
Once the
add-your-processtool is complete, your service will be accessible in the Process Playground catalogue within 5 minutes.You can also save your configuration by giving it a name
(1)and clicking on save(8)! Do not forget to save just before launching(9)to have the full configuration saved.
Configuration (2): where do you deploy from?
The Configuration section (2) allows you to initiate where you will deploy your process. There are three options available to you: from an existing Docker image, from a Docker file or from a Python environment.
Deploy from an Existing Docker Image
If you already built and pushed a Docker image to a registry, you’re in the right place.
Once you start the add-your-process tool, fill in the Configuration form fields:
Enable the Container image is already built option by setting it to
trueSpecify your image URL in the Image tag field
Example: if your image is on Docker Hub, the URL will look like
docker.io/myaccount/my-image-name:tag.
With this information, add-your-process will create a Helm Chart based on the image provided.
Deploy from a Dockerfile
If your project is hosted in a remote Git repository and can be built with Docker using a Dockerfile, you can also deploy it this way.
Set the Container image is already built to
falseSpecify the path to your Dockerfile from the root of your repository.
Example: if your Dockerfile is in a folder named
production, the path should beproduction/Dockerfile. If your Dockerfile is at root of the repository, the path should beDockerfile.
With this information, add-your-process will build a Docker image from the Dockerfile and create a Helm chart based on this image.
Deploy from a Python environment
If your project is hosted in a remote Git repository and uses only Python scripts, you can create a conda_environment.yaml file inside your project:
name: my-environment
channels:
- conda-forge
- defaults
dependencies:
- python=3.13
- pip
- pip:
- requests
This file allows you to declare all the information required for your script to run properly, especially the list of dependencies.
Then you will need to reference the path of your conda_environment.yaml and specify the entry point of your Python code:
Specify the entry point of your Python code in EntrypointPath
(1)Example: if your
main.pyfile is located in a folder namedproduction, the path should be:production/main.pyIf your script is at the root of the repository, the path should be:
my_script.pyReference the path of your
conda_environment.yamlin EnvironmentPath(2)Example: if your environment file is located in a folder named
production, the path should be:production/my_env_file.yaml
If your file is at the root of the repository, the path should be:
my_env_file.yaml
With this information, add-your-process will copy your code and integrate it into an official Python image, installing all necessary dependencies.
Metadata (3)
Whether you’re deploying from a Docker image, a Dockerfile or a Python environment, you must fill in some metadata about your service. In the Metadata form (3), complete the following fields:
Process name (capital letters are not allowed)
Process version
Process description
Icon URL
Homepage URL
You can also add keywords to improve search visibility within the service catalogue.
⚠️ If you want to replace an existing process with add-your-process, make sure you use a version number higher than the old version so that the replacement can take place.
For example, if the current version of the process is 1.1.2, you must have at least version 1.1.3, 1.2.0, or 2.0.0. In the Metadata section (1) Yyou can customize the version number of your process (2):
Git (4)
You now need to provide Git repository access information in the Git section (4). There are two possibilities:
If your repository is public:
Enable Add git config inside your environment in the Git section
(4)of the form to enable internal Git configurationProvide the repository URL in Repository
If your repository is private, provide your authentication information in the corresponding fields:
Name
Email
Token (personal access token)
By default, the tool will pull files from the main branch of your repository. If you need to target another branch, specify its name in the Branch field.
Copernicus Marine Credentials (5)
If your process requires access to a Copernicus Marine account, check this box (5). The Copernicus connection variables will be automatically injected when the process starts.
📌 Note: don't forget to set up your Copernicus Marine credentials in Your Account
Other environment variables (6)
All environment variables required for the process to run properly can be defined here. When the process starts, these values can be customized by the user who wants to run the process.
Resources (7)
Here you can define the resources on which the process will run. Your process will have at least the requested resources (1) and never more than its limits (2).
Once everything is ready, don't forget to save your add-your-process configuration by changing the friendly name (1) and clicking save (8).
You can then launch (9) the tool, and your process will be visible in the Process Playground catalogue within 5 minutes.
⚠️ Once you have deployed your process on EDITO, you may encounter the following error when attempting to launch it:
Onyxia API Error: Put /api/my-lab/app Response:500
In this case, we recommend opening a notebook with an “edit” role in order to view the exact error and resolve it more easily (it might be a wrong path in your add-your-process form for example).
You are now ready to start and use your process!
What's next?
If you want to deploy a process on EDITO in a more advanced way, you can also create your own Helm chart!
If you have any questions, problems, or suggestions, please feel free to contact us via chat using the widget available at the bottom right of the page.