Skip to main content

Add your own process on EDITO

Learn how to add your own computation as a process on EDITO using a pre-configured tool: add-your-process.

Context

On EDITO, we define a process this way: a process is a remote function that generates data, such as data transformation, pre/post-processing, reanalysis, forecasts, detections, What-If scenarios, quality controls. It can be piped, scheduled, or triggered on-demand. Its inputs and outputs locations can be configured at runtime. In opposite to a service, it is not interactive during execution (it does not host a web server or UI).

In order to have a broad compatibility, we chose to follow standards, and in the case of computation, the OGC API - process correspond to our needs. Their definition of a process is the following:

"The OGC API - Processes standard supports the wrapping of computational tasks into executable processes that can be offered by a server through a Web API and be invoked by a client application. The standard specifies a processing interface to communicate over a RESTful protocol using JavaScript Object Notation (JSON) encodings. Typically, these processes execute well-defined algorithms that ingest vector and/or coverage data to produce new datasets.”

⚠️ WARNING ⚠️

Even if the process is running, it may be deleted during platform maintenance. We strongly recommend saving the process configuration at the time of creation using the “Save” button to facilitate restart in case of an outage, and saving your code to a remote repository or in your “My Files” space. The process content will use resources from your temporary storage, not the permanent storage corresponding to the content in “My Files”.

Getting started

First thing first, you will need to have a container image hosted on a public repository. This image should run a container exposing environment variables to determine inputs and outputs location, if they are needed. We strongly recommand to follow the 12-factor app methodology.

In this tutorial, we will take the Coral bleaching detection process as an example. At the time being, it only has one input parameter determining if the process is launched as a small demonstration (small resources needed to run).

And as you can see, this project satisfies the minimal requirement for hosting a process on the datalab, which is having a public container image available with environment variables if an input is needed.

💡 EDITO Pro Tip: before using add-your-process, you can use the Generic Process to run your code without creating a tile in the EDITO catalogue. The code is fetched from your GitLab/GitHub repository or your Docker image, making it easier for you to manage your versions before pushing the final result via add-your-process.

Deploy your process using add-your-process

⚠️ The tool add-your-process is an early feature. Please, contact the user support to be authorized to access the contribution tools.

add-your-process is a tool available in the Contribution tab of the Datalab Process Catalogue. It will help you create and register a new EDITO process from a Docker image, a Dockerfile, or a Python configuration file. It automates your Helm Chart creation (metadata setup, code retrieval) and publication into the Datalab Playground Process Catalogue.

📌 Note on how to integrate data into your process:

  1. If the data is yours: enter the URL of your file in your code, you will have to enable the visibility of your file from the File Explorer by clicking the eye icon

  2. If each user use their own data: your code must read the link and the S3 bucket via an environment variable. You can refer to these articles (Interact with your personal EDITO storage and Write data to EDITO storage) for more details

💡 EDITO Pro Tips :

  • Once the add-your-process tool is complete, your service will be accessible in the Process Playground catalogue within 5 minutes.

  • You can also save your configuration by giving it a name (1) and clicking on save (8)! Do not forget to save just before launching (9) to have the full configuration saved.

Configuration (2): where do you deploy from?

The Configuration section (2) allows you to initiate where you will deploy your process. There are three options available to you: from an existing Docker image, from a Docker file or from a Python environment.

Deploy from an Existing Docker Image

If you already built and pushed a Docker image to a registry, you’re in the right place.

Once you start the add-your-process tool, fill in the Configuration form fields:

  1. Enable the Container image is already built option by setting it to true

  2. Specify your image URL in the Image tag field

    Example: if your image is on Docker Hub, the URL will look like docker.io/myaccount/my-image-name:tag.

With this information, add-your-process will create a Helm Chart based on the image provided.

Deploy from a Dockerfile

If your project is hosted in a remote Git repository and can be built with Docker using a Dockerfile, you can also deploy it this way.

  1. Set the Container image is already built to false

  2. Specify the path to your Dockerfile from the root of your repository.

    Example: if your Dockerfile is in a folder named production, the path should be production/Dockerfile. If your Dockerfile is at root of the repository, the path should be Dockerfile.

With this information, add-your-process will build a Docker image from the Dockerfile and create a Helm chart based on this image.

Deploy from a Python environment

If your project is hosted in a remote Git repository and uses only Python scripts, you can create a conda_environment.yaml file inside your project:

name: my-environment

channels:
  - conda-forge
  - defaults

dependencies:
  - python=3.13
  - pip

  - pip:
      - requests

This file allows you to declare all the information required for your script to run properly, especially the list of dependencies.

Then you will need to reference the path of your conda_environment.yaml and specify the entry point of your Python code:

  1. Specify the entry point of your Python code in EntrypointPath (1)

    Example: if your main.py file is located in a folder named production, the path should be: production/main.py

    If your script is at the root of the repository, the path should be: my_script.py

  2. Reference the path of your conda_environment.yaml in EnvironmentPath(2)

    Example: if your environment file is located in a folder named production, the path should be: production/my_env_file.yaml
    If your file is at the root of the repository, the path should be:
    ​my_env_file.yaml

With this information, add-your-process will copy your code and integrate it into an official Python image, installing all necessary dependencies.

Metadata (3)

Whether you’re deploying from a Docker image, a Dockerfile or a Python environment, you must fill in some metadata about your service. In the Metadata form (3), complete the following fields:

  • Process name (capital letters are not allowed)

  • Process version

  • Process description

  • Icon URL

  • Homepage URL

You can also add keywords to improve search visibility within the service catalogue.

⚠️ If you want to replace an existing process with add-your-process, make sure you use a version number higher than the old version so that the replacement can take place.

For example, if the current version of the process is 1.1.2, you must have at least version 1.1.3, 1.2.0, or 2.0.0. In the Metadata section (1) Yyou can customize the version number of your process (2):

Git (4)

You now need to provide Git repository access information in the Git section (4). There are two possibilities:

  1. If your repository is public:

    1. Enable Add git config inside your environment in the Git section (4) of the form to enable internal Git configuration

    2. Provide the repository URL in Repository

  2. If your repository is private, provide your authentication information in the corresponding fields:

    1. Name

    2. Email

    3. Token (personal access token)

By default, the tool will pull files from the main branch of your repository. If you need to target another branch, specify its name in the Branch field.

Copernicus Marine Credentials (5)

If your process requires access to a Copernicus Marine account, check this box (5). Don't forget to set the Copernicus Marine environment variables (COPERNICUSMARINE_SERVICE_USERNAME and COPERNICUSMARINE_SERVICE_PASSWORD) in your code so that the process can use them.

The Copernicus connection variables will be automatically injected when the process starts: once the process has been created, users who launch it will be able to enter their Copernicus Marine credentials in a dedicated section.

📌 Note: don't forget to set up your Copernicus Marine credentials in Your Account

Other environment variables (6)

If your project requires environment variables, you can define them in this section.

All environment variables required for the process to run properly can be defined here. Just complete the name, the description and the DefaultValue.

When the process starts, these values can be customized by the user who wants to run the process. For example, this could be the save path for the outputs.

Resources (7)

Here you can define the resources on which the process will run. Your process will have at least the requested resources (1) and never more than its limits (2).

Once everything is ready, don't forget to save your add-your-process configuration by changing the friendly name (1) and clicking save (8).

📌 Note: the friendly name must be less than 48 characters long and can only contain “-” (dash); it cannot contain “_” (underscore) or spaces.

You can then launch (9) the tool, and your process will be visible in the Process Playground catalogue within 5 minutes.

⚠️ Once you have deployed your process on EDITO, you may encounter the following error when attempting to launch it:

Onyxia API Error: Put /api/my-lab/app Response:500

In this case, we recommend opening a notebook with an “edit” role in order to view the exact error and resolve it more easily (it might be a wrong path in your add-your-process form for example).

You are now ready to start and use your process!

Start Using Your Process

Once the forms are complete, launch the tool. When it completes successfully, it will automatically create your process.

Your process will be available within 5 minutes in the Process Playground catalogue. You can search by name to find it easily.

Configure environment variables

If your process needs environment variables, they will appear in the Inputs form.

You’re now ready to start and use your process!

Add your process to the Process Catalogue

⚠️ ⚠️ Make sure you have filled out the metadata section in add-your-process correctly so that the Chart generated by EDITO works properly.

Once the process is fully operational and you want to submit it to the official Process Catalogue, you can proceed as follows (only if you used add-your-process; if you created your own Helm Chart, the workflow is slightly different):

  1. Open a Jupyter Notebook with "edit" role

  2. In the service's commands, add the Helm repo, using helm repo add process-playground https://gitlab.mercator-ocean.fr/api/v4/projects/2064/packages/helm/process-playground

  3. Enter the command line helm pull process-playground/NAME_OF_YOUR_PROCESS --untar

Now you have the folder ready to be pushed to the right repository.

To add the folder to the right repo you need to:

  1. Pull it using git (https://gitlab.mercator-ocean.fr/pub/edito-infra/process-helm-charts)

  2. Create a branch

  3. Add your folder in the right category

  4. Do the merge request. The technical team will have to validate it afterwards, you will receive a notification once it's done.

📌 Note: if necessary, the support team can download the Chart generated by add-your-process for you and send it to you. We are also currently setting up a solution to submit the MR on your behalf when you request it.

What's next?

If you want to deploy a process on EDITO in a more advanced way, you can also create your own Helm chart!

If you have any questions, problems, or suggestions, please feel free to contact us via chat using the widget available at the bottom right of the page.

Related Articles
Add your own service on EDITO
Add your service with Helm
Add your process with Helm
Run code on EDITO or on a remote SSH target
Deploy your web service using Generic
Did this answer your question?