Context
WEkEO's EOCanvas is a service that provides serverless processing for all Copernicus data to users. The eocanvas Python module or API enables users to process and analyze data without needing to download any files to their local directories. In general, all data available through the Harmonized Data Access (HDA) API can be processed using the EOCanvas service. The processes are executed in the cloud, so only the final results are downloaded to the local system! 😃
EOCanvas is specifically designed for working with Earth Observation data. As a result, it relies on specialized Earth Observation data analysis software, already widely used within the user community, to deliver these processes to users.
Currently, two software packages are implemented in EOCanvas:
Any process that can be performed within these software packages can also be applied using EOCanvas. Existing workflows from projects using these tools can easily be converted into serverless workflows by simply providing the SNAP graph or configuration file to the EOCanvas API.
You can download this notebook to follow along the article:
Or go to Jupyter Catalogue to access the notebook via the WEkEO JupyterHub or GitLab:
📌 Note: Discover more about EOCanvas on Read The Docs by visiting the following link: https://eocanvas.readthedocs.io/en/latest/
New Features (Released 15th November 2024):
The latest EO Canvas release introduces exciting new features to enhance usability and flexibility for users:
Support for S3 Buckets: Users can now leverage S3 buckets as both input sources and output destinations.
EODATA Bucket Compatibility: Data can be directly accessed from the EODATA bucket as an input source. (Note: A known issue with SNAP and EODATA input is under investigation.)
WEkEO Drive Integration: Full support for the WEkEO Drive, enabling seamless use as both input and output storage solutions.
Documentation and Examples: Updated guides for external storage are now available, and example notebooks for using S3 and WEkEO Drive will be released soon.
Explore these new capabilities to enhance your EO Canvas experience!
Set up EOCanvas for use
Step 1. Load required modules
To get started, make sure that you have installed the necessary packages for your project:
pip install hda -U
pip install eocanvas[hda]
These commands will install HDA and EOcanvas.
After the installation, you can import the required libraries:
from eocanvas import API, Credentials
from eocanvas.api import Input, Config, ConfigOption
from eocanvas.processes import SnapProcess
from eocanvas.snap.graph import Graph
Step 2. Set up credentials
You will need to provide your WEkEO credentials. If you do not have an account yet, you can register here.
Replace <your_user_name> and <your_password> with your WEkEO login information:
c = Credentials(username="<your_user_name>", password="<your_password>")
c.save()
Your credentials will be saved and automatically loaded when needed. Once saved, the credentials file will be written to your system at the following location:
C:\Users\your_username\.hdarc
Inputs to EOCanvas Serverless Functions
To process data using the EOCAnvas Serverless Functions, there are 3 necessary inputs to provide to the function:
Data query
Processing workflow
Configuration parameters
Let's take a look at each of the inputs using a simple example: let's say we want to use the Serverless Function to create a subset of a Sentinel-3 WFR tile (1) over a small area of interest over the Baltic Sea (2):
Step 1. Data query
The data is given to the Serverless Function as an url. The url is the reference location of the data in the WEkEO HDA.
To get the url, you need to make a request for the data using the WEkEO HDA:
from hda import Client
c = Client()
q = {
"dataset_id": "EO:EUM:DAT:SENTINEL-3:OL_2_WFR___",
"dtstart": "2024-08-06T08:58:00.000Z",
"dtend": "2024-08-06T09:00:00.000Z",
"timeliness": "NT"
}
r = c.search(q)
url = r.get_download_urls()[0]
inputs = Input(key="img1", url=url)
The url is the first input to the Serverless Function.
Step 2. Processing workflow
The Serverless Functions take workflows coming from common Earth Observation data software as input, e.g. the software SNAP.
You can build processing workflows using the SNAP desktop software. The graphs will look like this in the Graphical User Interface. You can also view the xml graph of the processing workflow. The xml graph is then the input to the Serverless Functions:
You can find the xml Graph for subsetting the area of interest in the file subset_olci.xml:
Let's now import the file as Graph input.
graph = Graph.from_uri("input_graphs/subset_olci.xml")💡WEkEO Pro Tip: use of SNAP is not mandatory. This article is intended to be used by as many people as possible. If you have a use case you'd like to implement, please contact us by chat or at support@wekeo.eu so that we can help you set it up. In the future, sample graphs will be available to cover as many cases as possible.
Step 3. Configuration parameters
Depending on how the data input looks like, there are several pre-processing steps necessary, e.g. unzipping. This is defined in the Config input to the Serverless Functions:
config = Config(key="img1", options=ConfigOption(uncompress=True, sub_path="xfdumanifest.xml"))
📌Note: in this example, the xfdumanifest.xml file is in the .zip file. Depending on the data you want, you may or may not have to unzip it.
Step 4. Executing the process
Now everything is in place to launch the Serverless Function and run the process:
process = SnapProcess(snap_graph=graph, eo_config=config, eo_input=inputs)
process.prepare_inputs()
When you call process.run, the process will be submitted. You can specify the directory where the results should be downloaded by setting the download_dir parameter.
The function will block execution until the results are fully downloaded. Depending on the complexity of the process, this may take some time:
process.run(download_dir="result")
Step 5. Examine the result
The result is stored in the directory and can be examined using common Python tools, such as xarray:
import os
result = os.listdir("result")[0]
result
You'll get the file name like this:
'76c4f06e-75cc-11ef-9aa4-0ea904598d5c.nc'
A preview of the data can then be displayed using matplotlib:
import xarray as xr
xr.open_dataset("result/"+result).CHL_NN.plot()
And voilàààà!
You have successfully subsetted a Sentinel-3 scene using the WEkEO EOCanvas Serverless Functions! 💪
Explore the Functions of the API
You can get more information about the processes of the Serverless Function using the following commands:
api = API()
Get an overview of the EOCanvas Serverless Functions API in this notebook:
Or go to Jupyter Catalogue to access the notebook via the WEkEO JupyterHub or GitLab:
Which processes are available in the Serverless Functions?
Which processes are available in the Serverless Functions?
processes = api.get_processes() processes
Output:
[Process(api=<eocanvas.api.API object at 0x000001B63AB64DD0>, process_id='dataTailor', version='3.4.0', title='EUMETSAT Data Tailor', description='The EUMETSAT Data Tailor provides format conversion and basic product customisation capabilities for a set of EUMETSAT products in native formats.', inputs={'epct_input': {'title': 'eo_inputs Title', 'description': 'Named inputs provided to the EUMETSAT Data Tailor tool in JSON encoding.', 'minOccurs': 0, 'maxOccurs': 1, 'schema': {'$ref': 'http://named_inputs'}}, 'epct_chain': {'title': 'EUMETSAT Data Tailor chain file', 'description': 'Customisation file describing the chain of customisation to apply to the epct_input files.', 'minOccurs': 0, 'maxOccurs': 1, 'schema': {'$ref': 'https://string'}}}), Process(api=<eocanvas.api.API object at 0x000001B63AB64DD0>, process_id='snap-function', version='10.0.0', title='snap-function', description='Function based on the ESA SNAP tool version 10.', inputs={'eo_config': {'title': 'Configuration for eo_input', 'description': 'Configuration for the eo_input in JSON format. They instruct the function on how to deal with the eo_input (e.g., uncompress).', 'minOccurs': 0, 'maxOccurs': 1, 'schema': {'$ref': 'https://string'}}, 'snap_graph': {'title': 'ESA SNAP GPT graph', 'description': 'A base64-encoded gpt graph. It should contain placeholders referring to the eo_input field.', 'minOccurs': 0, 'maxOccurs': 1, 'schema': {'$ref': 'https://string'}}, 'eo_input': {'title': 'eo_inputs Title', 'description': "Named inputs provided to the ESA SNAP tool in JSON encoding. They should match the placeholders in the SNAP's GPT graph.", 'minOccurs': 0, 'maxOccurs': 1, 'schema': {'$ref': 'http://named_inputs'}}})]You can see that two processes are available as of now:
the dataTailor function
the SNAP function
Along with the definition of the functions, you see their description and the inputs each function takes.
Which processes did I submit to the EOCanvas?
Which processes did I submit to the EOCanvas?
Using get_job() the api will return all of the submitted jobs.
jobs = api.get_jobs() jobs
Output:
[Job(api=<eocanvas.api.API object at 0x000001B63AB64DD0>, job_id='7cb62b31-e28e-5d9d-aa0c-4e510c17a44e', status='successful', started='2024-09-18:14.40.21', created='2024-09-18:14.40.21', updated='2024-09-18T14:44:55.340603913Z', finished='2024-09-18T14:44:54Z'), Job(api=<eocanvas.api.API object at 0x000001B63AB64DD0>, job_id='6ebd4434-9b53-569b-990e-9c7f7b4fc8be', status='failed', started='2024-08-20:15.52.42', created='2024-08-20:15.52.42', updated='2024-08-20T15:55:57.253617813Z', finished='2024-08-20T15:55:56Z'), Job(api=<eocanvas.api.API object at 0x000001B63AB64DD0>, job_id='f135490a-c8b4-573a-a6f3-49b26cc85dc9', status='successful', started='2024-08-20:12.50.03', created='2024-08-20:12.50.03', updated='2024-08-20T12:52:21.256719423Z', finished='2024-08-20T12:52:21Z'), Job(api=<eocanvas.api.API object at 0x000001B63AB64DD0>, job_id='ed0d1bd6-0efa-5cc5-934b-fc3a81f7a219', status='successful', started='2024-08-20:12.34.51', created='2024-08-20:12.34.51', updated='2024-08-20T12:37:29.254227006Z', finished='2024-08-20T12:37:28Z'), Job(api=<eocanvas.api.API object at 0x000001B63AB64DD0>, job_id='cc467fb5-0c74-5d2f-871e-240785724310', status='failed', started='2024-08-20:12.27.23', created='2024-08-20:12.27.23', updated='2024-08-20T12:29:03.249389979Z', finished='2024-08-20T12:29:02Z'), Job(api=<eocanvas.api.API object at 0x000001B63AB64DD0>, job_id='45091bb6-c788-5b2c-8f0d-2e683c9bdb64', status='failed', started='2024-08-20:12.26.53', created='2024-08-20:12.26.53', updated='2024-08-20T12:27:45.248828489Z', finished='2024-08-20T12:27:44Z')]
How do I get the Job Logs of one specific Job?
How do I get the Job Logs of one specific Job?
# select the job you want to get the logs for
job = '7cb62b31-e28e-5d9d-aa0c-4e510c17a44e'
api.get_job_logs(job=job)
You can now see the full logs of the process.
This is where you'll find error information if your request was unsuccessful.
How do I download the results from a job which has been launched in the past?
How do I download the results from a job which has been launched in the past?
To download the results again, you can manually trigger the download using the job id:
#get and inspect the results
results = api.get_job_results(job=job)
results
Output:
[Result(api=<eocanvas.api.API object at 0x000001B63AB64DD0>, href='/download/result/359', title='wks3:///7cb62b31-e28e-5d9d-aa0c-4e510c17a44e/output/76c4f06e-75cc-11ef-9aa4-0ea904598d5c.nc', rel='result')]
Then run:
results[0].download(download_dir="result")
Output:
Downloading result\76c4f06e-75cc-11ef-9aa4-0ea904598d5c.nc
Discover more notebooks
At WEkEO, we continually develop notebooks to help you explore and utilize Earth Observation data more effectively. Each notebook provides hands-on guidance for specific tasks and workflows, enhancing your understanding of EO data processing and visualization. Below are some of the notebooks we've created, designed to support various analytical needs:
Using the EUMETSAT Data Tailor with EOCanvas
Dive into the functionalities of EO Canvas DataTailor. This notebooks shows a simple example of how the EUMETSAT Data Tailor processing chains can be executed using the EOCavas.Creating an OLCI Data Cube with EO Canvas Workflow
This notebook guides you through processing Sentinel-3 OLCI satellite imagery by reprojecting, resampling, and working with multiple inputs to merge outputs into a cohesive data cube. You’ll also learn to visualize the data cube using the xcube Viewer for efficient analysis.Extracting Pixel Values with EO Canvas Workflow
In this notebook, you'll discover how to extract pixel values, modify workflows for custom locations, and visualize these values on a map for focused analysis.The input to the EOCanvas functions can either come from WEkEO’s HDA API or a remote object storage. This notebook provides an example of how to use data stored in S3 buckets as input for EOCanvas functions. It also demonstrates how to send results back to an S3 bucket, enabling seamless processing without the need to download data to your local machine.
What's next?
Feel free to check these articles that might be of interest for you:
If you have any question or feedback, feel free to contact us:
through a chat session available in the bottom right corner of the page
via our contact webpage
via e-mail to our support team (supportATwekeo.eu)
Regardless of how you choose to contact us, you will first be put in touch with our AI Agent Neo. At any time, you can reach a member of the WEkEO User Support team by clicking on "talk to a person" via chat, or by naturally requesting it in reply to Neo's email.