All Collections
WEkEO Harmonized Data Access
HDA API Client & Rest
How to use the WEkEO Harmonized Data Access API REST with cURL?
How to use the WEkEO Harmonized Data Access API REST with cURL?

Let's see how to use the WEkEO HDA API Rest with cURL!

Alexandre avatar
Written by Alexandre
Updated over a week ago

Context


The Harmonized Data Access (HDA) API allows uniform access to the whole WEkEO catalogue, including subsetting and downloading functionalities (to learn more: How to download WEkEO data?).

The HDA API is REST-based and is published at this base URL:

https://gateway.prod.wekeo2.eu/hda-broker.

πŸ“ŒNote: all endpoints defined in this article are relative to this URL.

We introduce here usage examples of this API in WEkEO's JupyterHub service (accessible from Sign in > Dashboard > JupyterHub).

How to use cURL?


cURL is a command line tool that you can use to transfer data via network protocols.

You can access cURL in several ways:

  • In WEkEO JupyterHub: open a terminal, by default cURL is installed on the base environment

  • In a local terminal: chances are you have cURL installed by default. If not, you can follow this link to install it

⚠️ Commands have only been tested in Linux.

Step by step Guide


Step 1. Authentication

All calls to the HDA API require an access token, which can be obtained through a call to GET /gettoken. You'll need to provide your credentials, where <credentials> is the base64-encoded version of the string <username>:<password>.

For instance, if john is your username and 123123 your password, credentials would be am9objoxMjMxMjM= (you can use the online tool base64encode to perform this conversion):

$ curl -X GET --header 'Authorization: Basic <credentials>' 'https://wekeo-broker.prod.wekeo2.eu/databroker/gettoken'

The HDA API will respond with a token, valid for 1 hour:

{ "access_token": "xxxxxxxx-yyyy-zzzz-xxxx-yyyyyyyyyyyy" }

⚠️ All other calls to the HDA API must include the access token. Whenever you see <access_token> in a command, replace it with your token.

Step 2. Accepting the terms and conditions

Before data can be accessed, the Copernicus Terms and Conditions must be accepted. This needs to be done only once:

$ curl --request PUT --header 'accept: application/json' --header 'authorization: <access_token>' --data 'accepted=true' https://wekeo-broker.prod.wekeo2.eu/databroker/termsaccepted/Copernicus_General_License

Step 3. Subsetting datasets

Every dataset in the WEkEO catalogue can be subsetted through multiple attributes. You can obtain the full list of subsetting attributes for a particular dataset through a call to GET /querymetadata/{datasetId}.

For instance, to get the attributes for EEA's HPPV dataset on Vegetation Phenology and Productivity:

$ curl -X 'GET'  'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/queryable/EO%3AEEA%3ADAT%3ACLMS_HRVPP_VPP'  -H 'accept: application/json'  -H 'Authorization: Bearer <access_token>'

Old cURL command (click me)

$ curl -X GET --header 'Authorization: <access_token>' 'https://wekeo-broker.prod.wekeo2.eu/databroker/querymetadata/EO:EEA:DAT:CLMS_HRVPP_VPP'

πŸ’‘WEkEO Pro Tip: if your datasetId contains ":" characters, replace them with the following characters: "%3A". In our case, the datasetId EO:EEA:DAT:CLMS_HRVPP_VPP becomes EO%3AEEA%3ADAT%3ACLMS_HRVPP_VPP.

Result of the command (click me)

{
"type": "object",
"title": "Querable",
"properties": {
"dataset_id": {
"title": "dataset_id",
"type": "string",
"oneOf": [
{
"const": "EO:EEA:DAT:CLMS_HRVPP_VPP",
"title": "EO:EEA:DAT:CLMS_HRVPP_VPP"
}
]
},
"itemsPerPage": {
"title": "Items PerPage",
"type": "string",
"pattern": "^[0-9]*$"
},
"startIndex": {
"title": "Start Index",
"type": "string",
"pattern": "^[0-9]*$"
},
"uid": {
"title": "Uid",
"type": "string",
"pattern": "[\\w-]+"
},
"productType": {
"title": "Product Type",
"type": "array",
"uniqueItems": true,
"items": {
"oneOf": [
{
"const": "MINV",
"title": "MINV"
},
{
"const": "MAXD",
"title": "MAXD"
},
{
"const": "LENGTH",
"title": "LENGTH"
},
{
"const": "SOSD",
"title": "SOSD"
},
{
"const": "QFLAG",
"title": "QFLAG"
},
{
"const": "EOSV",
"title": "EOSV"
},
{
"const": "TPROD",
"title": "TPROD"
},
{
"const": "MAXV",
"title": "MAXV"
},
{
"const": "AMPL",
"title": "AMPL"
},
{
"const": "SOSV",
"title": "SOSV"
},
{
"const": "LSLOPE",
"title": "LSLOPE"
},
{
"const": "EOSD",
"title": "EOSD"
},
{
"const": "RSLOPE",
"title": "RSLOPE"
},
{
"const": "SPROD",
"title": "SPROD"
}
]
}
},
"platformSerialIdentifier": {
"title": "Platform SerialIdentifier",
"type": "string",
"oneOf": [
{
"const": "S2A, S2B",
"title": "S2A, S2B"
}
]
},
"tileId": {
"title": "Tile Id",
"type": "string",
"pattern": "[\\w-]+"
},
"productVersion": {
"title": "Product Version",
"type": "string",
"pattern": "[\\w-]+"
},
"productGroupId": {
"title": "Product GroupId",
"type": "string",
"oneOf": [
{
"const": "S2A, S2B",
"title": "S2A, S2B"
}
]
},
"processingDate": {
"title": "Processing Date",
"type": "string",
"format": "date-time",
"default": ""
},
"start": {
"title": "Start Date",
"type": "string",
"format": "date-time",
"default": ""
},
"end": {
"title": "End Date",
"type": "string",
"format": "date-time",
"default": ""
},
"bbox": {
"title": "Bbox",
"type": "array",
"minItems": 4,
"maxItems": 4,
"items": [
{
"type": "number",
"maximum": -180,
"minimum": 180
},
{
"type": "number",
"maximum": -90,
"minimum": 90
},
{
"type": "number",
"maximum": -180,
"minimum": 180
},
{
"type": "number",
"maximum": -90,
"minimum": 90
}
]
}
},
"required": [
"dataset_id"
]
}

The HDA API responds with definitions for multiple variables you can use to subset (e.g. productType).

Here's a query based on these variables, which we'll use next:

{
"dataset_id": "EO:EEA:DAT:CLMS_HRVPP_VPP",
"productType": [
"TPROD"
],
"start": "2018-01-01T00:00:00.000Z",
"end": "2019-01-01T00:00:00.000Z",
"bbox": [
-0.58712594,
45.91075967748305,
3.25581995,
48.65066374786067
],
"itemsPerPage": 1,
"startIndex": 0
}

πŸ“ŒNote: an other way to create an API request is via the WEkEO Data Viewer. You can follow the article How to download WEkEO data? to learn how.

Step 4. Requesting data

Now let's create a command to retrieve the results linked to the query created above.

To do this we use POST /dataaccess/search:

$ curl -X 'POST' \
'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/search' \
-H 'accept: application/json' \
-H 'Authorization: Bearer <access_token>' \
-H 'Content-Type: application/json' \
-d '{
"dataset_id": "EO:EEA:DAT:CLMS_HRVPP_VPP",
"productType": [
"TPROD"
],
"start": "2018-01-01T00:00:00.000Z",
"end": "2019-01-01T00:00:00.000Z",
"bbox": [
-0.58712594,
45.91075967748305,
3.25581995,
48.65066374786067
],
"itemsPerPage": 1,
"startIndex": 0
}'

Old cURL command (click me)

$ curl -X POST \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: <access_token>' \
-d '{
"datasetId": "EO:EEA:DAT:CLMS_HRVPP_VPP",
"boundingBoxValues": [
{ "name": "bbox",
"bbox": [ -0.5871259395913329, 45.91075967748305, 3.255819949392474, 48.65066374786067 ] }
],
"dateRangeSelectValues": [
{
"name": "temporal_interval",
"start": "2018-01-01T00:00:00.000Z",
"end": "2019-01-01T00:00:00.000Z"
}
],
"stringChoiceValues": [
{ "name": "productType", "value": "TPROD"},
{"name": "productGroupId", "value": "s1"}
]
}' 'https://wekeo-broker.prod.wekeo2.eu/databroker/datarequest'

Use the jobId to poll the GET /datarequest/status/{jobId} endpoint:

$ curl -X GET --header 'Authorization: <access_token>' 'https://wekeo-broker.prod.wekeo2.eu/databroker/datarequest/status/FQ6_njWjE9RD7GCeYXyy6ejJ-Kc'

At this point you can retrieve the list of results calling the GET /datarequest/jobs/{jobId}/result endpoint:

$ curl -X GET --header 'Authorization: <access_token>' 'https://wekeo-broker.prod.wekeo2.eu/databroker/datarequest/jobs/FQ6_njWjE9RD7GCeYXyy6ejJ-Kc/result'

This returns an initial result with numerous information about the downloadable item:

{
"properties":{"itemsPerPage":1,"startIndex":0,"totalResults":92},
"features":[{"type":"Feature",
"geometry":{"type":"Polygon",
"coordinates":[[[-1.707468,46.0462655],[-1.7299243,45.0581973],[-0.3364958,45.0341901],[-0.2894559,46.0214217],[-1.707468,46.0462655]]]},
"properties":{
"startdate":"2018-01-01T00:00:00Z",
"enddate":"2018-12-31T23:59:59Z",
"location":"s3://hr-vpp-products-vpp-v01-2018/CLMS/Pan-European/Biophysical/VPP/v01/2018/s1/VPP_2018_S2_T30TXR-010m_V101_s1_TPROD.tif",
"size":119048027,
"dataprovider":"EEA"},
"id":"VPP_2018_S2_T30TXR-010m_V101_s1_TPROD",
"bbox":[-1.7299243,45.0341901,-0.2894559,46.0462655]}]
}

You'll notice that only one of 92 items is displayed, as shown in the first line. To display more, or display a different one, you can modify the last items in the query using the itemsPerPage and startIndex parameters.

πŸ” For example, if you want to display 5 items per page and you want to display the 2nd page, then you can set: "itemsPerPage": 5 and "startIndex": 1.

The parameters required for downloading are as follows:

  • dataset_id

  • location

  • id

Step 5. Get the download ID

To retrieve the download ID for the item returned above, we create a request POST dataaccess/download in which we fill in the fields with the previous mandatory parameters:

$ curl -X 'POST' \
'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download' \
-H 'accept: application/json' \
-H 'Authorization: Bearer <access_token>' \
-H 'Content-Type: application/json' \
-d '{ "cacheable": true, "searchMetadata": "string", "dataset_id": "EO:EEA:DAT:CLMS_HRVPP_VPP", "product_id": "VPP_2018_S2_T30TXR-010m_V101_s1_TPROD", "location": "s3://hr-vpp-products-vpp-v01-2018/CLMS/Pan-European/Biophysical/VPP/v01/2018/s1/VPP_2018_S2_T30TXR-010m_V101_s1_TPROD.tif" }'

Old cURL command (click me)

$ curl -X POST \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: <access_token>' \
-d '{
"jobId": "FQ6_njWjE9RD7GCeYXyy6ejJ-Kc",
"uri": "hr-vpp-products-vpp-v01-2018/CLMS/Pan-European/Biophysical/VPP/v01/2018/s1/VPP_2018_S2_T30TXR-010m_V101_s1_TPROD.tif"
}' 'https://wekeo-broker.prod.wekeo2.eu/databroker/dataorder'

You get the following download ID as the download_id parameter:

{"Location":"http://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download/6647672a2a44a9b2c0e4e5be",
"download_id":"6647672a2a44a9b2c0e4e5be"}

Step 6. Download data

Finally, we make the download request using the download_id in the request GET /dataacces/download/{download_id}:

$ curl -X 'GET' \
'http://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download/6647672a2a44a9b2c0e4e5be' \
-H 'accept: application/json' \
-H 'Authorization: Bearer <access_token>' \
--output "VPP_2018_S2_T30TXR-010m_V101_s1_TPROD.tif"

Old cURL command (click me)

$ curl -X GET --header 'Authorization: <access_token>' 'https://wekeo-broker.prod.wekeo2.eu/databroker/dataorder/status/0fW0ZecwsR-kKxRd13vsnjJpXOQ'

The download starts, and the downloaded files appear in your working directory:

  % Total  % Received % Xferd  Average Speed  Time   Time   Time  Current
Dload Upload Total Spent Left Speed
100 113M 0 113M 0 0 15.0M 0 --:--:-- 0:00:07 --:--:-- 20.0M

πŸ’‘WEkEO Pro Tip: by default, the file name is the download_id, but you can use --output as we did in the command above to specify another name. For information, the original file name can be found at the end of the location parameter.

After a short while, your files should have been downloaded to your working directory! πŸ˜ƒ

⚠️ There is a limitation of Request and Orders. More details in this article.

Swagger UI


The WEkEO HDA Broker UI is an interface explaining in details all the different routes you can call in the HDA API:

By browsing the categories, you'll see the different available calls, including the commands we made in this article. By clicking on one of them, you'll see all the results you can get.

Currently, to authenticate on this broker, you can go through the old Swagger to retrieve your token and then enter it in the following field under the new broker:

πŸ’‘WEkEO Pro Tip: you can run a command from the WEkEO HDA Broker UI by clicking the Try it out button.

What's next?


These articles might be of interest for you:

We are user-driven and we implement users' suggestions, so feel free to contact us:

  • through a chat session available in the bottom right corner of the page

  • via e-mail to our support team (supportATwekeo.eu)

Did this answer your question?