Setup ground truth geoparquet generation pipeline using RDM API

[kvantricht]

Create a function and an openEO job:

The function:

Creates a GeoParquet of points samples and matching metadata:

• Either from a local dataset of polygons/points (WorldCereal Harmonized from @cbutsko)
• Either from a spatial/temporal extent used to query the RDM

The function implements a sampling of points from polygons with a strategy defined by the R&D team (start with centroid). And returns a GeoParquet of points with minimal metadata (GT code, SampleID, ...)

OpenEO job:

Takes as input a GeoParquet of points to sample with additional metadata (GT, sampleID...)

From the collections of Sentinel1 and Sentinel2 patches, create a job that from input points samples:

• Sentinel 1 and Sentinel 2 data from the local stac collections
• METEO data from the pre-processed STAC collection available on CDSE bucket
• DEM data from global collection available on CDSE

Then performs processing such as monthly composites. Finally, concatenate GT and other metadata in a post-job action, returning a final GeoParquet with all the sample information alongside monthly-composited signal data and METEO/DEM.

[VincentVerelst]

Example of interaction with RDM API: https://github.com/WorldCereal/worldcereal-classification/blob/extra-demos/notebooks/utils.py#L384

[VincentVerelst]

Additional note: the GeoParquet of points with minimal metadata (GT code, SampleID, ...) should be uploaded to somewhere like Artifactory, so that it can be read with openeo.

[kvantricht]

Additional note: the GeoParquet of points with minimal metadata (GT code, SampleID, ...) should be uploaded to somewhere like Artifactory, so that it can be read with openeo.

based on what? Extractions that are finished?

[VincentVerelst]

Additional note: the GeoParquet of points with minimal metadata (GT code, SampleID, ...) should be uploaded to somewhere like Artifactory, so that it can be read with openeo.

based on what? Extractions that are finished?

Well I think there are two options:

• Create the point sample geoparquet file with STAC metadata. Then we can load_stac from disk
• Create the point sample geoparquet and directly load the Parquet file in. This is only possible with load_url, which only works over HTTP

[VincentVerelst]

After discussions with the openeo devs, we have decided to use the load_url process to load the reference data in openeo. For this two features need to be implemented on the VITO/CDSE backend:

• Support merge_cubes for vectorcubes Open-EO/openeo-geopyspark-driver#860
• Support spatiotemporal and property filtering in load_url Open-EO/openeo-geopyspark-driver#859

[GriffinBabe]

Also depends on

• Develop script to re-build STAC catalogue #151

I'm taking care of this this week

[HansVRP]

The decision has been made to develop this on the python client side.
It is very specific to world cereal, so it will end up in the world cereal repo.

Underlying technology to evaluate is duckdb/pandas.

requirement: location to store the geoparquet. (need to be accessible by terrascope back-end)
--> DIscuss with Peter/Jeroen

What defines success: Speed and efficiency