Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Save_asset_per_band: assets in different s3 folders + band-specific filename #877

Open
JorisCod opened this issue Sep 23, 2024 · 2 comments
Assignees

Comments

@JorisCod
Copy link

JorisCod commented Sep 23, 2024

The goal of this issue is to save assets to different folders, each with a specific name according to the band name.

Specific use case in mind: saving of monthly composites in different s3 folders (one folder/month)

Prefix example:
vito/products/2020/'<MONTH'>/

Filename example:
LCFM_LSF-MONTHLY_<DATE_START>_<DATE_END>29TNE_V001-SATIO_B<BAND_NAME>M.tif

So what's parametric is month, band_name, resolution, date_start and date_end. Resolution can be considered fixed (*). Month and band_name can be derived as follows:

band_name = openeo_band_name.split('')[-1]
month = openeo_band_name.split('
')[0]

(alternatively with fixed indices).

date_start and date_end are harder as they are to be derived from month:

import calendar
from datetime import datetime

def get_start_and_end_dates(year: int, month: int):
    # Get the first day of the month
    start_date = datetime(year, month, 1)

    # Get the last day of the month
    last_day = calendar.monthrange(year, month)[1]
    end_date = datetime(year, month, last_day)

    return start_date, end_date

# Example usage
year = 2020
month = 2
start_date, end_date = get_start_and_end_dates(year, month)

print("Start date:", start_date.strftime('%Y%m%d'))
print("End date:", end_date.strftime('%Y%m%d'))

The solution might be to provide a list of prefixes and filenames to use, both lists having the same length as the number of bands. This is quite general as well as flexible.

'* An ideal workflow is to have 1 job for all resolutions. This is actually the goal of main workflow of LCFM, to have a workflow for both 10m and 20m bands.

@JorisCod JorisCod changed the title Save_asset_per_band: save assets in different s3 folders Save_asset_per_band: assets in different s3 folders + band-specific filename Sep 23, 2024
@jdries
Copy link
Contributor

jdries commented Sep 24, 2024

@JeroenVerstraelen to plan in next sprint
a list or dictionary mapping band names to preferred asset name is indeed a way forward.
Allowing subdirectory in prefix is (hopefully) also possible.

@bossie
Copy link
Collaborator

bossie commented Sep 30, 2024

Related to Open-EO/openeo-geotrellis-extensions#317 in that this also accepts a dictionary mapping band names to scale, offset etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants