Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test local_batch_job.py with a simple process graph #839

Open
JeroenVerstraelen opened this issue Aug 20, 2024 · 3 comments
Open

test local_batch_job.py with a simple process graph #839

JeroenVerstraelen opened this issue Aug 20, 2024 · 3 comments
Assignees

Comments

@JeroenVerstraelen
Copy link
Contributor

EPIC: #806

  • test local_batch_job.py with a simple 1+2 process graph, see if output is written correctly to local job directory.
    • add this as a test
    • This has to work inside the docker container for local deployments
@jdries
Copy link
Contributor

jdries commented Sep 20, 2024

configuration for credentials:
https://github.com/Open-EO/openeo-geotrellis-extensions/blob/f0f81fdb504293601b4d4939cc7f1d2c3eb6c1b4/openeo-geotrellis/src/main/scala/org/openeo/geotrellis/CustomizableHttpRangeReaderProvider.scala#L44
Java system property: "http.credentials.file"
File contents:

{
"services.terrascope.be": {
  "username": "username",
  "password": "plain_text_pw"
}
}

services.terrascope.be is the hostname that is used in asset urls for which basic authentication is required.

@EmileSonneveld
Copy link
Contributor

EmileSonneveld commented Sep 20, 2024

Update:

  • Running a docker command will run the process graph in the current directory and output the results in "./out"
    TODO: make input and output configurable. Maybe with a stub python client local 'connection' object? EDIT: Can pass path to process_graph.json now.

  • Running docker image required 'sudo'. TODO: possible as local user? Output files should be written with current user permission. Seems to be root at the moment. EDIT: no sudo needed anymore.

  • Logs are shown in console while running, and json structured logs are written to out/openeo.log

  • Implementation now exists out of Dockerfile, entrypoint.sh and test_run_graph.py. TODO: find a better place to store those files EDIT: Code now here
    openeo_docker_local.zip

@jdries
Copy link
Contributor

jdries commented Sep 20, 2024

Nice, this seems to almost work as intended.
One of the most important todo's for the deadline is documentation.
The basic page for eoepca is here:
https://eoepca.readthedocs.io/projects/processing/en/latest/design/processing-engine/openeo/
But it actually links to:
https://github.com/Open-EO/openeo-geopyspark-driver/blob/master/README.md

Can you update the markdown to explain the current steps for running batch job in docker? (Or put it in a separate one if too much.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants