Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

t.rast.seasons: new temporal module to get seasonally aggregated data #920

Closed
wants to merge 10 commits into from

Conversation

lucadelu
Copy link
Contributor

@lucadelu lucadelu commented Jun 5, 2023

This new addons run t.rast.aggregate.ds with a temporal dataset of season and it is able to get seasonal aggregate values

@veroandreo veroandreo changed the title added new temporal module to get seasonal aggregated data t.rast.seasons: new temporal module to get seasonally aggregated data Jun 5, 2023
@veroandreo veroandreo added the new addon PR contains a new addon or issue proposes one label Jun 5, 2023
@veroandreo
Copy link
Contributor

@lucadelu you are trying to merge into grass7 branch, should go into grass8 I believe

Copy link
Contributor

@veroandreo veroandreo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments and edits added here and there. My main concern is that the outputs are maps instead of a strds with astronomical season aggregation pattern.

src/temporal/t.rast.seasons/Makefile Outdated Show resolved Hide resolved
src/temporal/t.rast.seasons/t.rast.seasons.html Outdated Show resolved Hide resolved
src/temporal/t.rast.seasons/t.rast.seasons.html Outdated Show resolved Hide resolved
src/temporal/t.rast.seasons/t.rast.seasons.html Outdated Show resolved Hide resolved
src/temporal/t.rast.seasons/t.rast.seasons.py Outdated Show resolved Hide resolved
src/temporal/t.rast.seasons/t.rast.seasons.py Outdated Show resolved Hide resolved
src/temporal/t.rast.seasons/t.rast.seasons.py Outdated Show resolved Hide resolved
src/temporal/t.rast.seasons/t.rast.seasons.py Outdated Show resolved Hide resolved
@lucadelu
Copy link
Contributor Author

lucadelu commented Jun 6, 2023

@lucadelu you are trying to merge into grass7 branch, should go into grass8 I believe

As I said in other pull request on my production system I still have GRASS 7. :-(
I'll update for GRASS8 too

lucadelu and others added 2 commits June 6, 2023 08:07
@lucadelu
Copy link
Contributor Author

lucadelu commented Jun 6, 2023

Some comments and edits added here and there. My main concern is that the outputs are maps instead of a strds with astronomical season aggregation pattern.

t.rast.aggregate.ds already return a strds for each year, but it make sense to have one with all the maps together. Going to implement it



if __name__ == "__main__":
options, flags = gscript.parser()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While this is in the old examples, unless you need options and flags as globals (which is a bad practice anyway), there is no reason to have this here. It should be in main.


if __name__ == "__main__":
options, flags = gscript.parser()
sys.exit(main())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to above, unless there main is returning a return code, no reason to pass it to sys.exit. Just put main() here.

from datetime import datetime

import grass.temporal as tgis
import grass.script as gscript
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While less unique, in new code we mostly adopted import grass.script as gs. grass.script is used as lot, so it can be short. It is not a requirement, so if there would be potential for confusion in a particular file, it can gscript.

else:
out_strds = basename

seasons_name = ["spring", "summer", "autumn", "winter"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is list, so plural would make sense season_names or seasons. The loop variable seas can then be an actual word like season_name or season.

if strds.find("@") >= 0:
id_ = strds
else:
id_ = strds + "@" + gscript.gisenv()["MAPSET"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

f-string here.

process_queue = pymod.ParallelModuleQueue(int(nprocs))

# create t.rast.aggregate.ds module to be copied
mod = pymod.Module("t.rast.aggregate.ds")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documentation says:

Setting run_ to False is important, otherwise a parallel processing is not possible

Comment on lines 205 to 214
for year in years:
remod = pymod.Module("t.remove")
remod.inputs.inputs = f"sample_seasons_{year}@{mapset}"
remod.flags.r = True
remod.flags.f = True
remod.flags.quiet = True
if output_name:
listmaps = pymod.Module("t.rast.list")
listmaps.inputs.input = f"{out_strds_{year}}"
listmaps.inputs.columns = ["name"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be a cleanup function registered with atexit to avoid a prematurely terminated process leaving data behind.

out_maps = []
# remove space time vector datasets
for year in years:
remod = pymod.Module("t.remove")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or is that only when other parameters are supplied? (Then disregard my comment about parallel processing.)

@neteler
Copy link
Member

neteler commented Dec 22, 2023

As I said in other pull request on my production system I still have GRASS 7. :-(
I'll update for GRASS8 too

@lucadelu can we switch the base to grass8? Because G7 is effectively end of life.

@lucadelu
Copy link
Contributor Author

As I said in other pull request on my production system I still have GRASS 7. :-(
I'll update for GRASS8 too

@lucadelu can we switch the base to grass8? Because G7 is effectively end of life.

Ok, I updated the script to be work properly in GRASS8 not I'm going to rebase it

@lucadelu
Copy link
Contributor Author

@wenzeslaus I updated the code according to you suggestions, thanks a lot.

Running the script it works properly, but now I have a problem compiling the source code to run the tests

g.extension extension=t.rast.aggregate.seasons url=/home/lucadelu/github/grass-addons/src/temporal/t.rast.aggregate.seasons/
Compiling...
Exception ignored in atexit callback: <function cleanup at 0x7f42a0ef1e40>
Traceback (most recent call last):
  File "/tmp/grass8-lucadelu-59800/tmpnh6krtpq/t.rast.aggregate.seasons/scripts/t.rast.aggregate.seasons", line 95, in cleanup
    for year in remove_years:
                ^^^^^^^^^^^^
  File "/tmp/grass8-lucadelu-59800/tmpnh6krtpq/t.rast.aggregate.seasons/scripts/t.rast.aggregate.seasons", line 95, in cleanup
    for year in remove_years:
                ^^^^^^^^^^^^
  File "/usr/lib/python3.11/bdb.py", line 90, in trace_dispatch
    return self.dispatch_line(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/bdb.py", line 115, in dispatch_line
    if self.quitting: raise BdbQuit
                      ^^^^^^^^^^^^^
bdb.BdbQuit: 
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
OSError: [Errno 9] Bad file descriptor
Installing...
Updating extensions metadata file...
Updating extension modules metadata file...


It remains stuck at this point, do you have any suggestions?

@lucadelu
Copy link
Contributor Author

@wenzeslaus I updated the code according to you suggestions, thanks a lot.

Running the script it works properly, but now I have a problem compiling the source code to run the tests

g.extension extension=t.rast.aggregate.seasons url=/home/lucadelu/github/grass-addons/src/temporal/t.rast.aggregate.seasons/
Compiling...
Exception ignored in atexit callback: <function cleanup at 0x7f42a0ef1e40>
Traceback (most recent call last):
  File "/tmp/grass8-lucadelu-59800/tmpnh6krtpq/t.rast.aggregate.seasons/scripts/t.rast.aggregate.seasons", line 95, in cleanup
    for year in remove_years:
                ^^^^^^^^^^^^
  File "/tmp/grass8-lucadelu-59800/tmpnh6krtpq/t.rast.aggregate.seasons/scripts/t.rast.aggregate.seasons", line 95, in cleanup
    for year in remove_years:
                ^^^^^^^^^^^^
  File "/usr/lib/python3.11/bdb.py", line 90, in trace_dispatch
    return self.dispatch_line(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/bdb.py", line 115, in dispatch_line
    if self.quitting: raise BdbQuit
                      ^^^^^^^^^^^^^
bdb.BdbQuit: 
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
OSError: [Errno 9] Bad file descriptor
Installing...
Updating extensions metadata file...
Updating extension modules metadata file...

It remains stuck at this point, do you have any suggestions?

My fault, there was a pdb.set_trace() returning the error, however there is still something wrong during the creation of the new temporal dataset, I'm going to investigate

@neteler
Copy link
Member

neteler commented Dec 28, 2023

@lucadelu can we switch the base to grass8? Because G7 is effectively end of life.

Ok, I updated the script to be work properly in GRASS8 not I'm going to rebase it

Does this mean that you need it (only?) in GRASS7? And then someone to forward-port it to GRASS8 in a new PR? Just to understand.

@lucadelu
Copy link
Contributor Author

@lucadelu can we switch the base to grass8? Because G7 is effectively end of life.

Ok, I updated the script to be work properly in GRASS8 not I'm going to rebase it

Does this mean that you need it (only?) in GRASS7? And then someone to forward-port it to GRASS8 in a new PR? Just to understand.

I don't know where the "not" is coming from :-)

However I tried to rebase and I got some errors, so I'll close this pull request and I'll do new one based on GRASS 8 branch

@lucadelu
Copy link
Contributor Author

replaced by #1003

@lucadelu lucadelu closed this Dec 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
new addon PR contains a new addon or issue proposes one
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants