Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests for the test suite #468

Open
wants to merge 39 commits into
base: draft
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
a3710ad
Clarify no-data value / null for most processes #480
m-mohr Jan 3, 2024
d19039a
Clarify NaN in comparisons, clarify character encoding issues
m-mohr Jan 3, 2024
281584a
Updates according to PR review
m-mohr Jan 5, 2024
35891d9
Add first tests from process examples
m-mohr Oct 12, 2023
1beb1df
Add tests and docs
m-mohr Oct 13, 2023
7946206
Add more tests
m-mohr Oct 24, 2023
5855071
More tests, clean-up
m-mohr Oct 25, 2023
c56913b
Tests for arrays
m-mohr Oct 26, 2023
0051c8b
Remove optional tests in favor of returns or throws
m-mohr Oct 30, 2023
96512d7
EditorConfig + fix invalid files
m-mohr Oct 30, 2023
d08f5e5
Add id and experimental flag to test files
m-mohr Oct 30, 2023
5553bf0
Add tests for comparison processes
m-mohr Oct 30, 2023
d0bed66
Add tests for reducers etc.
m-mohr Oct 31, 2023
25a0ed1
Add further tests
m-mohr Nov 5, 2023
9f8e8b5
title converted to comments, fix parameter name of exp in tests
m-mohr Dec 7, 2023
c0da9ab
Fix test for tan
m-mohr Dec 7, 2023
bd7e06e
Add tests for array_apply, more NaN tests for comparisons, add requir…
m-mohr Dec 11, 2023
2c448d4
add array_apply and array_filter tests, fix several math tests
m-mohr Dec 11, 2023
7bddb43
update readme
m-mohr Dec 11, 2023
fc07d68
Add first datacube test
m-mohr Dec 11, 2023
c1927e3
Fix apply test
m-mohr Dec 12, 2023
6514eb2
Add profile levels
m-mohr Dec 12, 2023
e26b038
Fix test in apply
m-mohr Dec 12, 2023
4787fb7
Add tests for reduce_dimension and apply_dimension
m-mohr Dec 12, 2023
622dc65
Add more tests
m-mohr Dec 13, 2023
de71911
Fix various test cases
m-mohr Dec 14, 2023
2bc57e9
Special no-data handling, fix tests
m-mohr Dec 14, 2023
5ab0d8b
Add filter_bands and filter_temporal
m-mohr Dec 14, 2023
932de0f
Add mask, merge_cubes, refactor datacube object
m-mohr Dec 15, 2023
b13cba5
Mark experimental processes
m-mohr Dec 15, 2023
1466504
Add additional test for apply_dimension
m-mohr Dec 19, 2023
1046067
Add tests for apply_kernel
m-mohr Dec 22, 2023
54f8fdc
Add tests for aggregate_temporal
m-mohr Dec 22, 2023
16440e2
Add tests for aggregate_temporal_period, fix aggregate_temporal
m-mohr Dec 22, 2023
768b209
Add tests for filter_bbox, filter_spatial and mask_polygon
m-mohr Jan 2, 2024
86b5e07
Use nodata type in tests properly
m-mohr Jan 3, 2024
91ad78a
Add changelog entry
m-mohr Jan 3, 2024
d364059
Fix invalid datetimes
m-mohr Jan 17, 2024
664b89d
Apply suggestions from code review
m-mohr Jan 25, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,11 @@ indent_style = spaces
indent_size = 4
insert_final_newline = true
trim_trailing_whitespace = true

[*.json5]
charset = utf-8
end_of_line = crlf
indent_style = spaces
indent_size = 2
insert_final_newline = true
trim_trailing_whitespace = true
22 changes: 16 additions & 6 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,23 +6,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## Unreleased / Draft

### Added

- Unit Tests (see folder `tests`, moved specification tests and CI tools to `dev`)

### Changed

- `clip`: Throw an exception if min > max [#472](https://github.com/Open-EO/openeo-processes/issues/472)
- Clarified for various mathematical functions the defined input and output ranges.
Mention that `NaN` is returned outside of the defined input range where possible.
- Clarified for several comparison processes how `NaN` values have to be handled.
- Clarified for various processes the handling of no-data values and `null`, see also the [implementation guide](meta/implementation.md#no-data-value). [#480](https://github.com/Open-EO/openeo-processes/issues/480)
- Added a [section about character encodings to the implementation guide](meta/implementation.md#character-encoding).
Removed any character encoding related wording from the process specifications itself.
- Added a uniqueness contraint to various array-typed parameters (e.g. lists of dimension names or labels)
- `array_interpolate_linear`: Apply interpolation to NaN and no-data values.
- `clip`: Throw an exception if min > max. [#472](https://github.com/Open-EO/openeo-processes/issues/472)

### Fixed

- Clarified for various mathematical functions the defined input and output ranges. Mention that `NaN` is returned outside of the defined input range where possible.
- `aggregate_temporal` and `aggregate_temporal_period`: Clarified that the process throws a `DimensionNotAvailable` exception when no temporal dimension exists.
- `aggregate_temporal_period`: Removed unused exception `DistinctDimensionLabelsRequired`
- `aggregate_temporal_period`: Clarified that the definition of weeks follows ISO 8601
- `divide`: Clarified behavior for division by 0
- `aggregate_temporal_period`: Removed unused exception `DistinctDimensionLabelsRequired`.
- `aggregate_temporal_period`: Clarified that the definition of weeks follows ISO 8601.
- `divide`: Clarified behavior for division by 0.
- `between`: Clarify that `null` is passed through.
- `eq` and `neq`: Explicitly set the minimum value for the `delta` parameter.
- `filter_bbox`, `load_collection`, `load_stac`: Clarified that the bounding box is reprojected to the CRS of the spatial data cube dimensions if required.
- `filter_spatial`: Clarified that masking is applied using the given geometries. [#469](https://github.com/Open-EO/openeo-processes/issues/469)
- `mod`: Clarified behavior for y = 0
- `mod`: Clarified behavior for y = 0.
- `sqrt`: Clarified that NaN is returned for negative numbers.

## [2.0.0-rc.1] - 2023-05-25
Expand Down
14 changes: 1 addition & 13 deletions absolute.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"id": "absolute",
"summary": "Absolute value",
"description": "Computes the absolute value of a real number `x`, which is the \"unsigned\" portion of `x` and often denoted as *|x|*.\n\nThe no-data value `null` is passed through and therefore gets propagated.",
"description": "Computes the absolute value of a real number `x`, which is the \"unsigned\" portion of `x` and often denoted as *|x|*.\n\nNo-data values are passed through.",
"categories": [
"math"
],
Expand All @@ -28,12 +28,6 @@
}
},
"examples": [
{
"arguments": {
"x": 0
},
"returns": 0
},
{
"arguments": {
"x": 3.5
Expand All @@ -45,12 +39,6 @@
"x": -0.4
},
"returns": 0.4
},
{
"arguments": {
"x": -3.5
},
"returns": 3.5
}
],
"links": [
Expand Down
4 changes: 2 additions & 2 deletions add.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"id": "add",
"summary": "Addition of two numbers",
"description": "Sums up the two numbers `x` and `y` (*`x + y`*) and returns the computed sum.\n\nNo-data values are taken into account so that `null` is returned if any element is such a value.\n\nThe computations follow [IEEE Standard 754](https://ieeexplore.ieee.org/document/8766229) whenever the processing environment supports it.",
"description": "Sums up the two numbers `x` and `y` (*`x + y`*) and returns the computed sum.\n\nNo-data values are taken into account so that the no-data value is returned if any element is such a value.\n\nThe computations follow [IEEE Standard 754](https://ieeexplore.ieee.org/document/8766229) whenever the processing environment supports it.",
"categories": [
"math"
],
Expand Down Expand Up @@ -88,4 +88,4 @@
"result": true
}
}
}
}
2 changes: 1 addition & 1 deletion aggregate_temporal.json
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@
},
{
"name": "reducer",
"description": "A reducer to be applied for the values contained in each interval. A reducer is a single process such as ``mean()`` or a set of processes, which computes a single value for a list of values, see the category 'reducer' for such processes. Intervals may not contain any values, which for most reducers leads to no-data (`null`) values by default.",
"description": "A reducer to be applied for the values contained in each interval. A reducer is a single process such as ``mean()`` or a set of processes, which computes a single value for a list of values, see the category 'reducer' for such processes. Intervals may not contain any values, which leads to a no-data value for most reducers by default.",
"schema": {
"type": "object",
"subtype": "process-graph",
Expand Down
2 changes: 1 addition & 1 deletion aggregate_temporal_period.json
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@
},
{
"name": "reducer",
"description": "A reducer to be applied for the values contained in each period. A reducer is a single process such as ``mean()`` or a set of processes, which computes a single value for a list of values, see the category 'reducer' for such processes. Periods may not contain any values, which for most reducers leads to no-data (`null`) values by default.",
"description": "A reducer to be applied for the values contained in each period. A reducer is a single process such as ``mean()`` or a set of processes, which computes a single value for a list of values, see the category 'reducer' for such processes. Periods may not contain any values, which leads to a no-data value for most reducers by default.",
"schema": {
"type": "object",
"subtype": "process-graph",
Expand Down
6 changes: 3 additions & 3 deletions all.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"id": "all",
"summary": "Are all of the values true?",
"description": "Checks if **all** of the values in `data` are true. If no value is given (i.e. the array is empty) the process returns `null`.\n\nBy default all no-data values are ignored so that the process returns `null` if all values are no-data, `true` if all values are true and `false` otherwise. Setting the `ignore_nodata` flag to `false` takes no-data values into account and the array values are reduced pairwise according to the following truth table:\n\n```\n || null | false | true\n----- || ----- | ----- | -----\nnull || null | false | null\nfalse || false | false | false\ntrue || null | false | true\n```\n\n**Remark:** The process evaluates all values from the first to the last element and stops once the outcome is unambiguous. A result is ambiguous unless a value is `false` or all values have been taken into account.",
"description": "Checks if **all** of the values in `data` are true. If no value is given (i.e. the array is empty) the process returns the no-data value (or `null`).\n\nBy default all no-data values are ignored so that the process returns the no-data value (or `null`) if all values are no-data, `true` if all values are true and `false` otherwise. Setting the `ignore_nodata` flag to `false` takes no-data values into account and the array values are reduced pairwise according to the following truth table:\n\n```\n || no-data | false | true\n------- || ------- | ----- | -------\nno-data || no-data | false | no-data\nfalse || false | false | false\ntrue || no-data | false | true\n```\n\n**Remark:** The process evaluates all values from the first to the last element and stops once the outcome is unambiguous. A result is ambiguous unless a value is `false` or all values have been taken into account.",
"categories": [
"logic",
"reducer"
Expand All @@ -22,7 +22,7 @@
},
{
"name": "ignore_nodata",
"description": "Indicates whether no-data values are ignored or not and ignores them by default.",
"description": "Indicates whether no-data values are ignored or not. Ignores them by default.",
"schema": {
"type": "boolean"
},
Expand Down Expand Up @@ -131,4 +131,4 @@
"returns": null
}
]
}
}
2 changes: 1 addition & 1 deletion and.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"id": "and",
"summary": "Logical AND",
"description": "Checks if **both** values are true.\n\nEvaluates parameter `x` before `y` and stops once the outcome is unambiguous. If any argument is `null`, the result will be `null` if the outcome is ambiguous.\n\n**Truth table:**\n\n```\nx \\ y || null | false | true\n----- || ----- | ----- | -----\nnull || null | false | null\nfalse || false | false | false\ntrue || null | false | true\n```",
"description": "Checks if **both** values are true.\n\nEvaluates parameter `x` before `y` and stops once the outcome is unambiguous. If any argument is a no-data value, the result will be the no-data value whenever the outcome is ambiguous.\n\n**Truth table:**\n\n```\nx \\ y || no-data | false | true\n------- || ------- | ----- | -------\nno-data || no-data | false | no-data\nfalse || false | false | false\ntrue || no-data | false | true\n```",
"categories": [
"logic"
],
Expand Down
6 changes: 3 additions & 3 deletions any.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"id": "any",
"summary": "Is at least one value true?",
"description": "Checks if **any** (i.e. at least one) value in `data` is `true`. If no value is given (i.e. the array is empty) the process returns `null`.\n\nBy default all no-data values are ignored so that the process returns `null` if all values are no-data, `true` if at least one value is true and `false` otherwise. Setting the `ignore_nodata` flag to `false` takes no-data values into account and the array values are reduced pairwise according to the following truth table:\n\n```\n || null | false | true\n----- || ---- | ----- | ----\nnull || null | null | true\nfalse || null | false | true\ntrue || true | true | true\n```\n\n**Remark:** The process evaluates all values from the first to the last element and stops once the outcome is unambiguous. A result is ambiguous unless a value is `true`.",
"description": "Checks if **any** (i.e. at least one) value in `data` is `true`. If no value is given (i.e. the array is empty) the process returns the no-data value (or `null`).\n\nBy default all no-data values are ignored so that the process returns the no-data value (or `null`) if all values are no-data, `true` if at least one value is true and `false` otherwise. Setting the `ignore_nodata` flag to `false` takes no-data values into account and the array values are reduced pairwise according to the following truth table:\n\n```\n || no-data | false | true\n------- || ------- | ------- | ----\nno-data || no-data | no-data | true\nfalse || no-data | false | true\ntrue || true | true | true\n```\n\n**Remark:** The process evaluates all values from the first to the last element and stops once the outcome is unambiguous. A result is ambiguous unless a value is `true`.",
"categories": [
"logic",
"reducer"
Expand All @@ -22,7 +22,7 @@
},
{
"name": "ignore_nodata",
"description": "Indicates whether no-data values are ignored or not and ignores them by default.",
"description": "Indicates whether no-data values are ignored or not. Ignores them by default.",
"schema": {
"type": "boolean"
},
Expand Down Expand Up @@ -131,4 +131,4 @@
"returns": null
}
]
}
}
4 changes: 2 additions & 2 deletions apply_kernel.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"id": "apply_kernel",
"summary": "Apply a spatial convolution with a kernel",
"description": "Applies a 2D convolution (i.e. a focal operation with a weighted kernel) on the horizontal spatial dimensions (axes `x` and `y`) of a raster data cube.\n\nEach value in the kernel is multiplied with the corresponding pixel value and all products are summed up afterwards. The sum is then multiplied with the factor.\n\nThe process can't handle non-numerical or infinite numerical values in the data cube. Boolean values are converted to integers (`false` = 0, `true` = 1), but all other non-numerical or infinite values are replaced with zeroes by default (see parameter `replace_invalid`).\n\nFor cases requiring more generic focal operations or non-numerical values, see ``apply_neighborhood()``.",
"description": "Applies a 2D convolution (i.e. a focal operation with a weighted kernel) on the horizontal spatial dimensions (axes `x` and `y`) of a raster data cube.\n\nEach value in the kernel is multiplied with the corresponding pixel value and all products are summed up afterwards. The sum is then multiplied with the factor.\n\nThe process can't handle non-numerical or infinite numerical values in the data cube. Boolean values are converted to integers (`false` = 0, `true` = 1), but all other non-numerical, NaN, no-data, or infinite values are replaced with zeroes by default (see parameter `replace_invalid`).\n\nFor cases requiring more generic focal operations or non-numerical values, see ``apply_neighborhood()``.",
"categories": [
"cubes",
"math > image filter"
Expand Down Expand Up @@ -70,7 +70,7 @@
},
{
"name": "replace_invalid",
"description": "This parameter specifies the value to replace non-numerical or infinite numerical values with. By default, those values are replaced with zeroes.",
"description": "This parameter specifies the value to replace non-numerical, NaN, no-data, or infinite numerical values with. By default, those values are replaced with zeroes.",
"schema": {
"type": "number"
},
Expand Down
2 changes: 1 addition & 1 deletion apply_neighborhood.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"id": "apply_neighborhood",
"summary": "Apply a process to pixels in a n-dimensional neighborhood",
"description": "Applies a focal process to a data cube.\n\nA focal process is a process that works on a 'neighborhood' of pixels. The neighborhood can extend into multiple dimensions, this extent is specified by the `size` argument. It is not only (part of) the size of the input window, but also the size of the output for a given position of the sliding window. The sliding window moves with multiples of `size`.\n\nAn overlap can be specified so that neighborhoods can have overlapping boundaries. This allows for continuity of the output. The overlap region must be included in the data cube or array returned by `process`, but any changed values will be ignored. The missing overlap at the borders of the original data cube is made available as no-data (`null`) in the sub-data cubes.\n\nThe neighborhood size should be kept small enough, to avoid running beyond computational resources, but a too-small size will result in a larger number of process invocations, which may slow down processing. Window sizes for spatial dimensions typically range from 64 to 512 pixels, while overlaps of 8 to 32 pixels are common.\n\nFor the special case of 2D convolution, it is recommended to use ``apply_kernel()``.",
"description": "Applies a focal process to a data cube.\n\nA focal process is a process that works on a 'neighborhood' of pixels. The neighborhood can extend into multiple dimensions, this extent is specified by the `size` argument. It is not only (part of) the size of the input window, but also the size of the output for a given position of the sliding window. The sliding window moves with multiples of `size`.\n\nAn overlap can be specified so that neighborhoods can have overlapping boundaries. This allows for continuity of the output. The overlap region must be included in the data cube or array returned by `process`, but any changed values will be ignored. The missing overlap at the borders of the original data cube is made available as no-data values in the sub-data cubes.\n\nThe neighborhood size should be kept small enough, to avoid running beyond computational resources, but a too-small size will result in a larger number of process invocations, which may slow down processing. Window sizes for spatial dimensions typically range from 64 to 512 pixels, while overlaps of 8 to 32 pixels are common.\n\nFor the special case of 2D convolution, it is recommended to use ``apply_kernel()``.",
"categories": [
"cubes"
],
Expand Down
2 changes: 1 addition & 1 deletion arccos.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"id": "arccos",
"summary": "Inverse cosine",
"description": "Computes the arc cosine of `x`. The arc cosine is the inverse function of the cosine so that *`arccos(cos(x)) = x`*.\n\nWorks on radians only.\nThe no-data value `null` is passed through and therefore gets propagated. `NaN` is returned for values outside of the allowed range.",
"description": "Computes the arc cosine of `x`. The arc cosine is the inverse function of the cosine so that *`arccos(cos(x)) = x`*.\n\nWorks on radians only.\nNo-data values are passed through. `NaN` is returned for values < -1 and > 1.",
"categories": [
"math > trigonometric"
],
Expand Down
2 changes: 1 addition & 1 deletion arcosh.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"id": "arcosh",
"summary": "Inverse hyperbolic cosine",
"description": "Computes the inverse hyperbolic cosine of `x`. It is the inverse function of the hyperbolic cosine so that *`arcosh(cosh(x)) = x`*.\n\nThe no-data value `null` is passed through and therefore gets propagated. `NaN` is returned for values outside of the allowed range.",
"description": "Computes the inverse hyperbolic cosine of `x`. It is the inverse function of the hyperbolic cosine so that *`arcosh(cosh(x)) = x`*.\n\nNo-data values are passed through. `NaN` is returned for values outside of the allowed range.",
"categories": [
"math > trigonometric"
],
Expand Down
2 changes: 1 addition & 1 deletion arcsin.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"id": "arcsin",
"summary": "Inverse sine",
"description": "Computes the arc sine of `x`. The arc sine is the inverse function of the sine so that *`arcsin(sin(x)) = x`*.\n\nWorks on radians only.\nThe no-data value `null` is passed through and therefore gets propagated. `NaN` is returned for values < -1 and > 1.",
"description": "Computes the arc sine of `x`. The arc sine is the inverse function of the sine so that *`arcsin(sin(x)) = x`*.\n\nWorks on radians only.\nNo-data values are passed through. `NaN` is returned for values < -1 and > 1.",
"categories": [
"math > trigonometric"
],
Expand Down
2 changes: 1 addition & 1 deletion arctan.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"id": "arctan",
"summary": "Inverse tangent",
"description": "Computes the arc tangent of `x`. The arc tangent is the inverse function of the tangent so that *`arctan(tan(x)) = x`*.\n\nWorks on radians only.\nThe no-data value `null` is passed through and therefore gets propagated.",
"description": "Computes the arc tangent of `x`. The arc tangent is the inverse function of the tangent so that *`arctan(tan(x)) = x`*.\n\nWorks on radians only.\nNo-data values are passed through.",
"categories": [
"math > trigonometric"
],
Expand Down
Loading
Loading