From eed04d6ddb670e102cbe7b032a5c6dc207c32379 Mon Sep 17 00:00:00 2001 From: tb06904 <141412860+tb06904@users.noreply.github.com> Date: Thu, 1 Aug 2024 16:38:54 +0000 Subject: [PATCH 1/5] document types handling --- .../query/gremlin/custom-features.md | 68 +++++++++++++++---- 1 file changed, 53 insertions(+), 15 deletions(-) diff --git a/docs/user-guide/query/gremlin/custom-features.md b/docs/user-guide/query/gremlin/custom-features.md index 861aa5c855..4572248afa 100644 --- a/docs/user-guide/query/gremlin/custom-features.md +++ b/docs/user-guide/query/gremlin/custom-features.md @@ -1,15 +1,27 @@ -# Custom Gremlin Features In Gaffer +# Custom Features in GafferPop The GafferPop implementation provides some extra features on top of the standard Tinkerpop framework that you can utilise in your queries. These are likely specific to how a Gaffer graph operates and may not be available in other graph technologies that support Gremlin queries. +## Adding Options to Queries + +In standard Gremlin syntax it is possible to add additional key value variables +into a query via a [`with()`](https://tinkerpop.apache.org/docs/current/reference/#with-step) +step. This feature is utilised to allow some custom properties to be passed in +for Gaffer specific options. + +!!! tip + Please see the [reference guide](../../../reference/gremlin-guide/gaffer-options.md) + for a full list of available options. + ## NamedOperations in Gremlin The [GafferPopNamedOperationService](https://gchq.github.io/Gaffer/uk/gov/gchq/gaffer/tinkerpop/service/GafferPopNamedOperationService.html) -allows for the running of Gaffer [Named Operations](../../../administration-guide/named-operations.md) using Tinkerpop. -Users can run Named Operations and add new Named Operations, deleting Named Operations is not currently possible with Tinkerpop. +allows for the running of Gaffer [Named Operations](../../../administration-guide/named-operations.md) +using Tinkerpop. Users can run Named Operations and add new Named Operations, +deleting Named Operations is not currently possible with Tinkerpop. !!! example "" Add a simple Named Operation that returns a count of all elements in your graph. @@ -66,27 +78,53 @@ Users can also run any existing or added Named Operations that are stored in the g.call("namedoperation", params).toList(); ``` -## Adding Options to Queries +## Custom Types and Functions -In standard Gremlin syntax it is possible to add additional key value variables -into a query via a [`with()` step](https://tinkerpop.apache.org/docs/current/reference/#with-step). -This feature is utilised to allow some custom properties to be passed in -for Gaffer specific options: +In Gaffer there are various [additional types](../../../reference/properties-guide/properties.md) +you may want to use to represent properties. These types may not be compatible +with the [GraphSON standard](https://tinkerpop.apache.org/docs/current/dev/io/#graphson-3d0) +used by Tinkerpop meaning they cannot be serialised or parsed by a Gremlin +client. To resolve this, measures have been added to GafferPop so that all custom +Gaffer types will be converted to their string representation when output via +this interface. + +This conversion to string works both ways for some types too. The current +supported two way conversions are outlined in the table, all other types will +**only** be converted from custom type to string so can not be submitted in a +Gremlin query. -| Key | Example | Description | +| Type | Input | Output | | --- | --- | --- | -| `operationOptions` | `g.with("operationOptions", "gaffer.federatedstore.operation.graphIds:graphA").V()` | Allows passing options to the underlying Gaffer Operations, this is the same as the `options` field on a standard JSON query. | -| `getAllElementsLimit` | `g.with("getAllElementsLimit", 100).V()` | Limits the amount of elements returned if performing an unseeded query e.g. a `GetAllElements` operation. | -| `hasStepFilterStage` | `g.with("hasStepFilterStage", "PRE_AGGREGATION").V()` | Controls which phase the filtering from a Gremlin `has()` stage is applied to the results. | -| `cypher` | `g.with("cypher", "MATCH (p:person) RETURN p").call()` | Translates the given Cypher query to Gremlin and executes it on the Graph. | +| [`TypeSubTypeValue`](https://gchq.github.io/Gaffer/uk/gov/gchq/gaffer/types/TypeSubTypeValue.html) | `g.V("[type=thetype,subType=thesubtype,value=thevalue]")` | `"TypeSubTypeValue[type=thetype,subType=thesubtype,value=thevalue]"` | + +Some additional functions and predicates are available in GafferPop that are +not present in standard Gremlin syntax. Currently these only include the +extensions provided by [`OpenCypher`](https://github.com/opencypher/cypher-for-gremlin/tree/master/tinkerpop/cypher-gremlin-extensions) +but may be expanded in the future. Please see the reference guide for a full +list. + +!!! note + To use custom functions and predicates via Gremlin python you must submit + the query as a Groovy script as they likely will not have python bindings + e.g. + + ```python + results = client.submit("g.V().values('count').map(cypherToString()).toList()") + results.all().result() + ``` + + See the [Tinkerpop documentation](https://tinkerpop.apache.org/docs/current/reference/#gremlin-python-scripts) + for more information. ## Gaffer Gremlin Explain With the Gaffer REST API there are additional endpoints to provide explanations of how a given Gremlin query maps to Gaffer operation chains. The two endpoints are: -- `/rest/gremlin/explain` - Accepts plain string Gremlin and will return JSON explain. -- `/rest/gremlin/cypher/explain` - Accepts plain OpenCypher querys and will return JSON explain. +- `/rest/gremlin/explain` - Accepts plain string Gremlin Groovy script and will + return JSON explain. +- `/rest/gremlin/cypher/explain` - Accepts plain string OpenCypher querys and + will return JSON explain. !!! warning Please be aware that in order to provide an explanation your submitted query will From f0f2b80c32da63e25e452a6debab836fd811f8e4 Mon Sep 17 00:00:00 2001 From: tb06904 <141412860+tb06904@users.noreply.github.com> Date: Thu, 1 Aug 2024 16:39:16 +0000 Subject: [PATCH 2/5] move options to reference guide --- .../gremlin-guide/custom-functions.md | 0 .../gremlin-guide/custom-predicates.md | 0 .../reference/gremlin-guide/gaffer-options.md | 62 +++++++++++++++++++ docs/user-guide/query/gremlin/gremlin.md | 9 --- mkdocs.yml | 6 +- 5 files changed, 67 insertions(+), 10 deletions(-) create mode 100644 docs/reference/gremlin-guide/custom-functions.md create mode 100644 docs/reference/gremlin-guide/custom-predicates.md create mode 100644 docs/reference/gremlin-guide/gaffer-options.md diff --git a/docs/reference/gremlin-guide/custom-functions.md b/docs/reference/gremlin-guide/custom-functions.md new file mode 100644 index 0000000000..e69de29bb2 diff --git a/docs/reference/gremlin-guide/custom-predicates.md b/docs/reference/gremlin-guide/custom-predicates.md new file mode 100644 index 0000000000..e69de29bb2 diff --git a/docs/reference/gremlin-guide/gaffer-options.md b/docs/reference/gremlin-guide/gaffer-options.md new file mode 100644 index 0000000000..bd50731fb4 --- /dev/null +++ b/docs/reference/gremlin-guide/gaffer-options.md @@ -0,0 +1,62 @@ +# Custom Options for Gremlin + +This page details the available options you can pass in a Gremlin +`with()` step when using the GafferPop API for querying. + +## Operation Options + +Key: `operationOptions` + +Allows passing options to the underlying Gaffer Operations, this is the same as +the `options` field on a standard JSON query. This will be applied to all +operations in the query. + +!!! example + + ```groovy + g.with("operationOptions", "gaffer.federatedstore.operation.graphIds:graphA").V().toList() + ``` + +## GetAllElements Limit + +Key `getAllElementsLimit` + +Limits the amount of elements returned if performing an unseeded query e.g. a +`GetAllElements` operation. This will override the default for the current +query see the [admin guide](../../administration-guide/gaffer-deployment/gremlin.md#configuring-the-gafferpop-library) +for more detail on setting up defaults. + +!!! example + + ```groovy + g.with("getAllElementsLimit", 100).V().toList() + ``` + +## Has Step Filter Stage + +Key: `hasStepFilterStage` + +Controls which phase the filtering from a Gremlin `has()` stage is applied to +the results. This will apply to all `has()` steps in the query and override the +default, see the [admin guide](../../administration-guide/gaffer-deployment/gremlin.md#configuring-the-gafferpop-library) +for more detail on setting up defaults. + +!!! example + + ```groovy + g.with("hasStepFilterStage", "PRE_AGGREGATION").V().has("count" P.gt(1)).toList() + ``` + +## Cypher Query + +Key: `cypher` + +Translates the given Cypher query to Gremlin and executes it on the Graph. This +can be used in combination with other `with()` steps as they will be protected +from translation. + +!!! example + + ```groovy + g.with("cypher", "MATCH (p:person) RETURN p").call().toList() + ``` diff --git a/docs/user-guide/query/gremlin/gremlin.md b/docs/user-guide/query/gremlin/gremlin.md index b5752f7d02..1f2534dcc2 100644 --- a/docs/user-guide/query/gremlin/gremlin.md +++ b/docs/user-guide/query/gremlin/gremlin.md @@ -255,12 +255,3 @@ a table of how different parts are mapped is as follows: | Vertex | Vertex with default label of `id` | | Entity | Vertex | | Edge | Edge | - -In Gafferpop Edge ID's must be made up of a list containing either source and destination -IDs, e.g. `[source, dest]`, or source, label and destination, e.g. `[source, label, dest]`. -In a seeded query these should be formatted like so `g.E("[source, dest]")` or -`g.E(["[source1, dest1]","[source2, label, dest2]"])`. - -Note that if using TypeSubTypeValue for seeds or property values these must be in the -format `t:type|st:subtype|v:value`. - diff --git a/mkdocs.yml b/mkdocs.yml index 097bc136ec..05af44d627 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -99,7 +99,7 @@ nav: - 'Using CSV Data': 'user-guide/query/gaffer-syntax/import-export/csv.md' - Apache Gremlin: - 'Gremlin in Gaffer': 'user-guide/query/gremlin/gremlin.md' - - 'Custom Features': 'user-guide/query/gremlin/custom-features.md' + - 'GafferPop Features': 'user-guide/query/gremlin/custom-features.md' - 'GafferPop Limitations': 'user-guide/query/gremlin/gremlin-limits.md' - 'Graph Schemas': 'user-guide/schema.md' - Developer Guide: @@ -219,6 +219,10 @@ nav: - 'Misc Operations': 'reference/operations-guide/misc.md' - Endpoints: - 'Endpoints Guide': 'reference/endpoints-guide/endpoints.md' + - Gremlin: + - 'Gaffer Options': 'reference/gremlin-guide/gaffer-options.md' + - 'Custom Predicates': 'reference/gremlin-guide/custom-predicates.md' + - 'Custom Functions': 'reference/gremlin-guide/custom-functions.md' theme: name: material logo: assets/icon.svg From 0d151c9c04e32cba60c324929b1fbe8b533c774d Mon Sep 17 00:00:00 2001 From: tb06904 <141412860+tb06904@users.noreply.github.com> Date: Fri, 2 Aug 2024 16:21:25 +0000 Subject: [PATCH 3/5] set up custom functions and predicates pages --- .../gremlin-guide/custom-functions.md | 21 +++++++++++++++++++ .../gremlin-guide/custom-predicates.md | 9 ++++++++ .../query/gremlin/custom-features.md | 4 +++- 3 files changed, 33 insertions(+), 1 deletion(-) diff --git a/docs/reference/gremlin-guide/custom-functions.md b/docs/reference/gremlin-guide/custom-functions.md index e69de29bb2..731d342a3a 100644 --- a/docs/reference/gremlin-guide/custom-functions.md +++ b/docs/reference/gremlin-guide/custom-functions.md @@ -0,0 +1,21 @@ +# GafferPop Custom Functions + +This page details each additional function that is available in +the Gaffer Gremlin API (GafferPop). These functions may be exclusive +to GafferPop so are likely not available in other databases that +support the Gremlin query language. + +## OpenCypher Extensions + +The following are additional functions provided by the [OpenCypher extensions](https://github.com/opencypher/cypher-for-gremlin/tree/master/tinkerpop/cypher-gremlin-extensions) +for Gremlin. + +## cypherToString + +Can be used to cast the input value to a `String` type. + +!!! example + + ```groovy + g.V().values('age').map(cypherToString()).toList() + ``` diff --git a/docs/reference/gremlin-guide/custom-predicates.md b/docs/reference/gremlin-guide/custom-predicates.md index e69de29bb2..f5219cd597 100644 --- a/docs/reference/gremlin-guide/custom-predicates.md +++ b/docs/reference/gremlin-guide/custom-predicates.md @@ -0,0 +1,9 @@ +# GafferPop Custom Predicates + +This page details each additional predicate that is available in +the Gaffer Gremlin API (GafferPop). These predicates may be exclusive +to GafferPop so are likely not available in other databases that +support the Gremlin query language. + +!!! note + Currently no custom predicates in GafferPop. diff --git a/docs/user-guide/query/gremlin/custom-features.md b/docs/user-guide/query/gremlin/custom-features.md index 4572248afa..8ad3d6b160 100644 --- a/docs/user-guide/query/gremlin/custom-features.md +++ b/docs/user-guide/query/gremlin/custom-features.md @@ -101,7 +101,9 @@ Some additional functions and predicates are available in GafferPop that are not present in standard Gremlin syntax. Currently these only include the extensions provided by [`OpenCypher`](https://github.com/opencypher/cypher-for-gremlin/tree/master/tinkerpop/cypher-gremlin-extensions) but may be expanded in the future. Please see the reference guide for a full -list. +list of [predicates](../../../reference/gremlin-guide/custom-predicates.md) +and [functions](../../../reference/gremlin-guide/custom-functions.md) +available. !!! note To use custom functions and predicates via Gremlin python you must submit From db43a68415539c47f110ae95c801ce61cf81c931 Mon Sep 17 00:00:00 2001 From: tb06904 <141412860+tb06904@users.noreply.github.com> Date: Mon, 5 Aug 2024 14:22:42 +0000 Subject: [PATCH 4/5] document the custom cypher functions --- .../gremlin-guide/custom-functions.md | 263 +++++++++++++++++- .../query/gremlin/custom-features.md | 11 +- 2 files changed, 272 insertions(+), 2 deletions(-) diff --git a/docs/reference/gremlin-guide/custom-functions.md b/docs/reference/gremlin-guide/custom-functions.md index 731d342a3a..37b0f79a1c 100644 --- a/docs/reference/gremlin-guide/custom-functions.md +++ b/docs/reference/gremlin-guide/custom-functions.md @@ -5,6 +5,8 @@ the Gaffer Gremlin API (GafferPop). These functions may be exclusive to GafferPop so are likely not available in other databases that support the Gremlin query language. +All examples featured on this page assume the [Tinkerpop modern dataset](https://tinkerpop.apache.org/docs/current/images/tinkerpop-modern.png). + ## OpenCypher Extensions The following are additional functions provided by the [OpenCypher extensions](https://github.com/opencypher/cypher-for-gremlin/tree/master/tinkerpop/cypher-gremlin-extensions) @@ -17,5 +19,264 @@ Can be used to cast the input value to a `String` type. !!! example ```groovy - g.V().values('age').map(cypherToString()).toList() + g.V().values('age').map(cypherToString()) + ``` + + Result: + + ```text + ['27', '29', '35', '32'] + ``` + +## cypherToBoolean + +Can be used to cast the input value to a `Boolean` type. Will convert `String` +representations e.g. "true" or "false", otherwise return a Cypher null value. + +!!! example + + ```groovy + g.V().values('name').map(cypherToBoolean()) + ``` + + Result: + + ```text + [' cypher.null', + ' cypher.null', + ' cypher.null', + ' cypher.null', + ' cypher.null', + ' cypher.null'] + ``` + +## cypherToInteger + +Can be used to cast the input value to an `Integer` type. Note if value is +a `Float` or `Double` rounding may be applied. + +!!! example + + ```groovy + g.E().values('weight').map(cypherToInteger()) + ``` + + Result: + + ```text + [0, 0, 0, 1, 1, 0] + ``` + +## cypherToFloat + +Can be used to cast the input value to a `Float` type. + +!!! example + + ```groovy + g.V().values('age').map(cypherToFloat()) + ``` + + Result: + + ```text + [27.0, 29.0, 35.0, 32.0] + ``` + +## cypherRound + +Can be used to round numerical input values. + +!!! example + + ```groovy + g.E().values('weight').map(cypherRound()) + ``` + + Result: + + ```text + [0, 0, 0, 1, 1, 1] + ``` + +## cypherProperties + +Used to extract only the properties from the input `Vertex` or `Edge`. +Similar to running an `elementMap()` or `propertyMap()` step. + +!!! example + + ```groovy + g.V().map(cypherProperties()) + ``` + + Result: + + ```json + [{'name': 'vadas', 'age': 27}, + {'name': 'ripple', 'lang': 'java'}, + {'name': 'marko', 'age': 29}, + {'name': 'peter', 'age': 35}, + {'name': 'josh', 'age': 32}, + {'name': 'lop', 'lang': 'java'}] + ``` + +## cypherSize + +Can be used to return the size of a `Collection` or `String`. + +!!! example + + ```groovy + g.E().values('weight').fold().map(cypherSize()) + ``` + + Result: + + ```text + [6] + ``` + +## cypherPlus + +Can be used to add two inputs together to act like the plus operator +in Cypher e.g. `RETURN 1 + 2`. + +Note this requires a slightly more complex input of pair list e.g. `[1, 2]` to +use in a Gremlin query. The example below uses a `project()` step to pair each +'age' with the constant `1` so that each age is incremented by 1 in the result. + +!!! example + + ```groovy + g.V().hasLabel('person') + .project('result') + .by(__.project('a', 'b') + .by(__.values('age')) + .by(__.constant(1)) + .select(values) + .map(cypherPlus())) + ``` + + Result: + + ```text + [{'result': 28}, {'result': 30}, {'result': 36}, {'result': 33}] + ``` + +## cypherReverse + +Can be used to reverse a `Collection` or `String`. + +!!! example + + ```groovy + g.V().values('name').map(cypherReverse()) + ``` + + Result: + + ```text + ['sadav', 'elppir', 'okram', 'retep', 'hsoj', 'pol'] + ``` + +## cypherSubstring + +Can be used to extract a substring from the result by using indexes for start +and or end point of the string. + +Note requires complex input pairs by using the project step for the input string +and index. The example below cuts the first two chars off the start of the +`name`. + +!!! example + + ```groovy + g.V().hasLabel('person') + .project('result') + .by(__.project('a', 'b') + .by(__.values('name')) + .by(__.constant(2)) + .select(values) + .map(cypherSubstring())) + ``` + + ```text + [{'result': 'das'}, {'result': 'rko'}, {'result': 'ter'}, {'result': 'sh'}] + ``` + +## cypherTrim + +Can be used to run a [`trim()`](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/String.html#trim()) +on a given string input, basically all leading and trailing whitespace is +removed. + +!!! example + + ```groovy + g.V().values('name').map(cypherTrim()) + ``` + + Result: + + ```text + ['vadas', 'ripple', 'marko', 'peter', 'josh', 'lop'] + ``` + +## cypherToUpper + +Can be used to uppercase the input string. + +!!! example + + ```groovy + g.V().values('name').map(cypherToUpper()) + ``` + + Result: + + ```text + ['VADAS', 'RIPPLE', 'MARKO', 'PETER', 'JOSH', 'LOP'] + ``` + +## cypherToLower + +Can be used to lowercase the input string. + +!!! example + + ```groovy + g.V().values('name').map(cypherToLower()) + ``` + + Result: + + ```text + ['vadas', 'ripple', 'marko', 'peter', 'josh', 'lop'] + ``` + +## cypherSplit + +Can be used to split an input string based on a given character. + +Note requires complex input pairs by using the project step for the input string +and split character. The example below splits on an `e`. + +!!! example + + ```groovy + g.V().hasLabel('person') + .project('result') + .by(__.project('a', 'b') + .by(__.values('name')) + .by(__.constant('e')) + .select(values) + .map(cypherSplit())) + ``` + + ```text + [{'result': ['vadas']}, + {'result': ['marko']}, + {'result': ['p', 't', 'r']}, + {'result': ['josh']}] ``` diff --git a/docs/user-guide/query/gremlin/custom-features.md b/docs/user-guide/query/gremlin/custom-features.md index 8ad3d6b160..96fbc6911a 100644 --- a/docs/user-guide/query/gremlin/custom-features.md +++ b/docs/user-guide/query/gremlin/custom-features.md @@ -78,7 +78,7 @@ Users can also run any existing or added Named Operations that are stored in the g.call("namedoperation", params).toList(); ``` -## Custom Types and Functions +## Custom Types In Gaffer there are various [additional types](../../../reference/properties-guide/properties.md) you may want to use to represent properties. These types may not be compatible @@ -97,6 +97,8 @@ Gremlin query. | --- | --- | --- | | [`TypeSubTypeValue`](https://gchq.github.io/Gaffer/uk/gov/gchq/gaffer/types/TypeSubTypeValue.html) | `g.V("[type=thetype,subType=thesubtype,value=thevalue]")` | `"TypeSubTypeValue[type=thetype,subType=thesubtype,value=thevalue]"` | +## Custom Functions and Predicates + Some additional functions and predicates are available in GafferPop that are not present in standard Gremlin syntax. Currently these only include the extensions provided by [`OpenCypher`](https://github.com/opencypher/cypher-for-gremlin/tree/master/tinkerpop/cypher-gremlin-extensions) @@ -111,6 +113,13 @@ available. e.g. ```python + from gremlin_python.driver import client + from gremlin_python.driver.serializer import GraphSONSerializersV3d0 + from gremlin_python.driver.driver_remote_connection import DriverRemoteConnection + + # Establish client connection + client = client.Client('ws://localhost:8080/gremlin', 'g', message_serializer=GraphSONSerializersV3d0()) + results = client.submit("g.V().values('count').map(cypherToString()).toList()") results.all().result() ``` From f1a0eb0267852751e66a1567872beb177605013a Mon Sep 17 00:00:00 2001 From: tb06904 <141412860+tb06904@users.noreply.github.com> Date: Mon, 12 Aug 2024 13:57:45 +0000 Subject: [PATCH 5/5] Apply suggestions from code review Co-authored-by: cn337131 <141730190+cn337131@users.noreply.github.com> Co-authored-by: wb36499 <166839644+wb36499@users.noreply.github.com> --- docs/reference/gremlin-guide/custom-functions.md | 2 +- docs/reference/gremlin-guide/gaffer-options.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/gremlin-guide/custom-functions.md b/docs/reference/gremlin-guide/custom-functions.md index 37b0f79a1c..a539f35bfe 100644 --- a/docs/reference/gremlin-guide/custom-functions.md +++ b/docs/reference/gremlin-guide/custom-functions.md @@ -183,7 +183,7 @@ Can be used to reverse a `Collection` or `String`. ## cypherSubstring Can be used to extract a substring from the result by using indexes for start -and or end point of the string. +and/or end point of the string. Note requires complex input pairs by using the project step for the input string and index. The example below cuts the first two chars off the start of the diff --git a/docs/reference/gremlin-guide/gaffer-options.md b/docs/reference/gremlin-guide/gaffer-options.md index bd50731fb4..4253d8b2a3 100644 --- a/docs/reference/gremlin-guide/gaffer-options.md +++ b/docs/reference/gremlin-guide/gaffer-options.md @@ -23,7 +23,7 @@ Key `getAllElementsLimit` Limits the amount of elements returned if performing an unseeded query e.g. a `GetAllElements` operation. This will override the default for the current -query see the [admin guide](../../administration-guide/gaffer-deployment/gremlin.md#configuring-the-gafferpop-library) +query, see the [admin guide](../../administration-guide/gaffer-deployment/gremlin.md#configuring-the-gafferpop-library) for more detail on setting up defaults. !!! example