From 2a4c5e8c466df558de3e0fe431812be774293333 Mon Sep 17 00:00:00 2001 From: ssd04 Date: Thu, 21 Sep 2023 18:00:45 +0300 Subject: [PATCH 1/2] ws connector - observer and event notifier --- docs/sdk-and-tools/notifier.md | 127 ++++++++++++++++++++++++++++++--- 1 file changed, 116 insertions(+), 11 deletions(-) diff --git a/docs/sdk-and-tools/notifier.md b/docs/sdk-and-tools/notifier.md index 97e8c4e0a..6d5727508 100644 --- a/docs/sdk-and-tools/notifier.md +++ b/docs/sdk-and-tools/notifier.md @@ -30,11 +30,11 @@ Set up at least one observer for each shard in order to handle all the events in ![img](/technology/notifier-overview.png) In the figure above: -- The observer nodes will push block events to Notifier instance, via HTTP POST requests. There are several endpoints for this: - - `/events/push` (POST) -> it will handle all events for each round - - `/events/revert` (POST) -> if there is a reverted block, the event will be +- The observer nodes will push block events to Notifier instance, via WebSocket or HTTP requests. There are several endpoints/topics for this: + - `block events` -> it will handle all events for each round + - `revert events` -> if there is a reverted block, the event will be pushed on this route - - `/events/finalized` (POST) -> when the block has been finalized, the events + - `finalized events` -> when the block has been finalized, the events will be pushed on this route - Notifier checks locker service (via Redis) and applies deduplication - Notifier will push events to RabbitMQ if enabled, or via Websockets. If Websockets will be enabled an additional endpoint will be available: @@ -48,13 +48,70 @@ In the figure above: ### Observer Client -On the observer side, there is a http client that will push block events to notifier service. +On the observer side, there is a WebSocket client that will push block events to notifier service. +There is also the HTTP Integration, which will be deprecated in the future. -In the observer node's configuration directory, `external.toml` config file can be configured -to enable notifier connector. The config file can be found +In the observer node's configuration directory, the `external.toml` config file can be configured +to enable events notifier connector via WebSocket or via HTTP integrations. The config file can be found [here](https://github.com/multiversx/mx-chain-go/blob/master/cmd/node/config/external.toml). -The supported config variables are as follows: +[comment]: # (mx-context-auto) + +#### WebSocket Integration + +This WebSocket integration is a generic one, and can be used for multiple outport driver integrations. +In case Elasticsearch integration is already being used with WebSocket connector, a separate config +section `HostDriversConfig` has to be set for event notifier. + +The corresponding config section for enabling the driver: + +```toml +[[HostDriversConfig]] + # This flag shall only be used for observer nodes + Enabled = true + + # This flag will start the WebSocket connector as server or client (can be "client" or "server") + Mode = "client" + + # URL for the WebSocket client/server connection + # This value represents the IP address and port number that the WebSocket client or server will use to establish a connection. + URL = "127.0.0.1:22111" + + # After a message will be sent it will wait for an ack message if this flag is enabled + WithAcknowledge = true + + # The duration in seconds to wait for an acknowledgment message, after this time passes an error will be returned + AcknowledgeTimeoutInSec = 60 + + # Currently, only "json" is supported. In the future, "gogo protobuf" could also be supported + MarshallerType = "gogo protobug" + + # The number of seconds when the client will try again to send the data + RetryDurationInSec = 5 + + # Sets if, in case of data payload processing error, we should block or not the advancement to the next processing event. Set this to true if you wish the node to stop processing blocks if the client/server encounters errors while processing requests. + BlockingAckOnError = true + + # Set to true to drop messages if there is no active WebSocket connection to send to. + DropMessagesIfNoConnection = false + + # Defines the payload version. Version will be changed when there are breaking + # changes on payload data. The receiver/consumer will have to know how to handle different + # versions. The version will be sent as metadata in the websocket message. + Version = 1 +``` + +In this case, observer node will act as client and events notifier service will act as a server. +`Mode` option should be set to `client`. It is important to have `WithAcknowledge` set to `true` +since observer node should continue only if there is an acknowledge that the event was processed +successfully. `MarshallerType` field has to be aligned with `DataMarshallerType` on events notifier +configuration file. + +[comment]: # (mx-context-auto) + +#### HTTP Integration + +For http integration, the supported config variables are as follows: - `Enabled`: signals whether a driver should be attached when launching the node. - `UseAuthorization`: signals whether to use authorization. For testing purposes it can be set to `false`. @@ -84,6 +141,10 @@ The corresponding config section for enabling the driver: Password = "" ``` +:::info +HTTP Integration will be deprecated in the future. +::: + :::tip Due to the possible high data volume, it's not recommended to use validators as nodes to push events to Notifier Service. @@ -99,6 +160,49 @@ delays due to outport driver. In the notifier configuration directory (`cmd/notifier/config`), there is the `config.toml` file that can be used to configure the service. +There are 2 ways to connect observer node with events notifier service: +- via WebSocket integration +- via HTTP integration (which will be deprecated in the future) + +[comment]: # (mx-context-auto) + +#### WebSocket Integration + +There is a separate config section `WebSocketConnector` that has to be aligned with +`HostDriversConfig` from observer node. + +```toml +[WebSocketConnector] + # Enabled will determine if websocket connector will be enabled or not + Enabled = false + + # URL for the WebSocket client/server connection + # This value represents the IP address and port number that the WebSocket client or server will use to establish a connection. + URL = "localhost:22111" + + # This flag describes the mode to start the WebSocket connector. Can be "client" or "server" + Mode = "server" + + # Possible values: json, gogo protobuf. Should be compatible with mx-chain-node outport driver config + DataMarshallerType = "gogo protobuf" + + # Retry duration (receive/send ack signal) in seconds + RetryDurationInSec = 5 + + # Signals if in case of data payload processing error, we should send the ack signal or not + BlockingAckOnError = false + + # After a message will be sent it will wait for an ack message if this flag is enabled + WithAcknowledge = true + + # The duration in seconds to wait for an acknowledgment message, after this time passes an error will be returned + AcknowledgeTimeoutInSec = 60 +``` + +[comment]: # (mx-context-auto) + +#### HTTP Integration + The supported config variables are: - `Port`: the port on which the http server listens on. Should be the same as the port in the `ProxyUrl` described above. @@ -123,6 +227,10 @@ The `ConnectorApi` section has to be aligned with the one from observer node: CheckDuplicates = true ``` +[comment]: # (mx-context-auto) + +#### Deduplication + If `CheckDuplicates` will be set to true, notifier service will try to connect to a redis instance. In this context, redis will be used as a locker service mechanism for deduplication. This is useful in scenarios when multiple observer nodes from same shard are used to send @@ -135,9 +243,6 @@ The `Redis` section includes the following parameters as described below: # The url used to connect to a pubsub server Url = "redis://localhost:6379/0" - # The pubsub channel used for publishing/subscribing - Channel = "pub-sub" - # The master name for failover client MasterName = "mymaster" From 433f1b758fae9706b996cec1db430162021f83ba Mon Sep 17 00:00:00 2001 From: ssd04 Date: Fri, 22 Sep 2023 15:49:26 +0300 Subject: [PATCH 2/2] notifier - fixes after review --- docs/sdk-and-tools/notifier.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/sdk-and-tools/notifier.md b/docs/sdk-and-tools/notifier.md index 6d5727508..c77969f4a 100644 --- a/docs/sdk-and-tools/notifier.md +++ b/docs/sdk-and-tools/notifier.md @@ -37,7 +37,7 @@ In the figure above: - `finalized events` -> when the block has been finalized, the events will be pushed on this route - Notifier checks locker service (via Redis) and applies deduplication -- Notifier will push events to RabbitMQ if enabled, or via Websockets. If Websockets will be enabled an additional endpoint will be available: +- Notifier will push events to RabbitMQ if enabled, or via Websocket. If Websocket will be enabled an additional endpoint will be available: - `/hub/ws` (GET) - this route can be used to manage the websocket connection subscription [comment]: # (mx-context-auto) @@ -59,7 +59,7 @@ to enable events notifier connector via WebSocket or via HTTP integrations. The #### WebSocket Integration -This WebSocket integration is a generic one, and can be used for multiple outport driver integrations. +The WebSocket integration is a generic one, and can be used for multiple outport driver integrations. In case Elasticsearch integration is already being used with WebSocket connector, a separate config section `HostDriversConfig` has to be set for event notifier. @@ -83,8 +83,8 @@ The corresponding config section for enabling the driver: # The duration in seconds to wait for an acknowledgment message, after this time passes an error will be returned AcknowledgeTimeoutInSec = 60 - # Currently, only "json" is supported. In the future, "gogo protobuf" could also be supported - MarshallerType = "gogo protobug" + # This flag defines the marshaller type. Currently supported: "json", "gogo protobuf" + MarshallerType = "gogo protobuf" # The number of seconds when the client will try again to send the data RetryDurationInSec = 5 @@ -310,7 +310,7 @@ There are multiple event types: - Finalized Block event: when the block is finalized In RabbitMQ there is a separate exchange for each event type. -In Websockets setup, there is a event type field in each message. +In Websocket setup, there is a event type field in each message. The WS event is defined as follows: