Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Websocket integration - events notifier #703

Merged
merged 2 commits into from
Oct 2, 2023
Merged

Websocket integration - events notifier #703

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2a4c5e8
ws connector - observer and event notifier
ssd04 Sep 21, 2023
433f1b7
notifier - fixes after review
ssd04 Sep 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
131 changes: 118 additions & 13 deletions docs/sdk-and-tools/notifier.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,14 +30,14 @@ Set up at least one observer for each shard in order to handle all the events in
![img](/technology/notifier-overview.png)

In the figure above:
- The observer nodes will push block events to Notifier instance, via HTTP POST requests. There are several endpoints for this:
- `/events/push` (POST) -> it will handle all events for each round
- `/events/revert` (POST) -> if there is a reverted block, the event will be
- The observer nodes will push block events to Notifier instance, via WebSocket or HTTP requests. There are several endpoints/topics for this:
- `block events` -> it will handle all events for each round
- `revert events` -> if there is a reverted block, the event will be
pushed on this route
- `/events/finalized` (POST) -> when the block has been finalized, the events
- `finalized events` -> when the block has been finalized, the events
bogdan-rosianu marked this conversation as resolved.
Show resolved Hide resolved
will be pushed on this route
- Notifier checks locker service (via Redis) and applies deduplication
- Notifier will push events to RabbitMQ if enabled, or via Websockets. If Websockets will be enabled an additional endpoint will be available:
- Notifier will push events to RabbitMQ if enabled, or via Websocket. If Websocket will be enabled an additional endpoint will be available:
- `/hub/ws` (GET) - this route can be used to manage the websocket connection subscription

[comment]: # (mx-context-auto)
Expand All @@ -48,13 +48,70 @@ In the figure above:

### Observer Client

On the observer side, there is a http client that will push block events to notifier service.
On the observer side, there is a WebSocket client that will push block events to notifier service.
There is also the HTTP Integration, which will be deprecated in the future.

In the observer node's configuration directory, `external.toml` config file can be configured
to enable notifier connector. The config file can be found
In the observer node's configuration directory, the `external.toml` config file can be configured
to enable events notifier connector via WebSocket or via HTTP integrations. The config file can be found
[here](https://github.com/multiversx/mx-chain-go/blob/master/cmd/node/config/external.toml).

The supported config variables are as follows:
[comment]: # (mx-context-auto)

#### WebSocket Integration
bogdan-rosianu marked this conversation as resolved.
Show resolved Hide resolved

The WebSocket integration is a generic one, and can be used for multiple outport driver integrations.
In case Elasticsearch integration is already being used with WebSocket connector, a separate config
section `HostDriversConfig` has to be set for event notifier.

The corresponding config section for enabling the driver:

```toml
[[HostDriversConfig]]
# This flag shall only be used for observer nodes
Enabled = true

# This flag will start the WebSocket connector as server or client (can be "client" or "server")
Mode = "client"

# URL for the WebSocket client/server connection
# This value represents the IP address and port number that the WebSocket client or server will use to establish a connection.
URL = "127.0.0.1:22111"

# After a message will be sent it will wait for an ack message if this flag is enabled
WithAcknowledge = true

# The duration in seconds to wait for an acknowledgment message, after this time passes an error will be returned
AcknowledgeTimeoutInSec = 60

# This flag defines the marshaller type. Currently supported: "json", "gogo protobuf"
MarshallerType = "gogo protobuf"

# The number of seconds when the client will try again to send the data
RetryDurationInSec = 5

# Sets if, in case of data payload processing error, we should block or not the advancement to the next processing event. Set this to true if you wish the node to stop processing blocks if the client/server encounters errors while processing requests.
BlockingAckOnError = true

# Set to true to drop messages if there is no active WebSocket connection to send to.
DropMessagesIfNoConnection = false

# Defines the payload version. Version will be changed when there are breaking
# changes on payload data. The receiver/consumer will have to know how to handle different
# versions. The version will be sent as metadata in the websocket message.
Version = 1
```

In this case, observer node will act as client and events notifier service will act as a server.
`Mode` option should be set to `client`. It is important to have `WithAcknowledge` set to `true`
since observer node should continue only if there is an acknowledge that the event was processed
successfully. `MarshallerType` field has to be aligned with `DataMarshallerType` on events notifier
configuration file.

[comment]: # (mx-context-auto)

#### HTTP Integration

For http integration, the supported config variables are as follows:

- `Enabled`: signals whether a driver should be attached when launching the node.
- `UseAuthorization`: signals whether to use authorization. For testing purposes it can be set to `false`.
Expand Down Expand Up @@ -84,6 +141,10 @@ The corresponding config section for enabling the driver:
Password = ""
```

:::info
HTTP Integration will be deprecated in the future.
:::

:::tip
Due to the possible high data volume, it's not recommended to use validators as nodes
to push events to Notifier Service.
Expand All @@ -99,6 +160,49 @@ delays due to outport driver.
In the notifier configuration directory (`cmd/notifier/config`), there is the `config.toml`
file that can be used to configure the service.

There are 2 ways to connect observer node with events notifier service:
- via WebSocket integration
- via HTTP integration (which will be deprecated in the future)

[comment]: # (mx-context-auto)

#### WebSocket Integration

There is a separate config section `WebSocketConnector` that has to be aligned with
`HostDriversConfig` from observer node.

```toml
[WebSocketConnector]
# Enabled will determine if websocket connector will be enabled or not
Enabled = false

# URL for the WebSocket client/server connection
# This value represents the IP address and port number that the WebSocket client or server will use to establish a connection.
URL = "localhost:22111"

# This flag describes the mode to start the WebSocket connector. Can be "client" or "server"
Mode = "server"

# Possible values: json, gogo protobuf. Should be compatible with mx-chain-node outport driver config
DataMarshallerType = "gogo protobuf"

# Retry duration (receive/send ack signal) in seconds
RetryDurationInSec = 5

# Signals if in case of data payload processing error, we should send the ack signal or not
BlockingAckOnError = false

# After a message will be sent it will wait for an ack message if this flag is enabled
WithAcknowledge = true

# The duration in seconds to wait for an acknowledgment message, after this time passes an error will be returned
AcknowledgeTimeoutInSec = 60
```

[comment]: # (mx-context-auto)

#### HTTP Integration

The supported config variables are:
- `Port`: the port on which the http server listens on. Should be the same
as the port in the `ProxyUrl` described above.
Expand All @@ -123,6 +227,10 @@ The `ConnectorApi` section has to be aligned with the one from observer node:
CheckDuplicates = true
```

[comment]: # (mx-context-auto)

#### Deduplication

If `CheckDuplicates` will be set to true, notifier service will try to connect to a redis
instance. In this context, redis will be used as a locker service mechanism for deduplication.
This is useful in scenarios when multiple observer nodes from same shard are used to send
Expand All @@ -135,9 +243,6 @@ The `Redis` section includes the following parameters as described below:
# The url used to connect to a pubsub server
Url = "redis://localhost:6379/0"

# The pubsub channel used for publishing/subscribing
Channel = "pub-sub"

# The master name for failover client
MasterName = "mymaster"

Expand Down Expand Up @@ -205,7 +310,7 @@ There are multiple event types:
- Finalized Block event: when the block is finalized

In RabbitMQ there is a separate exchange for each event type.
In Websockets setup, there is a event type field in each message.
In Websocket setup, there is a event type field in each message.

The WS event is defined as follows:

Expand Down
Loading