Glue together NATS JetStream, MinIO blob store and p2panda gossip

[adzialocha]

This PR implements the logic to coordinate all components to play nicely together as described in: https://whimsical.com/rhio-JtQuJKYkfaepHGDsFL7qBC and fixes a couple of bugs which have been made on the way here.

For blob import this introduces a simple NATS Core subject rhio.import where file paths are sent to. A reply can be specified which will contain the resulting blob hash.

---

🚀	This description was created by Ellipsis for commit 2209169

Summary:

This PR integrates NATS JetStream, MinIO, and p2panda into rhio, introduces rhio.import subject, updates configurations, fixes bugs, and enhances log level handling.

Key points:

• Integrates NATS JetStream, MinIO, and p2panda into rhio.
• Introduces rhio.import subject for blob import.
• Updates Cargo.lock for latest p2panda dependencies.
• Adds announce_blob function in rhio-client/src/lib.rs.
• Modifies rhio-client/src/main.rs for blob announcements.
• Updates rhio-core/src/extensions.rs to include blob_hash.
• Adds create_blob_announcement in rhio-core/src/operation.rs.
• Updates rhio/config.example.toml for stream configurations.
• Refactors rhio/src/blobs/actor.rs for file imports/exports to MinIO.
• Modifies rhio/src/nats/actor.rs and rhio/src/nats/consumer.rs for JetStream events.
• Enhances rhio/src/node/actor.rs for NATS and p2panda operations.
• Updates rhio/src/panda/actor.rs and rhio/src/panda/mod.rs for gossip events and operation ingestion.
• Updates configuration in rhio/src/config.rs.
• Fixes bugs encountered during development.
• Coordinates components as per the design in the whimsical link.
• Updates rhio/src/tracing.rs to handle log levels more flexibly.

---

Generated with ❤️ by ellipsis.dev

[ellipsis-dev]

👍 Looks good to me! Reviewed everything up to 8f6436d in 49 seconds

More details

• Looked at 1893 lines of code in 21 files
• Skipped 0 files when reviewing.
• Skipped posting 5 drafted comments based on config settings.

1. rhio-client/src/lib.rs:39

• Draft comment:
  The separation of announce_blob and publish methods is a good design choice, as it clearly distinguishes between different types of operations.
• Reason this comment was not posted:
  Confidence changes required: 0%
  The announce_blob method in the Client struct is similar to the publish method, but it uses create_blob_announcement instead of create_message. This is a good design choice as it separates the logic for different types of operations.

2. rhio/src/blobs/actor.rs:123

• Draft comment:
  The use of a loop to handle ImportBlobEvent in on_import_file is a robust approach to ensure all events are processed, including error handling.
• Reason this comment was not posted:
  Confidence changes required: 0%
  The on_import_file method in BlobsActor uses a loop to handle ImportBlobEvent. This is a good approach to ensure that all events are processed correctly, including handling errors.

3. rhio/src/blobs/actor.rs:215

• Draft comment:
  Checking the response status code after completing the multipart upload is a good practice to ensure the upload was successful.
• Reason this comment was not posted:
  Confidence changes required: 0%
  The on_export_blob_minio method in BlobsActor checks the response status code after completing the multipart upload. This is a good practice to ensure that the upload was successful.

4. rhio/src/node/actor.rs:223

• Draft comment:
  Decoding and validating operations from NATS messages in on_nats_message is crucial for ensuring data integrity before further processing.
• Reason this comment was not posted:
  Confidence changes required: 0%
  The on_nats_message method in NodeActor decodes and validates operations from NATS messages. This is a critical step to ensure data integrity before processing further.

5. rhio/src/node/actor.rs:258

• Draft comment:
  Forwarding operations to the NATS server in on_operation ensures persistence and communication to other processes, maintaining consistency.
• Reason this comment was not posted:
  Confidence changes required: 0%
  The on_operation method in NodeActor forwards operations to the NATS server. This ensures that operations are persisted and communicated to other processes, which is essential for maintaining consistency.

Workflow ID: wflow_B6PyYSXCOdCETFHQ

---

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

⌛ 3 days left in your free trial, upgrade for $20/seat/month or contact us.

[ellipsis-dev]

👍 Looks good to me! Incremental review on 5988609 in 24 seconds

More details

• Looked at 28 lines of code in 2 files
• Skipped 0 files when reviewing.
• Skipped posting 2 drafted comments based on config settings.

1. rhio/src/blobs/actor.rs:128

• Draft comment:
  Unnecessary use of to_path_buf(). path is already a PathBuf.
• Reason this comment was not posted:
  Confidence changes required: 50%
  The use of to_path_buf() is unnecessary since path is already a PathBuf. This can be simplified.

2. rhio/src/blobs/actor.rs:132

• Draft comment:
  Avoid using #[allow(clippy::never_loop)]. Refactor the code to avoid using a loop if it's not needed.
• Reason this comment was not posted:
  Confidence changes required: 50%
  The #[allow(clippy::never_loop)] attribute is used to suppress a warning for a loop that never loops. However, this is not a best practice as it can lead to confusion. Consider refactoring the code to avoid using a loop if it's not needed.

Workflow ID: wflow_NHtHQHZo7noFOgAT

---

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

⌛ 3 days left in your free trial, upgrade for $20/seat/month or contact us.

[ellipsis-dev]

👍 Looks good to me! Incremental review on 9449186 in 34 seconds

More details

• Looked at 168 lines of code in 6 files
• Skipped 0 files when reviewing.
• Skipped posting 2 drafted comments based on config settings.

1. rhio/src/blobs/actor.rs:128

• Draft comment:
  The never_loop clippy allowance is used here. Ensure to revisit this when more enum cases are added to handle potential looping scenarios.
• Reason this comment was not posted:
  Confidence changes required: 50%
  The on_import_file method in rhio/src/blobs/actor.rs uses a loop with a never_loop clippy allowance. This is a known issue, but it should be addressed in the future to ensure the code is robust when more enum cases are added.

2. rhio/src/nats/mod.rs:114

• Draft comment:
  Consider refining the wildcard subscription pattern rhio.* in spawn_control_handler to avoid potential performance issues with many matching subjects.
• Reason this comment was not posted:
  Confidence changes required: 50%
  The spawn_control_handler function in rhio/src/nats/mod.rs uses a wildcard subscription to rhio.*. This could potentially lead to performance issues if there are many subjects matching this pattern. Consider refining the subscription pattern if possible.

Workflow ID: wflow_1OV2qCydm4Yqw4Ff

---

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

⌛ 3 days left in your free trial, upgrade for $20/seat/month or contact us.

[ellipsis-dev]

👍 Looks good to me! Incremental review on b80d4b7 in 36 seconds

More details

• Looked at 12 lines of code in 1 files
• Skipped 0 files when reviewing.
• Skipped posting 1 drafted comments based on config settings.

1. rhio/src/nats/consumer.rs:248

• Draft comment:
  The comment here suggests uncertainty about using num_pending to determine the initial stream height. Verify if this is the correct field or if adjustments (e.g., +1) are needed. This might affect the initialization logic.
• Reason this comment was not posted:
  Confidence changes required: 50%
  The code in question is related to the initialization of a NATS JetStream consumer. The comment on line 248 suggests uncertainty about whether the correct field is being used to determine the initial stream height. The field num_pending is used, which represents the number of pending messages for the consumer. This seems appropriate for determining how many messages need to be processed initially. However, the comment suggests there might be a need to adjust this value, possibly by adding 1 or using a different field. Without further context or documentation, it's difficult to determine if this is indeed a bug, but the comment indicates a potential issue that should be verified.

Workflow ID: wflow_Tu2qyHjQfNlC27Cw

---

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

⌛ 3 days left in your free trial, upgrade for $20/seat/month or contact us.

[ellipsis-dev]

👍 Looks good to me! Incremental review on 5b707fc in 16 seconds

More details

• Looked at 114 lines of code in 1 files
• Skipped 0 files when reviewing.
• Skipped posting 1 drafted comments based on config settings.

1. rhio/config.example.toml:25

• Draft comment:
  Typo in comment: "ndoe" should be "node".
• Reason this comment was not posted:
  Confidence changes required: 10%
  The comment on line 25 contains a typo that should be corrected for clarity.

Workflow ID: wflow_11ARplWazXZ7pv9K

---

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

⌛ 3 days left in your free trial, upgrade for $20/seat/month or contact us.

[ellipsis-dev]

👍 Looks good to me! Incremental review on a42fbee in 18 seconds

More details

• Looked at 27 lines of code in 1 files
• Skipped 0 files when reviewing.
• Skipped posting 0 drafted comments based on config settings.

Workflow ID: wflow_qzFlN5BjtR1DrfLR

---

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

⌛ 3 days left in your free trial, upgrade for $20/seat/month or contact us.

[ellipsis-dev]

👍 Looks good to me! Incremental review on 82c6fb2 in 20 seconds

More details

• Looked at 19 lines of code in 1 files
• Skipped 0 files when reviewing.
• Skipped posting 1 drafted comments based on config settings.

1. README.md:102

• Draft comment:
  Typo: Replace "it's" with "its" for correct possessive form.

1. Inform rhio to import and encode a file from the file system into the MinIO database. Send its path by publishing to the NATS Core subject `rhio.import`. The resulting hash is displayed in the server's logs, a [reply subject](https://docs.nats.io/nats-concepts/core-nats/reqreply) can also be specified where the resulting hash will be sent to.

• Reason this comment was not posted:
  Confidence changes required: 10%
  The possessive pronoun "it's" is incorrectly used instead of the correct form "its".

Workflow ID: wflow_TrpI401CD730oQaz

---

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

⌛ 3 days left in your free trial, upgrade for $20/seat/month or contact us.

[ellipsis-dev]

👍 Looks good to me! Incremental review on 86afe11 in 19 seconds

More details

• Looked at 27 lines of code in 2 files
• Skipped 0 files when reviewing.
• Skipped posting 1 drafted comments based on config settings.

1. rhio/config.example.toml:26

• Draft comment:
  Typo in comment: "ndoe" should be "node".
• Reason this comment was not posted:
  Confidence changes required: 10%
  The typo in the comment should be corrected for clarity.

Workflow ID: wflow_PYovQFVLYFFO3saq

---

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

⌛ 3 days left in your free trial, upgrade for $20/seat/month or contact us.

[ellipsis-dev]

👍 Looks good to me! Incremental review on 788c1ad in 40 seconds

More details

• Looked at 64 lines of code in 4 files
• Skipped 0 files when reviewing.
• Skipped posting 0 drafted comments based on config settings.

Workflow ID: wflow_xHY9NIQ1OKFexrai

---

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

⌛ 3 days left in your free trial, upgrade for $20/seat/month or contact us.

[ellipsis-dev]

👍 Looks good to me! Incremental review on bcf50a3 in 39 seconds

More details

• Looked at 238 lines of code in 3 files
• Skipped 0 files when reviewing.
• Skipped posting 1 drafted comments based on config settings.

1. rhio/src/network/actor.rs:6

• Draft comment:
  The change from futures_lite to futures_util is appropriate and consistent with the rest of the codebase.
• Reason this comment was not posted:
  Confidence changes required: 0%
  The change from futures_lite to futures_util is consistent with the rest of the codebase and should not cause issues.

Workflow ID: wflow_OiUkDZwS4MXfqjzA

---

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

⌛ 3 days left in your free trial, upgrade for $20/seat/month or contact us.

[ellipsis-dev]

❌ Changes requested. Incremental review on 31b4136 in 26 seconds

More details

• Looked at 92 lines of code in 1 files
• Skipped 0 files when reviewing.
• Skipped posting 0 drafted comments based on config settings.

Workflow ID: wflow_FoQyEsFiVha5RDH0

---

Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

⌛ 3 days left in your free trial, upgrade for $20/seat/month or contact us.

[ellipsis-dev]

❌ Changes requested. Incremental review on 901b2f4 in 26 seconds

More details

• Looked at 61 lines of code in 2 files
• Skipped 0 files when reviewing.
• Skipped posting 0 drafted comments based on config settings.

Workflow ID: wflow_mDmfJCQHAgvGDYZN

---

Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

⌛ 3 days left in your free trial, upgrade for $20/seat/month or contact us.

[mycognosist]

I've offered some suggested changes for grammar and typos.

Currently playing with everything on my local machine to see if I have any other suggestions.

[ellipsis-dev]

👍 Looks good to me! Incremental review on a99f371 in 21 seconds

More details

• Looked at 79 lines of code in 1 files
• Skipped 0 files when reviewing.
• Skipped posting 1 drafted comments based on config settings.

1. README.md:104

• Draft comment:
  The code block indentation is inconsistent. Ensure consistent indentation for better readability.
• Reason this comment was not posted:
  Confidence changes required: 10%
  The README file contains a minor formatting issue in the code block indentation.

Workflow ID: wflow_KOCV6ZxIuWgNm7CD

---

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

⌛ 3 days left in your free trial, upgrade for $20/seat/month or contact us.

[ellipsis-dev]

👍 Looks good to me! Incremental review on 19bf675 in 34 seconds

More details

• Looked at 205 lines of code in 7 files
• Skipped 0 files when reviewing.
• Skipped posting 2 drafted comments based on config settings.

1. rhio/src/network/actor.rs:156

• Draft comment:
  Redundant return statement. The Err variant is already being returned, so the return keyword is unnecessary.

Err(anyhow!("attempted to send operation on unknown topic {topic:?}"))

• Reason this comment was not posted:
  Confidence changes required: 50%
  The code in rhio/src/network/actor.rs has a redundant return statement in the on_broadcast function. The Err variant of the Result is already being returned, so the return keyword is unnecessary.

2. rhio/src/tracing.rs:37

• Draft comment:
  Use String::new() instead of String::default() for clarity and idiomatic code.

String::new()

• Reason this comment was not posted:
  Confidence changes required: 20%
  The code in rhio/src/tracing.rs uses String::default() which is equivalent to String::new(). Using String::new() is more idiomatic and clear.

Workflow ID: wflow_jX39ciQrjomF6VpC

---

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

⌛ 3 days left in your free trial, upgrade for $20/seat/month or contact us.

[ellipsis-dev]

👍 Looks good to me! Incremental review on e4f757b in 15 seconds

More details

• Looked at 13 lines of code in 1 files
• Skipped 0 files when reviewing.
• Skipped posting 1 drafted comments based on config settings.

1. README.md:39

• Draft comment:
  The change from $ cp config.example.toml config.toml to cp config.example.toml config.toml is correct as the dollar sign is not needed in the command example.
• Reason this comment was not posted:
  Confidence changes required: 0%
  The change in the README.md file is a correction of a command syntax, which is a valid and necessary change.

Workflow ID: wflow_AUQlF0Cu6SqB1gdd

---

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

⌛ 3 days left in your free trial, upgrade for $20/seat/month or contact us.

[ellipsis-dev]

👍 Looks good to me! Incremental review on c99ed86 in 22 seconds

More details

• Looked at 29 lines of code in 2 files
• Skipped 0 files when reviewing.
• Skipped posting 1 drafted comments based on config settings.

1. rhio/src/tracing.rs:31

• Draft comment:
  Consider simplifying the else block by directly using the filter if Level::from_str fails, without the intermediate format! call.
• Reason this comment was not posted:
  Confidence changes required: 50%
  The use of strip_prefix is a good improvement for clarity and safety. However, the else block could be improved for clarity and efficiency. The Level::from_str is used to check if the filter is a valid level, and if not, it defaults to using the filter as-is. This could be more clearly expressed by directly using the filter if Level::from_str fails, without the intermediate format! call.

Workflow ID: wflow_lDcLnJfYAR5gkYIr

---

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

⌛ 3 days left in your free trial, upgrade for $20/seat/month or contact us.

[ellipsis-dev]

👍 Looks good to me! Incremental review on 971cb01 in 28 seconds

More details

• Looked at 110 lines of code in 3 files
• Skipped 0 files when reviewing.
• Skipped posting 2 drafted comments based on config settings.

1. rhio/src/nats/actor.rs:14

• Draft comment:
  The documentation for wait_for_ack could be clearer. Consider rephrasing to explicitly state that this flag should be true for JetStream messages to ensure delivery and persistence, and false for NATS Core messages which are "fire and forget".
• Reason this comment was not posted:
  Confidence changes required: 50%
  The publish method in Nats and NatsActor is being called with a wait_for_ack parameter, which is correctly set based on the context (e.g., true for JetStream and false for NATS Core). However, the documentation comments in ToNatsActor::Publish and the on_publish method could be improved for clarity.

2. rhio/src/nats/mod.rs:89

• Draft comment:
  The publish method's wait_for_ack parameter is correctly used, but ensure that its purpose is clearly documented in the method's comments for future maintainability.
• Reason this comment was not posted:
  Confidence changes required: 33%
  The publish method in Nats and NatsActor is being called with a wait_for_ack parameter, which is correctly set based on the context (e.g., true for JetStream and false for NATS Core). However, the documentation comments in ToNatsActor::Publish and the on_publish method could be improved for clarity.

Workflow ID: wflow_dm1dzeh4el6knlAh

---

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

⌛ 3 days left in your free trial, upgrade for $20/seat/month or contact us.

[ellipsis-dev]

👍 Looks good to me! Incremental review on 32ef526 in 23 seconds

More details

• Looked at 14 lines of code in 1 files
• Skipped 0 files when reviewing.
• Skipped posting 1 drafted comments based on config settings.

1. rhio/src/network/actor.rs:207

• Draft comment:
  Good change to use while let Ok(joined_topic) = joined_rx.recv().await for handling channel closure gracefully.
• Reason this comment was not posted:
  Confidence changes required: 0%
  The change from a loop to a while let Ok loop is a good improvement for handling the joined_rx.recv() result. It ensures that the loop exits gracefully if the channel is closed, which is a more robust approach.

Workflow ID: wflow_SgxcTOk48uotPqos

---

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

⌛ 3 days left in your free trial, upgrade for $20/seat/month or contact us.

[ellipsis-dev]

👍 Looks good to me! Incremental review on 5736a8b in 18 seconds

More details

• Looked at 14 lines of code in 1 files
• Skipped 0 files when reviewing.
• Skipped posting 1 drafted comments based on config settings.

1. rhio/src/network/actor.rs:207

• Draft comment:
  Good change from loop to while let Ok to handle channel receiving more gracefully and prevent potential infinite loops.
• Reason this comment was not posted:
  Confidence changes required: 0%
  The change from a loop to a while let Ok loop is a good improvement for handling the channel receiving logic. It prevents potential infinite loops if the channel is closed unexpectedly.

Workflow ID: wflow_XaqJ442IPPVKCCGp

---

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

⌛ 3 days left in your free trial, upgrade for $20/seat/month or contact us.

[sandreae]

Fantastic work! I ran through the demo steps (https://gist.github.com/sandreae/b75226ca6f2bb52e1ad471aa3ce190a6) and everything worked as expect 🌟

[ellipsis-dev]

👍 Looks good to me! Incremental review on 2209169 in 21 seconds

More details

• Looked at 26 lines of code in 1 files
• Skipped 0 files when reviewing.
• Skipped posting 1 drafted comments based on config settings.

1. rhio/config.example.toml:64

• Draft comment:
  The comment about 'external_topic' being required by p2panda is repeated from the PR description. Ensure consistency across documentation and comments to avoid redundancy.
• Reason this comment was not posted:
  Confidence changes required: 20%
  The comment provides additional context about the configuration options, which is useful for users. However, the comment about the 'external_topic' is repeated in the PR description, and it seems like a minor change. The rest of the configuration file seems to be well-documented and doesn't require further comments.

Workflow ID: wflow_RjlTj2RHSKGnvcWA

---

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

⌛ 3 days left in your free trial, upgrade for $20/seat/month or contact us.