Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimised minio implementation of required blob store traits #51

Open
sandreae opened this issue Aug 20, 2024 · 0 comments
Open

Optimised minio implementation of required blob store traits #51

sandreae opened this issue Aug 20, 2024 · 0 comments
Labels

Comments

@sandreae
Copy link
Contributor

sandreae commented Aug 20, 2024

We're currently using the filesystem blob store to calculate the bao-tree hashes for the file. This is the only way to retrieve the blob hash right now as implementing it directly in an S3 store is tricky.

The downside with this approach is that we are limited by storage: If we're transferring 7 TB of data to a MinIO database (with a lot of storage), then the transport and batching itself will be done efficiently (not much memory will be occupied), but with that extra-step we temporarily need 7 TB of space for the rhio process itself.

Since rhio is currently in an experimental phase, this shouldn't be a real-world issue right now.

In the future we want to do all of this inside of MinIO / S3 and skip loading the file onto the file-system first, this will allow us to keep the resources for the rhio process low (not much storage required)

@sandreae sandreae added the rhio label Aug 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant