Recommended sharding point? #9817
Unanswered
danthegoodman1
asked this question in
Q&A
Replies: 1 comment
-
We don't really recommend sharding over multiple Materialize instances yourself, since it's non-trivial to handle consistency and other knick-knacks of distributed dataflow execution. If you want to give horizontal scalability a try, though, you can play around with this prototype! |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I am wondering at what point you all would recommend sharding to multiple materialize processes while there is no clustering features.
For example, if a topic is doing 10Gb/s, that probably should be handled by it's own shard? Or maybe we have to look at CPU limits first, i.e. we will hit CPU bottlenecks before bandwidth bottlenecks?
Obviously we lose some features sharding, but for inrelated streams I imagine this would be ok.
Beta Was this translation helpful? Give feedback.
All reactions