You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Today Bento only supports PLAIN and DELTA_LENGTH_BYTE_ARRAY encodings, and you can only use a single encoding for all data types.
We should instead support configuring a different encoding for each field.
For example, in a use case where you are transforming a Kafka topic into a set of Parquet files, you would want to use RLE_DICTIONARY for the topic names and partitions. You'd probably also want to use DELTA_BINARY_PACKED for the offsets and record timestamps.
The text was updated successfully, but these errors were encountered:
Relevant docs: https://warpstreamlabs.github.io/bento/docs/components/processors/parquet_encode/
Today Bento only supports PLAIN and DELTA_LENGTH_BYTE_ARRAY encodings, and you can only use a single encoding for all data types.
We should instead support configuring a different encoding for each field.
For example, in a use case where you are transforming a Kafka topic into a set of Parquet files, you would want to use RLE_DICTIONARY for the topic names and partitions. You'd probably also want to use DELTA_BINARY_PACKED for the offsets and record timestamps.
The text was updated successfully, but these errors were encountered: