-
Notifications
You must be signed in to change notification settings - Fork 334
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Default Configuration for RocksDB MaxManifestFileSize #1706
Add Default Configuration for RocksDB MaxManifestFileSize #1706
Conversation
@@ -321,7 +321,8 @@ These properties define Samza's storage mechanism for efficient [stateful stream | |||
|stores.**_store-name_**.<br>rocksdb.keep.log.file.num|2|The number of RocksDB LOG files (including rotated LOG.old.* files) to keep.| | |||
|stores.**_store-name_**.<br>rocksdb.metrics.list|(none)|A list of [RocksDB properties](https://github.com/facebook/rocksdb/blob/master/include/rocksdb/db.h#L409) to expose as metrics (gauges).| | |||
|stores.**_store-name_**.<br>rocksdb.delete.obsolete.files.period.micros|21600000000|This property specifies the period in microseconds to delete obsolete files regardless of files removed during compaction. Allowed range is up to 9223372036854775807.| | |||
|stores.**_store-name_**.<br>rocksdb.max.manifest.file.size|18446744073709551615|This property specifies the maximum size of the MANIFEST data file, after which it is rotated. Default value is also the maximum, making it practically unlimited: only one manifest file is used.| |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The number was outdated, and it should be 1024^3 which is 1073741824
- The RocksDB version for Samza was upgraded from
5.7.3
to6.5.3
at 0189b5d - In
5.7.3
, themax_manifest_file_size
defaults to uint64 max which is18446744073709551615
(https://github.com/facebook/rocksdb/blob/v5.7.3/include/rocksdb/options.h#L547) - In
6.5.3
, themax_manifest_file_size
defaults to 1GB which is1024*1024*1024
(https://github.com/facebook/rocksdb/blob/v6.5.3/include/rocksdb/options.h#L623)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks. Some small comments.
samza-core/src/main/java/org/apache/samza/config/StorageConfig.java
Outdated
Show resolved
Hide resolved
samza-core/src/main/java/org/apache/samza/config/StorageConfig.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the match
Some minor readability comments
docs/learn/documentation/versioned/jobs/samza-configurations.md
Outdated
Show resolved
Hide resolved
|
||
static final String DEFAULT_ROCKSDB_MAX_MANIFEST_FILE_SIZE = | ||
STORE_PREFIX + DEFAULT_STORE_NAME + ".rocksdb.max.manifest.file.size"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
for the default
config, lets not follow the same format as the config with named stores in case of default
table name collision
Changes
stores.default.rocksdb.max.manifest.file.size
in Samza allowing to set the default manifest file size for all stores.stores.default.rocksdb.max.manifest.file.size
is1G
(1024*1024*1024
), which is the default value in RocksDB v7.8.3 that is currently used by Samza https://github.com/facebook/rocksdb/blob/v7.8.3/include/rocksdb/options.h#L786Tests
TestStorageConfig.java