Version 2.1.4

A patch release that bumps dependencies to latest versions

• Update copyright notice to 2022 (#255)
• Change docker base image to eclipse-temurin:11-jre-focal (#254)
• Bump protobuf-java to 3.19.4 (#253)
• Bump jackson to 2.12.6 (2.12.6)
• Bump kinesis client to 1.14.7 (#251)

Version 2.1.3

Fix partition format in example hocon (#247)
Clean up terminated shards before expiry (#248)

Version 2.1.2

Bug fixes

Bump amazon-kinesis-client to 1.14.5 (#245)

Version 2.1.1

Bug fixes

Exclude transitive dependencies of hadoop (#243)
Bump commons-collections to 3.2.2 (#242)
Bump elephant-bird-core to 4.17 (#241)
Remove log4j (#240)

Version 2.1.1-rc2

Exlude transitive dependencies of hadoop (close #234)

Version 2.1.1-rc1

Exlude transitive dependencies of hadoop (close #234)

Version 2.1.0

This version adds a feature to give complete control over the partitioning of S3 directories via the configuration hocon. It also fixes a major bug that could cause the loader to hang during kinesis scaling events.

Bug fixes and performance improvements

• Optimise fromEnriched function (#227)
• Fix dateFormat partitioning in output path (#236)
• Fix premature shutdown of HTTP connection pool (#231)
• Fix duplicate statsd metrics when loading lzo files (#229)

Under the hood

• Update readme (#239)
• Integrate lacework (#232)
• Use sbt-dynver (#233)
• Add Twitter Maven repository (#234)
• Bump amazon-kinesis-client to 1.14.4 (#230)

Version 2.0.0

A new major releasing bringing StatsD observability, Sentry and config refactoring.

• Upgrade guide
• Discourse announcement

New features

• Report metrics to StatsD (#216)
• Integrate Sentry (close #218)
• Use AdoptOpenJDK 11 as docker base image (#224)
• Attach jar files to Github releases (#197)

API changes

• Redesign config file structure (#214)
• Drop NSQ support (#211)

Under the hood

• Use snowplow-badrows (#215)
• Use sbt-tpolecat (#222)
• Harmonize module structure (#210)
• Extend copyright notice to 2021 (#219)
• Apply automated code-formatting (#221)
• Point to Snowplow roadmap in the README (#220)
• Replace joda-time by java-time (#202)
• Replace scopt by decline (#201)
• Remove json4s dependency (#200)

Dependency bumps

• Bump Scala to 2.13.6 (#223)
• Bump hadoop-common to 2.7.7 (#208)
• Bump jackson to 2.9.10.8 (#207)
• Bump amazon-kinesis-client to 1.14.2 (#206)
• Bump sbt-native-packager to 1.8.1 (#205)
• Bump pureconfig to 0.14.1 (#203)
• Bump base-debian to 0.2.2 (#199)
• Bump sbt to 1.5.2 (#198)

Version 1.0.0 (2020-12-22)

This version comes with minor bug fixes (#180 and #192) and 2 new features:

• #173: files on S3 now contain the time in the prefix, making it possible to partition data in Athena.
• #193: it's now possible to disable KCL metrics that are automatically sent to CloudWatch, in order to save money. To do so, this parameter needs to be added to the config.

Documentation for this component can be found on our docs website.

Bug fixes / new features

• Handle case where partitionRecords is empty (#192)
• Use random partition key for bad rows (#180)
• Add sink timestamp section into file names (#173)
• Make it possible to disable KCL CloudWatch metrics (#193)

Misc

• Set log level of records serialized to DEBUG (#174)
• Migrate from Travis to GH actions (#181)
• Use hadoop-lzo 0.4.20 from Snowplow Bintray maven (#183)
• Add coveralls integration (#195)
• Use log4j-over-slf4j for log4j logs (#194)
• Use sbt-scoverage plugin (#164)
• Use sbt-scalafmt plugin (#163)
• Bump sbt to 1.4.4 (#167)
• Bump specs2-core to 4.10.5 (#189)
• Bump iglu-core-json4s to 1.0.0 (#188)
• Bump pureconfig to 0.14.0 (#187)
• Bump json4s-jackson to 3.6.10 (#190)
• Bump cats-core to 2.3.0 (#186)
• Bump scopt to 4.0.0 (#185)
• Bump Scala tracker to 0.7.0 (#184)
• Bump Scala to 2.13 (#182)

Version 0.7.0 (2020-02-24)

An initial release of data partitioning by schema

Features

Add partitioning for self-describing types (#146)
Remove colons from NSQ filenames (#125)
Allow configurable and dynamic s3 path (#134)
Allow override of kinesis and s3 endpoints in configuration (#131)
Update flags to resemble ones used in GCS loader config (#157)
Add sbt-native-packager and Docker config (#152)
Remove curly braces from transformed directory pattern (#137)
Document that maxTimeout is in milliseconds (#129)

Dependencies

Bump sbt-assembly to 0.14.10 (#156)
Replace scalaz by cats (#148)
Bump SBT to 1.2.8 (#151)

Misc

Extend copyright notice to 2019 (#154)
Change Travis distribution to Trusty (#155)
Add Docker Hub credentials to .travis.yml (#153)
Move Twitter repo to https (#127)
Remove vagrant setup (#140)