Merge pull request #641 from FgForrest/undertow-backup

Time-travel implementation, backup & restore process, last Undertow based API implementation

README.md

@@ -149,6 +149,12 @@ That's it.
 
 ## Prerequisities
 
+To checkout Git repository on Windows you need to have long paths enabled:
+
+```shell
+git config --system core.longpaths true
+```
+
 evitaDB requires and is tested on OpenJDK 17.
 
 Java applications support multiple platforms depending on the
@@ -231,4 +237,4 @@ In order code to be accepted it will fulfill following criteria:
 
 [Icons sourced at Icons8.com](https://icons8.com/)
 
-[//]: # (https://icons8.com/icon/set/github/carbon-copy--static--white)
+[//]: # (https://icons8.com/icon/set/github/carbon-copy--static--white)

documentation/user/en/operate/configure.md

@@ -39,7 +39,6 @@ transaction:                                      # [see Transaction configurati
   transactionMemoryRegionCount: 256
   walFileSizeBytes: 16MB
   walFileCountKept: 8
-  maxQueueSize: 1K
   flushFrequencyInMillis: 1s
 
 cache:                                            # [see Cache configuration](#cache-configuration)
@@ -441,6 +440,14 @@ This section contains configuration options for the storage layer of the databas
         <p>Minimal file size threshold for compaction. If the file size is lower, the file will not be compacted even 
             if the share of active records is lower than the minimal share.</p>
     </dd>
+    <dt>timeTravelEnabled</dt>
+    <dd>
+        <p>**Default:** `true`</p>
+        <p>When set to true, the data files are not removed immediately after compacting, but are kept on disk as long 
+        as there is history available in the WAL log. This allows a snapshot of the database to be taken at any point 
+        in the history covered by the WAL log. From the snapshot, the database can be restored to the exact point in 
+        time with all the data available at that time.</p>
+    </dd>
 </dl>
 
 ## Transaction configuration
@@ -481,12 +488,6 @@ This section contains configuration options for the storage layer of the databas
         <p>Number of WAL files to keep. Increase this number in combination with `walFileSizeBytes` if you want to
             keep longer history of changes.</p>
     </dd>
-    <dt>maxQueueSize</dt>
-    <dd>
-        <p>**Default:** `1K`</p>
-        <p>Size of the catalog queue for parallel transaction. If there are more transaction than the number of free 
-            threads in the pool, the transaction are queued. If the queue is full, the transaction is rejected.</p>
-    </dd>
     <dt>flushFrequencyInMillis</dt>
     <dd>
         <p>**Default:** `1s`</p>

documentation/user/en/operate/observe.md

@@ -430,3 +430,9 @@ in [Logback](https://logback.qos.ch/index.html) this can be done using `%X{trace
 	<pattern>%d{HH:mm:ss.SSS} %-5level %logger{10} C:%X{traceId} R:%X{spanId} - %msg%n</pattern>
 </encoder>
 ```
+
+## Reference documentation
+
+<MDInclude>[Java Flight Recorder events](/documentation/user/en/operate/reference/jfr-events.md)</MDInclude>
+
+<MDInclude>[Metrics](/documentation/user/en/operate/reference/metrics.md)</MDInclude>

documentation/user/en/operate/reference/jfr-events.md

@@ -0,0 +1,143 @@
+### Java Flight Recorder (JFR) Events
+
+#### API / GraphQL / Instance / Schema
+
+<dl>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/externalApi/graphql/metric/event/instance/BuiltEvent.java</SourceClass> GraphQL instance built</dt>
+  <dd>Event that is fired when a GraphQL instance is built.</dd>
+</dl>
+
+#### API / gRPC
+
+<dl>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/externalApi/grpc/metric/event/ProcedureCalledEvent.java</SourceClass> gRPC procedure called</dt>
+  <dd>Event that is fired when a gRPC procedure is called.</dd>
+</dl>
+
+#### Cache
+
+<dl>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/cache/AnteroomRecordStatisticsUpdatedEvent.java</SourceClass> Anteroom statistics updated</dt>
+  <dd>Event that is fired in regular intervals to update statistics about records waiting in anteroom.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/cache/AnteroomWastedEvent.java</SourceClass> Anteroom wasted</dt>
+  <dd>Event that is fired when an entire anteroom contents were thrown away.</dd>
+</dl>
+
+#### ExternalAPI / GraphQL / Request
+
+<dl>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/externalApi/graphql/metric/event/request/ExecutedEvent.java</SourceClass> GraphQL request executed</dt>
+  <dd>Event that is fired when a GraphQL request is executed.</dd>
+</dl>
+
+#### ExternalAPI / REST / Instance / Schema
+
+<dl>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/externalApi/rest/metric/event/instance/BuiltEvent.java</SourceClass> REST API instance built</dt>
+  <dd>Event that is fired when a REST API instance is built.</dd>
+</dl>
+
+#### ExternalAPI / REST / Request
+
+<dl>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/externalApi/rest/metric/event/request/ExecutedEvent.java</SourceClass> REST request executed</dt>
+  <dd>Event that is fired when a REST request is executed.</dd>
+</dl>
+
+#### Query
+
+<dl>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/query/EntityEnrichEvent.java</SourceClass> Entity fetched</dt>
+  <dd>Event that is fired when an entity is enriched directly.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/query/EntityFetchEvent.java</SourceClass> Entity fetched</dt>
+  <dd>Event that is fired when an entity is fetched directly.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/query/FinishedEvent.java</SourceClass> Catalog finished</dt>
+  <dd>Event that is fired when a query is finished.</dd>
+</dl>
+
+#### Session
+
+<dl>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/session/ClosedEvent.java</SourceClass> Session closed</dt>
+  <dd>Event that is fired when a session is closed.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/session/KilledEvent.java</SourceClass> Session killed</dt>
+  <dd>Event that is fired when a session is killed due to timeout.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/session/OpenedEvent.java</SourceClass> Session opened</dt>
+  <dd>Event that is fired when a session is started.</dd>
+</dl>
+
+#### Storage
+
+<dl>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/storage/CatalogStatisticsEvent.java</SourceClass> Catalog flushed</dt>
+  <dd>Event that is fired when a new catalog version is flushed.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/storage/DataFileCompactEvent.java</SourceClass> OffsetIndex compaction</dt>
+  <dd>Event that is fired when an OffsetIndex file is compacted.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/storage/EvitaDBCompositionChangedEvent.java</SourceClass> Evita composition changed</dt>
+  <dd>Event that is fired when evitaDB composition changes.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/storage/ObservableOutputChangeEvent.java</SourceClass> ObservableOutput buffers</dt>
+  <dd>Event that is fired when an ObservableOutput buffer count is changed.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/storage/OffsetIndexFlushEvent.java</SourceClass> OffsetIndex flushed to disk</dt>
+  <dd>Event that is fired when an OffsetIndex file is flushed.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/storage/OffsetIndexHistoryKeptEvent.java</SourceClass> OffsetIndex last record kept</dt>
+  <dd>Event that is fired when history data kept in memory change.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/storage/OffsetIndexNonFlushedEvent.java</SourceClass> OffsetIndex non-flushed records</dt>
+  <dd>Event that is fired when non flushed record count changes in offset index.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/storage/OffsetIndexRecordTypeCountChangedEvent.java</SourceClass> OffsetIndex record type count changed</dt>
+  <dd>Event that is fired when number of records of a particular type in OffsetIndex file changes.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/storage/ReadOnlyHandleClosedEvent.java</SourceClass> File read handles closed</dt>
+  <dd>Event that is fired when a file read handle is closed.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/storage/ReadOnlyHandleOpenedEvent.java</SourceClass> File read handles opened</dt>
+  <dd>Event that is fired when a new file read handle is opened.</dd>
+</dl>
+
+#### System
+
+<dl>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/system/BackgroundTaskFinishedEvent.java</SourceClass> Background task finished</dt>
+  <dd>Event that is fired when a background task is finished.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/system/BackgroundTaskRejectedEvent.java</SourceClass> Background task rejected</dt>
+  <dd>Event that is fired when a background task is rejected due to full queues.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/system/BackgroundTaskStartedEvent.java</SourceClass> Background task started</dt>
+  <dd>Event that is fired when a background task is started.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/system/BackgroundTaskTimedOutEvent.java</SourceClass> Background task timed out</dt>
+  <dd>Event that is fired when a background task was timed out and was canceled.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/system/EvitaStartedEvent.java</SourceClass> Evita started</dt>
+  <dd>Event that is fired when evitaDB instance is started.</dd>
+</dl>
+
+#### Transaction
+
+<dl>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/transaction/CatalogGoesLiveEvent.java</SourceClass> Catalog goes live</dt>
+  <dd>Event that is fired when a catalog goes live (transactional).</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/transaction/IsolatedWalFileClosedEvent.java</SourceClass> Isolated WAL file closed</dt>
+  <dd>Event that is fired when a file for isolated WAL storage is closed and deleted.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/transaction/IsolatedWalFileOpenedEvent.java</SourceClass> Isolated WAL file opened</dt>
+  <dd>Event that is fired when new file is opened for isolated WAL storage.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/transaction/NewCatalogVersionPropagatedEvent.java</SourceClass> New catalog version propagated</dt>
+  <dd>Event that is fired when a new catalog version is propagated shared view.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/transaction/OffHeapMemoryAllocationChangeEvent.java</SourceClass> Off-heap memory allocation change</dt>
+  <dd>Event that is fired when off-heap memory allocation changes.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/transaction/TransactionAcceptedEvent.java</SourceClass> Transaction accepted</dt>
+  <dd>Event that is fired when a transaction passed conflict resolution stage.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/transaction/TransactionAppendedToWalEvent.java</SourceClass> Transaction appended to WAL</dt>
+  <dd>Event that is fired when a transaction passed conflict resolution stage.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/transaction/TransactionFinishedEvent.java</SourceClass> Transaction finished</dt>
+  <dd>Event that is fired when a transaction is finished either by commit or rollback and corresponding session is closed. This also includes waiting for transaction reaching requested stage of processing.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/transaction/TransactionIncorporatedToTrunkEvent.java</SourceClass> Transaction incorporated to trunk</dt>
+  <dd>Event that is fired when a transaction was incorporated into a shared data structures.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/transaction/TransactionProcessedEvent.java</SourceClass> Transaction processed and visible</dt>
+  <dd>Event that is fired when a transaction reached the shared view.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/transaction/TransactionQueuedEvent.java</SourceClass> Transaction waiting in queue</dt>
+  <dd>Event that is fired in each transaction processing stage to reflect the time transaction waited in the queue before it was picked up for processing.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/transaction/TransactionStartedEvent.java</SourceClass> Transaction started</dt>
+  <dd>Event that is fired when a transaction is started.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/transaction/WalCacheSizeChangedEvent.java</SourceClass> WAL cache size changed</dt>
+  <dd>Event that is fired when a shared WAL location cache size is changed.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/transaction/WalRotationEvent.java</SourceClass> WAL rotated</dt>
+  <dd>Event that is fired when a shared WAL is rotated.</dd>
+  <dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/transaction/WalStatisticsEvent.java</SourceClass> WAL statistics</dt>
+  <dd>Event that is fired when a catalog is loaded and WAL examined.</dd>
+</dl>
+