Write Ahead Logging
- Separate checkpoint is not required for recovery.
- Lightweight logging for insert, update. Lower logging overhead when execute transactions.
- When recovery, only redo the log item with "committing" status. Other consistent items are discarded.
- Minimize log file size.
Logging manager controls all backend loggers and frontend loggers. It provides interfaces to query and manage these loggers. During Peloton startup, logging manager reads configuration file for log settings. Such as which logging protocol should be used.
The backend loggers are thread-local instances. It is responsible of collecting all logs generated in the work thread.
The frontend logger is a global singleton instance. Logging manager ensures all newly created backend loggers are linked to the frontend loggers. Then the frontend logger continuously collects logs from all registered backend loggers and flush log records to log file or other persistent store.
- invalid
- Standby -- Bootstrap
- Recovery -- Optional recovery
- Logging -- Collect data and flush when commit
- Terminate -- Collect any remaining data and flush
- Sleep -- Disconnect backend loggers and frontend logger from manager
Only tuple header information is recorded in logs. We use database ID, table ID and tuple offset to refer tuple data in tuple data storage.
All modifications are directly applied to NVRAM without waiting for logs to be flushed.
We rely on MVCC to ensure consistency. So although the updated data is written to persistent storage immediately, they are not visible before all log items are flushed. When all logs get persisted, DBMS sets the version of the database to make all pending updates visible.
We control tuple update visibility by using commit bits (insert bit and delete bit). Only after logs flushed, logging thread starts to change commit bits in the tuples.
5 steps transaction workflow when doing WBL:
Extended logging workflow
A typical Peloton logging log file structure:
Ensure previous run is not interrupt during changing commit bits. Otherwise, redo changing commit bits.
last_record = final log item in the log file;
If (last_record != flush_done_log) {
`return; // recover done`
} else {
`Scan file backward;`
`Set the file cursor after previous commit marked log.`
`Start read all logs from the cursor to the end of file.`
`Re-mark all tuples in these log records.`
}
cd build/tests
./aries_logging_test
./peloton_logging_test
writing_logfile: execute transaction and write logs.
recovery: turn off logging and write more dummy records. Then reset database and recovery.
-h --help : Print help message
-t --tuple-count : Tuple count.
-b --backend-count : Backend count. Only work for execute transaction.
-c --check-tuple-count : Check tuple count. Turn off when test performance.
-r --redo-all-logs : Force redo all logs. Execute transaction will fail because commit is not done
-d --dir : log file dir.