Skip to content

NatLibFi/melinda-rest-api-commons

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Shared modules for microservices of Melinda rest api import system

NPM Version Node Version

Shared modules for microservices of Melinda rest api import system

Constants

QUEUE_ITEM_STATE

QueueItemState flow for bulk jobs (multiple records, no validations):

microservice states
http -> VALIDATOR.UPLOADING -> VALIDATOR.PENDING_QUEUING ->
validator VALIDATOR.QUEUING_IN_PROGRESS -> IMPORTER.IN_QUEUE ->
importer IMPORTER.IMPORTING -> IMPORT_JOB_STATEs* -> DONE
any any -> ERROR if errored

*IMPORT_JOB_STATEs: EMPTY or IN_QUEUE, PROCESSING -> final: DONE, ERROR, ABORT

QueueItemState flow for prio jobs (single record):

microservice states
http -> VALIDATOR.PENDING_VALIDATION ->
validator VALIDATOR.VALIDATING -> IMPORTER.IN_QUEUE ->
importer IMPORTER.IMPORTING -> IMPORT_JOB_STATEs* -> DONE
http any -> ABORT if job has stayed in an active state too long
validator VALIDATOR.VALIDATING -> DONE for noop (no-operation) jobs
any any -> ERROR if errored

*IMPORT_JOB_STATEs: EMPTY or IN_QUEUE, PROCESSING -> final: DONE, ERROR, ABORT

CHUNK_SIZE

Chunck size to execution time ratio (From file to queue: ~450 records in 15sec)

Total records Chunk size Chunks Execution time Time / chunk Operation
9248 100 93 ~597 sec ~6.4 sec Update
Executed on test server using bridge to docker server and back to server

Common confs for aleph-record-load-api

P_manage_18 name Node name Prio update Prio create Bulk update Bulk create Description
p_active_library pActiveLibrary params params params params Library to use
p_input_file pInputFile generated generated generated generated Source file location
p_reject_file pRejectFile generated generated generated generated Log file for rejected records
p_log_file pLogFile generated generated generated generated Log file for updated/created record ids
p_old_new pOldNew OLD NEW OLD NEW Method of operation. Either NEW or OLD
p_fix_type pFixType API API INSB INSB Aleph fix routine code
p_check_references pCheckReferences
p_update_f pUpdateF FULL FULL FULL FULL Indexing action
p_update_type pUpdateType REP REP REP REP REP or APP (REPlace or APPend)
p_update_mode pUpdateMode M M M M User mode. Either M (Multi-user) or S (Single-user)
p_char_conv pCharConv Character conversion to apply
p_merge_type pMergeType Merge/Preferred routine
p_cataloger_in pCatalogerIn params params params params Value which is written to CAT fields
p_cataloger_level_x pCatalogerLevelX Cataloger lever
p_z07_priority_year pZ07PriorityYear 1998 1990 2099 2099 Override indexing priority
p_redirection_field pRedirectionField

License and copyright

Copyright (c) 2020-2024 University Of Helsinki (The National Library Of Finland)

This project's source code is licensed under the terms of MIT or any later version.