waltti-apc-anonymizer

Anonymize accurate APC messages and output GTFS Realtime OccupancyStatus levels.

To anonymize the APC messages that have been augmented with vehicle journey metadata, match the vehicle with its anonymization profile and sample from the anonymization profile.

This repository has been created as part of the Waltti APC project.

Development

1. Create a suitable .env file for configuration. Check below for the configuration reference.

2. Create any necessary secrets that the .env file points to.

3. Install dependencies:

   npm install

4. Run linters and tests and build:

   npm run check-and-build

5. Load the environment variables:

   set -a
   source .env
   set +a

6. Run the application:

   npm start

Docker

You can use the Docker image tvvlmj/waltti-apc-anonymizer:edge. Check out the available tags.

Configuration

Environment variable	Required?	Default value	Description
ACCEPTED_DEVICE_MAP	✅ Yes		A map from vehicles to counting device IDs. Only vehicles with multiple counting devices installed are included. Also only those counting devices are included which should be used for publishing the anonymized APC results. The map is given in the form of a stringified JSON array of strings in the shape [[uniqueVehicleId1, systemId1], [uniqueVehicleId2, systemId2], ...] like the output of Map.prototype.entries(). An example could be [[\"fi:jyvaskyla:6714_503\",\"c5e96843-e820-4837-8eef-6176be4b4c4e\"],[\"fi:jyvaskyla:6714_529\",\"6dd41f2e-841f-44a0-b5f8-a108847dc4a2\"]].
AUTHORITY_MAP	✅ Yes		A map from Waltti OpenData Authority IDs to feed publisher IDs. The map is given in the form of a stringified JSON array of strings in the shape [[walttiOpenDataAuthority1, feedPublisher1], [walttiOpenDataAuthority2, feedPublisher2], ...] like the output of Map.prototype.entries(). The feed publisher ID is a unique ID for the authority or the GTFS feed publisher whose APC data will be handled. The format is <country-code>:<name> where <country-code> follows a lowercase version of ISO 3166-1 alpha-2 and <name> is unique within the country. An example could be fi:jyvaskyla. The corresponding Waltti OpenData Authority IDs are listed here: https://opendata.waltti.fi/docs#gtfs-static-packages . An example of the whole list could be [[\"221\",\"fi:kuopio\"],[\"209\",\"fi:jyvaskyla\"]].
HEALTH_CHECK_PORT	❌ No	8080	Which port to use to respond to health checks.
PINO_LOG_LEVEL	❌ No	info	The level of logging to use. One of "fatal", "error", "warn", "info", "debug", "trace" or "silent".
PULSAR_APC_CONSUMER_TOPICS_PATTERN	✅ Yes		The topic pattern to consume APC vehicle messages from.
PULSAR_APC_SUBSCRIPTION	✅ Yes		The name of the subscription for reading messages from PULSAR_APC_CONSUMER_TOPICS_PATTERN.
PULSAR_BLOCK_IF_QUEUE_FULL	❌ No	true	Whether the send operations of the producer should block when the outgoing message queue is full. If false, send operations will immediately fail when the queue is full.
PULSAR_COMPRESSION_TYPE	❌ No	ZSTD	The compression type to use in the topic where messages are sent. Must be one of Zlib, LZ4, ZSTD or SNAPPY.
PULSAR_OAUTH2_AUDIENCE	✅ Yes		The OAuth 2.0 audience.
PULSAR_OAUTH2_ISSUER_URL	✅ Yes		The OAuth 2.0 issuer URL.
PULSAR_OAUTH2_KEY_PATH	✅ Yes		The path to the OAuth 2.0 private key JSON file.
PULSAR_PRODUCER_TOPIC	✅ Yes		The topic to send messages to.
PULSAR_PROFILE_READER_TOPIC	✅ Yes		The topic to read vehicle anonymization profile messages from. As we are using a Reader, the topic must have some retention configured, e.g. a week. Otherwise the messages might be deleted before reading.
PULSAR_PROFILE_READER_NAME	✅ Yes		The name of the reader for reading messages from PULSAR_PROFILE_READER_TOPIC.
PULSAR_SERVICE_URL	✅ Yes		The service URL.
PULSAR_TLS_VALIDATE_HOSTNAME	❌ No	true	Whether to validate the hostname on its TLS certificate. This option exists because some Apache Pulsar hosting providers cannot handle Apache Pulsar clients setting this to true.
VEHICLE_PROFILE_MAP	✅ Yes		Temporary.