pipe-vms

This repository contains the VMS pipeline, a dataflow pipeline that computes normalized positions for all countries and stores them in partitioned tables to use with v3.

---

---

Project structure

The project contains the next structure

├── README.md
├── docker-compose.yaml
├── nx.json
├── package-lock.json
├── package.json
└── packages
    ├── libs
    ├── pipe-vms-ingestion
    └── ...

At the root level, the repository contains common config files or dependencies like NX

• README.MD: Project basic documentation
• docker-compose.yaml: Docker configuration to run the projects locally.
• nx.json: Configuration related to NX library
• package.json and package-lock.json: NodeJS dependencies
• packages: This folder contains all the code related to the pipe-vms projects

Looking the package folder in detail, the repository contains 3 different folders

• libs: Common/shared code between all the projects

---

---

Requirements

• Docker
• NodeJS

Optional, if you do not use Docker, you need to install these dependencies

• Python 3.9
• Poetry

---

Usage

Execute one project

docker compose up [service-name] [--build]

Example:

docker compose up pipe-vms-ingestion --build

Note: you can omit --build flag if you do not need to rebuild the docker image

---

Create a release and push it to GitHub

This process will launch a Cloud Build trigger and generate the Docker image.

If you do not do this before, you need to install the NX dependencies

npm install

Once the NX dependencies were installed, then you can execute the create-release.sh script

./create-release [project-name] [tag-version]

For example:

./create-release pipe-vms-ingestion 1.0.0

This script will open an editor, and you can add release notes to there. Once the tag is created you have to create the release in https://github.com/GlobalFishingWatch/pipe-vms/releases/new selecting the tag that was just created. We recommend using the Generate releease notes button in the New Release page to generate the release notes.

One trigger will be launching via CloudBuild and generate a docker image like this:

gcr.io/world-fishing-827/github.com/globalfishingwatch/pipe-vms-ingestion@1.0.0

---

Add a new project

npx nx generate @nxlv/python:poetry-project [project_name] \
--projectType application \
--description='Project description' \
--packageName=[project-name] \
--moduleName=[project_name]

Example:

npx nx generate @nxlv/python:poetry-project vms_ingestion \
--projectType application \
--description='VMS Ingestion processes' \
--packageName=pipe-vms-ingestion \
--moduleName=vms_ingestion

---

Add a new library

npx nx generate @nxlv/python:poetry-project [library_name] \
--projectType library \
--description='Library description' \
--packageName=[library-name] \
--moduleName=[library_name] \
--directory=libs

Example:

npx nx generate @nxlv/python:poetry-project gfw-logger \
--projectType library \
--description='Library to reuse the GFW Logger' \
--packageName=gfw-logger \
--moduleName=gfw_logger \
--directory=libs

---

Add a local library with a project

In order to use a shared library in the projects, you need to link the library with the project.

npx nx run [project-name]:add --name [lib-name] --local

Example:

npx nx run publication-vms-ingestion:add --name libs-logger --local

Note: NX adds a prefix for each project based on the folder in which they are located. For instance, publication for publication projects, core for core projects or libs for libs. This is something that you need to take into account to link libs with projects.