Ἴρις App ( Iris )

[ This repository is for archiving the source code of internal software that i made for my previous company to manage envvars&secrets with Google Secret Manager ]

A Simple API Endpoints - Tools to create and manage secret & environment variables for your applications (Google Cloud Platform) using Google Secret Manager (GSM).

Authors

• fhmlsml

API Reference

Docs ( Swagger UI & ReDoc )

Access the Swagger UI & Redoc

• http://localhost:8000/v1/docs

• http://localhost:8000/v1/redoc

Usage / Examples (v1)

[ Users ]

Register User & Generate Token (If token expired)

  POST /v1/register

  POST /v1/generate

Curl

curl -X 'POST' \
  'http://{base_url}/v1/register' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{                    
  "email": "me@fhmisml.moe",
  "password": "yourpersonalsecretpassword"
}'

curl -X 'POST' \
  'http://{base_url}/v1/generate' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "email": "me@fhmisml.moe",
  "password": "yourpersonalsecretpassword"
}'

[ Services ]

List, Get, and Update Secret & Environment Variables

  GET /v1/list?lang={lang}

  GET /v1/get?service={service_name}

  POST /v1/update?service={service_name}

Curl

curl -X 'GET' \
  'http://{base_url}/v1/list?lang={lang}' \
  -H 'accept: application/json'

curl -X 'GET' \
  'http://{base_url}/v1/get?service={service_name}' \
  -H 'accept: application/json' \
  -H 'token: your-jwt-token'

curl -X 'POST' \
  'http://{base_url}/v1/update?service={service_name}' \
  -H 'accept: application/json' \
  -H 'token: your-jwt-token' \
  -H 'Content-Type: application/json' \
  -d '{
  "data1": "val1",
  "data2": "val2",
  "data3": "val3"
}'

Parameter	Name	Type	Description
query	lang	string	Required.
query & header	service & token	string	Required. Service Name & JWT
query & header	service & token	string	Required. Service Name & JWT

Ops Team

Operations Team Only

  POST /v1/ops/assign

  POST /v1/ops/new-secret

  PUT /v1/ops/metadata

  DELETE /v1/ops/delete

  GET /v1/synchronize

Parameter	Name	Type	Description
-	-	-	-
query	label_type & label_lang	string	Required. label_type & label_lang
query	service & token	string	Required. Service Name & JWT
query	lang	string	Required.
query	lang	string	Required.

Implementation

After adding new env or secret in irisapp or Google Secret Manager directly, your applications or services need to access the secret that you have added.

To do that, you need to use Google Secret Manager Library, for example in python :

--> For further information, check this link, choose your preferred language, and follow the instructions

import os
import json

from google.cloud import secretmanager

client = secretmanager.SecretManagerServiceClient()

name = f"projects/{os.getenv('PROJECT_ID')}/secrets/app-your-service-name/versions/latest"

# Access the secret version.
response = client.access_secret_version(request={"name": name})

# the json_string_data will be -> {"data1": "val1", "data2": "val2", "data3": 1234}
json_string_data = response.payload.data.decode("UTF-8")    # -> if this is not decoded, json_string_data is a BYTE type

# json_string_data is a STRING, convert to json (or actually dict in python)
json_data = json.loads(json_string_data)    # -> json_data variable is a JSON

# loop the json_data and set the environment variables
for key, value in json_data.items():
    # set the env vars
    os.environ[key] = str(value)

No need to worry about permissions, all apps/services that have been deployed have an ability/permissions to access env/secret data that you passed in irisapp or Google Secret Manager directly and they have a PROJECT_ID environment variable. The PROJECT_ID environment variable is dynamic based on where the app is deployed.
*If you are wondering how all apps / services have permission to access secret, the answer is secretmanager.secretAccessor roles is attached to the resources.

The PROJECT_ID environment variable on each services is set by kubernetes configmap yaml manifest from each service repository and the value in the configmap will be substituted by Jenkinsfile in CI/CD workflow. It would be nice if it uses Workload Identity.

*side notes:
By enabling workload-identity at the cluster level has downtime to the control plane (no editing of the cluster possible but existing workloads are unaffected)
By enabling workload-identity at the Node Pool level recreates nodes (gke automatically cordons and recreates nodes).

Contributing & Run Locally

In order to run this app locally & for development purposes, you need to use your own Google Cloud Platform account for development and generate the JSON key from Service Account.

Requirements to run this app locally & development :

• Google Cloud Platform
• Docker ( version 20+ is recommended )
• Docker Compose
• Linux shell / Bash ( if you're using windows or other os)

Generate Secret Manager Service Account Key

One of the methods to grant access your local machine to GCP in order to run this app is you need to generate Service Account key with Secret Manager Admin roles permission.

If you don't really know how it works, learn how to generate SA JSON key in GCP or you can ask DevOps / Ops team to generate it.

Clone the project

  $ git clone https://github.com/fhmlsml/iris-engine
  
  or

  $ git clone git@github.com:fhmlsml/iris-engine.git

Edit deploy-local.sh and copy & rename your service-acc.json and copy to SA directory

Remember, you need to edit deploy-local.sh this will deploy standalone database in your local system as docker container and after editing those values in deploy-local.sh follow the steps below :

  $ cd iris-engine

  # edit deploy-local.sh
  # copy and rename your service account to SA directory, json key filename must be "sa.json" or you can change from docker-compose-local.yaml file
  $ mv /your/path/to/your/serviceaccount.json ./SA/sa.json

Substitute env var with envsubst and run iris with docker-compose

  $ source deploy-local.sh

  ## run from stdin
  $ envsubst < docker-compose-local.yaml | docker-compose -f - up --build -d

  ## After the db and app containers are running, you need to run database migration for the first time and one time only
  ## Since from docker-compose-local.yaml already mount the volume (./alembic/versions:/app/alembic/versions)
  $ alembic upgrade head

Stop the service

  $ docker-compose -f docker-compose-local.yaml down

  # no need to worry, your volume / persistent data still exist
  # you can check it with `docker volume ls`

Database Migration

If you are modifying the auth/models.py file, you need to run migration with Alembic since FastAPI uses SQLAlchemy. Please keep track of these migrations file in alembic/versions directory to persist the current users and service in irisapp.

After you modify the auth/models.py you have to commit your changes with alembic autogenerate (you have to do this command inside the irisapp container since it's already mounted from docker-compose, the migration file will persist to your local system and ready to push in repository):

$ alembic revision --autogenerate -m "your commit messages"

Then apply the changes :

$ alembic upgrade head

if you encounter any issues, do a rollback :

$ alembic downgrade -1

# or

$ alembic downgrade <identifier> # randomly generated unique identifier, you can check with `alembic history`

CI/CD

This project uses 1 main branch, it will trigger Cloud Build pipeline if create a new tag and it will deploy to dev, stg, and production environment, no need to worry about failure or something.
The image repository for this project is Artifact Registry and the cloud build worker currently managed in x-project.

License

-- license --