Skip to content

Latest commit

 

History

History
81 lines (79 loc) · 5.28 KB

running_special_job.md

File metadata and controls

81 lines (79 loc) · 5.28 KB
Raw

Running a Special Job

A recent update to the Hosts table in the Host-Inventory database required updating every host in the table. Making such change was tricky for the Production deployment because it posed a chicken-and-egg problem. The update could be made by executing a job created by a new ClowdJobInvocation (CJI) using a special image that is different from the current image. The problem was that the ClowdApp operator did not allow CJIs to use a different image than the one used by the parent/main clowdapp, host-inventory, and that is the reason for this document.

The problem was solved by hard-coding the new image in a new CJI definition in the Clowdapp template and then updating the SaaS file to explicitly use an IMAGE_TAG for deploying the parent clowdapp, host-inventory.

Steps

  1. Create/update the script as required by the job and create a pull request, which should automatically build a Quay image via continuous integration, e.g. quay.io/cloudservices/insights-inventory:bca5615 
    - name: just-host-synchronizer
  podSpec:
    image: quay.io/cloudservices/insights-inventory:bca5615
    restartPolicy: OnFailure
    command:
      - /bin/sh
      - -c
      - python3 host_synchronizer.py
    env:
      - name: INVENTORY_LOG_LEVEL
        value: ${LOG_LEVEL}
  2. Update the ClowdJobInvocation definition file to use the new job name, just-host-synchronizer. 
        apiVersion: v1
    kind: Template
    metadata:
    name: insights-host-inventory-cjis
    objects:
    - apiVersion: cloud.redhat.com/v1alpha1
    kind: ClowdJobInvocation
    metadata:
        labels:
        app: host-inventory
        name: just-host-synchronizer-${SYNCHRONIZER_RUN_NUMBER}
    spec:
        appName: host-inventory
        jobs:
        - just-host-synchronizer
    parameters:
    - name: SYNCHRONIZER_RUN_NUMBER
    value: '1'
  3. Create a pull request to get these updates into host-inventory.
  4. From the target Kubernetes cluster, get the image tag of the currently deployed service. e.g. a873421. All services in host-inventory use the same image.
  5. Clone the app-interface repo.
  6. Update the host-inventory SaaS file.
    • Change the ref value of the target cluster/namespace with the commit SHA from Step 4. For this write up, the ref in Production cluster was used.
    • Under parameters, set IMAGE_TAG to the one currently deployed in the target cluster, so that running the new job does not affect the current state of the application.
  7. Create a merge request (MR) to get the changes from the previous step merged.
  8. Clone the app-interface repo again to add the new CJI.
  9. Update the CJI in the host-inventory SaaS file to set the ref like in step 7, and the job run number in the CJI definition in the target cluster in the SaaS file.
  10. Create another MR to get the latest changes merged.
  11. After the MR is merged, monitor the new job in a terminal window:
    • connect to the cluster and run 
      $ kubectl get cji -n host-inventory-prod
NAME                        COMPLETED
events-topic-rebuilder      false
just-host-synchronizer-15   false
$
    • Get the pod name: 
      kubectl get pods | grep just-host-synchronizer # note job name in the pod name
host-inventory-just-host-synchronizer-hg4rqrg-9hxh8   0/1   Running   0   10m
    • Check the log: 
      kubectl logs host-inventory-just-host-synchronizer-hg4rqrg-9hxh8 -n host-inventory-prod
  12. After the CJI completes the job execution, expect the following 
        [aarif@damaan insights-host-inventory]$ kubectl get cji -n host-inventory-prod
    NAME                        COMPLETED
    events-topic-rebuilder      false
    just-host-synchronizer-15   true
    $
    $
    $ kubectl get po | grep just-host-synchronizer
    host-inventory-just-host-synchronizer-hg4rqrg-9hxh8   0/1   Completed   0   20h
    $