Scan Images for CVEs Prior to Pushing to Image Repository within a CI/CD using Tekton and 

Buildah | Open Container Initiative (OCI)| Trivy | Storage Drivers | CI/CD | Tekton | OpenShift Pipelines | Kubernetes | Vulnerabilities

There is a LOT of power in the Open Container Initiative. The OCI is a governance for the runtime specification and the image specification. This article will reference the Image Specification by example. The image specification output is set up in three main sections. 1. The image Manifest, 2. the Filesystem (layers in serialization format), and 3. Image configuration. You will not need to know everything about this format, but it is a good reference to understand how the Trivy scanner is able to work. So let’s begin.

This article will be utilizing a Tekton Task for building the CI/CD portion. Buildah which is a Redhat/IBM product will be utilized for building our Dockerfile. It is recommended to understand and review Tekton prior to starting this article. Tekton is a Cloud-Native Continuous Integration/Continuous Deployment tool by Google. To learn more about this you can look at the documentation. Ok, let’s create our task.

Step 1. Create the Task for Building, Scanning, and Pushing Up

Create a file called buildah-task.yml and add the following:

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  namespace: <your-namespace>
spec:
  params:
    - default: 'false'
      description: Identifies weather or not to use TLS for the VAPO Quay
      name: VAPO_QUAY_TLSVERIFY
      type: string
    - default: '<your-image-location>'
      description: Identifies the destination image location and tag
      name: TAG_IMAGE_NAME
      type: string
    - default: Add Image Repository
      description: Identifies the Base Images
      name: IMAGE_BASE
      type: string
    - default: Add Image Name
      description: Identifies the Image Name
      name: IMAGE_NAME
      type: string
    - default: 1.0.0
      description: The tag for the image
      name: IMAGE_TAG
      type: string
    - default: >-   registry.redhat.io/rhel8/buildah@sha256:180c4d9849b6ab0e5465d30d4f3a77765cf0d852ca1cb1efb59d6e8c9f90d467
      description: Identifies the Buildah Image
      name: BUILDAH_IMAGE
      type: string
    - default: vfs
      description: Buildah Storage Driver
      name: BUILDAH_STORAGE_DRIVER
      type: string
    - default: /build/<docker-filename>
      description: Identifies the location and name of the dockerfile
      name: DOCKERFILE_NAME
      type: string
    - default: .
      description: Identifies the Docker context root location
      name: DOCKER_CONTEXT_ROOT
      type: string
    - default: oci
      name: STANDARD_IMAGE_FORMAT
      type: string
    - default: /workspace/<repository git pipeline resource name>
      description: Identifies the location of the source code within the workspace
      name: WORKING_DIRECTORY
      type: string
    - default: 'quay.io/ksummersill2/trivy:0.17.2'
      description: Identifies the Trivy Image to use for scanning the OCI spec of the image
      name: TRIVY_IMAGE
      type: string
    - default: '0'
      description: Identifies if Trivy should stop the pipeline
      name: EXIT_CODE
      type: string
    - default: 'os,library'
      description: Identifies the types of vulnerabilities
      name: VUL_TYPE
      type: string
    - default: 'HIGH,CRITICAL'
      description: Identifies the severity level for the Trivy scan
      name: SEVERITY_LVL
      type: string
  resources:
    inputs:
      - description: Identifies the Pipeline Resource to Clone the Source Code
        name: source
        type: git
  steps:
    - image: $(params.BUILDAH_IMAGE)
      name: build-image
      resources: {}
      script: >
        #!/usr/bin/env bash

buildah bud --format=$(params.STANDARD_IMAGE_FORMAT)
        --storage-driver=$(params.BUILDAH_STORAGE_DRIVER) -f
        $(params.DOCKERFILE_NAME) -t app:1.0.0 .

buildah push --format=$(params.STANDARD_IMAGE_FORMAT)
        --storage-driver=$(params.BUILDAH_STORAGE_DRIVER) app:1.0.0
        oci:/workspace/source/image

buildah tag --storage-driver=$(params.BUILDAH_STORAGE_DRIVER) app:1.0.0
        $(params.TAG_IMAGE_NAME)
      volumeMounts:
        - mountPath: /var/lib/containers
          name: varlibcontainers
      workingDir: $(params.WORKING_DIRECTORY)
    - image: 'quay.io/ksummersill2/ubuntu-wget:1.0.0'
      name: get-latest-cve-findings
      resources: {}
      script: >
        mkdir -p /tekton/home/.cache/trivy/db

wget -O /tekton/home/.cache/trivy/db/trivy-offline.db.tgz
        https://github.com/aquasecurity/trivy-db/releases/latest/download/trivy-offline.db.tgz
        --no-check-certificate

cd /tekton/home/.cache/trivy/db

tar xvf trivy-offline.db.tgz
    - image: $(params.TRIVY_IMAGE)
      name: initial-oci-scan
      resources: {}
      script: >
        trivy image --skip-update --input /workspace/source/image
        --exit-code=$(params.EXIT_CODE) --severity=$(params.SEVERITY_LVL)
        --vuln-type=$(params.VUL_TYPE) --format=table >
        /workspace/source/cve-report.txt
      volumeMounts:
        - mountPath: /var/lib/containers
          name: varlibcontainers
      workingDir: $(params.WORKING_DIRECTORY)
    - image: 'quay.io/ksummersill2/ubuntu-wget:1.0.0'
      name: send-cve-report-to-nexus3
      resources: {}
      script: >
        curl -v -u <nexus-username>:<nexus-password> --upload-file
        /workspace/source/cve-report.txt <nexus-repo-location>
    - image: $(params.BUILDAH_IMAGE)
      name: push-image-to-vapo-quay-dev
      resources: {}
      script: >
        buildah push --storage-driver=$(params.BUILDAH_STORAGE_DRIVER)
        --tls-verify=$(params.VAPO_QUAY_TLSVERIFY) $(params.TAG_IMAGE_NAME)
      volumeMounts:
        - mountPath: /var/lib/containers
          name: varlibcontainers
      workingDir: $(params.WORKING_DIRECTORY)
  volumes:
    - emptyDir: {}
      name: varlibcontainers

Now there is A LOT going on here so let's break it down.

There are actually 5 steps within this task. They are 1. Build Image, 2. Get Latest CVE Findings, 3. Initial OCI Scan, 4. Send CVE Report to Nexus, 5. Push Image to Quay. So let's break down each one.

1. Build Image

So this particular step of course builds the image, but if you look close it actually does more.

buildah bud --format=$(params.STANDARD_IMAGE_FORMAT)
        --storage-driver=$(params.BUILDAH_STORAGE_DRIVER) -f
        $(params.DOCKERFILE_NAME) -t app:1.0.0 .

This part of the steps utilizes the VFS driver with OCI format to build the image using the command “bud”. Bud is a command within Buildah that allows the capability to build an image from a “Dockerfile”. The Image then tags to “app:1.0.0”.

Ok create so we have an image but now it must be converted in a particular format.

buildah push --format=$(params.STANDARD_IMAGE_FORMAT)
        --storage-driver=$(params.BUILDAH_STORAGE_DRIVER) app:1.0.0
        oci:/workspace/source/image

This next step does just that. This next part utilizes Buildah push to push the image into the OCI Image Specification format into a specific directory called /workspace/source/image. Yes! that is correct this pushes the entire image of app:1.0.0 into a directory within the Tekton workspace. Just wait it get better.

So to prepare the image to later be pushed it is Tagged.

buildah tag --storage-driver=$(params.BUILDAH_STORAGE_DRIVER) app:1.0.0 $(params.TAG_IMAGE_NAME)

So this step just tags the image. Nothing special. Note, you must specify the Storage Driver or it will NOT find the image. Let's move on.

2. Get Latest CVE Findings

This next part utilizes an AirGap approach with Trivy. If you wish to learn more about the AirGap Approach, then read my article here. https://ksummersill.medium.com/setting-up-trivy-for-airgap-approach-within-ci-cd-3d429da21de4

mkdir -p /tekton/home/.cache/trivy/db
wget -O /tekton/home/.cache/trivy/db/trivy-offline.db.tgz
        https://github.com/aquasecurity/trivy-db/releases/latest/download/trivy-offline.db.tgz
        --no-check-certificate
cd /tekton/home/.cache/trivy/db
tar xvf trivy-offline.db.tgz

This step basically creates a directory called /tekton/home/.cache/trivy/db. This is where the offline Database will be pushed to. Next, the Offline database is grabbed from the Official Trivy Github repo and then pushed into the created directory. Then the Tar file is extracted within that directory for utilization by the next step.

3. Initial OCI Scan

So this is where all the magic occurs. Most CVE scans are conducted after the image is within the Repo. But, wait what if you can scan the image for vulnerabilities prior to pushing up. I mean we can do that with Source Code, why not images?

trivy image --skip-update --input /workspace/source/image
        --exit-code=$(params.EXIT_CODE) --severity=$(params.SEVERITY_LVL)
        --vuln-type=$(params.VUL_TYPE) --format=table >
        /workspace/source/cve-report.txt

Well, the code above does just that. Remember that we utilize the buildah push to send an OCI formatted image using the OCI image specification to /workspace/source/image. Well, guess what! You can actually scan this with Trivy by using:

trivy image --skip-update --input <location of image>

Note you must utilize the skip-update or trivy will try to use the regular approach of pull down the database. The only reason that the AirGap approach was used was because of restrictions within the environment i was working in. So now the input field is the key in order to identify the folder where the image is located.

Now the “exit-code” allows the capability to identify whether or not if “findings” are located to make the CI/CD pipeline fail or to move forward. This is done by using 0 to move forward or 1 to keep going. Of course, you also have the severity levels and vulnerability types to choose from on how this “gate check” is conducted. Severity Levels are Critical, High, Medium, and Lows. Whereas the Vulnerability types are Operating systems (os) or Libraries. This is usually if you have an application push into the image or you want to check for dependencies other than that of the Operating System. Trivy does a great job of this.

4. Pushing the Scan Results to Nexus

Now there is nothing special about this next step except that you can push the results for your security team within a centralized area.

curl -v -u <nexus-username>:<nexus-password> --upload-file
        /workspace/source/cve-report.txt <nexus-repo-location>

5. Finally Push the Image

Once you have verified all the gate checks required, you can now push the image to an image repository.

buildah push --storage-driver=$(params.BUILDAH_STORAGE_DRIVER)
--tls-verify=$(params.VAPO_QUAY_TLSVERIFY) $(params.TAG_IMAGE_NAME)

Well, that is pretty much it. Of course, if you want to run this Task, then that will be done by either a Task Run or Pipeline Run. If you would like me to continue this article on how to implement, then please message me and I will see about continuing how to implement this further. Great! Hope this helped anyone trying to figure out how to scan an image prior to pushing up using OCI compliant Image Specification.