Cristian Marius Tiutiu, Bikram Gupta, and Anish Singh Walia
Kubescape is a Kubernetes open-source tool developed by Armosec used for risk analysis, security compliance, RBAC visualizer, and image vulnerabilities scanning. In addition, Kubescape is able to scan Kubernetes manifests to detect potential configuration issues that expose your deployments to the risk of attack. It can also scan Helm charts, detect RBAC (role-based-access-control) violations, performs risk score calculations and shows risk trends over time.
Kubescape key features:
Kubescape can be run in different ways:
Kubescape is using different frameworks to detect misconfigurations such as:
Is Kubescape free?
Yes, the tooling and community edition is free forever, except the cloud portal backend implementation and maybe some other advanced features. There is also a limitation on the maximum number of worker nodes you can scan per cluster (up to 10). Your scan reports data retention is limited to one month in the Armo cloud portal.
See pricing plans for more information.
Is Kubescape open source?
Yes, the tooling for sure is. You can visit the Armo GitHub home page to find more details about each component implementation. The cloud portal backend implementation is not open source.
In this guide you will use Kubescape to perform risk analysis for your Kubernetes applications supply chain (container images, Kubernetes YAML manifests). Then, you will learn how to take the appropriate action to remediate the situation. Finally, you will learn how to integrate Kubescape in a CI/CD pipeline to scan for vulnerabilities in the early stages of development.
To complete all steps from this guide, you will need:
DOKS
cluster running Kubernetes version >=1.21
that you have access to. For additional instructions on configuring a DigitalOcean Kubernetes cluster, see: How to Set Up a DigitalOcean Managed Kubernetes Cluster (DOKS).Kubernetes
interaction. Follow these instructions to connect to your cluster with kubectl
and doctl
.You can manually scan for vulnerabilities via the kubescape
command line interface. The kubescape CLI is designed to be used in various scripts and automations. A practical example is in a CI/CD pipeline implemented using various tools such as Tekton, Jenkins, GitHub Workflows, etc.
Kubescape is designed to scan a whole Kubernetes cluster from ground up (workloads, containers, etc). If desired, you can limit scans to a specific namespace as well. Other features include host scanning (worker nodes), local or remote repositories scanning (e.g. GitHub), detect misconfigurations in Kubernetes YAML manifests or Helm charts. Various frameworks can be selected via the framework
command, such as ArmoBest, NSA, MITRE, etc.
When kubescape CLI is invoked, it will download (or update) the known vulnerabilities database on your local machine. Then, it will start the scanning process and report back issues in a specific format. By default it will print a summary table using the standard output or the console. Kubescape can generate reports in other formats as well, such as JSON, HTML, SARIF, etc.
You can opt to push the results to the Armosec Cloud Portal via the --submit
flag to store and visualize scan results later.
Note: It’s not mandatory to submit scan results to the Armosec cloud portal. The big advantage of using the portal is visibility because it gives you access to a nice dashboard where you can check all scan reports and the overall risk score. It also helps you on the long term with investigations and remediation hints.
Some examples to try with Kubescape CLI:
Scan a whole Kubernetes cluster and generate a summary report in the console (standard output):
kubescape scan
Use a specific namespace only for scanning:
kubescape scan --include-namespaces microservices
Exclude specific namespaces from scanning:
kubescape scan --exclude-namespaces kube-system,kube-public
Scan a specific namespace and submit results to the Armosec cloud portal:
kubescape scan --include-namespaces default --submit
Perform cluster scan using a specific framework (e.g. NSA):
kubescape scan framework nsa --exclude-namespaces kube-system,kube-public
Kubescape is able to scan your Kubernetes cluster hosts (or worker nodes) for OS vulnerabilities as well. To enable this feature, you need to pass the --enable-host-scan
flag to the kubescape CLI. When this flag is enabled, kubescape deploys sensors
in your cluster. Sensors are created using Kubernetes DaemonSets which deploy Pods on each node of your cluster to scan for known vulnerabilities. After the scan process is completed, the sensors are removed from your cluster (including the associated Kubernetes resources).
Kubescape CLI provides help pages for all available options. Below command can be used to print the main help page:
kubescape --help
The output looks similar to:
OutputKubescape is a tool for testing Kubernetes security posture. Docs: https://hub.armo.cloud/docs
Usage:
kubescape [command]
Available Commands:
completion Generate autocompletion script
config Handle cached configurations
delete Delete configurations in Kubescape SaaS version
download Download controls-inputs,exceptions,control,framework,artifacts
help Help about any command
list List frameworks/controls will list the supported frameworks and controls
scan Scan the current running cluster or yaml files
submit Submit an object to the Kubescape SaaS version
version Get current version
...
Each kubescape CLI command (or subcommand) has an associated help page as well which can be accessed via kubescape [command] --help
.
Please visit the official kubescape CLI documentation page for more examples.
Armosec provides a nice cloud based portal where you can upload your Kubescape scan results and perform risk analysis. This is pretty useful because you will want to visualize and inspect each scan report, take the appropriate action to remediate the situation, and then run the scan again to check results. By having a good visual representation for each report and the associated risk score helps you on the long term with the investigations and iterations required to fix the reported security issues.
You can create an account for free limited to 10 worker nodes and 1 month of data retention which should be sufficient in most cases (e.g. for testing or development needs). You can read more about how to create the kubescape
cloud account on the official documentation page.
Once you have the account created, an unique user ID is generated which you can use to upload scan results for that specific account. For example, you may have a specific automation such as a CI/CD pipeline where you need to upload scan results, hence the associated user ID is required to distinguish between multiple tenants.
For each scan report uploaded to your Armosec cloud account, a new history record is added containing the list of issues found and the associated risk score. This way you can get trends and the associated graphs showing risk score evolution over time. Also, a list with topmost security issues is generated as well in the main dashboard.
The below picture illustrates these features:
On each scan, kubescape verifies your resources for potential security risks using internal controls. A Kubescape Control is a concept used by the kubescape
tool to denote the tests used under the hood to check for a particular aspect of your cluster (or resources being scanned). Going further, a framework is a collection of controls or tests used internally to scan your particular resource(s) for issues. So, depending on what framework you use, a different suite of checks is performed (still, some tests share some things in common). Finally, depending on the risk factor associated with each test the final score is computed.
The final score is a positive number ranging from 0 to 100%. A lower value indicates the best score, whereas a higher value indicates the worst. So, if you want to be on the safe side you should aim for the lowest value possible. In practice, a risk score equal to or lower than 30% should be a good starting point.
Another useful feature provided by the Armosec cloud portal is security issues remediation assistance. It means, you receive a recommendation about how to fix each security issue found by the kubescape scanner. This is very important because it simplifies the process and closes the loop for each iteration that you need to perform to fix each reported security issue.
The below picture illustrates this process better:
For each reported security issue, there is a wrench tool icon displayed, which you can click on and get remediation assistance:
Next, a new window opens, giving you details about each affected Kubernetes object, highlighted in green color:
You can click on each control such as C-0018
, C-0030
, C-0086
, etc. and investigate the highlighted issues. You will be presented with suggestions about how to fix each security issue. What’s left is to follow the hints and fix each security issue.
The Armo cloud portal offers the possibility to trigger cluster scans from web interface as well if the Armo cloud components Helm chart is deployed in your DOKS cluster (discussed in the next step). Both configuration and image scanning can be triggered via a one button click in the portal. In order for this feature to work, you need to wait for the Armo cloud components to finish scanning your cluster in the background, and upload the results.
Triggering a configuration scanning is done by navigating to the configuration scanning page and clicking on the Scan button. The below picture shows how to accomplish this task:
You can also set or modify the current schedule for automatic scanning if desired by clicking on the Schedule button in the pop-up window that appears after clicking the Scan button. Using the same window, you can select which control frameworks to use for scanning. The below picture shows how to accomplish the tasks:
Kubescape can be configured to automatically scan your entire Kubernetes cluster at a specific interval of time, or each time a new application image is deployed. You need to deploy Armo Cluster Components in your Kubernetes cluster using Helm to achieve this functionality. An Armosec Cloud Portal account is needed as well to upload and inspect the results.
The Armo Helm chart installs cron jobs that trigger a vulnerability scan both for the entire Kubernetes cluster and container images. Each cron job interval is configurable in the Helm values file.
Steps to deploy kubescape
in your Kubernetes cluster using Helm:
Helm
repo and list the available charts
:helm repo add armo https://armosec.github.io/armo-helm/
helm repo update armo
helm search repo armo
The output looks similar to the following:
OutputNAME CHART VERSION APP VERSION DESCRIPTION
armo/armo-cluster-components 1.7.15 v1.7.15 ARMO Vulnerability Scanning
Note:
The chart of interest is armo/armo-cluster-components
, which will install Armo components in your Kubernetes cluster. Please visit the armo-helm repository page, for more details about this chart.
kubescape config view
The output looks similar to:
{
"accountID": "c952b81f-77d5-4afb-80cc-59b59ec2sdfr"
}
Note:
If you have never used kubescape
CLI to submit scan results to the Armosec cloud portal, the above command won’t work. In this case, you need to log in to the portal and get the account ID from there as explained here.
Helm
- a dedicated armo-system
namespace will be created as well (make sure to replace the <>
placeholders accordingly):ARMO_KUBESCAPE_CHART_VERSION="1.7.15"
helm install armo armo/armo-cluster-components \
--version "$ARMO_KUBESCAPE_CHART_VERSION" \
--namespace armo-system \
--create-namespace \
--set clusterName="$(kubectl config current-context)" \
--set accountGuid=<YOUR_ARMO_ACCOUNT_ID>
Note:
A specific version for the armo-cluster-components
Helm chart is used. In this case, 1.7.15
was picked, which maps to the 1.7.15
release of Armo cluster components (see the output from Step 1.
). It’s good practice, in general, to lock on a specific version. This helps to have predictable results, and allows versioning control via Git
.
Now check if all the Armo cluster components deployments are up and running:
kubectl get deployments -n armo-system
The output looks similar to:
OutputNAME READY UP-TO-DATE AVAILABLE AGE
armo-collector 1/1 1 1 5d6h
armo-kubescape 1/1 1 1 5d6h
armo-notification-service 1/1 1 1 5d6h
armo-vuln-scan 1/1 1 1 5d6h
armo-web-socket 1/1 1 1 5d6h
All Armo cluster components should be up and running.
Finally, after a few minutes, you should be able to see your cluster scan reports available in the cloud portal, such as:
Configuration scanning results:
Image scanning results:
RBAC visualizer results:
For more information, please visit the cluster vulnerability scanning page from the official documentation.
You can change the behavior of the Armo cluster components chart by editing the Helm values file provided in this guide.
The following settings can be changed:
armoScanScheduler
and armoKubescapeScanScheduler
values.triggerNewImageScan
value.The full list of values that can be customized to your needs is available in the official Helm chart values file.
To apply changes, you need to upgrade the current Helm chart version via (make sure to replace the <>
placeholders accordingly):
ARMO_KUBESCAPE_CHART_VERSION="1.7.15"
helm upgrade armo armo/armo-cluster-components \
--version "$ARMO_KUBESCAPE_CHART_VERSION" \
--namespace armo-system \
--set clusterName="$(kubectl config current-context)" \
--set accountGuid=<YOUR_ARMO_ACCOUNT_ID> \
-f <YOUR_CUSTOM_HELM_VALUES_FILE_HERE>
How do you benefit from embedding a security compliance scanning tool in your CI/CD pipeline and avoid unpleasant situations in a production environment?
It all starts at the foundation level, where software development starts. In general, you will want to use a dedicated environment for each stage. So, in the early stages of development when application code changes very often, you should use a dedicated development environment (called the lower environment usually). Then, the application gets more and more refined in the QA environment, where QA teams perform manual and/or automated testing. Next, if the application gets the QA team’s approval, it is promoted to the upper environments, such as staging, and finally into production. In this process, where the application is promoted from one environment to another, a dedicated pipeline runs, which continuously scans application artifacts and computes the security risk score. If the score doesn’t meet a specific threshold, the pipeline fails immediately and application artifacts promotion to production is stopped in the early stages.
So, the security scanning tool (e.g., kubescape) acts as a gatekeeper, stopping unwanted artifacts in your production environment from the early stages of development. In the same manner, upper environments pipelines use kubescape
to allow or forbid application artifacts from entering the final production stage.
In this step, you will learn how to create and test a sample CI/CD pipeline with integrated vulnerability scanning via GitHub workflows. To learn the fundamentals of using Github Actions with DigitalOcean Kubernetes, refer to this tutorial.
The pipeline provided in the following section builds and deploys the game-2048-example application from the DigitalOcean kubernetes-sample-apps repository.
At a high-level overview, the example CI/CD workflow provided in the Kubernetes-sample-apps repo is comprised of the following stages:
The below diagram illustrates each job from the pipeline and the associated steps with actions (only relevant configuration is shown):
Note:
kustomize
-based projects, it’s best to render the final manifest via the kubectl kustomize </path/to/kustomization_/file>
command in order to capture and scan everything (including remote resources). On the other hand, it can be hard to identify which Kubernetes resource needs to be patched. This is due to the fact that the resulting manifest file is comprised of all resources to be applied. This is how Kustomize works - it gathers all configuration fragments from each overlay and applies them over a base to build the final compound.Kubescape
to scan the entire folder where you keep your customize
configurations (the current guide relies on this approach). This way, it’s easier to identify what resource needs to be fixed in your repository. Remote resources used by kustomize need to be fixed upstream. Also, Kubernetes secrets and ConfigMaps generated via kustomize
are not captured.How do you fail the pipeline if a certain security compliance level is not met?
Kubescape CLI provides a flag named --fail-threshold
for this purpose. This flag correlates with the overall risk score computed after each scan. You can fail or pass the pipeline based on the threshold value and stop application deployment if conditions are not met.
The below picture illustrates the flow for the example CI/CD pipeline used in this guide:
Please follow the below steps to create and test the kubescape
CI/CD GitHub workflow provided in the kubernetes-sample-apps GitHub repository:
DIGITALOCEAN_ACCESS_TOKEN
- holds your DigitalOcean account token.DOCKER_REGISTRY
- holds your DigitalOcean docker registry name including the endpoint (e.g. registry.digitalocean.com/sample-apps
).DOKS_CLUSTER
- holds your DOKS cluster name. You can run the following command to get your DOKS cluster name: doctl k8s cluster list --no-header --format Name
.ARMOSEC_PORTAL_ACCOUNT_ID
- holds your Armo portal user account ID - run: kubescape config view
to get the ID. If that doesn’t work you can find more info here.SLACK_WEBHOOK_URL
- holds your Slack incoming webhook URL used for kubescape scan notifications.A new entry should appear in below list after clicking the Run Workflow green button. Select the running workflow to observe pipeline progress:
The pipeline will fail and stop when the kubescape-nsa-security-check job runs. This is expected because the default threshold value of 30
for the overall risk score is lower than the desired value. You should also receive a Slack notification with details about the workflow run:
In the next step, you will learn how to investigate the kubescape
scan report to fix the issues, lower the risk score, and pass the pipeline.
Whenever the risk score value threshold is not met, the game-2048 GitHub workflow will fail and a Slack notification is sent with additional details.
The game-2048 workflow runs one security check (local image scanning is not supported) - Kubernetes manifests misconfiguration checks. The kubescape-nsa-security-check job is used for this purpose. The equivalent kubescape
command being used is - kubescape scan framework nsa /path/to/project/kubernetes/manifests
.
Below snippet shows the main logic of the kubescape-nsa-security-check job:
- name: Scan Kubernetes YAML files
run: |
kubescape scan framework nsa kustomize/ \
-t ${{ github.event.inputs.kubescape_fail_threshold || env.KUBESCAPE_FAIL_THRESHOLD }} \
--submit --account=${{ secrets.ARMOSEC_PORTAL_ACCOUNT_ID }}
working-directory: ${{ env.PROJECT_DIR }}
Above configuration tells kubescape CLI to start a new scan for all Kubernetes manifests present in the kustomize/
directory using the NSA framework. It also specifies what threshold level to use via the -t flag, and to submit final results to Armo cloud portal (the –submit flag in conjunction with –acount).
Thus, lowering the risk score value and passing the workflow consists of investigating and fixing issues reported by the kubescape-nsa-security-check job. Next, you will learn how to address security issues reported by this job.
To check the status report, you can click on the kubescape scan results link from the received Slack notification. Then, click on the REPOSITORIES SCAN scan button from the left menu in the Armo cloud portal. Now, click on the kubernetes-sample-apps entry from the list:
Next, click on the deployment.yaml entry, and then hit the wrench tool from the upper right part:
A new browser window opens showing in detail each control and description. You will also be presented with required actions to remediate the issue (highlighted in green color):
After collecting all the information from the scan report, you can go ahead and edit the deployment.yaml file from your repo (located in the game-2048-example/kustomize/resources
subfolder). The fixes are already in place, you just need to uncomment the last lines from the file. The final deployment.yaml
file should look like below:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: game-2048
spec:
replicas: 1
selector:
matchLabels:
app: game-2048
strategy:
type: RollingUpdate
template:
metadata:
labels:
app: game-2048
spec:
containers:
- name: backend
# Replace the `<>` placeholders with your docker registry info
image: registry.digitalocean.com/sample-apps/2048-game:latest
ports:
- name: http
containerPort: 8080
resources:
requests:
cpu: 100m
memory: 50Mi
limits:
cpu: 200m
memory: 100Mi
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true
allowPrivilegeEscalation: false
capabilities:
drop:
- all
Note : The C-0055 suggestions were omitted in this example for simplicity. You can read more about secure computing mode in Kubernetes here.
What changed? The following security fixes were applied:
readOnlyRootFilesystem
- runs container image in read-only (cannot alter files by kubectl exec
in the container).runAsNonRoot
- runs as the non root user defined by the USER directive from the game-2048 project Dockerfile.allowPrivilegeEscalation
- setting allowPrivilegeEscalation to false ensures that no child process of a container can gain more privileges than its parent.capabilities.drop
- To make containers more secure, you should provide containers with the least amount of privileges it needs to run. In practice, you drop everything by default, then add required capabilities step by step. You can read more about container security in this article written by Armosec.Finally, commit the changes for the deployment.yaml file and push to main branch. After manually triggering the workflow it should complete successfully this time:
You should also receive a green Slack notification from the kubescape
scan job. Navigate to the Armo portal link and check if the issues that you fixed recently are gone - there should be none reported.
A few final checks can be performed as well on the Kubernetes side to verify if the reported issues were fixed:
Check if the game-2048 deployment has a read-only (immutable) filesystem by writing the application index.html file:
kubectl exec -it deployment/game-2048 -n game-2048 -- /bin/bash -c "echo > /public/index.html"
The output looks similar to:
Output/bin/bash: /public/index.html: Read-only file system
command terminated with exit code 1
Check if the container runs as a non-root user (should print an integer number different than zero - e.g., 1000
):
kubectl exec -it deployment/game-2048 -n game-2048 -- id -u
If all checks pass, then you applied the required security recommendations successfully.
There are situations when you don’t want the final risk score to be affected by some reported issues that your team considers safe to ignore. Kubescape offers a built-in feature to manage exceptions and overcome this situation.
You can read more about this feature here.
A more efficient approach is where you integrate vulnerability scan tools directly in your favorite IDE (or Integrated Development Environment). This way, you can detect and fix security issues ahead of time in the software development cycle.
Kubescape offers support for IDE integration via the following extensions:
The above plugins will help you detect and fix issues in the early stages of development, thus eliminating frustration, costs, and security flaws in production systems. Also, it helps you to reduce the iterations and end human effort in the long run. As an example, for each reported security issue by your CI/CD automation you need to go back and fix the issue in your code, commit changes, wait for the CI/CD automation again, then repeat in case of failure.
You can read more about these features by navigating to the Kubescape documentation page, then search in the INTEGRATIONS section.
You can set the workflow to trigger automatically on each commit or PR against the main branch by uncommenting the following lines at the top of the game-2048-kubescape.yaml file:
on:
push:
branches: [ master ]
pull_request:
branches: [ master ]
After editing the file, commit the changes to your main branch, and you should be ready to go.
The vulnerability scan automation you have implemented so far is a good starting point but not perfect. Why?
One issue with the current approach is that you never know when new issues are reported for the assets you already deployed in your environments. In other words, you assessed the security risks and took the measures to fix the issues at one specific point in time - when your CI/CD automation was executed.
But what if new issues are reported meanwhile, and your application is vulnerable again?
The monitoring feature of Kubescape helps you address new vulnerabilities, which are constantly disclosed. When combined with the Slack integration, you can take immediate actions to fix newly disclosed issues that may affect your application in a production environment.
The Armo cloud portal supports Slack integration for sending real-time alerts after each cluster scan. This feature requires the Armo cloud components Helm chart to be installed in your DOKS cluster as explained in Step 3 - Configuring Kubescape Automatic Scans for DOKS.
By enabling Slack alerts, you will get important notifications about vulnerabilities detected in your DOKS cluster, such as:
First, you need to create a Slack App. Then, you need to give the following permissions to your Slack Bot in the OAuth & Permissions page:
channels:join
- Join public channels in a workspace.channels:read
- View basic information about public channels in a workspace.groups:read
- View basic information about private channels your Slack App has been added to.chat:write
- Send messages as @< Your Slack App Name >.im:read
- View basic information about direct messages your Slack App has been added to.mpim:read
- View basic information about group direct messages your Slack App has been added to.Next, navigate to the settings page of your Armo cloud portal account (top right gear icon). From there, select Integrations page, then Slack.
Now, paste your Slack Bot OAuth token (can be found in the OAuth & Permissions page from your Slack App page) in the Insert Token input field. Finally, select how to get notified and the Slack channel where alerts should be sent. Click on Set Notifications button and you’re set. Below picture illustrates the details:
After configuring the Slack integration, you should receive periodic notifications after each cluster scan on the designated channel:
If you receive notifications similar to the above, then you configured the Armosec Kubescape Slack integration successfully.
In this guide, you learned how to use one of the most popular Kubernetes vulnerability scanning tools - Kubescape. You also learned how to perform cluster and repository scanning (YAML manifests) using the Kubescape CLI. Then, you learned how to integrate the vulnerability scanning tool in a traditional CI/CD pipeline implemented using GitHub workflows.
Finally, you learned how to investigate vulnerability scan reports, apply fixes to remediate the situation, and reduce the risk score to a minimum via a practical example - the game-2048 application from the kubernetes-sample-apps repository.
You can learn more by reading the following additional resources:
Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!