> For the complete documentation index, see [llms.txt](https://smadi0x86-blog.gitbook.io/smadi0x86-playground/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://smadi0x86-blog.gitbook.io/smadi0x86-playground/certifications/certified-kubernetes-security/certified-kubernetes-security-4.md).

# Runtime Security

## Behavioral Analytics

We can use strace to see syscalls made when running commands in linux, for instance we can run `strace -cw ls` to display all calls in a table format.

The `/proc` directory has information and connections to processes and kernel, it contains files that don't exist, yet you can access them, you can check the pid of the service using `ps aux | grep etcd` for example and you want to access and go to `/proc/<pid>/root` to access this process files.

Also, you can run `strace -p <pid> -f -cw` to check process system calls.

#### Strace Kuberenetes etcd

* List syscalls
* Find Open Files
* Read Secret Values

To list syscalls we can get pid of etcd by running `ps aux | grep etcd` and then run `strace -p <etcd-pid> -cw -f` the -f is to follow all forks and -cw to display it as a table format.

For more accurate way to get pid, we can use `crictl ps | grep etcd` and then `crictl inspect <etcd-container-id | grep pid`

To find the open files we go to `/proc/<etcd-pid>` and cd to `fd` directory which has list of open files.

To read secret values, we need to create a test secret `k create secret generic credit-card --from-literal cc=11112222233333 -o yaml > credit-card-secret.yaml`

Then to find the secret, we can `ls -l` to find the symlink that points to the db, then we can run `cat 10 | strings | grep 111222 -A 5 -B 5`

#### Create Apache pod with a secret as env variable

Read that secret from the host file system.

* We can move to the node that the pod is running on, then get the pid of the httpd service.
* After that, we move to `/proc/<pid>`
* And we can see a file called `environ`, if we run cat on it we can see the list of environment variables the container is using.

## Falco

![Falco brand | Falco](https://falco.org/img/brand/falco-horizontal-color.svg)

#### Installing Falco on Worker Node

<https://falco.org/docs/install-operate/installation/>

Important: When installing falco, use the `kmod` driver installation instead of `eBPF`, this is needed so that falco.service can start

Go to `/etc/falco` to check default rules and config files for falco.

#### Use Falco to find malicious processes inside containers

When we exec into a pod, falco automatically detects this and gives information about it, we used `tail -f /var/log/syslog | grep falco` to get the information.

Also, here we will write to `/etc/passwd` and check how falco responds.

#### Viewing Falco Config

If we view `/etc/falco/falco_rules.yaml` we can search for common messages such as `A shell was spawned` to see the rules for it and the conditions.

Also, if we view `/etc/falco/k8s_audit_rules.yaml` we can see the conditions to check for commands sent to kubeapi-server.

Here we can see the list of paths that falco will watch and send us details about what happened.

#### Change Falco rule to get custom output format

<https://falco.org/docs/reference/rules/supported-fields/>

* Edit the `falco_rules.yaml file`, then copy the rule you wanna edit to `falco_rules.local.yaml` which overrides the `falco_rules.yaml`
* Go to [documentation ](https://falco.org/docs/reference/rules/supported-fields/)to check the correct output names.
* Replace output values and save.

## Immutability of containers at runtime

We must ensure container won't be modified during its lifetime.

#### Use startup probe to remove `touch` and `bash` from container

We can simply create a startupProbe section, we can take the `livenessProbe` definition from [https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ ](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)and change it to `startupProbe`.

Now we will ensure some directories are still writeable using `emptyDir` volume.

* We create an `httpd` pod with `securityContext.readOnlyRootFilesystem` set to true.
* Now, the pod will crash, due to the read-only restriction (`Read-only file system: AH00099: could not create /usr/local/apache2/logs/httpd.pid.Xf2wrE)` , so we must add an emptyDir volume.
* Add the emptyDir volume and change the mount path to the path of the [httpd.pid](http://httpd.pid) which is `/usr/local/apache2/logs/.`
* Now we can't write on system, but we can keep the `httpd` service run by writing to its directory.

**Important:** Make sure to use RBAC to ensure certain people can even edit pod specs.

## Auditing

Audit logs are important for compliance and know if someone accessed an important secret while it was not protected.

* When was last time that user X did access cluster Y?
* Does my CRD work proprely?

**Each API request can be recorded with an associated "stage", the known stages are:**

#### What data to store?

**An example of an audit policy:**

We can store logs as JSON in a seperate database server.

#### Configure api-server to store audit logs in JSON format

<https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/#audit-policy>

* Create an `audit` folder in `/etc/kubernetes` directory.
* Create a policy.yaml inside the audit folder.
* Paste a minimal metadata level policy file.
* Add the logging directory and path to policy.yaml and mount them as volumes in kube-apiserver definition.

#### Create a secret and investigate the JSON audit log

We created a test secret called `audit-secret` and ran `cat /var/log/kubernetes/audit/audit.log | grep audit-secret`

```
root@cks-master:/etc/kubernetes/audit# cat /var/log/kubernetes/audit/audit.log | grep -i audit-secret

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"f5760769-b784-4f9b-80fb-7a92727ff8db","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/default/secrets?fieldManager=kubectl-create\u0026fieldValidation=Strict","verb":"create","user":{"username":"kubernetes-admin","groups":["kubeadm:cluster-admins","system:authenticated"]},"sourceIPs":["10.156.0.2"],"userAgent":"kubectl/v1.29.2 (linux/amd64) kubernetes/4b8e819","objectRef":{"resource":"secrets","namespace":"default","name":"audit-secret","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":201},"requestReceivedTimestamp":"2024-06-28T12:55:03.810776Z","stageTimestamp":"2024-06-28T12:55:04.113629Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"kubeadm:cluster-admins\" of ClusterRole \"cluster-admin\" to Group \"kubeadm:cluster-admins\"","mutation.webhook.admission.k8s.io/round_0_index_0":"{\"configuration\":\"gatekeeper-mutating-webhook-configuration\",\"webhook\":\"mutation.gatekeeper.sh\",\"mutated\":false}"}}
```

#### Restrict logged data with an audit policy (IMPORTANT)

Restart kube-apiserver.