Runtime Security

Behavioral Analytics

We can use strace to see syscalls made when running commands in linux, for instance we can run strace -cw ls to display all calls in a table format.

The /proc directory has information and connections to processes and kernel, it contains files that don't exist, yet you can access them, you can check the pid of the service using ps aux | grep etcd for example and you want to access and go to /proc/<pid>/root to access this process files.

Also, you can run strace -p <pid> -f -cw to check process system calls.

Strace Kuberenetes etcd

List syscalls
Find Open Files
Read Secret Values

To list syscalls we can get pid of etcd by running ps aux | grep etcd and then run strace -p <etcd-pid> -cw -f the -f is to follow all forks and -cw to display it as a table format.

For more accurate way to get pid, we can use crictl ps | grep etcd and then crictl inspect <etcd-container-id | grep pid

To find the open files we go to /proc/<etcd-pid> and cd to fd directory which has list of open files.

To read secret values, we need to create a test secret k create secret generic credit-card --from-literal cc=11112222233333 -o yaml > credit-card-secret.yaml

Then to find the secret, we can ls -l to find the symlink that points to the db, then we can run cat 10 | strings | grep 111222 -A 5 -B 5

Create Apache pod with a secret as env variable

Read that secret from the host file system.

We can move to the node that the pod is running on, then get the pid of the httpd service.
After that, we move to /proc/<pid>
And we can see a file called environ, if we run cat on it we can see the list of environment variables the container is using.

Falco

Installing Falco on Worker Node

Important: When installing falco, use the kmod driver installation instead of eBPF, this is needed so that falco.service can start

Go to /etc/falco to check default rules and config files for falco.

Use Falco to find malicious processes inside containers

When we exec into a pod, falco automatically detects this and gives information about it, we used tail -f /var/log/syslog | grep falco to get the information.

Also, here we will write to /etc/passwd and check how falco responds.

Viewing Falco Config

If we view /etc/falco/falco_rules.yaml we can search for common messages such as A shell was spawned to see the rules for it and the conditions.

Also, if we view /etc/falco/k8s_audit_rules.yaml we can see the conditions to check for commands sent to kubeapi-server.

Here we can see the list of paths that falco will watch and send us details about what happened.

Change Falco rule to get custom output format

Edit the falco_rules.yaml file, then copy the rule you wanna edit to falco_rules.local.yaml which overrides the falco_rules.yaml
Replace output values and save.

Immutability of containers at runtime

We must ensure container won't be modified during its lifetime.

Use startup probe to remove `touch` and `bash` from container

Now we will ensure some directories are still writeable using emptyDir volume.

We create an httpd pod with securityContext.readOnlyRootFilesystem set to true.
Now, the pod will crash, due to the read-only restriction (Read-only file system: AH00099: could not create /usr/local/apache2/logs/httpd.pid.Xf2wrE) , so we must add an emptyDir volume.
Now we can't write on system, but we can keep the httpd service run by writing to its directory.

Important: Make sure to use RBAC to ensure certain people can even edit pod specs.

Auditing

Audit logs are important for compliance and know if someone accessed an important secret while it was not protected.

When was last time that user X did access cluster Y?
Does my CRD work proprely?

Each API request can be recorded with an associated "stage", the known stages are:

What data to store?

An example of an audit policy:

We can store logs as JSON in a seperate database server.

Configure api-server to store audit logs in JSON format

Create an audit folder in /etc/kubernetes directory.
Create a policy.yaml inside the audit folder.
Paste a minimal metadata level policy file.
Add the logging directory and path to policy.yaml and mount them as volumes in kube-apiserver definition.

Create a secret and investigate the JSON audit log

We created a test secret called audit-secret and ran cat /var/log/kubernetes/audit/audit.log | grep audit-secret

root@cks-master:/etc/kubernetes/audit# cat /var/log/kubernetes/audit/audit.log | grep -i audit-secret

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"f5760769-b784-4f9b-80fb-7a92727ff8db","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/default/secrets?fieldManager=kubectl-create\u0026fieldValidation=Strict","verb":"create","user":{"username":"kubernetes-admin","groups":["kubeadm:cluster-admins","system:authenticated"]},"sourceIPs":["10.156.0.2"],"userAgent":"kubectl/v1.29.2 (linux/amd64) kubernetes/4b8e819","objectRef":{"resource":"secrets","namespace":"default","name":"audit-secret","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":201},"requestReceivedTimestamp":"2024-06-28T12:55:03.810776Z","stageTimestamp":"2024-06-28T12:55:04.113629Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"kubeadm:cluster-admins\" of ClusterRole \"cluster-admin\" to Group \"kubeadm:cluster-admins\"","mutation.webhook.admission.k8s.io/round_0_index_0":"{\"configuration\":\"gatekeeper-mutating-webhook-configuration\",\"webhook\":\"mutation.gatekeeper.sh\",\"mutated\":false}"}}

Restrict logged data with an audit policy (IMPORTANT)

Restart kube-apiserver.

PreviousSupply Chain Security NextSystem Hardening

Last updated 10 months ago

Runtime Security

Behavioral Analytics

We can use strace to see syscalls made when running commands in linux, for instance we can run strace -cw ls to display all calls in a table format.

Also, you can run strace -p <pid> -f -cw to check process system calls.

Strace Kuberenetes etcd

List syscalls
Find Open Files
Read Secret Values

To list syscalls we can get pid of etcd by running ps aux | grep etcd and then run strace -p <etcd-pid> -cw -f the -f is to follow all forks and -cw to display it as a table format.

For more accurate way to get pid, we can use crictl ps | grep etcd and then crictl inspect <etcd-container-id | grep pid

To find the open files we go to /proc/<etcd-pid> and cd to fd directory which has list of open files.

To read secret values, we need to create a test secret k create secret generic credit-card --from-literal cc=11112222233333 -o yaml > credit-card-secret.yaml

Then to find the secret, we can ls -l to find the symlink that points to the db, then we can run cat 10 | strings | grep 111222 -A 5 -B 5

Create Apache pod with a secret as env variable

Read that secret from the host file system.

We can move to the node that the pod is running on, then get the pid of the httpd service.
After that, we move to /proc/<pid>
And we can see a file called environ, if we run cat on it we can see the list of environment variables the container is using.

Falco

Installing Falco on Worker Node

Important: When installing falco, use the kmod driver installation instead of eBPF, this is needed so that falco.service can start

Go to /etc/falco to check default rules and config files for falco.

Use Falco to find malicious processes inside containers

When we exec into a pod, falco automatically detects this and gives information about it, we used tail -f /var/log/syslog | grep falco to get the information.

Also, here we will write to /etc/passwd and check how falco responds.

Viewing Falco Config

If we view /etc/falco/falco_rules.yaml we can search for common messages such as A shell was spawned to see the rules for it and the conditions.

Also, if we view /etc/falco/k8s_audit_rules.yaml we can see the conditions to check for commands sent to kubeapi-server.

Here we can see the list of paths that falco will watch and send us details about what happened.

Change Falco rule to get custom output format

Edit the falco_rules.yaml file, then copy the rule you wanna edit to falco_rules.local.yaml which overrides the falco_rules.yaml
Go to to check the correct output names.
Replace output values and save.

Immutability of containers at runtime

We must ensure container won't be modified during its lifetime.

Use startup probe to remove `touch` and `bash` from container

We can simply create a startupProbe section, we can take the livenessProbe definition from and change it to startupProbe.

Now we will ensure some directories are still writeable using emptyDir volume.

We create an httpd pod with securityContext.readOnlyRootFilesystem set to true.
Now, the pod will crash, due to the read-only restriction (Read-only file system: AH00099: could not create /usr/local/apache2/logs/httpd.pid.Xf2wrE) , so we must add an emptyDir volume.
Add the emptyDir volume and change the mount path to the path of the which is /usr/local/apache2/logs/.
Now we can't write on system, but we can keep the httpd service run by writing to its directory.

Important: Make sure to use RBAC to ensure certain people can even edit pod specs.

Auditing

Audit logs are important for compliance and know if someone accessed an important secret while it was not protected.

When was last time that user X did access cluster Y?
Does my CRD work proprely?

Each API request can be recorded with an associated "stage", the known stages are:

What data to store?

An example of an audit policy:

We can store logs as JSON in a seperate database server.

Configure api-server to store audit logs in JSON format

Create an audit folder in /etc/kubernetes directory.
Create a policy.yaml inside the audit folder.
Paste a minimal metadata level policy file.
Add the logging directory and path to policy.yaml and mount them as volumes in kube-apiserver definition.

Create a secret and investigate the JSON audit log

We created a test secret called audit-secret and ran cat /var/log/kubernetes/audit/audit.log | grep audit-secret

root@cks-master:/etc/kubernetes/audit# cat /var/log/kubernetes/audit/audit.log | grep -i audit-secret

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"f5760769-b784-4f9b-80fb-7a92727ff8db","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/default/secrets?fieldManager=kubectl-create\u0026fieldValidation=Strict","verb":"create","user":{"username":"kubernetes-admin","groups":["kubeadm:cluster-admins","system:authenticated"]},"sourceIPs":["10.156.0.2"],"userAgent":"kubectl/v1.29.2 (linux/amd64) kubernetes/4b8e819","objectRef":{"resource":"secrets","namespace":"default","name":"audit-secret","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":201},"requestReceivedTimestamp":"2024-06-28T12:55:03.810776Z","stageTimestamp":"2024-06-28T12:55:04.113629Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"kubeadm:cluster-admins\" of ClusterRole \"cluster-admin\" to Group \"kubeadm:cluster-admins\"","mutation.webhook.admission.k8s.io/round_0_index_0":"{\"configuration\":\"gatekeeper-mutating-webhook-configuration\",\"webhook\":\"mutation.gatekeeper.sh\",\"mutated\":false}"}}

Restrict logged data with an audit policy (IMPORTANT)

Restart kube-apiserver.

PreviousSupply Chain Security NextSystem Hardening

Last updated 10 months ago

Behavioral Analytics

Strace Kuberenetes etcd

Create Apache pod with a secret as env variable

Falco

Installing Falco on Worker Node

Use Falco to find malicious processes inside containers

Viewing Falco Config

Change Falco rule to get custom output format

Immutability of containers at runtime

Use startup probe to remove touch and bash from container

Auditing

What data to store?

Configure api-server to store audit logs in JSON format

Create a secret and investigate the JSON audit log

Restrict logged data with an audit policy (IMPORTANT)

Behavioral Analytics

Strace Kuberenetes etcd

Create Apache pod with a secret as env variable

Falco

Installing Falco on Worker Node

Use Falco to find malicious processes inside containers

Viewing Falco Config

Change Falco rule to get custom output format

Immutability of containers at runtime

Use startup probe to remove touch and bash from container

Auditing

What data to store?

Configure api-server to store audit logs in JSON format

Create a secret and investigate the JSON audit log

Restrict logged data with an audit policy (IMPORTANT)

Use startup probe to remove `touch` and `bash` from container

Use startup probe to remove `touch` and `bash` from container