Kubernetes でどのような操作が行われたか(audit)のログを取得する方法をメモ。

ここでは例として secrets リソースの audit ログを取得してみる。

1. Audit Policy を作成する。

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - "RequestReceived"
rules:
  - level: Metadata
    resources:
    - group: ""
      resources: ["secrets"]
  - level: None

2. kube-apiserver の引数を設定する

次の5行を追加する。

    ...
    - --audit-policy-file=/etc/kubernetes/policies/audit.yaml
    - --audit-log-path=/var/log/kubernetes/audit.log
    - --audit-log-maxage=30
    - --audit-log-maxsize=200
    - --feature-gates=AdvancedAuditing=false
    ...

3. Policy ファイルとログの出力先をマウントする

   ...
   - mountPath: /etc/kubernetes/policies
      name: policies
      readOnly: true
    - mountPath: /var/log/kubernetes
      name: log
    ...

  ...
  - hostPath:
      path: /etc/kubernetes/policies
      type: DirectoryOrCreate
    name: policies
  - hostPath:
      path: /var/log/kubernetes
      type: DirectoryOrCreate
    name: log
  ...

適用し、 kubectl create secret generic test などを実行すると、次のようなログを取得できる。
今回は level に Metadata を指定したので request metadata (ユーザーや時間など) が記録されている。

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "47269892-49b9-47b3-918f-8c885ff89c6d",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/default/secrets",
  "verb": "create",
  "user": {
    "username": "system:serviceaccount:default:mrtc0",
    "uid": "0ef354ff-a8a6-11e9-9bf9-080027c2be11",
    "groups": [
      "system:serviceaccounts",
      "system:serviceaccounts:default",
      "system:authenticated"
    ]
  },
  "sourceIPs": [
    "172.16.20.1"
  ],
  "userAgent": "kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5",
  "objectRef": {
    "resource": "secrets",
    "namespace": "default",
    "name": "test",
    "apiVersion": "v1"
  },
  "responseStatus": {
    "metadata": {},
    "code": 201
  },
  "requestReceivedTimestamp": "2019-07-18T11:04:40.000127Z",
  "stageTimestamp": "2019-07-18T11:04:40.002140Z",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"member\" of ClusterRole \"member\" to ServiceAccount \"mrtc0/default\""
  }
}

level は https://kubernetes.io/docs/tasks/debug-application-cluster/audit/#audit-policy にあるとおり、4種類ある。
Request を指定した場合は次のように Request Body も含まれるようになる。

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Request",
  "auditID": "bb11ab38-50dc-4ecf-a615-48e3508888a8",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/default/secrets",
  "verb": "create",
  "user": {
    "username": "system:serviceaccount:default:mrtc0",
    "uid": "0ef354ff-a8a6-11e9-9bf9-080027c2be11",
    "groups": [
      "system:serviceaccounts",
      "system:serviceaccounts:default",
      "system:authenticated"
    ]
  },
  "sourceIPs": [
    "172.16.20.1"
  ],
  "userAgent": "kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5",
  "objectRef": {
    "resource": "secrets",
    "namespace": "default",
    "name": "test",
    "apiVersion": "v1"
  },
  "responseStatus": {
    "metadata": {},
    "code": 201
  },
  "requestObject": {
    "kind": "Secret",
    "apiVersion": "v1",
    "metadata": {
      "name": "test",
      "creationTimestamp": null
    },
    "type": "Opaque"
  },
  "requestReceivedTimestamp": "2019-07-18T11:05:48.157346Z",
  "stageTimestamp": "2019-07-18T11:05:48.171986Z",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"member\" of ClusterRole \"member\" to ServiceAccount \"mrtc0/default\""
  }
}

RequestResponse だと Response Body も含まれるようになる。

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "RequestResponse",
  "auditID": "48f8db9e-a8e2-4754-b260-160a2291a8ff",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/default/secrets",
  "verb": "create",
  "user": {
    "username": "system:serviceaccount:default:mrtc0",
    "uid": "0ef354ff-a8a6-11e9-9bf9-080027c2be11",
    "groups": [
      "system:serviceaccounts",
      "system:serviceaccounts:default",
      "system:authenticated"
    ]
  },
  "sourceIPs": [
    "172.16.20.1"
  ],
  "userAgent": "kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5",
  "objectRef": {
    "resource": "secrets",
    "namespace": "default",
    "name": "test",
    "apiVersion": "v1"
  },
  "responseStatus": {
    "metadata": {},
    "code": 201
  },
  "requestObject": {
    "kind": "Secret",
    "apiVersion": "v1",
    "metadata": {
      "name": "test",
      "creationTimestamp": null
    },
    "type": "Opaque"
  },
  "responseObject": {
    "kind": "Secret",
    "apiVersion": "v1",
    "metadata": {
      "name": "test",
      "namespace": "default",
      "selfLink": "/api/v1/namespaces/default/secrets/test",
      "uid": "2735fe19-a94c-11e9-97ba-080027c2be11",
      "resourceVersion": "41933",
      "creationTimestamp": "2019-07-18T11:06:55Z"
    },
    "type": "Opaque"
  },
  "requestReceivedTimestamp": "2019-07-18T11:06:55.195890Z",
  "stageTimestamp": "2019-07-18T11:06:55.199958Z",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"member\" of ClusterRole \"member\" to ServiceAccount \"mrtc0/default\""
  }
}

GKEでの設定が https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/gci/configure-helper.sh#L825 にあるので、これを参考に作っていけばよさそう。