# 从Mac机器上添加证书 $ ssh-copy-id root@172.16.132.10 The authenticity of host '172.16.132.10 (172.16.132.10)' can't be established. ECDSA key fingerprint is SHA256:4vwrFA2u0DwO8G0jCN+rqp3A3ZVf1oTIDb+LNG9M334. Are you sure you want to continue connecting (yes/no)? yes /usr/local/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/local/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@172.16.132.10's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'root@172.16.132.10'" and check to make sure that only the key(s) you wanted were added.
# 禁用 selinux $ setenforce 0 $ vim /etc/selinux/config SELINUX=disabled
$ docker version Client: Version: 1.12.6 API version: 1.24 Package version: docker-1.12.6-68.gitec8512b.el7.centos.x86_64 Go version: go1.8.3 Git commit: ec8512b/1.12.6 Built: Mon Dec 11 16:08:42 2017 OS/Arch: linux/amd64 $ docker info Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 0 Server Version: 1.12.6 Storage Driver: devicemapper Pool Name: docker-8:3-34809708-pool Pool Blocksize: 65.54 kB Base Device Size: 10.74 GB Backing Filesystem: xfs Data file: /dev/loop0 Metadata file: /dev/loop1 Data Space Used: 11.8 MB Data Space Total: 107.4 GB Data Space Available: 14.57 GB Metadata Space Used: 581.6 kB Metadata Space Total: 2.147 GB Metadata Space Available: 2.147 GB Thin Pool Minimum Free Space: 10.74 GB Udev Sync Supported: true Deferred Removal Enabled: true Deferred Deletion Enabled: true Deferred Deleted Device Count: 0 Data loop file: /var/lib/docker/devicemapper/devicemapper/data WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device. Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata Library Version: 1.02.140-RHEL7 (2017-05-03) Logging Driver: journald Cgroup Driver: systemd # 默认为systemd，不需要单独设置，否则在/etc/docker/daemon.json中设置 Plugins: Volume: local Network: host bridge overlay null Swarm: inactive Runtimes: docker-runc runc Default Runtime: docker-runc Security Options: seccomp selinux Kernel Version: 3.10.0-693.el7.x86_64 Operating System: CentOS Linux 7 (Core) OSType: linux Architecture: x86_64 Number of Docker Hooks: 3 CPUs: 1 Total Memory: 976.3 MiB Name: master ID: W4NY:E6NL:NV37:46ME:GYTH:Q3P7:VQKC:ONYH:YOIR:4TRK:6CAO:XBRB Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Insecure Registries: 127.0.0.0/8 Registries: docker.io (secure)
On each of your machines, install Docker. Version v1.12 is recommended, but v1.11, v1.13 and 17.03 are known to work as well. Versions 17.06+ might work, but have not yet been tested and verified by the Kubernetes node team.
# docker 1.12.6 中，不需要设置该参数 cmd 中已经包含 # 使用系统默认支持的方法 # Note: Make sure that the cgroup driver used by kubelet is the same as the one used by Docker. # To ensure compatability you can either update Docker, like so: $ mkdir -p /etc/docker/ $ cat << EOF > /etc/docker/daemon.json { "exec-opts": ["native.cgroupdriver=systemd"] } EOF
$ yum install -y kubelet kubeadm kubectl Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.163.com Resolving Dependencies --> Running transaction check ---> Package kubeadm.x86_64 0:1.9.1-0 will be installed --> Processing Dependency: kubernetes-cni for package: kubeadm-1.9.1-0.x86_64 ---> Package kubectl.x86_64 0:1.9.1-0 will be installed ---> Package kubelet.x86_64 0:1.9.1-0 will be installed --> Processing Dependency: socat for package: kubelet-1.9.1-0.x86_64 --> Running transaction check ---> Package kubernetes-cni.x86_64 0:0.6.0-0 will be installed ---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed --> Finished Dependency Resolution
Dependencies Resolved
Package Arch Version Repository Size ========================================================= Installing: kubeadm x86_64 1.9.1-0 kubernetes 16 M kubectl x86_64 1.9.1-0 kubernetes 8.9 M kubelet x86_64 1.9.1-0 kubernetes 17 M Installing for dependencies: kubernetes-cni x86_64 0.6.0-0 kubernetes 8.6 M socat x86_64 1.7.3.2-2.el7 base 290 k
$ kubeadm init --kubernetes-version=v1.9.1 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=172.16.132.10 --ignore-preflight-errors=Swap > install.log 2>&1 # 安装的信息 [init] Using Kubernetes version: v1.9.1 [init] Using Authorization modes: [Node RBAC] [preflight] Running pre-flight checks. [WARNING Swap]: running with swap on is not supported. Please disable swap [WARNING FileExisting-crictl]: crictl not found in system path [preflight] Starting the kubelet service [certificates] Generated ca certificate and key. [certificates] Generated apiserver certificate and key. [certificates] apiserver serving cert is signed for DNS names [node1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.16.132.10] [certificates] Generated apiserver-kubelet-client certificate and key. [certificates] Generated sa key and public key. [certificates] Generated front-proxy-ca certificate and key. [certificates] Generated front-proxy-client certificate and key. [certificates] Valid certificates and keys now exist in"/etc/kubernetes/pki" [kubeconfig] Wrote KubeConfig file to disk: "admin.conf" [kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf" [kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf" [kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf" [controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml" [controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml" [controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml" [etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml" [init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests". [init] This might take a minute or longer if the control plane images have to be pulled.
[apiclient] All control plane components are healthy after 31.501968 seconds [uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config"in the "kube-system" Namespace [markmaster] Will mark node node1 as master by adding a label and a taint [markmaster] Master node1 tainted and labelled with key/value: node-role.kubernetes.io/master="" [bootstraptoken] Using token: 047b97.bf92a2b4e89d9e0b [bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstraptoken] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [addons] Applied essential addon: kube-dns [addons] Applied essential addon: kube-proxy
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node as root:
$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE gcr.io/google_containers/kube-apiserver-amd64 v1.9.1 e313a3e9d78d 10 days ago 210.4 MB gcr.io/google_containers/kube-scheduler-amd64 v1.9.1 677911f7ae8f 10 days ago 62.7 MB gcr.io/google_containers/kube-controller-manager-amd64 v1.9.1 4978f9a64966 10 days ago 137.8 MB gcr.io/google_containers/kube-proxy-amd64 v1.9.1 e470f20528f9 10 days ago 109.1 MB gcr.io/google_containers/etcd-amd64 3.1.10 1406502a6459 4 months ago 192.7 MB gcr.io/google_containers/pause-amd64 3.0 99e59f495ffa 20 months ago 746.9 kB
安装中遇到的两个问题解释：
1 2 3 4
[WARNING Swap]: running with swap on is not supported. Please disable swap ---- 我们前面已经通过 kubelet 的命令行来指定 [WARNING FileExisting-crictl]: crictl not found in system path ---- crictl 工具是go开发的工具包，需要单独安装，但是安装成功后也遇到了不能解决的问题，因此可以忽略这个错误
The network must be deployed before any applications. Also, kube-dns, an internal helper service, will not start up before a network is installed. kubeadm only supports Container Network Interface (CNI) based networks (and does not support kubenet).
$ mkdir -p ~/k8s/ $ cd ~/k8s $ wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml $ kubectl apply -f kube-flannel.yml clusterrole "flannel" created clusterrolebinding "flannel" created serviceaccount "flannel" created configmap "kube-flannel-cfg" created daemonset "kube-flannel-ds" created
6: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN link/ether 9e:a4:0e:29:5c:cf brd ff:ff:ff:ff:ff:ff inet 10.244.0.0/32 scope global flannel.1 valid_lft forever preferred_lft forever inet6 fe80::9ca4:eff:fe29:5ccf/64 scope link valid_lft forever preferred_lft forever 7: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP qlen 1000 link/ether 0a:58:0a:f4:00:01 brd ff:ff:ff:ff:ff:ff inet 10.244.0.1/24 scope global cni0 valid_lft forever preferred_lft forever inet6 fe80::c83d:76ff:fe49:6232/64 scope link valid_lft forever preferred_lft forever
$ brctl show bridge name bridge id STP enabled interfaces cni0 8000.0a580af40001 no veth7f53d148 docker0 8000.0242eed2f41f no virbr0 8000.52540096686f yes virbr0-nic
如果过程中遇到问题可以使用以下命令清除设置的相关网络设备与运行时候的配置文件：
1 2 3 4 5
$ ifconfig cni0 down $ ip link delete cni0 $ ifconfig flannel.1 down $ ip link delete flannel.1 $ rm -rf /var/lib/cni/
$ cat nginx-deployment.yaml apiVersion: apps/v1beta2 # for versions before 1.8.0 use apps/v1beta1 kind: Deployment metadata: name: nginx-deployment spec: selector: matchLabels: app: nginx replicas: 2 # tells deployment to run 2 pods matching the template template: # create pods using pod definition in this template metadata: # unlike pod-nginx.yaml, the name is not included in the meta data as a unique name is # generated from the deployment name labels: app: nginx spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80
[preflight] Running pre-flight checks. [WARNING FileExisting-crictl]: crictl not found in system path [discovery] Trying to connect to API Server "172.16.132.10:6443" [discovery] Created cluster-info discovery client, requesting info from "https://172.16.132.10:6443" [discovery] Requesting info from "https://172.16.132.10:6443" again to validate TLS against the pinned public key [discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "172.16.132.10:6443" [discovery] Successfully established connection with API Server "172.16.132.10:6443"
This node has joined the cluster: * Certificate signing request was sent to master and a response was received. * The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the master to see this node join the cluster.
Data ==== ca.crt: 1025 bytes namespace: 11 bytes token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbi10b2tlbi10c3pqNSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImRkMWE1ODJhLWY5Y2QtMTFlNy04NWVjLTAwMGMyOTc1YmU4MSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTprdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbiJ9.iuqjGsXy9zohzCDLSCpd4RqUyFptSZ_Al8qEpGb_D46Gfscb8DvV24qLR6QF5ejZKh_3Oe4g42GRROLsy_he8Exlxs86YDA5505QptNMDcNOqJqvlk6y8hovLl8gIu6K70ND4q_i9pIxWLDOOUuYLuDO1re3Z0rUa0jZimXiayBXUjuzbJJYYlHL9SREIjxr4y1FTsFFnbZESCYmMNKcQSwhYyTrSyPA8XiiUm_k4aYVtvWqo84nRyxreZ7DH6Zg7YT57oy8DqXHC-GNXFGj7tmDFWzih1GFvTuFp0zqhkjtS1ZAFsSNLIvIwBhg7Aj-6LyDBE4RSUOJg5UiH2trYA
查询 dashboard 暴露出来的 NodePort，并使用上图的 token 信息输入：
1 2 3 4
$ kubectl get service -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 2h kubernetes-dashboard NodePort 10.111.165.62 <none> 443:31290/TCP 5m
$ kubectl get service -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE .... monitoring-grafana NodePort 10.103.229.66 <none> 80:32447/TCP 18m ...
$ wget https://dl.google.com/go/go1.9.2.linux-amd64.tar.gz $ tar -C /usr/local -xzf go1.9.2.linux-amd64.tar.gz $ export PATH=$PATH:/usr/local/go/bin $ go version go version go1.9.2 linux/amd64 $ yum install git -y $ mkdir -p $HOME/go/src $ export GOPATH=$HOME/go $ go get github.com/kubernetes-incubator/cri-tools $ cd$GOAPTH/src/github.com/kubernetes-incubator/cri-toolscri-tools/ && make $ cp $GOPATH/bin/crictl /usr/local/bin
1 2 3 4 5 6 7 8
$ kubeadm init --kubernetes-version=v1.9.1 --pod-network-cidr=10.0.0.0/16 --apiserver-advertise-address=172.16.132.10 [init] Using Kubernetes version: v1.9.1 [init] Using Authorization modes: [Node RBAC] [preflight] Running pre-flight checks. [preflight] Some fatal errors occurred: [ERROR Swap]: running with swap on is not supported. Please disable swap [ERROR CRI]: unable to check if the container runtime at "/var/run/dockershim.sock" is running: exit status 1 [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
$ kubeadm init --kubernetes-version=v1.9.1 --pod-network-cidr=10.0.0.0/16 --apiserver-advertise-address=172.16.132.10 --ignore-preflight-errors=Swap [init] Using Kubernetes version: v1.9.1 [init] Using Authorization modes: [Node RBAC] [preflight] Running pre-flight checks. [preflight] Some fatal errors occurred: [ERROR CRI]: unable to check if the container runtime at "/var/run/dockershim.sock" is running: exit status 1 [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
1 2 3 4 5 6 7 8 9
$ crictl ps
2018/01/10 11:16:54 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix /var/run/dockershim.sock: connect: no such file or directory"; Reconnecting to {/var/run/dockershim.sock <nil>}
FATA[0000] listing containers failed: rpc error: code = Unavailable desc = grpc: the connection is unavailable
CRI is complaining about the lack of the shim sock, which gets checked during kubeadm initpreflight: [ERROR CRI]: unable to check if the container runtime at "/var/run/dockershim.sock" is running: exit status 1. I don’t think this is crictl because kubeadm still complains about it even without crictl installed. FYI, CRI is a kubelet concept which is exposed as a JSON API, crictl/cri-tools is just a CLI to access kubelet’s CRI API.
Also, kubeadm only warns with the lack of crictl, but doesn’t require it. So, the additional dependencies only come into play when you want to use a different container runtime interface other than the dockershim that ships with kubelet. This is where you might use cri-o (formerly ocid), which must be manually built and additionally depends on runc.
Aside: on centos 7, runc v1.0.0 is available via extras, so a yum install runc should do the trick. i’m not sure about other centos versions or distros.
running with swap on is not supported
一个警告信息是 crictl not found in system path，另一个错误信息是 running with swap on is not supported. Please disable swap。因为我们前面已经修改了kubelet的启动参数，所以重新添加 –ignore-preflight-errors=Swap 参数忽略这个错误，重新运行。或者使用 swapoff -a 临时全局关闭，可能会影响运行的服务。
Jan 09 17:50:56 localhost.localdomain systemd[1]: Starting Docker Application Container Engine... Jan 09 17:50:56 localhost.localdomain dockerd-current[41393]: time="2018-01-09T17:50:56+08:00" level=fatal msg="unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: exec-opts: (from flag: [native.cgroupdriver=systemd], from file: [native.cgroupdriver=systemd])\n" Jan 09 17:50:56 localhost.localdomain systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE Jan 09 17:50:56 localhost.localdomain systemd[1]: Failed to start Docker Application Container Engine. Jan 09 17:50:56 localhost.localdomain systemd[1]: Unit docker.service entered failed state. Jan 09 17:50:56 localhost.localdomain systemd[1]: docker.service failed.
system:anonymous” cannot get path
不能访问 API-Server 遇到 system:anonymous" cannot get path:
# 设置 pod-reader role $ cat pod-reader.yaml kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: namespace: default name: pod-reader rules: - apiGroups: [""] # "" indicates the core API group resources: ["pods"] verbs: ["get", "watch", "list"]
# 绑定 role 到 user diwh $ cat roleBinding_diwh.yaml kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: read-pods namespace: default subjects: - kind: User name: diwh apiGroup: rbac.authorization.k8s.io roleRef: kind: Role name: pod-reader apiGroup: rbac.authorization.k8s.io
# 连接使用 # 不具备 configmap，期望失败 $ kubectl --server=https://172.16.132.100:6443 --certificate-authority=ca.crt --client-certificate=diwh.crt --client-key=diwh.key get configmap Error from server (Forbidden): configmaps is forbidden: User "diwh" cannot list configmaps in the namespace "default"
# 访问 pod，期望成功 $ kubectl --server=https://172.16.132.100:6443 --certificate-authority=ca.crt --client-certificate=diwh.crt --client-key=diwh.key get pods NAME READY STATUS RESTARTS AGE curl-545bbf5f9c-g28fr 1/1 Running 0 1d nginx-deployment-6c54bd5869-8t2xt 1/1 Running 0 1d nginx-deployment-6c54bd5869-hh2ft 1/1 Running 2 1d
