手把手从零搭建 k8s 集群系列（二）kubekey 搭建高可用集群
一、集群规划
1、节点规划
由于使用的虚拟机搭建，电脑资源有限，搭建4台服务器，2台master，2台worker，分别为：
| 系统类型 | IP地址 | 节点角色 | CPU | Memory | 硬盘 | Hostname |
|---|---|---|---|---|---|---|
| CentOS7.9 | 11.0.1.10 | master | 2C | 4G | 40G | k8s-master01 |
| CentOS7.9 | 11.0.1.11 | master | 2C | 4G | 40G | k8s-master01 |
| CentOS7.9 | 11.0.1.20 | worker | 2C | 3G | 40G | k8s-master01 |
| CentOS7.9 | 11.0.1.21 | worker | 2C | 4G | 40G | k8s-master01 |
说明：CPU必须为 2C+，master 内存必须为 4G+，否则安装kubesphere会报错，容器起不来。
2、集群规划
集群搭建规划，这里采用kubesphere kubekey 来进行安装，该方案安装简便快捷，详细安装步骤，请看官方文档：使用 KubeKey 内置 HAproxy 创建高可用集群（v3.3）
确定好kubekey、kubesphere、k8s版本后，其他组件包括(docker、helm) 会自动下载安装：
- kubekey 2.1.0
- kubesphere 3.1.1
- k8s 1.19.9
- Docker 20.10.8
- helm v3.6.3
3、其他（非必须）
虚拟机图形化界面非常耗资源，所以，需要默认开启为非图形化界面：
init 3 # 非图形化界面
init 5 # 图形化界面打开终端输入：vi /etc/inittab
到里面把默认启动级别改为3
# inittab is no longer used when using systemd.
#
# ADDING CONFIGURATION HERE WILL HAVE NO EFFECT ON YOUR SYSTEM.
#
# Ctrl-Alt-Delete is handled by /usr/lib/systemd/system/ctrl-alt-del.target
#
# systemd uses 'targets' instead of runlevels. By default, there are two main targets:
#
# multi-user.target: analogous to runlevel 3
# graphical.target: analogous to runlevel 5
#
# To view current default target, run:
# systemctl get-default
#
# To set a default target, run:
# systemctl set-default TARGET.target
id:3:initdefault:查看版本：
# runlevel
N 3
#grep initdefault /etc/inittab
id:3:initdefault:关闭防火墙、selinux、swap，重置iptables
# 关闭selinux
$ setenforce 0
$ sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config
# 关闭防火墙
$ systemctl stop firewalld && systemctl disable firewalld
# 设置iptables规则
$ iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat && iptables -P FORWARD ACCEPT
# 关闭swap
$ swapoff -a && free –h
# 关闭dnsmasq(否则可能导致容器无法解析域名)
$ service dnsmasq stop && systemctl disable dnsmasq二、安装集群
1、安装yum依赖
在 4 个节点执行以下命令：
yum install -y openssl socat conntrack ipset ebtables chrony ipvsadm2、确定kubekey 版本：
我们使用 kubekey 2.1.0 版本来安装 kubesphere 3.1.1 集群，k8s版本为 1.19.9
./kk version
version.BuildInfo{version:"2.1.0", gitCommit:"8c7bb1d6", GoVersion:"go1.17.9"}
3、下载kubekey（master01节点）
详细安装步骤，请看官方文档：使用 KubeKey 内置 HAproxy 创建高可用集群（v3.3）
mkdir -p /k8s/softwares/
cd /k8s/softwares/
curl -sfL https://get-kk.kubesphere.io | VERSION=v2.1.0 sh -下载：
[root@k8s-master01 softwares]# ls -l
total 70060
-rwxr-xr-x. 1 1001 121 54751232 May 5 20:52 kk
-rw-r--r--. 1 root root 16986290 Sep 10 20:52 kubekey-v2.1.0-linux-amd64.tar.gz为 kk 添加可执行权限：
chmod +x kk创建包含默认配置的示例配置文件。这里使用 Kubernetes v1.19.9作为示例。
./kk create config --with-kubesphere v3.1.1 --with-kubernetes v1.19.9查看生成的配置文件：
[root@k8s-master01 softwares]# ./kk create config --with-kubesphere v3.1.1 --with-kubernetes v1.19.9
[root@k8s-master01 softwares]# ls -l
total 70064
-rw-r--r--. 1 root root 3764 Sep 10 20:56 config-sample.yaml
-rwxr-xr-x. 1 1001 121 54751232 May 5 20:52 kk
-rw-r--r--. 1 root root 16986290 Sep 10 20:52 kubekey-v2.1.0-linux-amd64.tar.gz
[root@k8s-master01 softwares]# 修改后的 config-sample.yaml 配置文件：
[root@k8s-master01 softwares]# cat config-sample.yaml
apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Cluster
metadata:
name: sample
spec:
hosts:
- {name: k8s-master01, address: 11.0.1.10, internalAddress: 11.0.1.10, user: root, password: "123456"}
- {name: k8s-master02, address: 11.0.1.11, internalAddress: 11.0.1.11, user: root, password: "123456"}
- {name: k8s-node01, address: 11.0.1.20, internalAddress: 11.0.1.20, user: root, password: "123456"}
- {name: k8s-node02, address: 11.0.1.21, internalAddress: 11.0.1.21, user: root, password: "123456"}
roleGroups:
etcd:
- k8s-master01
- k8s-master02
control-plane:
- k8s-master01
- k8s-master02
worker:
- k8s-node01
- k8s-node02
controlPlaneEndpoint:
## Internal loadbalancer for apiservers
internalLoadbalancer: haproxy
domain: lb.kubesphere.local
address: ""
port: 6443
kubernetes:
version: v1.19.9
clusterName: cluster.local
autoRenewCerts: true
etcd:
type: kubekey
network:
plugin: calico
kubePodsCIDR: 10.233.64.0/18
kubeServiceCIDR: 10.233.0.0/18
## multus support. https://github.com/k8snetworkplumbingwg/multus-cni
multusCNI:
enabled: false
registry:
privateRegistry: ""
namespaceOverride: ""
registryMirrors: []
insecureRegistries: []
addons: []
---
apiVersion: installer.kubesphere.io/v1alpha1
kind: ClusterConfiguration
metadata:
name: ks-installer
namespace: kubesphere-system
labels:
version: v3.1.1
spec:
persistence:
storageClass: ""
authentication:
jwtSecret: ""
zone: ""
local_registry: ""
etcd:
monitoring: false
endpointIps: localhost
port: 2379
tlsEnable: true
common:
redis:
enabled: false
redisVolumSize: 2Gi
openldap:
enabled: false
openldapVolumeSize: 2Gi
minioVolumeSize: 20Gi
monitoring:
endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090
es:
elasticsearchMasterVolumeSize: 4Gi
elasticsearchDataVolumeSize: 20Gi
logMaxAge: 7
elkPrefix: logstash
basicAuth:
enabled: false
username: ""
password: ""
externalElasticsearchUrl: ""
externalElasticsearchPort: ""
console:
enableMultiLogin: true
port: 30880
alerting:
enabled: false
# thanosruler:
# replicas: 1
# resources: {}
auditing:
enabled: false
devops:
enabled: true
jenkinsMemoryLim: 2Gi
jenkinsMemoryReq: 1500Mi
jenkinsVolumeSize: 8Gi
jenkinsJavaOpts_Xms: 512m
jenkinsJavaOpts_Xmx: 512m
jenkinsJavaOpts_MaxRAM: 2g
events:
enabled: false
ruler:
enabled: true
replicas: 2
logging:
enabled: false
logsidecar:
enabled: true
replicas: 2
metrics_server:
enabled: false
monitoring:
storageClass: ""
prometheusMemoryRequest: 400Mi
prometheusVolumeSize: 20Gi
multicluster:
clusterRole: none
network:
networkpolicy:
enabled: false
ippool:
type: none
topology:
type: none
openpitrix:
store:
enabled: false
servicemesh:
enabled: false
kubeedge:
enabled: false
cloudCore:
nodeSelector: {"node-role.kubernetes.io/worker": ""}
tolerations: []
cloudhubPort: "10000"
cloudhubQuicPort: "10001"
cloudhubHttpsPort: "10002"
cloudstreamPort: "10003"
tunnelPort: "10004"
cloudHub:
advertiseAddress:
- ""
nodeLimit: "100"
service:
cloudhubNodePort: "30000"
cloudhubQuicNodePort: "30001"
cloudhubHttpsNodePort: "30002"
cloudstreamNodePort: "30003"
tunnelNodePort: "30004"
edgeWatcher:
nodeSelector: {"node-role.kubernetes.io/worker": ""}
tolerations: []
edgeWatcherAgent:
nodeSelector: {"node-role.kubernetes.io/worker": ""}
tolerations: []
[root@k8s-master01 softwares]# 开启内置高可用模式（已经配置）：
spec:
controlPlaneEndpoint:
##Internal loadbalancer for apiservers
internalLoadbalancer: haproxy
domain: lb.kubesphere.local
address: ""
port: 6443
4、开始安装
配置完成后，您可以执行以下命令来开始安装：
./kk create cluster -f config-sample.yaml执行过程：
[root@k8s-master01 softwares]# ./kk create cluster -f config-sample.yaml
_ __ _ _ __
| | / / | | | | / /
| |/ / _ _| |__ ___| |/ / ___ _ _
| \| | | | '_ \ / _ \ \ / _ \ | | |
| |\ \ |_| | |_) | __/ |\ \ __/ |_| |
\_| \_/\__,_|_.__/ \___\_| \_/\___|\__, |
__/ |
|___/
21:16:12 PDT [GreetingsModule] Greetings
21:16:12 PDT message: [k8s-node02]
Greetings, KubeKey!
21:16:12 PDT message: [k8s-master02]
Greetings, KubeKey!
21:16:13 PDT message: [k8s-master01]
Greetings, KubeKey!
21:16:13 PDT message: [k8s-node01]
Greetings, KubeKey!
21:16:13 PDT success: [k8s-node02]
21:16:13 PDT success: [k8s-master02]
21:16:13 PDT success: [k8s-master01]
21:16:13 PDT success: [k8s-node01]
21:16:13 PDT [NodePreCheckModule] A pre-check on nodes
21:16:14 PDT success: [k8s-master01]
21:16:14 PDT success: [k8s-master02]
21:16:14 PDT success: [k8s-node02]
21:16:14 PDT success: [k8s-node01]
21:16:14 PDT [ConfirmModule] Display confirmation form
+--------------+------+------+---------+----------+-------+-------+---------+-----------+--------+--------+------------+------------+-------------+------------------+--------------+
| name | sudo | curl | openssl | ebtables | socat | ipset | ipvsadm | conntrack | chrony | docker | containerd | nfs client | ceph client | glusterfs client | time |
+--------------+------+------+---------+----------+-------+-------+---------+-----------+--------+--------+------------+------------+-------------+------------------+--------------+
| k8s-master01 | y | y | y | y | y | y | y | y | y | | | y | | y | PDT 21:16:14 |
| k8s-master02 | y | y | y | y | y | y | y | y | y | | | y | | y | PDT 21:16:14 |
| k8s-node01 | y | y | y | y | y | y | y | y | y | | | y | | y | PDT 21:16:14 |
| k8s-node02 | y | y | y | y | y | y | y | y | y | | | y | | y | PDT 21:16:14 |
+--------------+------+------+---------+----------+-------+-------+---------+-----------+--------+--------+------------+------------+-------------+------------------+--------------+
This is a simple check of your environment.
Before installation, you should ensure that your machines meet all requirements specified at
https://github.com/kubesphere/kubekey#requirements-and-recommendations
Continue this installation? [yes/no]: yes
21:19:14 PDT message: [localhost]
downloading amd64 kubeadm v1.19.9 ...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 37.2M 100 37.2M 0 0 14.6M 0 0:00:02 0:00:02 --:--:-- 14.6M
21:19:17 PDT message: [localhost]
downloading amd64 kubelet v1.19.9 ...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 104M 100 104M 0 0 22.5M 0 0:00:04 0:00:04 --:--:-- 22.5M
21:19:23 PDT message: [localhost]
downloading amd64 kubectl v1.19.9 ...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 40.9M 100 40.9M 0 0 6103k 0 0:00:06 0:00:06 --:--:-- 9950k
21:19:30 PDT message: [localhost]
downloading amd64 helm v3.6.3 ...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 13.0M 100 13.0M 0 0 3473k 0 0:00:03 0:00:03 --:--:-- 3472k
21:19:35 PDT message: [localhost]
downloading amd64 kubecni v0.9.1 ...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
11 37.9M 11 4632k 0 0 564k 0 0:01:08 0:00:08 0:01:00 650k
:19:35 PDT message: [localhost]
downloading amd64 kubecni v0.9.1 ...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 37.9M 100 37.9M 0 0 793k 0 0:00:48 0:00:48 --:--:-- 1994k
21:20:24 PDT message: [localhost]
downloading amd64 crictl v1.22.0 ...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 17.8M 100 17.8M 0 0 5840k 0 0:00:03 0:00:03 --:--:-- 11.8M
21:20:27 PDT message: [localhost]
downloading amd64 etcd v3.4.13 ...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
100 16.5M 100 16.5M 0 0 2155k 0 0:00:07 0:00:07 --:--:-- 2337k
21:20:35 PDT message: [localhost]
downloading amd64 docker 20.10.8 ...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 58.1M 100 58.1M 0 0 26.9M 0 0:00:02 0:00:02 --:--:-- 26.9M
21:20:38 PDT success: [LocalHost]
21:20:38 PDT [ConfigureOSModule] Prepare to init OS
21:20:39 PDT success: [k8s-node01]
21:20:39 PDT success: [k8s-node02]
21:20:39 PDT success: [k8s-master02]
21:20:39 PDT success: [k8s-master01]
21:20:39 PDT [ConfigureOSModule] Generate init os script
21:20:39 PDT success: [k8s-node02]
21:20:39 PDT success: [k8s-master02]
21:20:39 PDT success: [k8s-master01]
21:20:39 PDT success: [k8s-node01]
21:20:39 PDT [ConfigureOSModule] Exec init os script
21:20:41 PDT stdout: [k8s-master02]
Permissive
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
kernel.pid_max = 65535
21:20:41 PDT stdout: [k8s-node01]
Permissive
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
kernel.pid_max = 65535
21:20:41 PDT stdout: [k8s-node02]
Permissive
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1遇到的问题：
1、未关闭防火墙
1.E0106 11:04:19.993753 1 reflector.go:131] pkg/mod/k8s.io/client-go@v0.0.0-20190411052641-7a6b4715b709/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.233.0.1:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.233.0.1:443: connect: no route to host这个应该是容器里10.96.0.1:443不通，一般是节点上开启了防火墙或者其他网络策略, 解决方法，需要再每个节点执行：
[root@k8s-node02 ~] systemctl stop kubelet
[root@k8s-node02 ~] iptables --flush
[root@k8s-node02 ~] iptables -tnat --flush
[root@k8s-node02 ~] systemctl start kubelet
[root@k8s-node02 ~] systemctl start docker
[root@k8s-node02 ~] kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f重启ks-installer
kubectl rollout restart deploy -n kubesphere-system ks-installer
./kk create cluster -f config-sample.yaml2、集群构建失败
The connection to the server lb.kubesphere.local:6443 was refused - did you specify the right host or port?: Process exited with status 1
23:00:58 PDT success: [k8s-master01]
23:00:58 PDT failed: [k8s-master02]
error: Pipeline[CreateClusterPipeline] execute failed: Module[KubernetesStatusModule] exec failed:
failed: [k8s-master02] [GetClusterStatus] exec failed after 3 retires: get kubernetes cluster info failed: Failed to exec command: sudo -E /bin/bash -c "/usr/local/bin/kubectl --no-headers=true get nodes -o custom-columns=:metadata.name,:status.nodeInfo.kubeletVersion,:status.addresses"
The connection to the server lb.kubesphere.local:6443 was refused - did you specify the right host or port?: Process exited with status 1
You have new mail in /var/spool/mail/rootThe connection to the server lb.kubesphere.local:6443 was refused验证：
# 清除集群
# ./kk delete cluster -f config-sample.yaml
# 查看节点信息
kubectl get nodes
# 查看集群信息
kubectl cluster-info问题
TASK [common : debug] **********************************************************
ok: [localhost] => {
"msg": [
"1. check the storage configuration and storage server",
"2. make sure the DNS address in /etc/resolv.conf is available",
"3. execute 'kubectl logs -n kubesphere-system -l job-name=minio-make-bucket-job' to watch logs",
"4. execute 'helm -n kubesphere-system uninstall ks-minio && kubectl -n kubesphere-system delete job minio-make-bucket-job'",
"5. Restart the installer pod in kubesphere-system namespace"
]
}
TASK [common : fail] ***********************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "It is suggested to refer to the above methods for troubleshooting problems ."}首先删除 minio helm del --purge ks-minio
进入installer pod bash kubectl exec -n kubesphere-system ks-installer-xx -it sh
执行minio 安装命令 /usr/local/bin/helm upgrade --install ks-minio /etc/kubesphere/minio-ha -f /etc/kubesphere/custom-values-minio.yaml --set fullnameOverride=minio --namespace kubesphere-system --wait --timeout 1800
观察minio状态，排查为何出错https://kubesphere.com.cn/forum/d/3127-devops
https://kubesphere.com.cn/forum/d/138-2-1-log
5、验证安装
运行以下命令查看安装日志。
kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f
若您看到以下信息，您的高可用集群便已创建成功。
[root@k8s-master01 ~]# kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f
2022-09-10T23:31:14-07:00 INFO : shell-operator latest
2022-09-10T23:31:14-07:00 INFO : Use temporary dir: /tmp/shell-operator
2022-09-10T23:31:14-07:00 INFO : Initialize hooks manager ...
2022-09-10T23:31:14-07:00 INFO : Search and load hooks ...
2022-09-10T23:31:14-07:00 INFO : Load hook config from '/hooks/kubesphere/installRunner.py'
2022-09-10T23:31:14-07:00 INFO : HTTP SERVER Listening on 0.0.0.0:9115
2022-09-10T23:31:15-07:00 INFO : Load hook config from '/hooks/kubesphere/schedule.sh'
2022-09-10T23:31:15-07:00 INFO : Initializing schedule manager ...
2022-09-10T23:31:15-07:00 INFO : KUBE Init Kubernetes client
2022-09-10T23:31:15-07:00 INFO : KUBE-INIT Kubernetes client is configured successfully
2022-09-10T23:31:15-07:00 INFO : MAIN: run main loop
2022-09-10T23:31:15-07:00 INFO : MAIN: add onStartup tasks
2022-09-10T23:31:15-07:00 INFO : QUEUE add all HookRun@OnStartup
2022-09-10T23:31:15-07:00 INFO : Running schedule manager ...
2022-09-10T23:31:15-07:00 INFO : MSTOR Create new metric shell_operator_live_ticks
2022-09-10T23:31:15-07:00 INFO : MSTOR Create new metric shell_operator_tasks_queue_length
2022-09-10T23:31:16-07:00 INFO : GVR for kind 'ClusterConfiguration' is installer.kubesphere.io/v1alpha1, Resource=clusterconfigurations
2022-09-10T23:31:16-07:00 INFO : EVENT Kube event '1554548d-1111-48bd-b877-1747435b2edc'
2022-09-10T23:31:16-07:00 INFO : QUEUE add TASK_HOOK_RUN@KUBE_EVENTS kubesphere/installRunner.py
2022-09-10T23:31:18-07:00 INFO : TASK_RUN HookRun@KUBE_EVENTS kubesphere/installRunner.py
2022-09-10T23:31:18-07:00 INFO : Running hook 'kubesphere/installRunner.py' binding 'KUBE_EVENTS' ...
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
PLAY [localhost] ***************************************************************
TASK [download : include_tasks] ************************************************
skipping: [localhost]
TASK [download : Downloading items] ********************************************
skipping: [localhost]
TASK [download : Synchronizing container] **************************************
skipping: [localhost]
TASK [kubesphere-defaults : KubeSphere | Setting images' namespace override] ***
skipping: [localhost]
TASK [kubesphere-defaults : KubeSphere | Configuring defaults] *****************
ok: [localhost] => {
"msg": "Check roles/kubesphere-defaults/defaults/main.yml"
}
TASK [preinstall : KubeSphere | Checking Kubernetes version] *******************
changed: [localhost]
TASK [preinstall : KubeSphere | Initing Kubernetes version] ********************
ok: [localhost]
TASK [preinstall : KubeSphere | Stopping if Kubernetes version is nonsupport] ***
ok: [localhost] => {
"changed": false,
"msg": "All assertions passed"
}
TASK [preinstall : KubeSphere | Checking StorageClass] *************************
changed: [localhost]
TASK [preinstall : KubeSphere | Stopping if StorageClass was not found] ********
skipping: [localhost]
TASK [preinstall : KubeSphere | Checking default StorageClass] *****************
changed: [localhost]
TASK [preinstall : KubeSphere | Stopping if default StorageClass was not found] ***
ok: [localhost] => {
"changed": false,
"msg": "All assertions passed"
}
TASK [preinstall : KubeSphere | Checking KubeSphere component] *****************
changed: [localhost]
TASK [preinstall : KubeSphere | Getting KubeSphere component version] **********
skipping: [localhost]
TASK [preinstall : KubeSphere | Getting KubeSphere component version] **********
skipping: [localhost] => (item=ks-openldap)
skipping: [localhost] => (item=ks-redis)
skipping: [localhost] => (item=ks-minio)
skipping: [localhost] => (item=ks-openpitrix)
skipping: [localhost] => (item=elasticsearch-logging)
skipping: [localhost] => (item=elasticsearch-logging-curator)
skipping: [localhost] => (item=istio)
skipping: [localhost] => (item=istio-init)
skipping: [localhost] => (item=jaeger-operator)
skipping: [localhost] => (item=ks-jenkins)
skipping: [localhost] => (item=ks-sonarqube)
skipping: [localhost] => (item=logging-fluentbit-operator)
skipping: [localhost] => (item=uc)
skipping: [localhost] => (item=metrics-server)
PLAY RECAP *********************************************************************
localhost : ok=8 changed=4 unreachable=0 failed=0 skipped=7 rescued=0 ignored=0
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
PLAY [localhost] ***************************************************************
TASK [download : include_tasks] ************************************************
skipping: [localhost]
TASK [download : Downloading items] ********************************************
skipping: [localhost]
TASK [download : Synchronizing container] **************************************
skipping: [localhost]
TASK [kubesphere-defaults : KubeSphere | Setting images' namespace override] ***
skipping: [localhost]
TASK [kubesphere-defaults : KubeSphere | Configuring defaults] *****************
ok: [localhost] => {
"msg": "Check roles/kubesphere-defaults/defaults/main.yml"
}
TASK [metrics-server : Metrics-Server | Getting metrics-server installation files] ***
skipping: [localhost]
TASK [metrics-server : Metrics-Server | Creating manifests] ********************
skipping: [localhost] => (item={'file': 'metrics-server.yaml'})
TASK [metrics-server : Metrics-Server | Checking Metrics-Server] ***************
skipping: [localhost]
TASK [metrics-server : Metrics-Server | Uninstalling old metrics-server] *******
skipping: [localhost]
TASK [metrics-server : Metrics-Server | Installing new metrics-server] *********
skipping: [localhost]
TASK [metrics-server : Metrics-Server | Waitting for metrics.k8s.io ready] *****
skipping: [localhost]
TASK [metrics-server : Metrics-Server | Importing metrics-server status] *******
skipping: [localhost]
PLAY RECAP *********************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=11 rescued=0 ignored=0
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
PLAY [localhost] ***************************************************************
TASK [download : include_tasks] ************************************************
skipping: [localhost]
TASK [download : Downloading items] ********************************************
skipping: [localhost]
TASK [download : Synchronizing container] **************************************
skipping: [localhost]
TASK [kubesphere-defaults : KubeSphere | Setting images' namespace override] ***
skipping: [localhost]
TASK [kubesphere-defaults : KubeSphere | Configuring defaults] *****************
ok: [localhost] => {
"msg": "Check roles/kubesphere-defaults/defaults/main.yml"
}
TASK [common : KubeSphere | Checking kube-node-lease namespace] ****************
changed: [localhost]
TASK [common : KubeSphere | Getting system namespaces] *************************
ok: [localhost]
TASK [common : set_fact] *******************************************************
ok: [localhost]
TASK [common : debug] **********************************************************
ok: [localhost] => {
"msg": [
"kubesphere-system",
"kubesphere-controls-system",
"kubesphere-monitoring-system",
"kubesphere-monitoring-federated",
"kube-node-lease",
"kubesphere-devops-system"
]
}
TASK [common : KubeSphere | Creating KubeSphere namespace] *********************
changed: [localhost] => (item=kubesphere-system)
changed: [localhost] => (item=kubesphere-controls-system)
changed: [localhost] => (item=kubesphere-monitoring-system)
changed: [localhost] => (item=kubesphere-monitoring-federated)
changed: [localhost] => (item=kube-node-lease)
changed: [localhost] => (item=kubesphere-devops-system)
TASK [common : KubeSphere | Labeling system-workspace] *************************
changed: [localhost] => (item=default)
changed: [localhost] => (item=kube-public)
changed: [localhost] => (item=kube-system)
changed: [localhost] => (item=kubesphere-system)
changed: [localhost] => (item=kubesphere-controls-system)
changed: [localhost] => (item=kubesphere-monitoring-system)
changed: [localhost] => (item=kubesphere-monitoring-federated)
changed: [localhost] => (item=kube-node-lease)
changed: [localhost] => (item=kubesphere-devops-system)
TASK [common : KubeSphere | Creating ImagePullSecrets] *************************
changed: [localhost] => (item=default)
changed: [localhost] => (item=kube-public)
changed: [localhost] => (item=kube-system)
changed: [localhost] => (item=kubesphere-system)
changed: [localhost] => (item=kubesphere-controls-system)
changed: [localhost] => (item=kubesphere-monitoring-system)
changed: [localhost] => (item=kubesphere-monitoring-federated)
changed: [localhost] => (item=kube-node-lease)
changed: [localhost] => (item=kubesphere-devops-system)
TASK [common : KubeSphere | Labeling namespace for network policy] *************
changed: [localhost]
TASK [common : KubeSphere | Getting Kubernetes master num] *********************
changed: [localhost]
TASK [common : KubeSphere | Setting master num] ********************************
ok: [localhost]
TASK [common : KubeSphere | Getting common component installation files] *******
changed: [localhost] => (item=common)
changed: [localhost] => (item=ks-crds)
TASK [common : KubeSphere | Creating KubeSphere crds] **************************
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/app.k8s.io_applications.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/application.kubesphere.io_helmapplications.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/application.kubesphere.io_helmapplicationversions.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/application.kubesphere.io_helmcategories.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/application.kubesphere.io_helmreleases.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/application.kubesphere.io_helmrepos.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/cluster.kubesphere.io_clusters.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/devops.kubesphere.io_devopsprojects.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/devops.kubesphere.io_pipelines.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/devops.kubesphere.io_s2ibinaries.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/devops.kubesphere.io_s2ibuilders.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/devops.kubesphere.io_s2ibuildertemplates.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/devops.kubesphere.io_s2iruns.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/iam.kubesphere.io_globalrolebindings.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/iam.kubesphere.io_globalroles.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/iam.kubesphere.io_groupbindings.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/iam.kubesphere.io_groups.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/iam.kubesphere.io_loginrecords.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/iam.kubesphere.io_rolebases.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/iam.kubesphere.io_users.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/iam.kubesphere.io_workspacerolebindings.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/iam.kubesphere.io_workspaceroles.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/network.kubesphere.io_ipamblocks.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/network.kubesphere.io_ipamhandles.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/network.kubesphere.io_ippools.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/network.kubesphere.io_namespacenetworkpolicies.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/quota.kubesphere.io_resourcequotas.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/servicemesh.kubesphere.io_servicepolicies.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/servicemesh.kubesphere.io_strategies.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/storage.kubesphere.io_provisionercapabilities.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/storage.kubesphere.io_storageclasscapabilities.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/tenant.kubesphere.io_workspaces.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/tenant.kubesphere.io_workspacetemplates.yaml)
TASK [common : KubeSphere | Creating Storage ProvisionerCapability] ************
changed: [localhost]
TASK [common : KubeSphere | Checking Kubernetes version] ***********************
changed: [localhost]
TASK [common : KubeSphere | Getting common component installation files] *******
changed: [localhost] => (item=snapshot-controller)
TASK [common : KubeSphere | Creating snapshot controller values] ***************
changed: [localhost] => (item={'name': 'custom-values-snapshot-controller', 'file': 'custom-values-snapshot-controller.yaml'})
TASK [common : KubeSphere | Removing old snapshot crd] *************************
changed: [localhost]
TASK [common : KubeSphere | Deploying snapshot controller] *********************
changed: [localhost]
TASK [common : KubeSphere | Checking openpitrix common component] **************
changed: [localhost]
TASK [common : include_tasks] **************************************************
skipping: [localhost] => (item={'op': 'openpitrix-db', 'ks': 'mysql-pvc'})
skipping: [localhost] => (item={'op': 'openpitrix-etcd', 'ks': 'etcd-pvc'})
TASK [common : Getting PersistentVolumeName (mysql)] ***************************
skipping: [localhost]
TASK [common : Getting PersistentVolumeSize (mysql)] ***************************
skipping: [localhost]
TASK [common : Setting PersistentVolumeName (mysql)] ***************************
skipping: [localhost]
TASK [common : Setting PersistentVolumeSize (mysql)] ***************************
skipping: [localhost]
TASK [common : Getting PersistentVolumeName (etcd)] ****************************
skipping: [localhost]
TASK [common : Getting PersistentVolumeSize (etcd)] ****************************
skipping: [localhost]
TASK [common : Setting PersistentVolumeName (etcd)] ****************************
skipping: [localhost]
TASK [common : Setting PersistentVolumeSize (etcd)] ****************************
skipping: [localhost]
TASK [common : KubeSphere | Checking mysql PersistentVolumeClaim] **************
changed: [localhost]
TASK [common : KubeSphere | Setting mysql db pv size] **************************
skipping: [localhost]
TASK [common : KubeSphere | Checking redis PersistentVolumeClaim] **************
changed: [localhost]
TASK [common : KubeSphere | Setting redis db pv size] **************************
skipping: [localhost]
TASK [common : KubeSphere | Checking minio PersistentVolumeClaim] **************
changed: [localhost]
TASK [common : KubeSphere | Setting minio pv size] *****************************
skipping: [localhost]
TASK [common : KubeSphere | Checking openldap PersistentVolumeClaim] ***********
changed: [localhost]
TASK [common : KubeSphere | Setting openldap pv size] **************************
skipping: [localhost]
TASK [common : KubeSphere | Checking etcd db PersistentVolumeClaim] ************
changed: [localhost]
TASK [common : KubeSphere | Setting etcd pv size] ******************************
skipping: [localhost]
TASK [common : KubeSphere | Checking redis ha PersistentVolumeClaim] ***********
changed: [localhost]
TASK [common : KubeSphere | Setting redis ha pv size] **************************
skipping: [localhost]
TASK [common : KubeSphere | Checking es-master PersistentVolumeClaim] **********
changed: [localhost]
TASK [common : KubeSphere | Setting es master pv size] *************************
skipping: [localhost]
TASK [common : KubeSphere | Checking es data PersistentVolumeClaim] ************
changed: [localhost]
TASK [common : KubeSphere | Checking openldap-ha status] ***********************
skipping: [localhost]
TASK [common : KubeSphere | Getting openldap-ha pod list] **********************
skipping: [localhost]
TASK [common : KubeSphere | Getting old openldap data] *************************
skipping: [localhost]
TASK [common : KubeSphere | Migrating openldap data] ***************************
skipping: [localhost]
TASK [common : KubeSphere | Disabling old openldap] ****************************
skipping: [localhost]
TASK [common : KubeSphere | Restarting openldap] *******************************
skipping: [localhost]
TASK [common : KubeSphere | Restarting ks-account] *****************************
skipping: [localhost]
TASK [common : KubeSphere | Importing openldap status] *************************
skipping: [localhost]
TASK [common : KubeSphere | Checking ha-redis] *********************************
changed: [localhost]
TASK [common : KubeSphere | Getting redis installation files] ******************
TASK [common : KubeSphere | Creating manifests] ********************************
changed: [localhost] => (item={'name': 'custom-values-minio', 'file': 'custom-values-minio.yaml'})
TASK [common : KubeSphere | Checking minio] ************************************
changed: [localhost]
TASK [common : KubeSphere | Deploying minio] ***********************************
changed: [localhost]
TASK [common : debug] **********************************************************
ok: [localhost] => {
"msg": [
"1. check the storage configuration and storage server",
"2. make sure the DNS address in /etc/resolv.conf is available",
"3. execute 'kubectl logs -n kubesphere-system -l job-name=minio-make-bucket-job' to watch logs",
"4. execute 'helm -n kubesphere-system uninstall ks-minio && kubectl -n kubesphere-system delete job minio-make-bucket-job'",
"5. Restart the installer pod in kubesphere-system namespace"
]
}
TASK [common : fail] ***********************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "It is suggested to refer to the above methods for troubleshooting problems ."}
PLAY RECAP *********************************************************************
localhost : ok=46 changed=40 unreachable=0 failed=1 skipped=75 rescued=0 ignored=0 安装之后，查看 hosts，kubekey 会自动写入hosts
[root@k8s-master02 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
11.0.1.10 k8s-master01
11.0.1.11 k8s-master02
11.0.1.20 k8s-node01
11.0.1.21 k8s-node02
# kubekey hosts BEGIN
11.0.1.10 k8s-master01.cluster.local k8s-master01
11.0.1.11 k8s-master02.cluster.local k8s-master02
11.0.1.20 k8s-node01.cluster.local k8s-node01
11.0.1.21 k8s-node02.cluster.local k8s-node02
127.0.0.1 lb.kubesphere.local
# kubekey hosts END
You have new mail in /var/spool/mail/root
[root@k8s-master02 ~]# 若您看到以下信息，您的高可用集群便已创建成功。
#####################################################
### Welcome to KubeSphere! ###
#####################################################
Console: http://11.0.1.10:30880
Account: admin
Password: P@88w0rd
NOTES：
1. After you log into the console, please check the
monitoring status of service components in
the "Cluster Management". If any service is not
ready, please wait patiently until all components
are up and running.
2. Please change the default password after login.
#####################################################
https://kubesphere.io 2020-xx-xx xx:xx:xx
#####################################################
登录后台出错：
http://11.0.1.10:30880/login
报这样的错：
request to http://ks-apiserver.kubesphere-system.svc/oauth/token failed, reason: connect ECONNREFUSED 10.233.60.122:80
解决方案,需要暴露ks-api端口：
https://kubesphere.io/zh/docs/v3.3/reference/api-docs/
[root@k8s-master02 ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-7fc49b8c4-g4cws 1/1 Running 3 3h50m
calico-node-6mnkg 1/1 Running 1 3h50m
calico-node-d7z9v 1/1 Running 1 3h50m
calico-node-gxtsw 1/1 Running 1 3h50m
calico-node-h4brp 1/1 Running 1 3h50m
coredns-86cfc99d74-9fprg 1/1 Running 2 3h50m
coredns-86cfc99d74-klwm5 1/1 Running 2 3h50m
haproxy-k8s-node01 1/1 Running 1 3h50m
haproxy-k8s-node02 1/1 Running 1 3h50m
init-pvc-39d10484-b8c3-4de4-9140-6266b43c73b3 0/1 Completed 0 3h33m
kube-apiserver-k8s-master01 1/1 Running 2 3h51m
kube-apiserver-k8s-master02 1/1 Running 3 3h50m
kube-controller-manager-k8s-master01 1/1 Running 9 3h51m
kube-controller-manager-k8s-master02 1/1 Running 9 3h50m
kube-proxy-5z9wk 1/1 Running 1 3h50m
kube-proxy-f9pmb 1/1 Running 1 3h50m
kube-proxy-lkx8h 1/1 Running 1 3h50m
kube-proxy-pv48j 1/1 Running 1 3h50m
kube-scheduler-k8s-master01 1/1 Running 7 3h51m
kube-scheduler-k8s-master02 1/1 Running 8 3h50m
nodelocaldns-ng72s 1/1 Running 2 3h50m
nodelocaldns-qrk2z 1/1 Running 1 3h50m
nodelocaldns-t6dgk 1/1 Running 1 3h50m
nodelocaldns-xhbld 1/1 Running 1 3h50m
openebs-localpv-provisioner-64fb84d4cc-bttbn 1/1 Running 14 3h50m
snapshot-controller-0 1/1 Running 1 3h36m
[root@k8s-master02 ~]# kubectl delete pod coredns-86cfc99d74-klwm5 -n kube-system
pod "coredns-86cfc99d74-klwm5" deleted
[root@k8s-master02 ~]#
[root@k8s-master02 ~]# kubectl delete pod coredns-86cfc99d74-9fprg -n kube-system
pod "coredns-86cfc99d74-9fprg" deleted
[root@k8s-master02 ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-7fc49b8c4-g4cws 1/1 Running 3 3h52m
calico-node-6mnkg 1/1 Running 1 3h52m
calico-node-d7z9v 1/1 Running 1 3h52m
calico-node-gxtsw 1/1 Running 1 3h52m
calico-node-h4brp 1/1 Running 1 3h52m
coredns-86cfc99d74-5dnxc 1/1 Running 0 16s
coredns-86cfc99d74-zgflm 1/1 Running 0 75s
haproxy-k8s-node01 1/1 Running 1 3h52m
haproxy-k8s-node02 1/1 Running 1 3h52m
init-pvc-39d10484-b8c3-4de4-9140-6266b43c73b3 0/1 Completed 0 3h35m
kube-apiserver-k8s-master01 1/1 Running 2 3h53m
kube-apiserver-k8s-master02 1/1 Running 3 3h52m
kube-controller-manager-k8s-master01 1/1 Running 9 3h53m
kube-controller-manager-k8s-master02 1/1 Running 9 3h52m
kube-proxy-5z9wk 1/1 Running 1 3h52m
kube-proxy-f9pmb 1/1 Running 1 3h52m
kube-proxy-lkx8h 1/1 Running 1 3h52m
kube-proxy-pv48j 1/1 Running 1 3h52m
kube-scheduler-k8s-master01 1/1 Running 7 3h53m
kube-scheduler-k8s-master02 1/1 Running 8 3h52m
nodelocaldns-ng72s 1/1 Running 2 3h52m
nodelocaldns-qrk2z 1/1 Running 1 3h52m
nodelocaldns-t6dgk 1/1 Running 1 3h52m
nodelocaldns-xhbld 1/1 Running 1 3h53m
openebs-localpv-provisioner-64fb84d4cc-bttbn 1/1 Running 14 3h52m
snapshot-controller-0 1/1 Running 1 3h38m
[root@k8s-master02 ~]# 问题排查
查看所有pod:
[root@k8s-master01 ~]# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-7fc49b8c4-89gzg 1/1 Running 8 9h
kube-system calico-node-6l4qp 1/1 Running 0 9h
kube-system calico-node-85qfv 1/1 Running 1 9h
kube-system calico-node-t7zmq 1/1 Running 0 9h
kube-system calico-node-ttd98 1/1 Running 0 9h
kube-system coredns-86cfc99d74-5cw5r 1/1 Running 2 9h
kube-system coredns-86cfc99d74-mk6wp 1/1 Running 2 9h
kube-system haproxy-k8s-node01 1/1 Running 0 9h
kube-system haproxy-k8s-node02 1/1 Running 0 9h
kube-system init-pvc-26d095bd-f725-46f0-9674-f86ca91ea26f 0/1 Completed 0 65m
kube-system init-pvc-e08bc790-bb4c-4a30-bac1-c8268c09e8ab 0/1 Completed 0 66m
kube-system kube-apiserver-k8s-master01 1/1 Running 16 9h
kube-system kube-apiserver-k8s-master02 0/1 Running 5 9h
kube-system kube-controller-manager-k8s-master01 0/1 Running 10 9h
kube-system kube-controller-manager-k8s-master02 0/1 Running 10 9h
kube-system kube-proxy-2rs56 1/1 Running 0 9h
kube-system kube-proxy-9272j 1/1 Running 0 9h
kube-system kube-proxy-bd9sx 1/1 Running 0 9h
kube-system kube-proxy-msxzn 1/1 Running 1 9h
kube-system kube-scheduler-k8s-master01 0/1 Running 9 9h
kube-system kube-scheduler-k8s-master02 0/1 CrashLoopBackOff 11 9h
kube-system nodelocaldns-cwj2s 1/1 Running 2 9h
kube-system nodelocaldns-ddvv9 1/1 Running 0 9h
kube-system nodelocaldns-f6vpl 1/1 Running 0 9h
kube-system nodelocaldns-rx8l6 1/1 Running 0 9h
kube-system openebs-localpv-provisioner-64fb84d4cc-ndc5k 0/1 CrashLoopBackOff 20 9h
kube-system snapshot-controller-0 1/1 Running 0 68m
kubesphere-controls-system default-http-backend-76d9fb4bb7-vqgwf 1/1 Running 0 64m
kubesphere-devops-system s2ioperator-0 1/1 Running 0 5m54s
kubesphere-system ks-apiserver-66d5fdd69b-vnc2s 1/1 Running 3 64m
kubesphere-system ks-apiserver-699b6fd77d-xlhwl 1/1 Running 0 5m54s
kubesphere-system ks-console-d6446bd77-s4l8v 1/1 Running 0 64m
kubesphere-system ks-console-d6446bd77-zzszx 1/1 Running 0 64m
kubesphere-system ks-controller-manager-549756ddd7-p2fcg 0/1 CrashLoopBackOff 5 5m54s
kubesphere-system ks-controller-manager-bb447cf6f-gcwgr 0/1 CrashLoopBackOff 6 64m
kubesphere-system ks-installer-66cb7455bb-b66td 1/1 Running 0 69m
kubesphere-system minio-f69748945-758dm 1/1 Running 0 66m
kubesphere-system openldap-0 0/1 Pending 0 66m
kubesphere-system redis-ha-haproxy-75575dcdd7-4jqsm 1/1 Running 11 68m
kubesphere-system redis-ha-haproxy-75575dcdd7-d8l59 1/1 Running 7 68m
kubesphere-system redis-ha-haproxy-75575dcdd7-hzhxl 1/1 Running 0 68m
kubesphere-system redis-ha-server-0 2/2 Running 4 68m
kubesphere-system redis-ha-server-1 0/2 Pending 0 65m
[root@k8s-master01 ~]# kubectl logs -f -n kubesphere-system ks-controller-manager-549756ddd7-p2fcg
Unable to connect to the server: net/http: TLS handshake timeout
You have new mail in /var/spool/mail/root查看资源是否充足：
[root@k8s-master01 ~]# free -h -s 2 -c 2
total used free shared buff/cache available
Mem: 1.8G 1.5G 82M 80M 239M 95M
Swap: 0B 0B 0B
total used free shared buff/cache available
Mem: 1.8G 1.5G 86M 80M 216M 81M
Swap: 0B 0B 0B解决方案
①、将master 内存调至 4G，worker 节点调至 3G+；
[root@k8s-master01 ~]# free -h -s 2 -c 2
total used free shared buff/cache available
Mem: 3.7G 1.8G 232M 15M 1.6G 1.6G
Swap: 0B 0B 0B
total used free shared buff/cache available
Mem: 3.7G 1.8G 232M 15M 1.6G 1.6G
Swap: 0B 0B 0B
[root@k8s-master01 ~]# ②、删除掉启动失败的pod
重启kubesphere的docker后发现ks-controller-manager-748c98775c-dn8th服务一直处于crashloopbackoff 状态。
删除对应的pod后，kubectl会自动重新部署这个服务。
kubectl delete pod openldap-0 -n kubesphere-system
kubectl delete pod redis-ha-server-1 -n kubesphere-system
kubectl delete pod init-pvc-26d095bd-f725-46f0-9674-f86ca91ea26f -n kube-system
kubectl delete pod init-pvc-e08bc790-bb4c-4a30-bac1-c8268c09e8ab -n kube-system
kubectl delete pod ks-controller-manager-5489dc9dd4-sp656 -n kubesphere-system
kubectl delete pod ks-controller-manager-5489dc9dd4-vr5lg -n kubesphere-systemhttps://blog.csdn.net/weixin_41927873/article/details/119177860
然后再查看所有的pod,可以看到所有的pod都启动成功了^_^
[root@k8s-master01 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master01 Ready master 9h v1.19.9
k8s-master02 Ready master 9h v1.19.9
k8s-node01 Ready worker 9h v1.19.9
k8s-node02 Ready worker 9h v1.19.9
[root@k8s-master01 ~]# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-7fc49b8c4-89gzg 1/1 Running 11 9h
kube-system calico-node-6l4qp 1/1 Running 1 9h
kube-system calico-node-85qfv 1/1 Running 2 9h
kube-system calico-node-t7zmq 1/1 Running 1 9h
kube-system calico-node-ttd98 1/1 Running 1 9h
kube-system coredns-86cfc99d74-5cw5r 1/1 Running 3 9h
kube-system coredns-86cfc99d74-mk6wp 1/1 Running 3 9h
kube-system haproxy-k8s-node01 1/1 Running 1 9h
kube-system haproxy-k8s-node02 1/1 Running 1 9h
kube-system kube-apiserver-k8s-master01 1/1 Running 18 9h
kube-system kube-apiserver-k8s-master02 1/1 Running 8 9h
kube-system kube-controller-manager-k8s-master01 1/1 Running 13 9h
kube-system kube-controller-manager-k8s-master02 1/1 Running 12 9h
kube-system kube-proxy-2rs56 1/1 Running 1 9h
kube-system kube-proxy-9272j 1/1 Running 1 9h
kube-system kube-proxy-bd9sx 1/1 Running 1 9h
kube-system kube-proxy-msxzn 1/1 Running 2 9h
kube-system kube-scheduler-k8s-master01 1/1 Running 14 9h
kube-system kube-scheduler-k8s-master02 1/1 Running 15 9h
kube-system nodelocaldns-cwj2s 1/1 Running 3 9h
kube-system nodelocaldns-ddvv9 1/1 Running 1 9h
kube-system nodelocaldns-f6vpl 1/1 Running 1 9h
kube-system nodelocaldns-rx8l6 1/1 Running 1 9h
kube-system openebs-localpv-provisioner-64fb84d4cc-ndc5k 1/1 Running 23 9h
kube-system snapshot-controller-0 1/1 Running 1 117m
kubesphere-controls-system default-http-backend-76d9fb4bb7-vqgwf 1/1 Running 1 112m
kubesphere-controls-system kubectl-admin-69b8ff6d54-cgdgk 1/1 Running 0 9m18s
kubesphere-devops-system ks-jenkins-65db765f86-4bbmj 1/1 Running 0 25m
kubesphere-devops-system s2ioperator-0 1/1 Running 0 25m
kubesphere-monitoring-system alertmanager-main-0 2/2 Running 0 23m
kubesphere-monitoring-system alertmanager-main-1 2/2 Running 0 23m
kubesphere-monitoring-system alertmanager-main-2 2/2 Running 0 23m
kubesphere-monitoring-system kube-state-metrics-67588479db-978hf 3/3 Running 0 24m
kubesphere-monitoring-system node-exporter-57sk7 2/2 Running 0 24m
kubesphere-monitoring-system node-exporter-ds5jq 2/2 Running 0 24m
kubesphere-monitoring-system node-exporter-mjnr7 2/2 Running 0 24m
kubesphere-monitoring-system node-exporter-qntqg 2/2 Running 0 24m
kubesphere-monitoring-system notification-manager-deployment-7bd887ffb4-cqd5f 1/1 Running 0 21m
kubesphere-monitoring-system notification-manager-deployment-7bd887ffb4-k7xln 1/1 Running 0 21m
kubesphere-monitoring-system notification-manager-operator-78595d8666-qrpvl 2/2 Running 0 22m
kubesphere-monitoring-system prometheus-k8s-0 3/3 Running 1 23m
kubesphere-monitoring-system prometheus-k8s-1 3/3 Running 1 23m
kubesphere-monitoring-system prometheus-operator-d7fdfccbf-2q85k 2/2 Running 0 24m
kubesphere-system ks-apiserver-655998d448-9rqpf 1/1 Running 0 20m
kubesphere-system ks-apiserver-655998d448-f7jjf 1/1 Running 0 20m
kubesphere-system ks-console-d6446bd77-s4l8v 1/1 Running 1 112m
kubesphere-system ks-console-d6446bd77-zzszx 1/1 Running 1 112m
kubesphere-system ks-controller-manager-5489dc9dd4-8t6vf 1/1 Running 0 8m19s
kubesphere-system ks-controller-manager-5489dc9dd4-jsxnz 1/1 Running 0 9m1s
kubesphere-system ks-installer-66cb7455bb-b66td 1/1 Running 1 118m
kubesphere-system minio-f69748945-758dm 1/1 Running 1 114m
kubesphere-system openldap-0 1/1 Running 1 14m
kubesphere-system openldap-1 1/1 Running 1 10m
kubesphere-system redis-ha-haproxy-75575dcdd7-4jqsm 1/1 Running 15 116m
kubesphere-system redis-ha-haproxy-75575dcdd7-d8l59 1/1 Running 8 116m
kubesphere-system redis-ha-haproxy-75575dcdd7-hzhxl 1/1 Running 1 116m
kubesphere-system redis-ha-server-0 2/2 Running 9 116m
kubesphere-system redis-ha-server-1 2/2 Running 0 13m
kubesphere-system redis-ha-server-2 2/2 Running 0 11m
[root@k8s-master01 ~]# 登录后端的管理后台：
http://11.0.1.10:30880/clusters/default/overview
账号：admin
密码：K8s@123456
相关文章：
使用 KubeKey 内置 HAproxy 创建高可用集群（v3.3）
知乎|灾备切换|Kubernetes核心架构与高可用集群详解（含100%部署成功的方案）
使用负载均衡器创建高可用集群(v3.1)
KubeKey简介
KubeKey 版本
kubesphere 3.1安装以及简单使用和常见问题解决(一)
kubekey 部署内置 haproxy k8s 高可用集群
kubekey 部署内置 haproxy k8s 高可用集群
为者常成，行者常至
自由转载-非商用-非衍生-保持署名（创意共享3.0许可证）
