K8S kubeadm 구축

1. Ubuntu 설치

• master node 1, worker node 2
• 최소 2core 이상
• OS - Ubuntu 20.04

## 기본 패키지 설치 & 방화벽 해제
sudo apt-get install -y openssh-server curl vim tree net-tools

systemctl stop firewalld
systemctl disable firewalld

## docker 설치
# 1) 필요 패키지 설치
sudo apt-get update
sudo apt install -y \
apt-transport-https \
ca-certificates \
curl \
gnupg \
lsb-release 

# 2) gpg key 발급
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

# 3) gpg key 저장
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# 4) docker 설치
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io
sudo systemctl enable docker
sudo systemctl start docker

# 5) docker 설치 확인
docker version
Client: Docker Engine - Community
 Version:           20.10.21
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        baeda1f
 Built:             Tue Oct 25 18:04:24 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.21
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       3056208
  Built:            Tue Oct 25 18:02:38 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.10
  GitCommit:        770bd0108c32f3fb5c73ae1264f7e503fe7b2661
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2022-11-21 14:41:38 KST; 2 weeks 0 days ago
     Docs: https://docs.docker.com
 Main PID: 30276 (dockerd)
    Tasks: 22
   Memory: 1.2G
   CGroup: /system.slice/docker.service
           └─30276 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

## kubernetes 설치를 위한 설정 및 패키지 설치
# 1) swap off (root 계정)
swapoff -a && sed -i '/swap/s/^/#/' /etc/fstab

# 2) 네트워크 환경 설정 : bridge network listen 가능하도록 환경 설정
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system

# 3) kubeadm, kubectl, kubelet 설치
sudo apt-get update && sudo apt-get install -y apt-transport-https ca-certificates curl
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update && sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

sudo systemctl daemon-reload
sudo systemctl restart kubelet

sudo systemctl start kubelet
sudo systemctl enable kubelet

# 4) docker daemon.json 파일 설정
cat <<EOF | sudo tee /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

sudo systemctl enable docker
sudo systemctl daemon-reload
sudo systemctl restart docker

####### master node에서 실행 #######
# control-palne 설치(master node에 해당)
# 1) 설치 및 cluster 구성을 위한 key 생성
sudo kubeadm init
~
~
kubeadm join 10.100.0.10:6443 --token wn8ngf.64qva8505daau8bl \
        --discovery-token-ca-cert-hash sha256:e1c33f1403aad434b37316d1cfffea8d8a5739cbd98246c2ac9df2653862e001
## kubeadm init 실행 후 가장 마지막에 worker node join을 위한 key가 위와 같은 형태로 나옴
## 해당 key를 따로 저장 및 복사해 둘 것!

## kubeadm init 트러블 슈팅 ## 

## 에러 메시지
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR CRI]: container runtime is not running: output: time="2020-11-25T12:58:32Z" level=fatal msg="getting status of runtime failed: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

sudo rm /etc/containerd/config.toml
sudo systemctl restart containerd
sudo kubeadm reset
sudo kubeadm init

################################

# 2) 환경 설정 
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
export KUBECONFIG=/etc/kubernetes/admin.conf

####### worker node에서 실행 #######
# worker node join
# 앞서 복사해둔 key 입력
kubeadm join 10.100.0.10:6443 --token wn8ngf.64qva8505daau8bl \
        --discovery-token-ca-cert-hash sha256:e1c33f1403aad434b37316d1cfffea8d8a5739cbd98246c2ac9df2653862e001

# CNI 설치
## CNI 종류는 다양하고 각각 장단점이 있으므로 각자 환경에 맞게 설치할 것
## 본 문서에서는 calico 설치
## calico 외에 weavenet, flannel 등이 있다

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/tigera-operator.yaml
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/custom-resources.yaml

# 치트시트
apt install -y bash-completion
echo 'source <(kubectl completion bash)' >>~/.bashrc
echo 'alias k=kubectl' >>~/.bashrc
echo 'complete -o default -F __start_kubectl k' >>~/.bashrc
source ~/.bashrc

or

kubectl completion bash | sudo tee /etc/bash_completion.d/kubectl > /dev/null

## kubernetes 설치 확인
kubectl get nodes
NAME      STATUS   ROLES    AGE   VERSION
master   Ready    <none>   14d   v1.23.0
worker1   Ready    <none>   14d   v1.23.0
kisti-5   Ready    <none>   14d   v1.23.0

 

2. CentOS 설치

• master node 1, worker node 2
• 최소 2core 이상
• OS - CentOS 7.8

## os에 필요한 기본 설치
sudo yum install -y tar bzip2 make automake gcc gcc-c++ \
pciutils elfutils-libelf-devel libglvnd-devel \
iptables firewalld bind-utils \
vim wget
sudo yum update -y

## yum kubernetes 저장소 추가
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

## permissive 모드로 SELinux 설정(효과적으로 비활성화)
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

## swap 끄기 && 방화벽 끄기(규칙 추가 대신)
sudo swapoff -a
sudo systemctl disable firewalld

## iptables 설정
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

## 도커 설치1
curl -fsSL https://get.docker.com | sudo sh
#--------or 도커 설치2
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install -y docker-ce

systemctl start docker
systemctl enable docker

## 도커 cgroup systemd로 변경
# 현재 cgroup 확인
docker info | grep -i cgroup
Cgroup Driver: cgroupfs
Cgroup Version: 1

# daemon.json 값을 통해 cgroup systemd로 변경
vim /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"]
}

# docker restart
systemctl restart docker

## kube tools 설치
sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
sudo systemctl enable --now kubelet
sudo systemctl enable docker
sudo systemctl start docker

## 쿠버네티스 구성
# 1) master node
kubeadm init

# 2) worker node
kubeadm join  ~~~~~~~~~~~

# 3) CNI 설치
# calico 설치
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/tigera-operator.yaml
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/custom-resources.yaml

# 4) 치트시트
yum install -y bash-completion
echo 'source <(kubectl completion bash)' >>~/.bashrc
echo 'alias k=kubectl' >>~/.bashrc
echo 'complete -o default -F __start_kubectl k' >>~/.bashrc
source ~/.bashrc

or

kubectl completion bash | sudo tee /etc/bash_completion.d/kubectl > /dev/null

 

3. kubernetes 삭제

kubeadm reset
systemctl stop kubelet
systemctl stop docker
rm -rf /var/lib/cni/
rm -rf /var/lib/kubelet/*
rm -rf /var/lib/etcd/
rm -rf /run/calico
rm -rf /etc/cni/
rm -rf /etc/kubernetes
rm -rf ~/.kube
apt-get purge kubeadm kubectl kubelet kubernetes-cni kube*
apt-get autoremove

4. kubernetes - pod test

# pod 란?
# Kubernetes에서 만들고 관리할 수 있는 배포 가능한 가장 작은 컴퓨팅 단위
# 쿠버네티스에서 최소 배포 단위로 하나 이상의 컨테이너를 포함
# 예제 참고 : https://kubernetes.io/docs/concepts/workloads/pods/

vi pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx:1.14.2
    ports:
    - containerPort: 80

kubectl apply -f pod.yaml

# 확인
kubectl get pods 
NAME    READY   STATUS    RESTARTS   AGE
nginx   1/1     Running   0          58s

kubectl describe pod nginx
Name:         nginx
Namespace:    default
Priority:     0
Node:         worker1/172.25.0.143
Start Time:   Mon, 05 Dec 2022 16:09:09 +0900
Labels:       <none>
Annotations:  cni.projectcalico.org/containerID: 79eb363e66fb21773fed89b8c2c2761c9b7655746279f2734b8124067aabadfd
              cni.projectcalico.org/podIP: 192.168.25.8/32
              cni.projectcalico.org/podIPs: 192.168.25.8/32
Status:       Running
IP:           192.168.25.8
IPs:
  IP:  192.168.25.8
Containers:
  nginx:
    Container ID:   docker://d6719865fc7370928aff129dfa520960e066c6b84ec49554199664609fa9aaac
    Image:          nginx:1.14.2
    Image ID:       docker-pullable://nginx@sha256:f7988fb6c02e0ce69257d9bd9cf37ae20a60f1df7563c3a2a6abe24160306b8d
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Mon, 05 Dec 2022 16:09:21 +0900
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gz574 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  kube-api-access-gz574:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  37m   default-scheduler  Successfully assigned default/nginx to worker1
  Normal  Pulling    37m   kubelet            Pulling image "nginx:1.14.2"
  Normal  Pulled     37m   kubelet            Successfully pulled image "nginx:1.14.2" in 10.798471063s
  Normal  Created    37m   kubelet            Created container nginx
  Normal  Started    37m   kubelet            Started container nginx

kubectl exec -it nginx -- bash   # pod 접속

5. single node kubernetes 설정

• 일반적으로 master node, 즉 control-plane role을 가진 쿠버네티스 노드는 pod 및 다른 리소스들을 올릴 수 없다.
• master node에 pod가 올라가지 못하도록 하는 설정(taint) 걸려있기 때문이다. 이를 해제하면 해당 노드 또한 pod 및 다른 리소스들을 배포할 수 있다.

## 방법 1. taint 해제 명령어
# taint 확인
kubectl describe node master | grep Taints
Taints:             node-role.kubernetes.io/master:NoSchedule

# taint 해제
kubectl taint nodes –all node-role.kubernetes.io/master-
node/master untainted


## 방법 2. node edit
kubectl edit node master
~~
metadata:
  labels:
~~
# line 21 - 삭제
    node-role.kubernetes.io/control-plane: ""
~
spec: 
# line 27~29 - 삭제
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/control-plane

6. token 생성

## kubernetes node join 시 필요한 token 확인 및 재생성 ##

## token 확인
kubeadm token list

## token 생성
# 1. 만료일 지정 없이 생성
kueadm token create
# 2. 만료일 지정 생성
# -ttl : 만료일 옵션 / 1w, 2h 등으로 설정 가능 및 0의 경우 만료기간 없음
kubeadm token create –ttl 0

## 새로 node join시 명령어 형식
kubeadm join <master node ip:6443> --token <token 값> --discovery-token-ca-cert-hash sha256:<hash 값>

## hash 값 확인
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

 

 

** Calico.yaml

calico.yaml

0.23MB