首页
首页
文章目录
  1. 简介
  2. 1、在kube-proxy和kube-controller-manager主机上安装ceph-common
  3. 2、初始化rbd pool “kube”
  4. 3、获取ceph的key并使用base64加密
  5. 4、创建storageclass
  6. 5、创建mongodb
  7. 检查mongodb状态
  8. 参考URL

k8s连接ceph操作记录

简介

前面文章写了如何直接通过服务器对接cephfs,今天来写如何使用k8s连接ceph

1、在kube-proxy和kube-controller-manager主机上安装ceph-common

1
yum install  -y ceph-common

2、初始化rbd pool “kube”

1
2
ceph osd pool create kube 2 2
rbd pool init kube

3、获取ceph的key并使用base64加密

1
2
cat ceph.client.admin.keyring
echo "key" | base64

获取cephkey 获取cephkey

4、创建storageclass

cephrbd_storageclass.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
apiVersion: v1
kind: Secret
metadata:
name: cephrbd-secret
namespace: kube-system
type: kubernetes.io/rbd
data:
key: QVFBam96ZGYyWHZWR3hBQWQ4aGF2UU9ZbEJ1NkJFanhtYjk1Unc9PQo=
---

apiVersion: v1
kind: Secret
metadata:
name: cephrbd-secret-user
type: kubernetes.io/rbd
data:
key: QVFBam96ZGYyWHZWR3hBQWQ4aGF2UU9ZbEJ1NkJFanhtYjk1Unc9PQo=

---

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: cephrbd
provisioner: kubernetes.io/rbd
parameters:
monitors: 10.10.77.144:6789,10.10.77.145:6789,10.10.77.146:6789
adminId: admin
adminSecretName: cephrbd-secret
adminSecretNamespace: kube-system
pool: kube
userId: admin
userSecretName: cephrbd-secret-user
userSecretNamespace: default
fsType: ext4
imageFormat: "2"
imageFeatures: "layering"

kubectl create -f cephrbd_storageclass.yaml

创建cephrbd 创建cephrbd

5、创建mongodb

mongodb.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
apiVersion: v1
kind: ServiceAccount
metadata:
name: mongo
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: read-pod-service-endpoint
rules:
- apiGroups:
- ""
resources:
- pods
- services
- endpoints
verbs:
- get
- list
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: system:serviceaccount:default:mongo
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: read-pod-service-endpoint
subjects:
- kind: ServiceAccount
name: mongo
namespace: default
---
apiVersion: v1
kind: Service
metadata:
name: mongo
labels:
name: mongo
spec:
ports:
- port: 27017
targetPort: 27017
clusterIP: None
selector:
role: mongo
---
apiVersion: v1
kind: Service
metadata:
name: mongo-service
labels:
name: mongo
spec:
ports:
- name: mongo-http
port: 27017
selector:
role: mongo
type: NodePort
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mongo
spec:
serviceName: "mongo"
replicas: 3
selector:
matchLabels:
role: mongo
template:
metadata:
labels:
role: mongo
environment: test
spec:
serviceAccountName: mongo
automountServiceAccountToken: true
terminationGracePeriodSeconds: 30
containers:
- name: mongo
image: dzero.com/base/mongo:4.4
command:
- mongod
args:
- "--replSet=rs0"
- "--bind_ip=0.0.0.0"
ports:
- containerPort: 27017
volumeMounts:
- name: mongo-persistent-storage
mountPath: /data/db
- name: mongo-sidecar
image: dzero.com/base/k8s-mongo-sidecar:latest
env:
- name: KUBERNETES_POD_LABELS
value: "role=mongo,environment=test"
- name: KUBERNETES_SERVICE_NAME
value: "mongo"
imagePullSecrets:
- name: myregistrykey
volumeClaimTemplates:
- metadata:
name: mongo-persistent-storage
spec:
storageClassName: cephrbd
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 2Gi

注: 这边用的是我自己的镜像仓库配置了自己的imagePullSecrets
创建的时候会自动创建pv pvc

ceph展示图 ceph展示图
pv\pvc展示图pv\pvc展示图

检查mongodb状态

1
2
kubectl exec -it mongo-0 -- mongo
rs.status()

mongodb正常效果mongodb正常效果

——-2020-08-27——-
修改的svc的yaml,修复之前无法通过NodePort连接问题

以上
End!

希望这篇文章能给你带来一些帮助。

参考URL

https://github.com/morphy2k/k8s-mongo-sidecar
https://hub.docker.com/_/mongo?tab=tags

支持一下
扫一扫,我会更有动力更新
  • 微信扫一扫
  • 支付宝扫一扫