安装配置

Helm 安装

集群安装

首先使用 Helm 安装 Rook 集群：

helm install --namespace rook-ceph rook-ceph rook-release/rook-ceph

kubectl apply -f rook-cluster.yaml

rook-cluster.yml 定义如下：

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    image: ceph/ceph:v14.2.5
    allowUnsupported: true
  dataDirHostPath: /opt/disks/vdb/rook-ceph
  mon:
    count: 3
  dashboard:
    enabled: true
  storage:
    useAllNodes: true
    useAllDevices: false
    directories:
      - path: /opt/disks/vdb/rook-ceph

然后启动 ceph 工具箱连接到 rook 集群：

# https://rook.io/docs/rook/master/ceph-toolbox.html
kubectl create -f rook-ceph-tools.yaml
kubectl -n rook-ceph get pod -l "app=rook-ceph-tools"
kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash

## 一些测试命令
# ceph status
# ceph osd status
# ceph df
# rados df

存储创建

首先我们创建 StorageClass：

apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: replicapool
  namespace: rook-ceph
spec:
  failureDomain: host
  replicated:
    size: 3
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rook-ceph-block
# Change "rook-ceph" provisioner prefix to match the operator namespace if needed
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
  # clusterID is the namespace where the rook cluster is running
  clusterID: rook-ceph
  # Ceph pool into which the RBD image shall be created
  pool: replicapool

  # RBD image format. Defaults to "2".
  imageFormat: "2"

  # RBD image features. Available for imageFormat: "2". CSI RBD currently supports only `layering` feature.
  imageFeatures: layering

  # The secrets contain Ceph admin credentials.
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph

  # Specify the filesystem type of the volume. If not specified, csi-provisioner
  # will set default as `ext4`.
  csi.storage.k8s.io/fstype: xfs

# Delete the rbd volume when a PVC is deleted
reclaimPolicy: Delete

典型的 PVC 如下：

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rook-example-pvc
spec:
  storageClassName: rook-ceph-block
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

手工管理不同节点上的存储

给节点打存储相关标签

方便创建指定服务器上的 PersistentVolume，为各节点打上标签

kubectl label nodes yourbiz-master 'yourdomain.com/bare-metal-storage-name=yourbiz-master'
kubectl label nodes yourbiz-slave000 'yourdomain.com/bare-metal-storage-name=yourbiz-slave000'
kubectl label nodes yourbiz-slave001 'yourdomain.com/bare-metal-storage-name=yourbiz-slave001'

创建指定节点上的 PV 和使用该 PV 的 PVC

关于 PV 概念，详见 Persistent Volumes，创建 PV，下面的配置文件创建了一个 PV，大小 50G，通过 selector 和 volumeMode 指明该存储位于 yourbiz-master 服务器上文件系统的 /opt/disks/vdb/gitlab-pvs/pv-gitlab-data/ 目录。

apiVersion: v1
kind: PersistentVolume
metadata:
  name: gitlab-data
  namespace: yourbiz-infra
spec:
  storageClassName: ""
  capacity:
    storage: 50Gi
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  hostPath:
    path: /opt/disks/vdb/gitlab-pvs/pv-gitlab-data/
    type: ""
  persistentVolumeReclaimPolicy: Retain
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: yourdomain.com/bare-metal-storage-name
              operator: In
              values:
                - yourbiz-master

可以创建一个 PVC 直接使用该 PV：

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: gitlab-data
  namespace: yourbiz-infra
spec:
  storageClassName: ""
  volumeName: gitlab-data
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi

这样，在 Deployment 中需要使用到 PVC 的地方可以直接引用该 PVC。

手动配置

首先安装 Operator：

$ git clone https://github.com/rook/rook

$ cd cluster/examples/kubernetes/ceph
$ kubectl create -f operator.yaml

# 查看 Operator 是否安装完毕
$ kubectl get pod -n rook-ceph-system

然后创建 Ceph 集群：

$ kubectl create -f cluster.yaml

# 查看 Ceph 集群
$ kubectl get pod -n rook-ceph

cluster.yaml 的参数定义如下：

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    # For the latest ceph images, see https://hub.docker.com/r/ceph/ceph/tags
    image: ceph/ceph:v13.2.2-20181023
  dataDirHostPath: /var/lib/rook # 数据盘目录
  mon:
    count: 3
    allowMultiplePerNode: true
  dashboard:
    enabled: true
  storage:
    useAllNodes: true
    useAllDevices: false
    config:
      databaseSizeMB: "1024"
      journalSizeMB: "1024"

Ceph 为我们提供了 UI 管控的界面：

$ kubectl get svc -n rook-ceph

# 外化接口
$ kubectl -n rook-ceph port-forward service/rook-ceph-mgr-dashboard 31631:7000

# 管理账户 admin，获取登录密码
$ kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o yaml | grep "password:" | awk '{print $2}' | base64 --decode

使用

创建 Pool

apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: replicapool # operator会监听并创建一个pool，执行完后界面上也能看到对应的pool
  namespace: rook-ceph
spec:
  failureDomain: host
  replicated:
    size: 3
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rook-ceph-block # 这里创建一个storage class, 在pvc中指定这个storage class即可实现动态创建PV
provisioner: ceph.rook.io/block
parameters:
  blockPool: replicapool
  # The value of "clusterNamespace" MUST be the same as the one in which your rook cluster exist
  clusterNamespace: rook-ceph
  # Specify the filesystem type of the volume. If not specified, it will use `ext4`.
  fstype: xfs
# Optional, default reclaimPolicy is "Delete". Other options are: "Retain", "Recycle" as documented in https://kubernetes.io/docs/concepts/storage/storage-classes/
reclaimPolicy: Retain

创建 PVC

在 cluster/examples/kubernetes 目录下，官方给了个 WordPress 的例子，可以直接运行一下：

$ kubectl create -f mysql.yaml
$ kubectl create -f wordpress.yaml

文件内容定义如下：

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
  labels:
    app: wordpress
spec:
  storageClassName: rook-ceph-block   # 指定storage class
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi  # 需要一个20G的盘

# ...
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim  # 指定上面定义的PVC

Ceph 访问与监控

cluster.yaml 里有配置，可配置成共享宿主机网络，这样外面可直接连接 ceph 集群：

network:
  # toggle to use hostNetwork
  hostNetwork: false

Ceph 集群监控

通过 prometheus operator 配合 rook 可以快速构建 ceph 集群的监控，sealyun 安装包中已经自带了 prometheus operator，所以直接干即可。注意这里是为 ceph 单独起了一个 prometheus，这样做挺好，因为毕竟可以缓解 prometheus 单点的压力：

cd cluster/examples/kubernetes/ceph/monitoring
kubectl create -f service-monitor.yaml
kubectl create -f prometheus.yaml
kubectl create -f prometheus-service.yaml

最近更新于 0001-01-01