Install

Ceph S3 / Rook (Preview)

Deploy the Rook kubernetes operator, default values will deactivate all cephfs related features:

With Helmfile:

  # operator only
  - name: rook-ceph
    namespace: kosmos-sds
    chart: ../../rook/rook-ceph
    labels:
      app: rook-ceph
    values:
      - ../../rook/values/values-operator.yaml

Directly with Helm:

helm upgrade --install --create-namespace --namespace kosmos-sds -f values/values-operator.yaml rook-ceph rook-ceph/

CEPH cluster

The following will deploy a single node CEPH cluster with S3 gateway on a local PV block device, avoiding the need of a full disk. It assume that the lvm local volume provisioner is deployed:

  # cluster
  - name: rook-ceph-cluster
    namespace: kosmos-sds
    chart: ../../rook/rook-ceph-cluster
    needs: [kosmos-sds/rook-ceph]
    labels:
      app: rook-ceph-cluster
    deleteWait: true
    values:
      - ../../rook/values/values-single-pvc.yaml
      - operatorNamespace: kosmos-sds
        ingress:
          dashboard:
            annotations:
              cert-manager.io/cluster-issuer: kosmos-ca-issuer
            host:
              name: ceph.{{ .StateValues.domain }}
              path: "/"
              pathType: Prefix
            tls:
              - hosts:
                - ceph.{{ .StateValues.domain }}
                secretName: cephsecret-tls
        cephClusterSpec:
          mon:
            count: 1
            allowMultiplePerNode: false
            volumeClaimTemplate:
              spec:
                storageClassName: lvm-provisioner
                resources:
                  requests:
                    storage: 1Gi
          storage:
            storageClassDeviceSets:
            - name: set1
              count: 1
              portable: false
              encrypted: false
              volumeClaimTemplates:
              - metadata:
                  name: data
                spec:
                  resources:
                    requests:
                      storage: 30Gi
                  storageClassName: lvm-provisioner
                  volumeMode: Block
                  accessModes:
                    - ReadWriteOnce

You should see something like:

NAME                                                READY   STATUS      RESTARTS   AGE
rook-ceph-crashcollector-ubuntu1-8bcf65d78-ckvv5    1/1     Running     0          63s
rook-ceph-exporter-ubuntu1-887dc67b6-tjrwl          1/1     Running     0          60s
rook-ceph-mgr-a-5d5869f54b-fchmr                    1/1     Running     0          2m17s
rook-ceph-mon-a-6bcbb57579-w5pv9                    1/1     Running     0          2m43s
rook-ceph-osd-0-7f7d58bbb4-khrxp                    1/1     Running     0          105s
rook-ceph-osd-prepare-set1-data-0p5c4v-pn7k4        0/1     Completed   0          116s
rook-ceph-osd-prepare-ubuntu1-6rslr                 0/1     Completed   0          116s
rook-ceph-rgw-ceph-objectstore-a-7d6b677574-vjlcv   1/1     Running     0          63s
rook-ceph-tools-576c7f89ff-jrhvx                    1/1     Running     0          2m58s

Connect to the dashboard with https://ceph.<your domain>:443

The following will deploy a single node CEPH cluster with S3 gateway on a full block device (/dev/vdb in the sample):

  # cluster on full disk (/dev/vdb in the sample)
  - name: rook-ceph-cluster
    namespace: kosmos-sds
    chart: ../../rook/rook-ceph-cluster
    needs: [kosmos-sds/rook-ceph]
    labels:
      app: rook-ceph-cluster
    deleteWait: true
    values:
      - ../../rook/values/values-single.yaml
      - operatorNamespace: kosmos-sds
        ingress:
          dashboard:
            annotations:
              cert-manager.io/cluster-issuer: kosmos-ca-issuer
            host:
              name: ceph.{{ .StateValues.domain }}
              path: "/"
              pathType: Prefix
            tls:
              - hosts:
                - ceph.{{ .StateValues.domain }}
                secretName: cephsecret-tls
        cephClusterSpec:
          storage: # cluster level storage configuration and selection
            useAllNodes: true
            useAllDevices: false
            deviceFilter: "^vdb$" # test: lsblk -dn -o NAME | grep -E "^vdb$"

Connect to the dashboard with https://ceph.<your domain>:443

Minio (Deprecated)

Add the following in your helmfile.yaml.gotmpl or use the up-to-date one provided in the delivery, il will install s3 operator, create a basic cluster and configure oidc part.

  - name: s3-operator
    namespace: kosmos-s3
    chart: ../../s3/operator
    wait: true
    waitForJobs: true
    labels:
      app: s3
    values:
      # No HA (replica 1)
      - ../../s3/values/values-operator.yaml

  - name: s3-secrets
    namespace: kosmos-s3
    chart: ../../s3/minio-secrets
    needs: [kosmos-iam/keycloak-cluster, kosmos-s3/s3-operator]
    labels:
      app: s3
    values:
      - oidcConfig:
          domain: {{ .StateValues.domain }}
        trustedCA: ref+k8s://v1/Secret/kosmos-system-restricted/kosmos-ca-secret/ca.crt

  - name: keycloakimporter-import-s3
    needs:
      - kosmos-s3/s3-secrets
      - kosmos-iam/keycloakimporter-init-realm-kosmos
    namespace: kosmos-iam
    chart: ../../keycloakimporter/keycloakimporter
    labels:
      app: s3
    values:
      - ./values_templates/s3/import-client-s3.yaml.gotmpl

  - name: s3-cluster
    namespace: kosmos-s3
    needs: [ kosmos-s3/s3-operator, kosmos-s3/s3-secrets, kosmos-iam/keycloakimporter-import-s3 ]
    chart: ../../s3/tenant
    wait: true
    waitForJobs: true
    labels:
      app: s3
    values:
      - ../../s3/values/values-tenant.yaml
      #- ../../s3/values/values-tenant-minimal.yaml
      - domain: {{ .StateValues.domain }}
      - ingress:
          console:
            enabled: true
          api:
            enabled: false