Question

Gluster FS shared volumes for Kubernetes Cluster

Posted on March 14, 2019
CentOS Kubernetes
Asked by gradlon

Hi,

Right now i am Testing DO Kubernetes and am starting to get my head around it. There is only one thing that i am missing. An easy way to add shared volumes between PODS.

I tried GlusterFS since I am Familiar with it and Kubernetes has build in support for it, but so fare i Failed to get it working.

The GlusterFS is fine but I get an Error when I try to mound it.

"All subvolumes are down. Going offline until atleast one of them comes back up. The message “E [MSGID: 101075] [common-utils.c:294:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known)”

I have read that Rook.io would be a alternative solution but am still hesitant.

Is there a way to use Gluster FS on DO Kubernetes ore are there any other solutions for my need Kind regards.

Submit an answer

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

gradlon • March 25, 2019

Accepted Answer

For all those of you wondering. I found the Solution. One caveat, the endpoint has to be in the same namespace as the pod Consuming it.

Create a Service for Gluster

---
apiVersion: v1
kind: Service
metadata:
  name: glusterfs-cluster
  namespace: global-storage
spec:
  ports:
  - port: 1

Create endpoint for your Service

---
apiVersion: v1
kind: Endpoints
metadata:
  name: glusterfs-cluster
  namespace: default
subsets:
  - addresses:
      - ip: 10.135.112.37
    ports:
      - port: 1
  - addresses:
      - ip: 10.135.53.65
    ports:
      - port: 1
  - addresses:
      - ip: 10.135.26.91
    ports:
      - port: 1

Consume Shared Volume

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: default
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.15.4
        ports:
        - containerPort: 80
        volumeMounts:
        - mountPath: /usr/share/nginx/html
          name: nfs-vol
        - mountPath: /data
          name: glusterfsvol
      volumes:
      - name: nfs-vol
        nfs:
          server: glusterfs.cloud.integral-systems.ch
          path: /globalshare
      - name: glusterfsvol
        glusterfs:
          endpoints: glusterfs-cluster
          path: globalshare
          readOnly: false

Hope this helps :)

Filidor Wiese • January 7, 2021

When applying this test configuration:

---
apiVersion: v1
kind: Service
metadata:
  name: glusterfs-cluster
  namespace: default
spec:
  ports:
    - port: 1

---
apiVersion: v1
kind: Endpoints
metadata:
  name: glusterfs-cluster
  namespace: default
subsets:
  - addresses:
      - ip: 10.x.x.x # IP of a droplet running glusterfs server
    ports:
      - port: 1

---
apiVersion: v1
kind: Pod
metadata:
  name: glusterfs
spec:
  containers:
    - name: glusterfs
      image: nginx
      volumeMounts:
        - mountPath: "/mnt/user/test"
          name: glusterfsvol
  volumes:
    - name: glusterfsvol
      glusterfs:
        endpoints: glusterfs-cluster
        path: test
        readOnly: false

I get the following error:

mount: /var/lib/kubelet/pods/some-uuid/volumes/kubernetes.io~glusterfs/glusterfsvol: unknown filesystem type 'glusterfs'.

It looks like the kubernetes worker nodes don’t have glusterfs-client installed, henceforth the glusterfs mount type is non-existent.

Did anyone solve this yet?

liveb4buy • May 15, 2019

@gradlon , Still its not working for me with the equivalent code (considering different namespace not default namespace) . Did you configure StorageClass or Init-Container in the Deployment level? Pls let me know . Thanks in advance.