Requirements

Hosts

For preparation of RPi, we can use this article.

• rpi-01
• rpi-02
• rpi-03

Architecture

Replicated across 3 nodes

Install

sudo apt-get -y install glusterfs-server

systemctl enable --now glusterd
systemctl status glusterd

System Configuration

Host file

sudo nano /etc/hosts

Firewall

# Create configuration file for UFW
sudo nano /etc/ufw/applications.d/ufw-glusterfs.conf

Add the following ruleset

[GlusterFS Daemon]
title=GlusterFS (Daemon)
description=GlusterFS Daemon
ports=24007/tcp

[GlusterFS Management]
title=GlusterFS (Management)
description=GlusterFS Management
ports=24008/tcp

[GlusterFS NFS Service]
title=GlusterFS (NFS Service)
description=GlusterFS NFS Service
ports=38465:38467/tcp

[GlusterFS]
title=GlusterFS
description=GlusterFS (10 bricks)
ports=49152:49161/tcp

Check the configuration is readable by following command

sudo ufw app list

Enable firewall rule

sudo ufw allow "GlusterFS Daemon"
sudo ufw allow "GlusterFS Management"
sudo ufw allow "GlusterFS"

Join

master node:

sudo su

gluster peer probe 10.x.x.x
gluster peer probe 10.x.x.y

gluster peer status

Mount partition

# Prepare file system
sudo mkfs.xfs -f /dev/sda1

# Record the name, UUID, and FSType of the partition to mount
sudo lsblk -f
# e.g. sda1, ########-##... , ext4


# create mount point
sudo mkdir -p /mnt/gfdata
sudo chown -R ben:ben /mnt/gfdata

# create mount config
sudo nano /etc/fstab

# edit fstab
# include UUID=########-##...  /mnt/gfdata    xfs      defaults,auto,users,rw,nofail,noatime 0 0

sudo mount -a


# Check mount
lsblk

Create gfvolume

# Create gluster volume named gfsvol1
# Only on master
gluster volume create gfsvol1 replica 3 rpi-01:/mnt/gfdata/data1 rpi-02:/mnt/gfdata/data1 rpi-03:/mnt/gfdata/data1

# Start gluster volume ONLY on master
gluster volume start gfsvol1

# create mount point on ALL nodes
sudo mkdir /mnt/gfsvolume1
sudo chown -R ben:ben /mnt/gfsvolume1
sudo chown 777 /mnt/gfsvolume1

# mount gluster volume
# all nodes
sudo mount -t glusterfs rpi-01:/gfsvol1 /mnt/gfsvolume1
sudo mount -t glusterfs rpi-02:/gfsvol1 /mnt/gfsvolume1
sudo mount -t glusterfs rpi-03:/gfsvol1 /mnt/gfsvolume1

# mount gluster volume on boot
# all nodes
sudo /etc/fstab
# rpi-03:/gfsvol1 /mnt/gfsvolume1 glusterfs defaults,_netdev 0 0

# all nodes
sudo mkdir /etc/systemd/system/srv.mount.d/
sudo nano /etc/systemd/system/srv.mount.d/override.conf

# add the following content
# [Unit]
# After=glusterfs-server.service
# Wants=glusterfs-server.service

Replacing Brick

If any brick has failed, we can use the following command to replace brick

sudo gluster volume replace-brick gfsvol1 rpi-03:/mnt/gfdata/data1 rpi-03:/mnt/usbdata/data1 commit force

Adding New Node

At the middle of deployment, we may have the need to add new node (with new brick) into the already existing brick. This section will walk us through how to add new node to the existing cluster.

Follow the Install and System Configuration section above

More info: How to add new brick to replicated GlusterFS volume on Linux – nixCraft (cyberciti.biz)