Recently I’ve been studying for my Red Hat RH300 course (RHCE and RHCSA 1 week course) and being me, I got onto something totally different while reading up on a topic and ended up reading about GlusterFS – something i’ve heard about in talks with customers, mainly around the “Hadoop”/”Big Data” scene which is becoming more and more fashionable as the days wear on.

So what is GlusterFS? Not being one who wants to regurgitate other peoples musings and brand them my own, below are links to 2 excellent guides which have taught me on the subject (both from Red Hat):

Basically, it is FUSE (Filesystem in User Space) based, and allows mass scalability across multiple “bricks”; where a brick is a folder on a Gluster peer. We as Gluster admins, can add multiple servers running gluster to a gluster cluster (i like it, i like it…) with each server known as a peer. Once all the peers are added to the cluster, we can then create volumes, using storage from any number of the nodes, in any format (from replication, span, etc).

Gluster overview

What this allows us to do in real world terms is have FS-level replication across multiple servers, and using “geo replication”, across servers across regions (i’m looking at you – Amazon Web Services…).

Without getting into the nitty gritty too much, it has many benefits such as not-requiring metadata servers, which is quite interesting and the slides go into the where/why in detail if thats of interest.

Architecting with GlusterFS

The beauty of GlusterFS is apparent immediately when you begin playing with it; its simplicity. Adding peers to the GlusterFS Global namespace is remarkably simple. Adding a new volume can be done with a few commands – it is very nice to use.

In our hypothetical environment, we have the following:

Server #1:

  • Hostname: “server001”
  • IP Address: 10.0.0.2/24
  • Operating System: RHEL 6.3 x64
  • SE Linux: Disabled
  • Storage Configuration: 4 x 1.5TB 7200rpm SATA HDD’s in LVM RAID-5 giving us a usable 4.5TB in a 3+1 configuration.

Server #2:

  • Hostname: “server002”
  • IP Address: 10.0.1.2/24
  • Operating System: RHEL 6.3 x64
  • SE Linux: Disabled
  • Storage Configuration: 4 x 1.5TB 7200rpm SATA HDD’s in LVM RAID-5 giving us a usable 4.5TB in a 3+1 configuration.

What we want to do here with these 2 server nodes, is to cluster them into a single GlusterFS Global namespace (across 2 subnets, so beware of L3 involvement in the form of routers/firewalls!), and replicate our storage between the two 4.5TB volumes. This can then allow the servers to be located behind a load balancer, or behind a router running SLB (Server Load Balancing), and to serve the data via CIFS/NFS from either node.

…or we can use it as a DR site, on a seperate location – and have Apache2’s directories replicated and simply have a heartbeat script that says “If server001 “httpd” != “started”, then start up my httpd service” and we can be safe in the knowledge the servers /var/www directory (for example) is exactly the same.

Setup and configuration

As above, I have denoted “Server 1” in blue, and “Server 2” in purple, just to aid with understanding.

1.1 Firstly, we need to enable a few repo’s on both Server 1 and Server 2. As we are using RHEL, we can do this with:

subscription-manager register

And follow the prompts.

1.2. If we dont have an active RHEL subscription, or we are using CentOS (in this RPM based example), then we can simply add the EPEL repo’s to “/etc/yum.repos.d/epel.repo” and then copy and paste the below:

[epel]
name=Extra Packages for Enterprise Linux 6 - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/6/$basearch
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-6&arch=$basearch
failovermethod=priority
exclude=libmodplug
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
 
[epel-source]
name=Extra Packages for Enterprise Linux 6 - $basearch - Source
#baseurl=http://download.fedoraproject.org/pub/epel/6/SRPMS
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-source-6&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
gpgcheck=1

1.3. Finally, after either of the above, run “yum check-update” to update our repositories so we can view the required packages.

2.1. We must now install a few packages, again on both systems (Server 1 and Server 2):

yum install fuse fuse-libs

These are the FUSE packages required for GlusterFS to install and run.  (FUSE Definition here).

2.2. Next, lets install the GlusterFS packages:

yum install glusterfs glusterfs-fuse glusterfs-geo-replication glusterfs-server

If we run into problems with GPG Keys (as I did in the installation), you can choose to import them, or simply for ease of experimentation, run yum with the “–nogpgcheck” option (this sits just after “install”).

2.3. Finally, lets set the GlusterFS service to start on boot, and start it up:

chkconfig glusterd on && service glusterd start

3. On both servers (Server 1 and Server 2) we need to open up the firewall to allow GlusterFS traffic to pass through. To do this, run the commands below:

iptables -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 24007:24047 -j ACCEPT 
iptables -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 111 -j ACCEPT 
iptables -A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport 111 -j ACCEPT 
iptables -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 38465:38467 -j ACCEPT

or you can edit the file “/etc/sysconfig/iptables” and add the rules in manually.

After this, dont forget to restart the iptables service so the new rules are loaded (service iptables restart).

4.1 Now that both machines have glusterd running, iptables opened, and the packages installed, we can begin to configure at the gluster layer.

4.2. On Server 1 run:

gluster peer probe 10.0.1.2

Where “10.0.1.2” is the IP of Server 2. If this completes successfully, we should now be able to run “gluster peer status” and see an output similar to:

[root@Server001 ~]# gluster peer status
Number of Peers: 1
Hostname: 10.0.1.2
Uuid: b47185a7-ea0e-4582-b058-336b4fc0b0e0
State: Peer in Cluster (Connected)

This shows that Server 1 and Server 2 are in a cluster namespace and peered. If we see an output like below:

[root@Server001 ~]# gluster peer status
Number of Peers: 1
Hostname: 10.0.1.2
Uuid: 00000000-0000-0000-0000-000000000000
State: Establishing Connection (Disconnected)

Then this generally means that the iptables havent been set-up correctly, or there is a firewall blocking connectivity between the two (or your routing is a mess!).

4.3. Now we are in a cluster, we can begin to carve up our individual GlusterFS servers into “bricks”, and add these bricks into GlusterFS Volumes. For ease of use, we are going to replicate a single brick across the two GlusterFS peers, “/media/lvm-volume001/apache” – which is an LVM volume mounted at the above address. We want any files stored in here, to be replicated across both GlusterFS peers (Server 1 and Server 2) for redunancy.

4.4. To do this, on Server 1 run:

gluster volume create gfsapache001 replica 2 transport tcp 10.0.0.2:/media/lvm-volume001/apache 10.0.1.2:/media/lvm-volume001/apache

This creates a GlusterFS volume called “gfsapache001”, that is of the type “replica”, across “2” peers, at the addresses/folders specified. You will notice the address:/folder format is very similar to NFS.

4.5. We must now specify client access-control to our new volume:

gluster volume set gfsapache001 auth.allow 10.0.*

Where the IP address/* is the subnet we wish to grant permission to view/edit our new volume.

4.6. Finally, we can start the new GlusterFS volume:

gluster volume start gfsapache001

4.7. We can also view the output of the volume (information, etc) using the command:

gluster volume info

4.8. NB: It may be obvious, but it is worth making it abundantly clear – as these 2 nodes (Server 1 + Server 2) are clustered now at the Gluster layer, we can run any of the above commands from either node and we can see the output.

5. Now that GlusterFS replication is set-up between the 2 folders at /media/lvm-volume001/apache, we can now do a multitude of options. We can edit “etc/exports” on the servers and export the folder via NFS, so that users can mount the folder as the apache directory on web servers, for example.

We can also edit “/etc/smb.conf” and export the folder as CIFS (for Windows users), so they can use it as a file store, for example. The possbilities are endless!

Closing thoughts

Playing with GlusterFS has been rather enjoyable, and a suprise given how easy and simple it is to use yet what it is able to achieve. I worked professionally in SAN/NAS, yet anyone can follow the above guide/online documents and create a really useful, really scalable storage solution in a matter of minutes. All it takes is perserverance, and a large wallet – given the current price per TB!

Notes:

One thing i did find with GlusterFS and my testing that i found odd (and hopefully someone can enlighten me on this), is in the above scenario where we have our volume replicated, etc but not mounted by a user, when we add a file to the folder via the local CLI i.e.

root@server001# touch /media/lvm-volume001/apache/newfile

The file didnt appear in the folder on Server002. However, if we mounted the volume locally on each server, i.e.

Server001# mount –t glusterfs localhost:/gfsapache001 /media/gluster
Server002# mount –t glusterfs localhost:/gfsapache001 /media/gluster

And then added a file to /media/gluster, the file was replicated. I am going to have another poke around when i get time to see if i can figure out why this is!