Simple Ceph cluster deployment for test or RADOS development purposes

1. Prerequisites

Basic Ceph terminology knowledge (e.g. Monitor, OSD, RADOS, RADOS Gateway, RBD, MDS and CephFS) is expected.

Reading, or at least skimming through, the Ceph Hammer or master architecture document is recommended.

At least 7 identical CentOS 7.2 x86_64 systems. This guide expects KVM VMs (disk devices are vdb, vdc, etc) but adapting to physical hardware should be straightforward.

• installation = Minimal, with a working chrony daemon (e.g. with reliable NTP sources)
• 5GB system disks, with swap = 512MiB, and / = 3GiB
• 2GiB of RAM and a vCPU per system
• a static IP per system

• all systems must be able to ping each other by their shortname ( hostname -s ). An identical /etc/hosts file containing the shortnames (not the FQDN of the cluster) can be used if DNS is unavailable.

2. Description

We are going to deploy:

• an admin node,
• a single monitor,
• two OSDs (storage nodes),
• a RADOS gateway (which provides S3-like access to Ceph),
• a MDS for CephFS,
• and a Ceph client system.

Monitors work in a PAXOS cluster, so three (or five) are mandatory for production purposes, but we can test with a single monitor.

At least three OSDs are needed for a production Ceph cluster as each object is written three times by default, but we are going to make do with two OSDs, and edit the configuration to take this pecularity into account.

3. Warning

The resulting test cluster is not suitable for production. Do not base any production on this howto, as:

• we deploy a single Monitor, not a PAXOS cluster,
• we only deploy two OSDs so objects will get two replica only (instead of three),

• we do not use proper, separate production and management Ethernet interfaces,

• no CRUSH (Controlled Replication Under Scalable Hashing) hierarchy is setup,

• no system hardening is done,

• we use a ceph-deploy user and expect real-world deployments to at least obfuscate that particular user name,

• no performance sizing is included.

4. RADOS Cluster Installation

4.1. System Names

• c7-ceph-admin
• c7-ceph-mon0
• c7-ceph-osd0
• c7-ceph-osd1

4.2. Firewall configuration

Monitors listen on tcp:6789 by default, so run on c7-ceph-mon0:

 # firewall-cmd --zone=public --add-port=6789/tcp --permanent
 # firewall-cmd --reload

OSDs listen on a range of ports, tcp:6800-7300 by default, so run on on c7-ceph-osd{0,1}:

# firewall-cmd --zone=public --add-port=6800-7300/tcp --permanent
# firewall-cmd --reload

4.3. Yum repository configuration

On all hosts, install the centos-release-ceph rpm from centos-extras:

# yum -y install --enablerepo=extras centos-release-ceph

4.4. ceph-deploy user

On all hosts, create a ceph-deploy user, with sudo/NOPASSWD capabilities:

# adduser ceph-deploy
# echo myPassword | passwd ceph-deploy --stdin
# cat << EOF >/etc/sudoers.d/ceph-deploy
ceph-deploy ALL = (root) NOPASSWD:ALL
Defaults:ceph-deploy !requiretty
EOF
# chmod 440 /etc/sudoers.d/ceph-deploy

Note: do not use a ceph user. Ceph daemons in Hammer run as root, but the ceph user is used for that purpose in Infernalis and later releases.

As ceph-deploy on ceph-admin, create a ssh key with a passphrase:

$ ssh-keygen -b 4096
(do not use an empty passphrase)
$ ssh-agent bash
$ ssh-add
$ for node in c7-ceph-admin c7-ceph-mon0 c7-ceph-osd0 c7-ceph-osd1 ; do ssh-copy-id $node ; done

4.5. Disable SELinux

Ceph Hammer requires SELinux to be disabled. Ceph Infernalis and later releases include a proper SELinux policy.

$ for node in c7-ceph-admin c7-ceph-mon0 c7-ceph-osd0 c7-ceph-osd1 ; do ssh $node sudo sed -i -e 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config  ; done
$ for node in c7-ceph-mon0 c7-ceph-osd0 c7-ceph-osd1  ; do ssh $node sudo reboot  ; done
$ sudo reboot

Log back in as ceph-deploy on ceph-admin:

$ ssh-agent bash
$ ssh-add

4.6. Package installation

As ceph-deploy on ceph-admin, install the Ceph RPM packages on the Monitor and OSDs.

$ sudo yum -y install ceph-deploy
$ ceph-deploy install --mon c7-ceph-mon0
$ ceph-deploy install --osd c7-ceph-osd0 c7-ceph-osd1

NB: Upstream's ceph-deploy configures ceph.com's yum repositories by default. The StorageSIG version of ceph-deploy does not as the StorageSIG Ceph repositories are expected to be enabled beforehand.

4.7. Cluster installation

As ceph-deploy on ceph-admin, create the cluster configuration file:

$ ceph-deploy new c7-ceph-mon0

Add configuration directives: 10GiB journal, 2 (normal _and_ minimum) replicas per object, etc.

$ cat << EOF >> ceph.conf
osd_journal_size = 10000
osd_pool_default_size = 2
osd_pool_default_min_size = 2
osd_crush_chooseleaf_type = 1
osd_crush_update_on_start = true
max_open_files = 131072
osd pool default pg num = 128
osd pool default pgp num = 128
mon_pg_warn_max_per_osd = 0
EOF

As ceph-deploy on ceph-admin, create the monitor:

$ ceph-deploy mon create-initial

As ceph-deploy on ceph-admin, install and configure the cluster command-line interface:

$ ceph-deploy install --cli c7-ceph-admin
$ ceph-deploy admin c7-ceph-admin

Since we are not doing an upgrade, switch CRUSH tunables to optimal:

$ sudo ceph osd crush tunables optimal

Now, attach two 12GiB disks to each OSD and reboot. These disks will show up as vdb and vdc.

As ceph-deploy on ceph-admin, erase vdb and vdc on c7-ceph-osd0 and c7-ceph-osd1:

$ ceph-deploy disk zap c7-ceph-osd0:vdb c7-ceph-osd1:vdb c7-ceph-osd0:vdc c7-ceph-osd1:vdc

Create a partition on vdc, filling all the disk, on c7-ceph-osd0 and c7-ceph-osd1. TODO

Add parted command here

As ceph-deploy on ceph-admin, install and configure the OSDs, using vdb as datastore (this is normally a RAID0 of big rotational disks) and vdc1 as its journal (normally a partition on a SSD):

$ ceph-deploy osd prepare c7-ceph-osd0:vdb:vdc1 c7-ceph-osd1:vdb:vdc1

As ceph-deploy on ceph-admin, activate the OSDs. Note that the previous step partitioned vdb, so we specify vdb1 as datastore when activating the OSDs:

$ ceph-deploy osd activate c7-ceph-osd0:vdb1:vdc1 c7-ceph-osd1:vdb1:vdc1

If you're fast enough, the following command should output HEALTH_WARN first, and a after a little while, HEALTH_OK:

$ sudo ceph -s

4.8. RADOS Cluster monitoring

Cluster health:

$ sudo ceph health
HEALTH_OK

Cluster status:

$ sudo ceph status
    cluster 02b3e99f-b8a6-4fa0-a2ea-f70c46340ee6
     health HEALTH_OK
     monmap e1: 1 mons at {c7-ceph-mon0=192.168.2.181:6789/0}
            election epoch 2, quorum 0 c7-ceph-mon0
     osdmap e12: 2 osds: 2 up, 2 in
      pgmap v202: 65 pgs, 2 pools, 179 kB data, 1 objects
            68740 kB used, 24486 MB / 24553 MB avail
                  65 active+clean

Monitor status:

$ sudo ceph mon_status
{"name":"c7-ceph-mon0","rank":0,"state":"leader","election_epoch":2,"quorum":[0],"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"02b3e99f-b8a6-4fa0-a2ea-f70c46340ee6","modified":"0.000000","created":"0.000000","mons":[{"rank":0,"name":"c7-ceph-mon0","addr":"192.168.2.181:6789\/0"}]}}

Listing the OSDs:

$ sudo ceph osd tree
ID WEIGHT  TYPE NAME             UP/DOWN REWEIGHT PRIMARY-AFFINITY 
-1 0.01999 root default                                            
-2 0.00999     host c7-ceph-osd0                                   
 0 0.00999         osd.0              up  1.00000          1.00000 
-3 0.00999     host c7-ceph-osd1                                   
 1 0.00999         osd.1              up  1.00000          1.00000 

Displaying free space:

$ sudo ceph df
GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED 
    24553M     24486M       68868k          0.27 
POOLS:
    NAME     ID     USED     %USED     MAX AVAIL     OBJECTS 
    rbd      0         0         0        12243M           0 

5. RADOS usage

5.1. Pool creation

To create a pool:

$ sudo ceph osd pool create mypool 1
pool 'mypool' created
$ sudo ceph osd lspools
0 rbd,4 mypool,
$ sudo ceph df
GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED 
    24553M     24486M       68912k          0.27 
POOLS:
    NAME       ID     USED     %USED     MAX AVAIL     OBJECTS 
    rbd        0         0         0        12243M           0 
    mypool     4         0         0        12243M           0 

5.2. Object Manipulation

To create an object and upload a file into that object:

$ echo "test data" > testfile
$ sudo rados put -p mypool testfile testfile
$ sudo rados -p mypool ls
testfile

To set a key/value pair to that object:

$ sudo rados -p mypool setomapval testfile mykey myvalue
$ sudo rados -p mypool getomapval testfile mykey
 (length 7) : 0000 : 6d 79 76 61 6c 75 65                            : myvalue

To download the file:

$ sudo rados get -p mypool testfile testfile2
$ md5sum testfile testfile2 
39a870a194a787550b6b5d1f49629236  testfile
39a870a194a787550b6b5d1f49629236  testfile2

6. Ceph client

6.1. System Name

• c7-ceph-client

6.2. Yum repository configuration

StorageSIG Ceph repositories are not public yet so this will not work. Pull builds tagged storage7-ceph-hammer-testing manually from cbs.centos.org instead.

Install the centos-release-ceph rpm from centos-extras:

# yum -y install --enablerepo=extras centos-release-ceph

6.3. Ceph packages

The following packages are necessary for C/C++, Python, and S3-like access to your Ceph cluster:

# yum -y install librados2-devel python-rados  python-boto

6.4. Configuring Ceph access

Coming soon.

7. RADOS C++ API

Coming soon.

8. RADOS Gateway Installation & Configuration

8.1. Installation

System name:

• c7-ceph-radosgw

Disable SELinux:

$ sudo sed -i -e 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
$ sudo reboot

Install the Apache web server and mod_ssl:

$ sudo yum -y install httpd mod_ssl openssl
$ sudo firewall-cmd --zone=public --add-port=80/tcp --add-port=443/tcp --permanent
$ sudo firewall-cmd --reload
success

Generate a self-signed certificate if no PKI is available in your test environment:

$ openssl genrsa -out ca.key 2048
$ openssl req -new -key ca.key -out ca.csr \
  -subj "/C=KI/ST=Phoenix Islands/L=Nikumaroro/O=NR16020/OU=Ren Tree/CN=$(hostname -f)"
$ openssl x509 -req -days 365 -in ca.csr -signkey ca.key -out ca.crt
$ sudo cp ca.crt /etc/pki/tls/certs
$ sudo cp ca.key /etc/pki/tls/private/ca.key
$ sudo cp ca.csr /etc/pki/tls/private/ca.csr

Where:

• /C=: 2 letter [ISO 3166 https://en.wikipedia.org/wiki/ISO_3166] country code (KI=Kiribati)

• /ST=: State or Province (Phoenix Islands)
• /L=: Locality (Nikumaroro)
• /O=: Organization Name (NR16020)
• /OU=: Organizational Unit Name (Ren Tree)
• /CN=: Common Name. Use the FQDN.

Configure the Apache web server:

$ sudo sed -i -e 's,^SSLCertificateFile.*,SSLCertificateFile\ /etc/pki/tls/certs/ca.crt,g' \
   /etc/httpd/conf.d/ssl.conf
$ sudo sed -i -e 's,SSLCertificateKeyFile.*,SSLCertificateKeyFile\ /etc/pki/tls/private/ca.key,g' \
   /etc/httpd/conf.d/ssl.conf

Start and enable Apache:

$ sudo service httpd configtest
Syntax OK
$ sudo systemctl start httpd
$ sudo systemctl enable httpd

As our cluster is a test/demo cluster, we are going to install a so-called Simple RADOS Gateway. Installing a Federated Gateway (with geographically distributed Ceph storage services) for fault-tolerance is out of scope of this document.

8.2. Simple RADOS Gateway Configuration

As ceph-deploy on c7-ceph-admin:

$ ceph-deploy install --rgw c7-ceph-radosgw
$ ceph-deploy rgw create c7-ceph-radosgw
(...)
[ceph_deploy.rgw][INFO  ] The Ceph Object Gateway (RGW) is now running on host c7-ceph-radosgw and default port 7480

As ceph-deploy on c7-ceph-radosgw:

$ sudo vi /etc/httpd/conf.d/rgw.conf

And insert the following:

<VirtualHost *:80>
ServerName localhost
DocumentRoot /var/www/html
ErrorLog /var/log/httpd/rgw_error.log
CustomLog /var/log/httpd/rgw_access.log combined
# LogLevel debug
RewriteEngine On
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]
SetEnv proxy-nokeepalive 1
ProxyPass / unix:///var/run/ceph/ceph.radosgw.gateway.fastcgi.sock|fcgi://localhost:9000/
</VirtualHost>

Edit the SSL VirtualHost configuration:

$ sudo vi /etc/httpd/conf.d/ssl.conf

and add at the end of the file, before the </VirtualHost> tag:

ErrorLog /var/log/httpd/rgw_error.log
CustomLog /var/log/httpd/rgw_access.log combined
# LogLevel debug
RewriteEngine On
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]
SetEnv proxy-nokeepalive 1
ProxyPass / unix:///var/run/ceph/ceph.radosgw.gateway.fastcgi.sock|fcgi://localhost:9000/

Restart httpd:

sudo systemctl restart httpd

Create a user:

$ sudo radosgw-admin user create --uid="testuser" --display-name="First User"

After a while (about 10 seconds on a SSD-backed Ceph cluster), the output should be like:

{
    "user_id": "testuser",
    "display_name": "First User",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [
        {
            "user": "testuser",
            "access_key": "ACCESS_KEY",
            "secret_key": "SECRET_KEY"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "temp_url_keys": []
}

Keep the values of ACCESS_KEY and SECRET_KEY.

8.3. Testing the Gateway using the S3 API

On c7-ceph-client, create a s3test.py file which contains:

import boto
import boto.s3.connection

access_key = 'ACCESS_KEY'
secret_key = 'SECRET_KEY'

conn = boto.connect_s3( aws_access_key_id = access_key,
                        aws_secret_access_key = secret_key,
                        host = 'c7-ceph-radosgw',
                        is_secure=True,
                        calling_format = boto.s3.connection.OrdinaryCallingFormat(),
                      )

bucket = conn.create_bucket('my-new-bucket')

for bucket in conn.get_all_buckets():
        print "{name}\t{created}".format( name = bucket.name,
                                          created = bucket.creation_date,
                                        )
bucket = conn.delete_bucket('my-new-bucket')

Launch it:

$ python s3test.py 
my-new-bucket   2016-01-30T22:05:30.000Z

At this point, you have a working S3 gateway. More information about python-boto can be found in the official tutorial.

9. MDS and CephFS installation

Coming soon.

10. Wrap Up

This page is inspired from the quick start guide of the Ceph Community.

Refer to the Ceph Community documentation home page for more information.