CEPH: SATA and SSD pools on the same server without editing crushmap

I had some free slots in two on my ceph nodes and I used them to set a new SSD only pool. Because the ssd OSDs are sharing the server with an existing SATA pool we have to do some additional steps. There are some good posts out there that explain how to set up two pools sharing the same server, but you have to edit manually the ceph crushmap. Although it’s not very difficult to do it in that way I achieve the same goal setting the crush location parameter for those OSDs. I’ve tested it in Hammer release.

First create a new root bucket for the ssd pool. This bucket will be used to set the ssd pool location using a crush rule.

ceph osd crush add-bucket ssds root

We already have some servers with SATA OSDs in production, but we have to add two new host buckets for the faked hostnames that we are going to use to set the ssd OSDs.

ceph osd crush add-bucket ceph-node1-ssd  host
ceph osd crush add-bucket ceph-node2-ssd  host

Move the host buckets to the ssds root:

ceph osd crush move ceph-node1-ssd root=ssds
ceph osd crush move ceph-node2-ssd root=ssds

In the ceph configuration file (ceph.conf) set the crush location for the SSD OSDs. This is necessary because the default location is always the hostname obtained with the command hostname -s

[osd.35]
host =  ceph-node1
osd_journal = /dev/disk/by-id/ata-INTEL_SSDSC2BB016T6_BTWA543204R11P6KGN-part1
crush_location =  root=ssds host=ceph-node1-ssd

You can check the location of the osd running this command:

$ ceph-crush-location --id 35 --type osd
 root=ssds host=ceph-node1-ssd

For each new ssd OSD move the osd to ssds root:

ceph osd crush add 35 1.5 root=ssds
ceph osd crush set osd.35 1.5 root=ssds host=ceph-node1-ssd

Create a new SSD pool:

ceph osd pool create ssdpool 128 128

Crate a crush rule in the ssds root:

ceph osd crush rule create-simple ssdpool ssds host

Finally assign the new rule to the ssdpool:

$ ceph osd pool set ssdpool crush_ruleset 4
 set pool 5 crush_ruleset to 4

Got! Now, we have a new only SSD pool:

$ ceph osd tree
ID  WEIGHT   TYPE NAME                       UP/DOWN REWEIGHT PRIMARY-AFFINITY 
-25  3.00000 root ssds                                                    
-26  1.50000     host ceph-node1-ssd                                          
 35  1.50000         osd.35                       up  1.00000          1.00000 
-27  1.50000     host ceph-node2-ssd                                          
 36  1.50000         osd.36                       up  1.00000          1.00000 
-21 48.22978 root sata                                                    
-24  6.50995     host ceph-node1                                              
  1  0.92999         osd.1                        up  1.00000          1.00000 
  5  0.92999         osd.5                        up  1.00000          1.00000 
 16  0.92999         osd.16                       up  1.00000          1.00000 
 17  0.92999         osd.17                       up  1.00000          1.00000 
 18  0.92999         osd.18                       up  1.00000          1.00000 
 19  0.92999         osd.19                       up  1.00000          1.00000 
 20  0.92999         osd.20                       up  1.00000          1.00000 
-22  6.50995     host ceph-node2                                              
 21  0.92999         osd.21                       up  1.00000          1.00000 
 22  0.92999         osd.22                       up  1.00000          1.00000 
 23  0.92999         osd.23                       up  1.00000          1.00000 
 24  0.92999         osd.24                       up  1.00000          1.00000 
 25  0.92999         osd.25                       up  1.00000          1.00000 
 26  0.92999         osd.26                       up  1.00000          1.00000 
 27  0.92999         osd.27                       up  1.00000          1.00000 
 -7 13.29996     host ceph-node3                                              
  0  1.89999         osd.0                        up  1.00000          1.00000 
  6  1.89999         osd.6                        up  1.00000          1.00000 
  9  1.89999         osd.9                        up  1.00000          1.00000 
 11  1.89999         osd.11                       up  1.00000          1.00000 
 14  1.89999         osd.14                       up  1.00000          1.00000 
 15  1.89999         osd.15                       up  1.00000          1.00000 
  2  1.89999         osd.2                        up  1.00000          1.00000 
-20  6.50995     host ceph-node4                                              
 28  0.92999         osd.28                       up  1.00000          1.00000 
 29  0.92999         osd.29                       up  1.00000          1.00000 
 30  0.92999         osd.30                       up  1.00000          1.00000 
 31  0.92999         osd.31                       up  1.00000          1.00000 
 32  0.92999         osd.32                       up  1.00000          1.00000 
 33  0.92999         osd.33                       up  1.00000          1.00000 
 34  0.92999         osd.34                       up  1.00000          1.00000 
-14 15.39998     host ceph-node5                                              
  3  2.20000         osd.3                        up  1.00000          1.00000 
  4  2.20000         osd.4                        up  1.00000          1.00000 
  7  2.20000         osd.7                        up  1.00000          1.00000 
  8  2.20000         osd.8                        up  1.00000          1.00000 
 10  2.20000         osd.10                       up  1.00000          1.00000 
 12  2.20000         osd.12                       up  1.00000          1.00000 
 13  2.20000         osd.13                       up  1.00000          1.00000 
[...]