Skip to main content

Tag: ceph

mkfs.xfs: cannot open /dev/[device]: Device or resource busy

sysadmin 1 Comment

If you hit this error trying to format a partition after you have installed a new disk in your server:

~# mkfs -t xfs -f -i size=2048 /dev/sdav2
mkfs.xfs: cannot open /dev/sdav2: Device or resource busy

Maybe you have multipath enabled and the new device has been detected. Run this command to see if multipath has detected it.

~# multipath -ll /dev/sdav
2796a30585a4b6e45 dm-12 ,
size=2.2T features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=0 status=active
| `- #:#:#:# sdav 66:48  active undef running

then to fix it, remove from multipath:

~# multipath -f /dev/sdav

And finally you can format it without any problem:

~# mkfs -t xfs -f -i size=2048 /dev/sdav2
meta-data=/dev/device            isize=2048   agcount=4, agsize=3277258 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=13109032, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=6400, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Cómo saber si los discos soportan la opción discard para liberar espacio

sysadmin Leave a comment

La opción discard de los diospositivos de bloques nos permite liberar el espacio de los discos de forma efectiva cuando en el sistema de ficheros borramos ficheros.
Para saber si tenemos la opción discard activada y podemos liberar espacio en el dispositivo se puede ejecutar este comando en linux:

$ sudo lsblk -o MOUNTPOINT,DISC-MAX,FSTYPE

En el caso de que los dispositivos no lo soporten aparecerá en la colúmna DISC-MAX 0B:

MOUNTPOINT DISC-MAX FSTYPE
/boot      0B      ext4
/          0B      ext4
/usr       0B      ext4
/var/tmp   0B      ext4
/var       0B      ext4
/home      0B      ext4

 

Con la opción discard activada nos aparecerá bajo la columna DISC-MAX el tamaño máximo de bytes descartables:

MOUNTPOINT DISC-MAX FSTYPE
/boot      1G       ext4
/          1G       ext4
/usr       1G       ext4
/var/tmp   1G       ext4
/var       1G       ext4
/home      1G       ext4

 

 

Otra opción posible es usando la opción -D del mismo comando que nos proporciona algo más de información:

$ sudo lsblk -D
NAME            DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO
sda                    0        4K       1G         0
├─sda1                 0        4K       1G         0
└─sda2                 0        4K       1G         0
  ├─vgsys-root         0        4K       1G         0
  ├─vgsys-usr          0        4K       1G         0
  ├─vgsys-tmp          0        4K       1G         0
  └─vgsys-var          0        4K       1G         0
sdb                    0        4K       1G         0
└─sdb1                 0        4K       1G         0
  └─vgdata-home        0        4K       1G         0
sr0                    0        0B       0B         0

Una vez que sabemos que el dispositivo soporta la opción DISCARD podemos ejecutar el comando fstrim para liberar espacio en el backend.

CEPH: SATA and SSD pools on the same server without editing crushmap

sysadmin 3 Comments

I had some free slots in two on my ceph nodes and I used them to set a new SSD only pool. Because the ssd OSDs are sharing the server with an existing SATA pool we have to do some additional steps. There are some good posts out there that explain how to set up two pools sharing the same server, but you have to edit manually the ceph crushmap. Although it’s not very difficult to do it in that way I achieve the same goal setting the crush location parameter for those OSDs. I’ve tested it in Hammer release.

First create a new root bucket for the ssd pool. This bucket will be used to set the ssd pool location using a crush rule.

ceph osd crush add-bucket ssds root

We already have some servers with SATA OSDs in production, but we have to add two new host buckets for the faked hostnames that we are going to use to set the ssd OSDs.

ceph osd crush add-bucket ceph-node1-ssd  host
ceph osd crush add-bucket ceph-node2-ssd  host

Move the host buckets to the ssds root:

ceph osd crush move ceph-node1-ssd root=ssds
ceph osd crush move ceph-node2-ssd root=ssds

In the ceph configuration file (ceph.conf) set the crush location for the SSD OSDs. This is necessary because the default location is always the hostname obtained with the command hostname -s 

[osd.35]
host =  ceph-node1
osd_journal = /dev/disk/by-id/ata-INTEL_SSDSC2BB016T6_BTWA543204R11P6KGN-part1
crush_location =  root=ssds host=ceph-node1-ssd

You can check the location of the osd running this command:

$ ceph-crush-location --id 35 --type osd
 root=ssds host=ceph-node1-ssd

For each new ssd OSD move the osd to ssds root:

ceph osd crush add 35 1.5 root=ssds
ceph osd crush set osd.35 1.5 root=ssds host=ceph-node1-ssd

Create a new SSD pool:

ceph osd pool create ssdpool 128 128

Crate a crush rule in the ssds root:

ceph osd crush rule create-simple ssdpool ssds host

Finally assign the new rule to the ssdpool:

$ ceph osd pool set ssdpool crush_ruleset 4
 set pool 5 crush_ruleset to 4

Got! Now, we have a new only SSD pool:

$ ceph osd tree
ID  WEIGHT   TYPE NAME                       UP/DOWN REWEIGHT PRIMARY-AFFINITY 
-25  3.00000 root ssds                                                    
-26  1.50000     host ceph-node1-ssd                                          
 35  1.50000         osd.35                       up  1.00000          1.00000 
-27  1.50000     host ceph-node2-ssd                                          
 36  1.50000         osd.36                       up  1.00000          1.00000 
-21 48.22978 root sata                                                    
-24  6.50995     host ceph-node1                                              
  1  0.92999         osd.1                        up  1.00000          1.00000 
  5  0.92999         osd.5                        up  1.00000          1.00000 
 16  0.92999         osd.16                       up  1.00000          1.00000 
 17  0.92999         osd.17                       up  1.00000          1.00000 
 18  0.92999         osd.18                       up  1.00000          1.00000 
 19  0.92999         osd.19                       up  1.00000          1.00000 
 20  0.92999         osd.20                       up  1.00000          1.00000 
-22  6.50995     host ceph-node2                                              
 21  0.92999         osd.21                       up  1.00000          1.00000 
 22  0.92999         osd.22                       up  1.00000          1.00000 
 23  0.92999         osd.23                       up  1.00000          1.00000 
 24  0.92999         osd.24                       up  1.00000          1.00000 
 25  0.92999         osd.25                       up  1.00000          1.00000 
 26  0.92999         osd.26                       up  1.00000          1.00000 
 27  0.92999         osd.27                       up  1.00000          1.00000 
 -7 13.29996     host ceph-node3                                              
  0  1.89999         osd.0                        up  1.00000          1.00000 
  6  1.89999         osd.6                        up  1.00000          1.00000 
  9  1.89999         osd.9                        up  1.00000          1.00000 
 11  1.89999         osd.11                       up  1.00000          1.00000 
 14  1.89999         osd.14                       up  1.00000          1.00000 
 15  1.89999         osd.15                       up  1.00000          1.00000 
  2  1.89999         osd.2                        up  1.00000          1.00000 
-20  6.50995     host ceph-node4                                              
 28  0.92999         osd.28                       up  1.00000          1.00000 
 29  0.92999         osd.29                       up  1.00000          1.00000 
 30  0.92999         osd.30                       up  1.00000          1.00000 
 31  0.92999         osd.31                       up  1.00000          1.00000 
 32  0.92999         osd.32                       up  1.00000          1.00000 
 33  0.92999         osd.33                       up  1.00000          1.00000 
 34  0.92999         osd.34                       up  1.00000          1.00000 
-14 15.39998     host ceph-node5                                              
  3  2.20000         osd.3                        up  1.00000          1.00000 
  4  2.20000         osd.4                        up  1.00000          1.00000 
  7  2.20000         osd.7                        up  1.00000          1.00000 
  8  2.20000         osd.8                        up  1.00000          1.00000 
 10  2.20000         osd.10                       up  1.00000          1.00000 
 12  2.20000         osd.12                       up  1.00000          1.00000 
 13  2.20000         osd.13                       up  1.00000          1.00000 
[...]

Changing osd configuration without restarting

sysadmin Leave a comment

If you want to change some settings in your OSDs but you don’t want to reboot them this is the way to do it. There are two ways to change the settings on the osd:

If you are in a server where a specific OSD is running on like the OSD.3 in the example, you can use ceph daemon command to get and set settings on the OSD:

# ceph daemon osd.3 config get osd_deep_scrub_interval
{
    "osd_deep_scrub_interval": "604800"
}

 ceph daemon osd.3 config set osd_deep_scrub_interval 4838400
{
"success": "osd_deep_scrub_interval = '4.8384e+06' "
}

If you want to change settings in remote OSDs or you want to change all OSDs at once you can use injectargs:

ceph tell osd.4 injectargs '--osd-deep-scrub-interval 4838400'
osd_deep_scrub_interval = '4.8384e+06'

To change all of them:

ceph tell osd.* injectargs '--osd-deep-scrub-interval 4838400'

Radosgw list current created users

sysadmin Leave a comment

I haven’t found in the Ceph Object Gateway docs a command to get the radosgw active users, so I was looking for it and finally inspecting the objects inside the rados gateway pools, I found the way to get the list:

# rados -p .users.uid ls | grep -v buckets
user1
user2
user3

And another way using radosgw-admin command (recently discovered): 

# radosgw-admin metadata list user
[
    "user1",
    "user2",
    "user3"

]

# radosgw-admin user info --uid=user1
{
    "user_id": "user1",
    "display_name": "User 1",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "temp_url_keys": []
}