Today I found some errors in my ceph cluster:
$ ceph health detail HEALTH_ERR 1 pgs inconsistent; 2 scrub errors pg 4.b is active+clean+inconsistent, acting [6,13,15] 2 scrub errors
Looking for this pg in the primary OSD:
$ grep 4.b /var/log/ceph/* /var/log/ceph/ceph-osd.6.log:2014-07-07 08:46:17.253344 7f49cdc55700 0 log [ERR] : 4.b shard 6: soid 35ea2f8b/rbd_data.18486a2ae8944a.0000000000002441/head//4 digest 2737967484 != known digest 1957685266 /var/log/ceph/ceph-osd.6.log:2014-07-07 08:46:17.253348 7f49cdc55700 0 log [ERR] : 4.b shard 13: soid 35ea2f8b/rbd_data.18486a2ae8944a.0000000000002441/head//4 digest 2737967484 != known digest 1957685266 /var/log/ceph/ceph-osd.6.log:2014-07-07 08:46:17.253494 7f49cdc55700 0 log [ERR] : 4.b deep-scrub 0 missing, 1 inconsistent objects /var/log/ceph/ceph-osd.6.log:2014-07-07 08:46:17.253499 7f49cdc55700 0 log [ERR] : 4.b deep-scrub 2 errors
I tryed ceph pg repair command on this pg:
$ ceph pg repair 4.b instructing pg 4.b on osd.6 to repair
And, after few minutes, the pg seems to be healthy again:
$ ceph health detail HEALTH_OK
And querying this pg with ceph pg 4.b query command
{ "state": "active+clean",
"epoch": 15555,
"up": [
6,
13,
15],
"acting": [
6,
13,
15],
"actingbackfill": [
"6",
"13",
"15"],
Looking for this pg in the ceph log again shows that this was fixed:
$ grep 4.b /var/log/ceph/* /var/log/ceph/ceph-osd.6.log:2014-07-07 09:43:10.062293 7f49cdc55700 0 log [ERR] : 4.b shard 6: soid 35ea2f8b/rbd_data.18486a2ae8944a.0000000000002441/head//4 digest 2737967484 != known digest 1957685266 /var/log/ceph/ceph-osd.6.log:2014-07-07 09:43:10.062306 7f49cdc55700 0 log [ERR] : 4.b shard 13: soid 35ea2f8b/rbd_data.18486a2ae8944a.0000000000002441/head//4 digest 2737967484 != known digest 1957685266 /var/log/ceph/ceph-osd.6.log:2014-07-07 09:43:10.062423 7f49cdc55700 0 log [ERR] : 4.b repair 0 missing, 1 inconsistent objects /var/log/ceph/ceph-osd.6.log:2014-07-07 09:43:10.062447 7f49cdc55700 0 log [ERR] : 4.b repair 2 errors, 2 fixed