Skip to main content

ceph HEALTH_ERR 1 pgs inconsistent; 2 scrub errors

 

Today I found some errors in my ceph cluster:

$ ceph health detail
HEALTH_ERR 1 pgs inconsistent; 2 scrub errors
pg 4.b is active+clean+inconsistent, acting [6,13,15]
2 scrub errors

Looking for this pg in the primary OSD:

$ grep 4.b /var/log/ceph/*
/var/log/ceph/ceph-osd.6.log:2014-07-07 08:46:17.253344 7f49cdc55700 0 log [ERR] : 4.b shard 6: soid 35ea2f8b/rbd_data.18486a2ae8944a.0000000000002441/head//4 digest 2737967484 != known digest 1957685266
/var/log/ceph/ceph-osd.6.log:2014-07-07 08:46:17.253348 7f49cdc55700 0 log [ERR] : 4.b shard 13: soid 35ea2f8b/rbd_data.18486a2ae8944a.0000000000002441/head//4 digest 2737967484 != known digest 1957685266
/var/log/ceph/ceph-osd.6.log:2014-07-07 08:46:17.253494 7f49cdc55700 0 log [ERR] : 4.b deep-scrub 0 missing, 1 inconsistent objects
/var/log/ceph/ceph-osd.6.log:2014-07-07 08:46:17.253499 7f49cdc55700 0 log [ERR] : 4.b deep-scrub 2 errors

 

I tryed ceph pg repair command on this pg:

$ ceph pg repair 4.b
instructing pg 4.b on osd.6 to repair

 

And, after few minutes, the pg seems to be healthy again:

$ ceph health detail
HEALTH_OK

 

And querying this pg with ceph pg 4.b query command

{ "state": "active+clean",
  "epoch": 15555,
  "up": [
        6,
        13,
        15],
  "acting": [
        6,
        13,
        15],
  "actingbackfill": [
        "6",
        "13",
        "15"],

 

Looking for this pg in the ceph log again shows that this was fixed:

$ grep 4.b /var/log/ceph/*
/var/log/ceph/ceph-osd.6.log:2014-07-07 09:43:10.062293 7f49cdc55700 0 log [ERR] : 4.b shard 6: soid 35ea2f8b/rbd_data.18486a2ae8944a.0000000000002441/head//4 digest 2737967484 != known digest 1957685266
/var/log/ceph/ceph-osd.6.log:2014-07-07 09:43:10.062306 7f49cdc55700 0 log [ERR] : 4.b shard 13: soid 35ea2f8b/rbd_data.18486a2ae8944a.0000000000002441/head//4 digest 2737967484 != known digest 1957685266
/var/log/ceph/ceph-osd.6.log:2014-07-07 09:43:10.062423 7f49cdc55700 0 log [ERR] : 4.b repair 0 missing, 1 inconsistent objects
/var/log/ceph/ceph-osd.6.log:2014-07-07 09:43:10.062447 7f49cdc55700 0 log [ERR] : 4.b repair 2 errors, 2 fixed

Notice: Undefined variable: wp_sh_class_name in /var/www/elkano.org/blog/wp-content/plugins/wp-syntaxhighlighter/wp-syntaxhighlighter.php on line 1002

Notice: Undefined variable: wp_sh_class_name in /var/www/elkano.org/blog/wp-content/plugins/wp-syntaxhighlighter/wp-syntaxhighlighter.php on line 1002

Warning: Use of undefined constant XML - assumed 'XML' (this will throw an Error in a future version of PHP) in /var/www/elkano.org/blog/wp-content/plugins/wp-syntaxhighlighter/wp-syntaxhighlighter.php on line 1048