{"id":875,"date":"2014-07-07T10:13:36","date_gmt":"2014-07-07T08:13:36","guid":{"rendered":"http:\/\/elkano.org\/blog\/?p=875"},"modified":"2014-07-07T10:13:36","modified_gmt":"2014-07-07T08:13:36","slug":"ceph-health_err-1-pgs-inconsistent-2-scrub-errors","status":"publish","type":"post","link":"https:\/\/elkano.org\/blog\/ceph-health_err-1-pgs-inconsistent-2-scrub-errors\/","title":{"rendered":"ceph HEALTH_ERR 1 pgs inconsistent; 2 scrub errors"},"content":{"rendered":"<p>&nbsp;<\/p>\n<p>Today I found some errors in my ceph cluster:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"null\" data-enlighter-theme=\"enlighter\" data-enlighter-linenumbers=\"false\">$ ceph health detail\r\nHEALTH_ERR 1 pgs inconsistent; 2 scrub errors\r\npg 4.b is active+clean+inconsistent, acting [6,13,15]\r\n2 scrub errors\r\n<\/pre>\n<p>Looking for this pg in the primary OSD:<\/p>\n<pre class=\"brush: text; gutter: false\">$ grep 4.b \/var\/log\/ceph\/*\r\n\/var\/log\/ceph\/ceph-osd.6.log:2014-07-07 08:46:17.253344 7f49cdc55700 0 log [ERR] : 4.b shard 6: soid 35ea2f8b\/rbd_data.18486a2ae8944a.0000000000002441\/head\/\/4 digest 2737967484 != known digest 1957685266\r\n\/var\/log\/ceph\/ceph-osd.6.log:2014-07-07 08:46:17.253348 7f49cdc55700 0 log [ERR] : 4.b shard 13: soid 35ea2f8b\/rbd_data.18486a2ae8944a.0000000000002441\/head\/\/4 digest 2737967484 != known digest 1957685266\r\n\/var\/log\/ceph\/ceph-osd.6.log:2014-07-07 08:46:17.253494 7f49cdc55700 0 log [ERR] : 4.b deep-scrub 0 missing, 1 inconsistent objects\r\n\/var\/log\/ceph\/ceph-osd.6.log:2014-07-07 08:46:17.253499 7f49cdc55700 0 log [ERR] : 4.b deep-scrub 2 errors<\/pre>\n<p>&nbsp;<\/p>\n<p>I tryed\u00a0ceph pg repair command on this pg:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"null\" data-enlighter-theme=\"enlighter\" data-enlighter-linenumbers=\"false\">$ ceph pg repair 4.b\r\ninstructing pg 4.b on osd.6 to repair<\/pre>\n<p>&nbsp;<\/p>\n<p>And, after few minutes, the pg seems to be healthy again:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"null\" data-enlighter-theme=\"enlighter\" data-enlighter-linenumbers=\"false\">$ ceph health detail\r\nHEALTH_OK\r\n<\/pre>\n<p>&nbsp;<\/p>\n<p>And querying this pg with <em>ceph pg 4.b query<\/em> command<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"null\" data-enlighter-theme=\"enlighter\" data-enlighter-linenumbers=\"false\">{ &quot;state&quot;: &quot;active+clean&quot;,\r\n  &quot;epoch&quot;: 15555,\r\n  &quot;up&quot;: [\r\n        6,\r\n        13,\r\n        15],\r\n  &quot;acting&quot;: [\r\n        6,\r\n        13,\r\n        15],\r\n  &quot;actingbackfill&quot;: [\r\n        &quot;6&quot;,\r\n        &quot;13&quot;,\r\n        &quot;15&quot;],\r\n<\/pre>\n<p>&nbsp;<\/p>\n<p>Looking for this pg in the ceph log again shows that this was fixed:<\/p>\n<pre class=\"brush: text; gutter: false\">$\u00a0grep 4.b \/var\/log\/ceph\/*\r\n\/var\/log\/ceph\/ceph-osd.6.log:2014-07-07 09:43:10.062293 7f49cdc55700 0 log [ERR] : 4.b shard 6: soid 35ea2f8b\/rbd_data.18486a2ae8944a.0000000000002441\/head\/\/4 digest 2737967484 != known digest 1957685266\r\n\/var\/log\/ceph\/ceph-osd.6.log:2014-07-07 09:43:10.062306 7f49cdc55700 0 log [ERR] : 4.b shard 13: soid 35ea2f8b\/rbd_data.18486a2ae8944a.0000000000002441\/head\/\/4 digest 2737967484 != known digest 1957685266\r\n\/var\/log\/ceph\/ceph-osd.6.log:2014-07-07 09:43:10.062423 7f49cdc55700 0 log [ERR] : 4.b repair 0 missing, 1 inconsistent objects\r\n\/var\/log\/ceph\/ceph-osd.6.log:2014-07-07 09:43:10.062447 7f49cdc55700 0 log [ERR] : 4.b repair 2 errors, 2 fixed\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"<p>&nbsp; Today I found some errors in my ceph cluster: $ ceph health detail HEALTH_ERR 1 pgs inconsistent; 2 scrub errors pg 4.b is active+clean+inconsistent, acting [6,13,15] 2 scrub errors Looking for this pg in the primary OSD: $ grep 4.b \/var\/log\/ceph\/* \/var\/log\/ceph\/ceph-osd.6.log:2014-07-07 08:46:17.253344 7f49cdc55700 0 log [ERR] : 4.b shard 6: soid 35ea2f8b\/rbd_data.18486a2ae8944a.0000000000002441\/head\/\/4 digest [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[120],"tags":[121,123,122],"_links":{"self":[{"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/posts\/875"}],"collection":[{"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/comments?post=875"}],"version-history":[{"count":9,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/posts\/875\/revisions"}],"predecessor-version":[{"id":884,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/posts\/875\/revisions\/884"}],"wp:attachment":[{"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/media?parent=875"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/categories?post=875"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/tags?post=875"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}