Ceph status shows HEALTH_ERR with MDSs report damaged metadata

Environment

Red Hat Ceph Storage 3.x and 4.x with CephFS

Issue

ceph -s showing HEALTH_ERR, with the following line:

1 MDSs report damaged metadata

The output of damage ls shows a result similar to the following:

# ceph tell mds.$filesystem:0 damage ls
[
    {
        "damage_type": "backtrace",
        "id": 2091078454,
        "ino": 1,
        "path": ""

Resolution

** ATTENTION **

Before applying this fix confirm:
- the `"damage_type"` is listed as `"backtrace"`.
- the `"ino"` value is 1, which indicates that the damaged metadata corresponds to the root of the filesystem.

In the following process, the $filesystem should be the name the CephFS Filesystem which has the damaged metadata. This is not the hostname of the MDS. Please modify the value to match the proper MDS in your environment.

To resolve this reported damaged metadata, proceed as follows:

1. Force repair of the root of the CephFS filesystem:

   # ceph tell mds.$filesystem:0 scrub start / force repair
   

2. Once the repair completes, delete the damage entry where $id is the "id" value from the damage ls output obtained earlier:

   # ceph tell mds.$filesystem:0 damage rm $id
   

3. Confirm the ceph -s output no longer shows damaged metadata and the health line shows HEALTH_OK

Root Cause

The backtrace information for the root inode was not written in older versions of CephFS, thus it will show up as damaged when scrubbed. The process above repairs inode 1 and writes the backtrace information.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.