Concept behind Ceph OSD weight

Environment

• Red Hat Ceph Storage

Issue

• Concept behind Ceph OSD weight
• OSDs weight configuration

Currently we are using 3TB hard drives for ceph OSDs and each of this OSDs has weight=1.
What is the general recommendation of weights configuration? 
Does it make sense to configure OSD weight that reflects capacity of the OSD (i.e. weight=4 for 4TB OSDs and weight=3 for 3TB OSDs)?

Resolution

• There are two types of weight for an osd one is the crush weight, and second is the OSD weight.
• By default, the ceph OSD startup init will use df to determine the size of the OSD. Upon starting, the crush weight will be set automatically based on the drive's usable space. e.g. a 2TB OSD may have a crush weight set of 1.6, whereas a 1TB drive may automatically update with a weight of 0.8. These weights are displayed in the LEFT weight column when dumping a CRUSH map.
• The OSD weight accepts values only between 0 and 1. An OSD weight of 0 is equivalent to the OSD being marked OUT of the cluster. This is the RIGHT-most column in a decompiled CRUSH map. It is essentially an override to the item's CRUSH weight, and will not affect the weights assigned to the buckets above the OSD in the crush map.
• In other words, if using the default startup scripts and traditional methods of deploying OSDs, then we SHOULD NOT NEED to modify the weights. If you wish to use a custom ratio, you may modify the crush weight. If you need to override or weight out a disk gradually, or correct for skewed placement, then you should use the OSD weight.
• References : Content from ceph.com is not included.osd-subsystem (search "ceph osd reweight {osd-num} {weight}")
• Some example from my test cluster:

#sudo ceph osd tree
# id	weight	type name	up/down	reweight
-1	0.9899	root default
-2	0.33		host node1
0	0.11			osd.0	up	1	
3	0.11			osd.3	up	1	
6	0.11			osd.6	up	1	
-3	0.33		host node2
1	0.11			osd.1	up	1	
4	0.11			osd.4	up	1	
7	0.11			osd.7	up	1	
-4	0.33		host node3
2	0.11			osd.2	up	1	
5	0.11			osd.5	up	1	
8	0.11			osd.8	up	1	

• In above given example osd of size 110G and all osds are up and in.
• In above given example , weight is OSD crush weight and reweight is OSD weight.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.