Robbat2 (robbat2) wrote,

Ceph: RBD resizing workarounds, Python API example

Some mis-reading the help for 'rbd resize' recently lead somebody to making an RBD volume extremely large, if the units are bytes, this command makes sense:
rbd resize --size $((155*1024*1024*1024)) $RBD_VOL_NAME
But, as it happens, the units are actually in MiB.

# rbd info $RBD_VOL_NAME
rbd image '$RBD_VOL_NAME':
 size 155 PB in 41607495680 objects
Oops, that's a bit too big, and I doubt your cluster has that much space for a single volume, even if it's sparsely allocated.

There is the --allow-shrink option to rbd resize, however on a volume this large, at least in the Hammer release, it will pretty much never return (it eventually times out).

Future work for RBD resize might include a much more intelligent resize command, that checks the highest non-null block, perhaps a reverse iterator looking for keys named at most the size of the device.
Making resize ask for confirmation with output of the new size, or have a dry-run option would also be helpful to avoid this problem.

So, as a workaround, I give you the following useful script, that manually changes the size omap field directly. It is wildly unsafe if you are already using anything past the target size, but it's fine otherwise.
You need to ensure nothing is using the volume when you apply this...

# Prerequisite: From Infernalis source or newer, src/pybind/, src/pybind/ 
# Make sure you have this fix:
from rados import Rados, ReadOpCtx, WriteOpCtx
import rbd
import struct
import ctypes
RBD_NAME = '...'
NEWSIZE = 155 * 1024 * 1024 * 1024 # 155 GiB

with Rados(conffile='/etc/ceph/ceph.conf') as r:
  print 'RADOS object', r
  print 'RADOS version', r.version()
  print 'Cluster Stats', r.get_cluster_stats()
  with r.open_ioctx('rbd') as ioctx:
    with rbd.Image(ioctx, RBD_NAME, read_only=True) as img:
      imgstat = img.stat()
      print imgstat
      header_name = imgstat['block_name_prefix'].replace('rbd_data.', 'rbd_header.', 1)

    with ReadOpCtx(ioctx) as read_op:
      kv_it = ioctx.get_omap_vals(read_op, "", "", 500)
      ioctx.operate_read_op(read_op, header_name)
      kv = dict(kv_it[0])
      print 'OMAP of', header_name, kv
    newkeys = ('size', )
    newvals = (struct.pack('<Q', NEWSIZE), ) # unsigned long long, little endian
    with WriteOpCtx(ioctx) as write_op:
      ioctx.set_omap(write_op, newkeys, newvals)
      ioctx.operate_write_op(write_op, header_name)
    with rbd.Image(ioctx, RBD_NAME, read_only=True) as img:
      imgstat = img.stat()
      print imgstat

Tags: ceph, python, rbd
  • Post a new comment


    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded