OpenIO : Add a New Volume to Existing Infrastructure




Background

You are archiving (a lot), and soon, your archiving platform become full. How to solve this ? Following the below procedure, you will add an extra volume to the OpenIO platform and continue to archive, again and again.

Let's add some storage

First, you need to add the storage on the node itself. This can be done with physical disks or adding a virtual one with your hypervisor. I'm supposing this step is done and the new device is already seen by the OS.

Step 1 - Create a new disk and add XFS file system on it

[root@be-openio-01 ~]# fdisk -l

[...]
Disk /dev/sdc: 107.4 GB, 107374182400 bytes, 209715200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 1048576 bytes


In our case, this is /dev/sdc, let's create a new partition with fdisk and add file system with mkfs.xfs

[root@be-openio-01 ~]# fdisk /dev/sdc
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x3a440cd0.

The device presents a logical sector size that is smaller than
the physical sector size. Aligning to a physical sector (or optimal
I/O) size boundary is recommended, or performance may be impacted.

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 
First sector (2048-209715199, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-209715199, default 209715199): 
Using default value 209715199
Partition 1 of type Linux and of size 100 GiB is set

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.


Syncing disks.
Let's add a file system on it
[root@be-openio-01 ~]# mkfs.xfs /dev/sdc1
meta-data=/dev/sdc1              isize=512    agcount=4, agsize=6553536 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=26214144, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=12799, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

mount new volume

Next, we need to create an additional mount point and mount the volume

[root@be-openio-01 ~]# mkdir /data2
[root@be-openio-01 ~]# mount /dev/sdc1 /data2
[root@be-openio-01 ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
[...]
/dev/sdc1                100G   33M  100G   1% /data2

We also need to add it to /etc/fstab so the volume will automatically be mounted at boot time.

/dev/sdc1       /data2 xfs defaults 1 1

Last step in the storage preparations tasks is creating the relevant folders and set permissions on them.

[root@be-openio-01 ~]# mkdir /data2/sds
[root@be-openio-01 ~]# mkdir /data2/sds/OPENIO
[root@be-openio-01 ~]# mkdir /data2/sds/OPENIO/rdir-2
[root@be-openio-01 ~]# mkdir /data2/sds/OPENIO/rawx-2
[root@be-openio-01 ~]# chown -R openio:openio /data2

Now, we have the new storage ready on our server. We can start telling OpenIO to make good use of it !

Step 2 - Modify the puppet manifest with new disk details

Open the openio.pp manifest you used for the initial deployment and :
  • Change the service name : rawx-0 -> rawx-1, ....
  • Increment the "num"
  • Adjust the documentRoot path with the new mount point you have created above
openiosds::rawx {'rawx-2':
  num => 2,
  port => 6203,
  ns => 'OPENIO',
  ipaddress => $ipaddr,
  documentRoot => '/data2/sds/OPENIO/rawx-2',
}
openiosds::rdir{'rdir-2':
  num => 2,
  port => 6303,
  ns => 'OPENIO',
  ipaddress => $ipaddr,
  db_path=> '/data2/sds/OPENIO/rdir-2',
}
openiosds::oioblobindexer{'oio-blob-indexer-rawx-2':
  num => 2,
  ns => 'OPENIO',
  volume=> '/data2/sds/OPENIO/rawx-2',
}

Re-apply puppet manifest

[root@be-openio-01 ~]# puppet apply --no-stringify_facts ~/openio.pp
Warning: Config file /etc/puppet/hiera.yaml not found, using Hiera defaults
Notice: Compiled catalog for be-openio-01 in environment production in 2.92 seconds
Warning: The package type's allow_virtual parameter will be changing its default value from false to true in a future release. If you do not want to allow virtual packages, please explicitly set allow_virtual to false.
[...]
   (at /usr/share/ruby/vendor_ruby/puppet/type.rb:816:in `set_default')
Notice: /Stage[main]/Main/Openiosds::Rdir[rdir-2]/Openiosds::Service[OPENIO-rdir-3]/File[/data2/sds/OPENIO/rdir-2]/owner: owner changed 'root' to 'openio'
Notice: /Stage[main]/Main/Openiosds::Rdir[rdir-2]/Openiosds::Service[OPENIO-rdir-3]/File[/data2/sds/OPENIO/rdir-2]/group: group changed 'root' to 'openio'
[...]
Notice: Finished catalog run in 3.53 seconds

Next, we need to initialise the newly created data volume with the bootstrap mechanism

[root@be-openio-01 ~]# openio volume admin bootstrap --oio-ns OPENIO
+--------------------+--------------------+---------------+---------------+
| Rdir               | Rawx               | Rdir location | Rawx location |
+--------------------+--------------------+---------------+---------------+
| 192.168.8.109:6010 | 192.168.8.110:6004 | be-openio-02  | be-openio-03  |
| 192.168.8.109:6010 | 192.168.8.110:6202 | be-openio-02  | be-openio-03  |
| 192.168.8.109:6010 | 192.168.8.139:6004 | be-openio-02  | be-openio-01  |
| 192.168.8.110:6010 | 192.168.8.109:6004 | be-openio-03  | be-openio-02  |
| 192.168.8.110:6010 | 192.168.8.139:6202 | be-openio-03  | be-openio-01  |
| 192.168.8.139:6302 | 192.168.8.109:6202 | be-openio-01  | be-openio-02  |
+--------------------+--------------------+---------------+---------------+

Lastly, restart the services

[root@be-openio-01 ~]# gridinit_cmd restart
DONE            OPENIO-oio-blob-indexer-3       Success
DONE            OPENIO-rawx-3   Success
DONE            OPENIO-rdir-3   Success
DONE            OPENIO-rawx-2   Success
DONE            OPENIO-rdir-2   Success
DONE            OPENIO-oio-blob-indexer-2       Success
DONE            OPENIO-memcached-0      Success
DONE            OPENIO-oioswift-0       Success
DONE            OPENIO-rawx-0   Success
DONE            OPENIO-conscience-0     Success
DONE            OPENIO-zookeeper-0      Success
DONE            OPENIO-oioproxy-0       Success
DONE            OPENIO-redissentinel-0  Success
DONE            OPENIO-rdir-0   Success
DONE            OPENIO-oio-blob-indexer-0       Success
DONE            OPENIO-oio-event-agent-0        Success
DONE            OPENIO-redis-0  Success
DONE            OPENIO-meta2-0  Success
DONE            OPENIO-meta1-0  Success
DONE            OPENIO-account-0        Success
DONE            OPENIO-conscienceagent-0        Success
DONE            OPENIO-meta0-0  Success
DONE            OPENIO-beanstalkd-0     Success

Never forget to unlock the repositories

[root@be-openio-01 ~]# openio cluster unlockall --oio-ns OPENIO
+---------+--------------------+----------+
| Type    | Service            | Result   |
+---------+--------------------+----------+
| rdir    | 192.168.8.109:6302 | unlocked |
| rdir    | 192.168.8.109:6010 | unlocked |
| rdir    | 192.168.8.110:6010 | unlocked |
| rdir    | 192.168.8.110:6302 | unlocked |
| rdir    | 192.168.8.139:6302 | unlocked |
| rdir    | 192.168.8.139:6010 | unlocked |
| account | 192.168.8.109:6009 | unlocked |
| account | 192.168.8.139:6009 | unlocked |
| account | 192.168.8.110:6009 | unlocked |
| rawx    | 192.168.8.109:6004 | unlocked |
| rawx    | 192.168.8.109:6202 | unlocked |
| rawx    | 192.168.8.110:6004 | unlocked |
| rawx    | 192.168.8.110:6202 | unlocked |
| rawx    | 192.168.8.139:6004 | unlocked |
| rawx    | 192.168.8.139:6202 | unlocked |
| meta2   | 192.168.8.109:6003 | unlocked |
| meta2   | 192.168.8.110:6003 | unlocked |
| meta2   | 192.168.8.139:6003 | unlocked |
| meta1   | 192.168.8.109:6002 | unlocked |
| meta1   | 192.168.8.110:6002 | unlocked |
| meta1   | 192.168.8.139:6002 | unlocked |
| meta0   | 192.168.8.109:6001 | unlocked |
| meta0   | 192.168.8.110:6001 | unlocked |
| meta0   | 192.168.8.139:6001 | unlocked |
+---------+--------------------+----------+

Check if new rawx is coming up (score should increase up to 100 by 5% increments)

[root@be-openio-01 ~]# openio --oio-ns OPENIO cluster list | grep rawx
| rawx    | 192.168.8.139:6203 | /data2/sds/OPENIO/rawx-2        | be-openio-01 | n/a   | True  |    99 |
| rawx    | 192.168.8.139:6202 | /data/sds/OPENIO/rawx-1         | be-openio-01 | n/a   | True  |    88 |
| rawx    | 192.168.8.139:6004 | /var/lib/oio/sds/OPENIO/rawx-0  | be-openio-01 | n/a   | True  |    86 |
| rawx    | 192.168.8.110:6004 | /var/lib/oio/sds/OPENIO/rawx-0  | be-openio-03 | n/a   | True  |    86 |
| rawx    | 192.168.8.110:6202 | /data/sds/OPENIO/rawx-1         | be-openio-03 | n/a   | True  |    89 |
| rawx    | 192.168.8.109:6202 | /data/sds/OPENIO/rawx-1         | be-openio-02 | n/a   | True  |    89 |
| rawx    | 192.168.8.109:6004 | /var/lib/oio/sds/OPENIO/rawx-0  | be-openio-02 | n/a   | True  |    82 |

Our new data repository is ready to be used !

Ideally, you need to redo the above procedure on each hosts in your OpenIO cluster to be sure data is balanced accordingly.


Comments

Post a Comment

Thank you for your message, it has been sent to the moderator for review...

What's hot ?

ShredOS : HDD degaussing with style

Image
Background Have you ever had the need to properly erase HDD or SSD before disposing them ? Or selling them ? You do not want to have someone else playing around with your data right ? What we usually do is using a bootable Linux Live CD and use dd style command to write random data on the disk right ? But what if you have 10 disks in your machine ? What if you need a report to proof the world that wiping is a success ? I came across this challenge recently.
Read more

Mac OS X : Display images in-line with terminal

Image
Backgound This time, this is a MacOS tip ! You have a list of files in the terminal window in front of you, some are pictures, some are texts. For text files, this is easy to quickly issue a "more" or "cat" so you immediately figure out what it is. For a picture, this is another story, you need an image viewer. Whatever the speed and simplicity of this editor, you need to load a 3rd party application and switch to it, this is not very handy. What if you could see this image directly from the terminal ? 
Read more