Integrating LVM with Hadoop and providing Elasticity to DataNode Storage.

ARTH TASK 7.1.A

TASK DESCRIPTION

🌀 7.1: Elasticity Task
🔅Integrating LVM with Hadoop and
providing Elasticity to DataNode Storage

Prerequisite for this Task-

>Knowledge of setting up hadoop cluster

>Knowledge of LVM

Lets Start — -

Here I have hadoop cluster already configured

First start the Name node using command-

hadoop-daemon.sh start namenode

And also to check whether it is started or not use command -

jps

To check the status of cluster use the command-

hadoop dfsadmin -report

Currently there is no data node connected.

Now attach a hardisk to the data node .

Here I am attaching a hardisk of 30 Gib

To attach disk —

1.Go to vm’s settings>storage

2. Click add hard disk icon

3. Click on create

4. Click next

5 .Click Next

6. Give the size and name and click on create.

7. Then select the hardisk you have created and click ok.

To check the hard disk use the command-

fdisk -l

As we can see a hardisk of 30Gib is attached with name /dev/sdc

Now,

Create a physical volume from disk using command-

pvcreate /dev/sdc

To see the physical volume created or not use the command-

pvdisplay /dev/sdc

As we can PV of size 30Gib is created.

Now create the Volume Group:

To create volume group use the command-

vgcreate <name of volume group> <name of disk>

vgcreate da1 /dev/sdc

To see the details of volume group use the command-

vgdisplay da1

Now to check that whether the physical volume is allocated or not use the command-

pvdisplay /dev/sdb

Here we can see that allocatabe is yes and vg name is da1.

Now Create a Logical Volume .

Here I am creating a logical Volume of 20Gib.

Use the command-

lvcreate — size 20Gib — name dl1 da1

To see the details use command-

lvdisplay da1/dl1

Here we can see that LV of size 20Gib is created.

Now format the Logical Volume

To format use command-

mkfs.ext4 /dev/da1/dl1

Now Mount the Logical Volume to directory of datanode

To mount use command-

mount /dev/da1/dl1 /dataNode1

To check whether it is mounted or not use the command -

df -hT

Now start the data node.

Use command-

hadoop-daemon.sh start datanode

Now lets check cluster report and see how much data is shared.

To check use command-

hadoopp dfsadmin -report

Here we can see that 20Gib space is shared.

Now lets increase the storage without stopping datanode.

For this we have extend the size of logical volume. Here I am extending it by 5Gib.

Use command-

lvextend — size +5g /dev/da1/dl1

Here We can see than size of LV has been Increased.

But if you check now the size of volume which is mounted with data node directory is still 20Gib because we have update the inode tabel .

Now we have to format the extended volume.

Use command-

resize2fs /dev/da1/dl1

Now lets check whether the size incresed or not using the command-

df -hT

Here we can see that size has been increased to 25Gib

Also lets check cluster report-

Here also 25Gib storage is shared.

That’s all.

Thanks for reading.

#arth #vimaldaga #righteducation #lvm

Keep learning and Sharing.