Rejigging my available storage

borgy95 · 05-12-2015, 05:27 AM

Hi, I've inherited a system from a previous admin, and as time has gone on the demands have outgrown the setup for storage. I need to take some action and adjust the space... I was hoping a wise admin here might sensecheck my thinking. Below is an overview of the situation:

Code:

[root@ati02 var]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   50G   37G   14G  73% /
devtmpfs                  40G     0   40G   0% /dev
tmpfs                     40G     0   40G   0% /dev/shm
tmpfs                     40G  281M   39G   1% /run
tmpfs                     40G     0   40G   0% /sys/fs/cgroup
/dev/sdb1                1.1T   74G  971G   7% /opt
/dev/sda1                492M  141M  352M  29% /boot
/dev/mapper/centos-home  225G   50G  175G  23% /home
[root@ati02 var]# cat /etc/fstab
# /etc/fstab
# Created by anaconda on Tue Mar  3 16:58:32 2015
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root /                       xfs     defaults        1 1
UUID=23b82019-8962-4b43-b5f3-257c6a54427c /boot                   xfs     defaults        1 2
/dev/mapper/centos-home /home                   xfs     defaults        1 2
/dev/mapper/centos-swap swap                    swap    defaults        0 0
/dev/sdb1       /opt    ext4    defaults        1 2
//uk-fs009/collector$   /mnt/logon      cifs    credentials=/home/.credentials  0 0

So my biggest concern is the / parition.

Code:

 Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   50G   37G   14G  73% /

This holds our splunk dbs and temporary log storage it was made way too small and has been causing problems. I've arleady limit db size and told splunk to move old data to /opt/splunk/..... which has 1.1tb of space total.
However we are about to get alot more data input in the coming days. I need to have my logs being written to /dev/sdb1. So i ran a

Code:

mount /dev/sdb1 /var/data/sfa

the hope being that the new logs being written to /var/data/sfa will write to the physical disk /dev/sdb1, is this a correct understanding? My assumption is a disk can be mounted to more than 1 directory and a subdirectory can write to a different disk that is parent directory is writing to?

Is it possible there will be performance changes? my inclination is to think there will be an imporvement since i'm spreading the load of more than one needle?

J

MensaWater · 05-12-2015, 07:36 AM

What is "/dev/mapper/centos-root"? Is it a logical volume? A meta-disk (software raid)? or something else?

Is "/dev/mapper/centos-home" the same type of device? (i.e. comprised of the same disks if meta-disk or using the same volume group if a VG?)

I'd probably focus on resizing centos-home down and increasing centos-root. Also if I were going through this exercise I'd probably try to move the non-OS components out of "/" into their own filesystem. Filling up / or /var can have really bad consequences.

Note that /dev/sdb and /dev/sda are individual disks (from the OS standpoint) so if they're not presented RAID LUNs from some other device (e.g. Dell PERC) or array (e.g. EMC or Hitachi) anything you put on them are single points of failure.

Also have you run fdisk -l (or parted -l) to see if the entire partitions of /dev/sda and /dev/sdb are in use by the two partitions /dev/sda1 and /dev/sdb1? It's possible you have unused space on the two devices.

You also don't say which version of CentOS you're using.

zhjim · 05-12-2015, 07:57 AM

Assuming you are using lvm go with shrinking home and enlarge root like MensaWater said.

Nother thing would be to shrink /dev/sdb1 to a sane lvl. Create a new partition and copy your /var/log directory over there and have it mounted through fstab. Maybe the whole /var directory could be moved. This would take splunk with it I assume.

Your assumptions about the rebind of /dev/sdb1 are right. This way you have /opt as well as the other directory write to this hdd.

borgy95 · 05-12-2015, 08:12 AM

Thanks for the information mensawater, i'm pretty amateur at handling storage, but I've got no one to pawn it off on anymore... I'm using centos7 minimal. The machine is using a hardware raid. To answer your questions see the following output below.

Code:

[root@ati02 cgi]# parted -l
Model: HP LOGICAL VOLUME (scsi)
Disk /dev/sda: 300GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End    Size   Type     File system  Flags
 1      1049kB  525MB  524MB  primary  xfs          boot
 2      525MB   300GB  299GB  primary               lvm


Model: HP LOGICAL VOLUME (scsi)
Disk /dev/sdb: 1200GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  1200GB  1200GB  primary  ext4


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/centos-home: 241GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End    Size   File system  Flags
 1      0.00B  241GB  241GB  xfs


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/centos-root: 53.7GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system  Flags
 1      0.00B  53.7GB  53.7GB  xfs


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/centos-swap: 4295MB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system     Flags
 1      0.00B  4295MB  4295MB  linux-swap(v1)


[root@ati02 cgi]# lvdisplay
  --- Logical volume ---
  LV Path                /dev/centos/swap
  LV Name                swap
  VG Name                centos
  LV UUID                Z2XywM-YAaL-GJy2-N8Rk-U70P-MfaF-9Ylj9t
  LV Write Access        read/write
  LV Creation host, time lagan.logica.co.uk, 2015-03-03 16:58:30 +0000
  LV Status              available
  # open                 2
  LV Size                4.00 GiB
  Current LE             1024
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

  --- Logical volume ---
  LV Path                /dev/centos/home
  LV Name                home
  VG Name                centos
  LV UUID                hzYqv2-fCXr-2MLD-02U9-UfZf-I5kf-sIOueo
  LV Write Access        read/write
  LV Creation host, time lagan.logica.co.uk, 2015-03-03 16:58:30 +0000
  LV Status              available
  # open                 1
  LV Size                224.88 GiB
  Current LE             57568
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

  --- Logical volume ---
  LV Path                /dev/centos/root
  LV Name                root
  VG Name                centos
  LV UUID                ZR4DPZ-CtWz-mX1T-1Asd-Rxj1-kdiA-sJX3lY
  LV Write Access        read/write
  LV Creation host, time lagan.logica.co.uk, 2015-03-03 16:58:31 +0000
  LV Status              available
  # open                 1
  LV Size                50.00 GiB
  Current LE             12800
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1

Based on what you said i am thinking to format dev/sda and dev/sdb - add them to the hardware raid, and from that increase /dev/centos/root volume so / and /var have a lot more space make the partition something like 300gb! /opt/splunk holds lots of dbs so this will also be a large parition ~300gb also. /home could be canabalised a a bit in the short term.

I believe #lvmanage is the command used for resizing? or can it all be done through #parted or #fdisk? any pointers or google references? I've not done this in a long time!

MensaWater · 05-12-2015, 09:16 AM

Actually it appears to me you have free space in the LVM (Logical Volume Manager) Volume Group (VG) named "centos".

For sda you have a second partition (/dev/sda2) that is labeled as being used for lvm:
2 525MB 300GB 299GB primary lvm

The total of the 3 LVs (Logical Volumes) in VG centos is ~278 GB whereas the size of the VG is 299 GB so you have about 20 you could add.

To verify that run "vgdisplay -v centos". That will show you all the details of the centos VG which should include the space allocated to it and how much is free, the 3 LVs and the PVs (physical volumes) which should just be /dev/sda2.

You can NOT blow away sda as your /boot partition is on /dev/sda1. (/boot is usually on a partition by itself).

There are commands for adding space to an LV (lvextend) and also usually ways to extend a filesystem but I don't know the tool for xfs.

zhjim · 05-12-2015, 09:22 AM

Carefull with the format! Also you have a raid controller installed it does not mean you actually have any assembled RAID disks. To me it seems like sda and sdb are ordinary hdds which are not in any raid mode. But actually the hardware does not matter really due to linux seeing two disk sda and sdb. This should be the starting point of the voyage.

Thus sda has two partitions. One is the /boot partition and one holds your lvm container. sdb is just one big chunck. If you can take the server offline you could copy the content of /opt (sdb1) to the / (root) partition on /dev/mapper/centos-root. Afterwards add sdb1 to the lvm. Then shrink and enlarge partitions to your liking.
You could also use sdb as your new primary disk but this could lead to some hassle with various parts of the system. fstab, grub or lilo.

For intermediate rescue just go with the resizing of your lvm partitions. As usual tldp.org has some on lvm http://tldp.org/HOWTO/LVM-HOWTO/. But there are more up to date ones on the net.

borgy95 · 05-12-2015, 12:24 PM

Thanks for all the help, guys, i know hwere i'm going now.. Unfortunately i cant extend the volume, i reckon the remain space is lost to formatting... I reckon this to be so because:

Code:

[root@ati02 ~]# vgextend centos /dev/sda2
  Physical volume '/dev/sda2' is already in volume group 'centos'
  Unable to add physical volume '/dev/sda2' to volume group 'centos'.

and

Code:

 lvextend -L+1G /dev/centos/root
  Extending logical volume root to 51.00 GiB
  Insufficient free space: 256 extents needed, but only 0 available
[root@ati02 ~]# lvextend -l +100%FREE /dev/centos/root
  New size (12800 extents) matches existing size (12800 extents)
  Run `lvextend --help' for more information.

Happily I just got news budget for a SAN is inbound... I'm sure to be back to ask what the hell I do with the big expensive tin.

@Zhijm thanks for the link,
@Mensawater thanks for talking me through a sensible approach so clearly.

MensaWater · 05-13-2015, 10:56 AM

You tried to vgextend which grows the VG itself. Since sda2 was already in the VG that failed.

You tried the lvextend which grows the LV and that failed saying you had 0 extents. That suggests something else is in the VG that we aren't seeing.

Did you run the vgdisplay -v I suggested?