OpenSolaris ZFS Storage Migration

Everything in Its Place - Moving and Reorganizing ZFS Storage

Order is good. Organizing, cataloging and putting everything in its place should also look nice. Everything should be easy to find, and fall into place naturally. My plan of migrating my OpenSolaris installation from one set of disks to another by doing a fresh install was a bit more complicated than I first planned. I followed my previous plan fairly closely, but found some stumbling points along the way. In the end, everything is back together, I have more space, and know a lot more about ZFS.

My biggest challenge was namespace collisions and also beadm (Boot Environment administration utility) got a bit confused, probably due to some metadata written in the old ZFS rpool or some other location causing it to think the old imported rpool was still usable as a boot environment. This prevented me from being able to get recursive snapshots, but I managed to find a work-around. Additionally, ZFS handles mount points more-or-less in an automatic manner which only exaggerated the namespace issues.

I also, in my opinion discovered a minor inconvenience in ZFS when dealing with zvols for swap and dump areas. This is related to taking recursive snapshots on the zpool which results in taking a snapshot of swap and dump. It is questionable whether there is any value in having a snapshot of the dump area, but there is no value I can see in having a snapshot of swap - ever.

The Plan

This entry will cover the following:

‣Moving swap from partitions on disk to a ZFS zvol
‣Importing a foreign rpool into an existing rpool
‣Resolving namespace colisions
‣Growing storage on a mirror
‣ZFS send/receive to migrate off the foreign rpool onto a new pool
‣Finishing up the mirroring process

That’s a lot...

The Environment

In order to recreate this experience, I’ll go back to a simulation using a VM in VirtualBox. The configuration I’ll use 1/10th the size for all the disk drives, but it will work just the same. Additionally, I have some file systems on the actual hardware I will not duplicate here. The system layout was done like in my previous post regarding disk repartitioning and other than swap now situated in a ZFS zvol, it’s pretty much the same.

This is the initial configuration of the virtual machine I will use for this demonstration. The main difference is that the two 100GB disks are on SATA port 0 and 1 and the two 50GB disks are on SATA port 2 and 3. These have the original OpenSolaris 2008.05 installation on them. I went on to rearrange the disks to look much closer to the physical hardware configuration for the migration and the final system. The physical hardware I’m using has dual SATA controllers with two ports each. I put one mirror on the first controller and the other mirror on the second controller. The next snapshot reflects this configuration.

I’m using the first 50GB disk as a boot disk and the second as it’s mirror. They will be on ports 0 and 2, while the 100GB disks which contain the existing rpool mirror reside on ports 1 and 3. Again, these disks are sized proportionately to the actual hardware I’m using a pair of 500GB disks and a pair of 1TB disks.

Let’s Do It!

So after shuffling the hardware around, we have the following configuration.

The pool oldrpool has OpenSolaris 2008.05 installed and some additional directories we will need to move into the xpool storage hierarchy which I will do using zfs send/receive. However, before we can access the file systems on oldrpool, we will need to take care of some namespace collisions which occur due to mount points.

The first step will be importing the pool by using the zpool import and importing it as its new name oldrpool. Another thing we want to do at this point is also upgrade the ZFS version of oldrpool.

First the standard disclaimer as these procedure are done as root:

WARNING! - Important Safety Tips

‣All the following commands I will use are done using the root user.
‣If you are not comfortable using root directly, then DO NOT TRY THIS.
‣If you are not 100% comfortable with what you are doing, DO NOT TRY THIS!
‣Use common sense, HAVE A BACKUP/RECOVERY PLAN BEFORE ATTEMPTING THIS!
‣Unless you really know what you are doing and are comfortable with everything here, DO NOT ATTEMPT THIS ON PRODUCTION SYSTEMS.
‣LAST, I CLAIM NO RESPONSIBILITY FOR YOUR ACTIONS. IF YOU SCREW UP, YOU ARE ON YOUR OWN, SORRY.

First, lets move swap around. This is fairly simple to do.

root@panacea:~# swap -l

swapfile dev swaplo blocks free

/dev/dsk/c4t0d0s1 50,1 8 1044216 1044216

Swap only resides on /dev/dsk/c4t0d0s1 and we will use that partition when we build the xpool I described in my earlier post.

We need to create a new swap partition, add it and delete the old one. As a last step, we must add the swap device to /etc/vfstab.

root@panacea:~# zfs create -b 4096 -V 512m rpool/swap

root@panacea:~# swap -a /dev/zvol/dsk/rpool/swap

root@panacea:~# swap -l

swapfile dev swaplo blocks free

/dev/dsk/c4t0d0s1 50,1 8 1044216 1044216

/dev/zvol/dsk/rpool/swap 182,2 8 1048568 1048568

root@panacea:~# swap -d /dev/dsk/c4t0d0s1

root@panacea:~# swap -l

swapfile dev swaplo blocks free

/dev/zvol/dsk/rpool/swap 182,2 8 1048568 1048568

This is pretty straightforward system administration, so I really won’t go into details here. What is important is we created a 512MB zvol vdev /dev/zvol/dsk/rpool/swap for swap which will be mirrored using the zfs create command. After this is done, we want to make sure to replace the original swap device in /etc/vfstab with the new one.

Now, let’s take a look at how the configuration is actually reported.

root@panacea:~# zpool status

pool: rpool

state: ONLINE

scrub: none requested

config:

NAME STATE READ WRITE CKSUM

rpool ONLINE 0 0 0

mirror ONLINE 0 0 0

c4t2d0s0 ONLINE 0 0 0

c4t0d0s0 ONLINE 0 0 0

errors: No known data errors

pool: xpool

state: ONLINE

scrub: none requested

config:

NAME STATE READ WRITE CKSUM

xpool ONLINE 0 0 0

mirror ONLINE 0 0 0

c4t2d0s1 ONLINE 0 0 0

c4t0d0s1 ONLINE 0 0 0

root@panacea:~# zfs list

NAME USED AVAIL REFER MOUNTPOINT

rpool 3.14G 11.6G 72K /rpool

rpool/ROOT 2.39G 11.6G 18K legacy

rpool/ROOT/opensolaris 2.39G 11.6G 2.26G /

rpool/dump 256M 11.6G 256M -

rpool/export 613K 11.6G 19K /export

rpool/export/home 594K 11.6G 19K /export/home

rpool/export/home/mps 575K 11.6G 575K /export/home/mps

rpool/swap 512M 12.1G 37.2M -

xpool 69K 34.2G 18K /xpool

root@panacea:~# zpool import

pool: rpool

id: 2921547533073523785

state: ONLINE

status: The pool was last accessed by another system.

action: The pool can be imported using its name or numeric identifier and

the '-f' flag.

see: http://www.sun.com/msg/ZFS-8000-EY

config:

rpool ONLINE

mirror ONLINE

c4t3d0s0 ONLINE

c4t1d0s0 ONLINE

The command zfs import shows the old rpool and even gives us a warning about it. Notice in the zfs list, I have attached the new mirrored pool, xpool, and I have the OpenSolaris 2008.11 installed in the attached rpool. By simply importing the 2008.05 rpool, we will end up with namespace collisions. This is because some of the metadata stored in the zfs file systems have conflicting mount points, even though we will import the old pool as a different name, these mount point still exist from when it was rpool.

To import the zpool we simply provide another pool name for ZSF like this:

root@panacea:~# zpool import rpool oldrpool

cannot import 'rpool': pool may be in use from other system, it was last accessed by panacea (hostid: 0x235a1388) on Wed May 20 13:30:08 2009

uroot@panacea:~# zpool import -f rpool oldrpool

cannot mount '/export': directory is not empty

cannot mount '/export/home': directory is not empty

Note I had to use the “-f” flag when importing because ZSF detects that the rpool was in use by another system. It even includes the hostid of the machine so, if we have a cross reference of hostid’s we can locate the system the pool came out of. Additionally, if this were a single disk of the mirror I’m importing, ZFS would import the pool and let me know that it is degraded and missing a mirror when I do a zpool status. So here’s what the system looks like now:

root@panacea:~# zpool status

pool: oldrpool

state: ONLINE

status: The pool is formatted using an older on-disk format. The pool can

still be used, but some features are unavailable.

action: Upgrade the pool using 'zpool upgrade'. Once this is done, the

pool will no longer be accessible on older software versions.

scrub: none requested

config:

NAME STATE READ WRITE CKSUM

oldrpool ONLINE 0 0 0

mirror ONLINE 0 0 0

c4t3d0s0 ONLINE 0 0 0

c4t1d0s0 ONLINE 0 0 0

errors: No known data errors

pool: rpool

state: ONLINE

scrub: none requested

config:

NAME STATE READ WRITE CKSUM

rpool ONLINE 0 0 0

mirror ONLINE 0 0 0

c4t2d0s0 ONLINE 0 0 0

c4t0d0s0 ONLINE 0 0 0

errors: No known data errors

pool: xpool

state: ONLINE

scrub: none requested

config:

NAME STATE READ WRITE CKSUM

xpool ONLINE 0 0 0

mirror ONLINE 0 0 0

c4t2d0s1 ONLINE 0 0 0

c4t0d0s1 ONLINE 0 0 0

errors: No known data errors

root@panacea:~# zpool upgrade oldrpool

This system is currently running ZFS pool version 13.

Successfully upgraded 'oldrpool' from version 10 to version 13

Migration Time

The zpool command shows the pool is now sucessfully imported and it gives additional information that we need to upgrade the version of ZFS on oldrpool. This was painless, but now there is no turning back. These disks will likely no longer boot OpenSolaris 2008.05. At this point the system looks like the diagram here.

But, there are bigger problems, we have namespace collisions all over the ZFS devices. Before we can continue we must eliminate the conflicts. This will involve a number of ZFS properties to be adjusted. Here’s the output of zfs list with the newly imported oldrpool.

root@panacea:~# zfs list

NAME USED AVAIL REFER MOUNTPOINT

oldrpool 2.29G 93.7G 55K /oldrpool

oldrpool/ROOT 2.29G 93.7G 18K /oldrpool/ROOT

oldrpool/ROOT/opensolaris 2.29G 93.7G 2.23G legacy

oldrpool/ROOT/opensolaris/opt 3.61M 93.7G 3.61M /opt

oldrpool/export 484K 93.7G 21K /export

oldrpool/export/backups 18K 93.7G 18K /export/backups

oldrpool/export/home 428K 93.7G 410K /export/home

rpool 3.14G 11.6G 72K /rpool

rpool/ROOT 2.39G 11.6G 18K legacy

rpool/ROOT/opensolaris 2.39G 11.6G 2.26G /

rpool/dump 256M 11.6G 256M -

rpool/export 615K 11.6G 21K /export

rpool/export/home 594K 11.6G 19K /export/home

rpool/export/home/mps 575K 11.6G 575K /export/home/mps

rpool/swap 512M 12.1G 37.2M -

xpool 69K 34.2G 18K /xpool

Namespace Collisions and Resolution

Above when I imported the oldrpool (which was rpool in a previous life), ZFS informed us some of the file systems were already mounted. These are mounted automatically when we import because the properties are set to mount. In fact, if the system were to reboot at this point with the oldrpool imported, it would fail to start back up because the mount services will fail. The mount points for /ROOT/opensolaris, /export and /export/home are all in conflict between rpool and oldrpool. Due to these conflicts, the mount services fail and the machine will not boot cleanly. Manual intervention is needed. However, we can fix this now.

There are some items of interest between the 2008.05 and the 2008.11 installation. The file system for /opt is gone in 2008.11 and an additional file system for my user ID was added in 2008.11. I assume the file system for each user is so the user can setup and administer their own snapshots independently of the entire /export/home file system. This would also make sense given the new “Time-Slider” functionality in 2008.11.

The mount point listed as “legacy” is managed by means other than ZFS. This could be in /etc/vfstab or other location. In the case of /ROOT/opensolaris, it is the root file system mounted by the kernel.

We can find out which mount points are actually active using the zfs mount command.

root@panacea:~# zfs mount

rpool/ROOT/opensolaris /

rpool/export /export

rpool/export/home /export/home

rpool/export/home/mps /export/home/mps

rpool /rpool

xpool /xpool

oldrpool/export/backups /export/backups

oldrpool /oldrpool

oldrpool/ROOT /oldrpool/ROOT

By this I can see that the mount points which didn’t conflict with existing mount points are mounted.

In order to get rid of the conflicts and give the ZFS file systems on oldrpool a place to mount we must go change some of the properties. We can get a complete list of the properties set on an individual file system and it’s settings using the command zfs get all shown here.

root@panacea:~# zfs get all oldrpool

NAME PROPERTY VALUE SOURCE

oldrpool type filesystem -

oldrpool creation Wed May 20 10:32 2009 -

oldrpool used 2.29G -

oldrpool available 93.7G -

oldrpool referenced 57K -

oldrpool compressratio 1.00x -

oldrpool mounted yes -

oldrpool quota none default

oldrpool reservation none default

oldrpool recordsize 128K default

oldrpool mountpoint /oldrpool local

oldrpool sharenfs off default

oldrpool checksum on default

oldrpool compression off default

oldrpool atime on default

oldrpool devices on default

oldrpool exec on default

oldrpool setuid on default

oldrpool readonly off default

oldrpool zoned off default

oldrpool snapdir hidden default

oldrpool aclmode groupmask default

oldrpool aclinherit restricted default

oldrpool canmount on default

oldrpool shareiscsi off default

oldrpool xattr on default

oldrpool copies 1 default

oldrpool version 3 -

oldrpool utf8only off -

oldrpool normalization none -

oldrpool casesensitivity sensitive -

oldrpool vscan off default

oldrpool nbmand off default

oldrpool sharesmb off default

oldrpool refquota none default

oldrpool refreservation none default

oldrpool primarycache all default

oldrpool secondarycache all default

The two primary properties we are concerned about are “mountpoint” and “canmount.” On the oldrpool/ROOT/opensolaris the “canmount” property is set to “noauto” which will prevent them from being auto mounted when the pool is imported or we issue a zfs mount -a command. Since we are migrating these over to the new system, there may be files or settings in files we wish to have in the new Solaris install. Examples of this might be items in /etc and possibly crontabs, or items installed in /opt there may be other items depending on your particular system.

So here’s very quickly how to fix the namespace problems and get all the file systems mounted under /oldrpool.

root@panacea:~# zfs mount

rpool/ROOT/opensolaris /

rpool/export /export

rpool/export/home /export/home

rpool/export/home/mps /export/home/mps

rpool /rpool

xpool /xpool

oldrpool/export/backups /export/backups

oldrpool /oldrpool

oldrpool/ROOT /oldrpool/ROOT

root@panacea:~# for fs in `zfs mount | grep oldrpool | sort -r | awk '{print $1}'`

> do

> zfs umount ${fs}

> done

root@panacea:~# zfs mount

rpool/ROOT/opensolaris /

rpool/export /export

rpool/export/home /export/home

rpool/export/home/mps /export/home/mps

rpool /rpool

xpool /xpool

root@panacea:~# for fs in `zfs list | grep oldrpool | awk '{print $1}'`

> do

> zfs set mountpoint=/${fs} ${fs}

> done

root@panacea:~# zfs mount -a

root@panacea:~# zfs mount

rpool/ROOT/opensolaris /

rpool/export /export

rpool/export/home /export/home

rpool/export/home/mps /export/home/mps

rpool /rpool

xpool /xpool

oldrpool /oldrpool

oldrpool/ROOT /oldrpool/ROOT

oldrpool/export /oldrpool/export

oldrpool/export/backups /oldrpool/export/backups

oldrpool/export/home /oldrpool/export/home

root@panacea:~# zfs list

NAME USED AVAIL REFER MOUNTPOINT

oldrpool 2.29G 93.7G 55K /oldrpool

oldrpool/ROOT 2.29G 93.7G 18K /oldrpool/ROOT

oldrpool/ROOT/opensolaris 2.29G 93.7G 2.23G /oldrpool/ROOT/opensolaris

oldrpool/ROOT/opensolaris/opt 3.61M 93.7G 3.61M /oldrpool/ROOT/opensolaris/opt

oldrpool/export 484K 93.7G 21K /oldrpool/export

oldrpool/export/backups 18K 93.7G 18K /oldrpool/export/backups

oldrpool/export/home 428K 93.7G 410K /oldrpool/export/home

rpool 3.14G 11.6G 72K /rpool

rpool/ROOT 2.39G 11.6G 18K legacy

rpool/ROOT/opensolaris 2.39G 11.6G 2.26G /

rpool/dump 256M 11.6G 256M -

rpool/export 619K 11.6G 21K /export

rpool/export/home 598K 11.6G 19K /export/home

rpool/export/home/mps 579K 11.6G 579K /export/home/mps

rpool/swap 512M 12.1G 37.2M -

xpool 69K 34.2G 18K /xpool

root@panacea:~# zfs set canmount=on oldrpool/ROOT/opensolaris

root@panacea:~# zfs set canmount=on oldrpool/ROOT/opensolaris/opt

root@panacea:~# zfs mount -a

root@panacea:~# zfs mount

rpool/ROOT/opensolaris /

rpool/export /export

rpool/export/home /export/home

rpool/export/home/mps /export/home/mps

rpool /rpool

xpool /xpool

oldrpool /oldrpool

oldrpool/ROOT /oldrpool/ROOT

oldrpool/export /oldrpool/export

oldrpool/export/backups /oldrpool/export/backups

oldrpool/export/home /oldrpool/export/home

oldrpool/ROOT/opensolaris /oldrpool/ROOT/opensolaris

oldrpool/ROOT/opensolaris/opt /oldrpool/ROOT/opensolaris/opt

There a lot going on here. Let’s review. First, we unmount all the file systems which come from oldrpool. This will avoid mount points being used or other conflicts when we change the mount points and remount them. This is done as a simple for loop. Next I verify that there are no more oldrpool file systems mounted. Then we change all the mount points to be mounted from /oldrpool by the name of the zfs file system. Also accomplished by a simple for loop on the command line.

The zfs set commands reset the “canmount” property from “noauto” to “on.” Previously, these mounts were done as legacy mounts and not handled by ZFS. Once these are set to “on” we can issue another mount -a and see everything is mounted correctly.

Take a Snapshot, Split and Grow...

So, now that everything is in a separate namespace, we need to split the old mirror and add one of its disks to the new pool - xpool.

There are several things which must be done at this point. We need to make a snapshot before using send and receive for transferring the ZFS filesystem. The snapshot will also be useful in case we hose up any of the original data in the imported filesystem as we re-organize it later.

Splitting the mirror which makes up oldrpool will leave us a bit exposed as there will only be one copy of the data. If some of the data or file systems can be destroyed or deleted, including old snapshots to reduce the required migration to less than the available 400GB, then this is ideal. In my actual physical migration, this was fortunately the case.

Also it is at this point where I discovered an issue with be’s (boot environments). For some reason, the new Solaris installation sees the boot environments on the imported oldrpool. This caused me problems in creating snapshots for some reason. I haven’t been able to find any documentation on this issue, but have been able to replicate it easily in a virtual machine. This prevents me from issuing a recursive snapshot on the ROOT on the oldrpool, but I was able to perform the snapshots manually.

The issue with boot environments is illustrated here.

root@panacea:~# beadm list -a

BE/Dataset/Snapshot Active Mountpoint Space Policy Created

------------------- ------ ---------- ----- ------ -------

opensolaris

rpool/ROOT/opensolaris NR / 2.39G static 2009-05-20 14:26

rpool/ROOT/opensolaris@install - - 133.17M static 2009-05-20 15:09

root@panacea:~# zpool import oldrpool

root@panacea:~# beadm list -a

BE/Dataset/Snapshot Active Mountpoint Space Policy Created

------------------- ------ ---------- ----- ------ -------

opensolaris

oldrpool/ROOT/opensolaris N /oldrpool/ROOT/opensolaris 2.29G static 2009-05-20 10:32

oldrpool/ROOT/opensolaris/opt - /oldrpool/ROOT/opensolaris/opt 3.61M static 2009-05-20 10:32

oldrpool/ROOT/opensolaris/opt@install - - 0 static 2009-05-20 11:16

oldrpool/ROOT/opensolaris@install - - 56.88M static 2009-05-20 11:16

rpool/ROOT/opensolaris NR / 2.39G static 2009-05-20 14:26

rpool/ROOT/opensolaris@install - - 133.17M static 2009-05-20 15:09

Since they are both contained under the OpenSolaris be, I’m a bit at a loss as to how to delete it. However, when I did the send/receive, it was gone. This may be some of my own confusion with how ZFS and be’s interact.

So we’ll do snapshots of both pools using zfs snapshot.

root@panacea:~# zfs snapshot -r rpool@migration

root@panacea:~# zfs snapshot -r oldrpool@migration

root@panacea:~# zfs list -t all

NAME USED AVAIL REFER MOUNTPOINT

oldrpool 2.23G 93.7G 57K /oldrpool

oldrpool@migration 0 - 57K -

oldrpool/ROOT 2.23G 93.7G 19K /oldrpool/ROOT

oldrpool/ROOT@migration 0 - 19K -

oldrpool/ROOT/opensolaris 2.23G 93.7G 2.23G /oldrpool/ROOT/opensolaris

oldrpool/ROOT/opensolaris@migration 0 - 2.23G -

oldrpool/ROOT/opensolaris/opt 3.61M 93.7G 3.61M /oldrpool/ROOT/opensolaris/opt

oldrpool/ROOT/opensolaris/opt@migration 0 - 3.61M -

oldrpool/export 450K 93.7G 21K /oldrpool/export

oldrpool/export@migration 0 - 21K -

oldrpool/export/backups 18K 93.7G 18K /oldrpool/export/backups

oldrpool/export/backups@migration 0 - 18K -

oldrpool/export/home 410K 93.7G 410K /oldrpool/export/home

oldrpool/export/home@migration 0 - 410K -

rpool 3.18G 11.6G 72K /rpool

rpool@migration 0 - 72K -

rpool/ROOT 2.39G 11.6G 18K legacy

rpool/ROOT@migration 0 - 18K -

rpool/ROOT/opensolaris 2.39G 11.6G 2.26G /

rpool/ROOT/opensolaris@install 133M - 2.21G -

rpool/ROOT/opensolaris@migration 450K - 2.26G -

rpool/dump 256M 11.6G 256M -

rpool/dump@migration 0 - 256M -

rpool/export 619K 11.6G 21K /export

rpool/export@migration 0 - 21K -

rpool/export/home 598K 11.6G 19K /export/home

rpool/export/home@migration 0 - 19K -

rpool/export/home/mps 579K 11.6G 579K /export/home/mps

rpool/export/home/mps@migration 0 - 579K -

rpool/swap 549M 12.1G 37.2M -

rpool/swap@migration 0 - 37.2M -

xpool 69K 34.2G 18K /xpool

Now to split one of the mirrors off and add it to the new zpool “xpool”. Before I do this, I will scrub the zpool just to make sure there are no problems with either mirror.

root@panacea:~# zpool scrub oldrpool

root@panacea:~# zpool status oldrpool

pool: oldrpool

state: ONLINE

scrub: scrub completed after 0h6m with 0 errors on Thu May 28 22:59:10 2009

config:

NAME STATE READ WRITE CKSUM

oldrpool ONLINE 0 0 0

mirror ONLINE 0 0 0

c4t3d0s0 ONLINE 0 0 0

c4t1d0s0 ONLINE 0 0 0

errors: No known data errors

Detach c4t3d0s0 from oldrpool and repartition it since it once had a swap partition from the original OpenSolaris 2008.05 installation.

From this point on, we are not protected in the event of disk failure, or any other potential data loss because we only have a single disk in the old pool. It would be really nice to add wither a disk to the mirror, or a temporary disk to the new pool for the storage migration. In that case, the new storage pool would have some minimal exposure but our original data would still reside in the original pool and be mirrored. The new pool could easily be reconstructed in the event of any problem.

root@panacea:~# zpool detach oldrpool c4t3d0s0

root@panacea:~# format

Searching for disks...done

AVAILABLE DISK SELECTIONS:

0. c4t0d0 <DEFAULT cyl 6524 alt 2 hd 255 sec 63>

/pci@0,0/pci8086,2829@d/disk@0,0

1. c4t1d0 <DEFAULT cyl 13051 alt 2 hd 255 sec 63>

/pci@0,0/pci8086,2829@d/disk@1,0

2. c4t2d0 <ATA-VBOXHARDDISK-1.0 cyl 6524 alt 2 hd 255 sec 63>

/pci@0,0/pci8086,2829@d/disk@2,0

3. c4t3d0 <DEFAULT cyl 13051 alt 2 hd 255 sec 63>

/pci@0,0/pci8086,2829@d/disk@3,0

Specify disk (enter its number): 3

selecting c4t3d0

[disk formatted]

FORMAT MENU:

disk - select a disk

type - select (define) a disk type

partition - select (define) a partition table

current - describe the current disk

format - format and analyze the disk

fdisk - run the fdisk program

repair - repair a defective sector

label - write label to the disk

analyze - surface analysis

defect - defect list management

backup - search for backup labels

verify - read and display labels

save - save new disk/partition definitions

inquiry - show vendor, product and revision

volname - set 8-character volume name

!<cmd> - execute <cmd>, then return

quit

format> pa

PARTITION MENU:

0 - change `0' partition

1 - change `1' partition

2 - change `2' partition

3 - change `3' partition

4 - change `4' partition

5 - change `5' partition

6 - change `6' partition

7 - change `7' partition

select - select a predefined table

modify - modify a predefined partition table

name - name the current table

print - display the current table

label - write partition map and label to the disk

!<cmd> - execute <cmd>, then return

quit

partition> pr

Current partition table (original):

Total disk cylinders available: 13051 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks

0 root wm 262 - 13050 97.97GB (12789/0/0) 205455285

1 swap wu 1 - 261 2.00GB (261/0/0) 4192965

2 backup wu 0 - 13050 99.98GB (13051/0/0) 209664315

3 unassigned wm 0 0 (0/0/0) 0

4 unassigned wm 0 0 (0/0/0) 0

5 unassigned wm 0 0 (0/0/0) 0

6 unassigned wm 0 0 (0/0/0) 0

7 unassigned wm 0 0 (0/0/0) 0

8 boot wu 0 - 0 7.84MB (1/0/0) 16065

9 unassigned wm 0 0 (0/0/0) 0

partition> 0

Part Tag Flag Cylinders Size Blocks

0 root wm 262 - 13050 97.97GB (12789/0/0) 205455285

Enter partition id tag[root]: unassigned

Enter partition permission flags[wm]:

Enter new starting cyl[262]: 0

Enter partition size[205455285b, 12789c, 12788e, 100319.96mb, 97.97gb]: 0

partition> 1

Part Tag Flag Cylinders Size Blocks

1 swap wu 1 - 261 2.00GB (261/0/0) 4192965

Enter partition id tag[swap]: unassigned

Enter partition permission flags[wu]: wm

Enter new starting cyl[1]: 0

Enter partition size[4192965b, 261c, 260e, 2047.35mb, 2.00gb]: 0

partition> label

Ready to label disk, continue? yes

partition> quit

FORMAT MENU:

disk - select a disk

type - select (define) a disk type

partition - select (define) a partition table

current - describe the current disk

format - format and analyze the disk

fdisk - run the fdisk program

repair - repair a defective sector

label - write label to the disk

analyze - surface analysis

defect - defect list management

backup - search for backup labels

verify - read and display labels

save - save new disk/partition definitions

inquiry - show vendor, product and revision

volname - set 8-character volume name

!<cmd> - execute <cmd>, then return

quit

format> quit

Next we will add the entire disk to “xpool”. This will append the disk to the pool extending it in size. This must be forced since xpool is mirrored and we are just appending a single disk to the pool. Remember, this is done using the zfs add command which is different from zfs attach.

root@panacea:~# zpool add -f xpool c4t3d0

root@panacea:~# zpool list

NAME SIZE USED AVAIL CAP HEALTH ALTROOT

oldrpool 97.5G 2.23G 95.3G 2% ONLINE -

rpool 15G 2.69G 12.3G 17% ONLINE -

xpool 134G 76.5K 134G 0% ONLINE -

With the disk added, we can now see it has grown by 100G, the size of oldrpool and the bit of swap space that was on the original disk.

Moving The Data With send/receive

Now that we have the new enlarged zpool, we can move the ZFS file systems from the oldrpool to the new. This is easily done using zfs send and receive. It works on the snapshot level, and is good for moving data not just across pools, but can move data across the network through a pipe and ssh, and even perform differential updates between any two snapshots as an incremental. This ability to do differential/incremental snapshot replication makes it an ideal tool for backups or replicating a machine for some basic level of disaster recovery.

Since we are moving an already existing pool and it’s file systems, that means the metadata will also be moved as well. This means the properties which control mounting the original pool will be transferred to the new file systems on the receiving end. Since they are mounted, receive will try to mount them here and fail. So, first all of oldrpool’s file systems must be unmounted.

root@panacea:~# for fs in `zfs list | grep oldrpool | awk '{print $1}' | sort -r`

> do

> zfs umount ${fs}

> done

root@panacea:~# zfs create xpool/oldpanacea

root@panacea:~# zfs send -R oldrpool@migration | zfs receive -F -d xpool/oldpanacea

After unmounting, I create a new file system to receive the base pool.

The send and receive is simple, the only thing here is that when dealing with an entire pool it must be forced into a new location.

Final Steps, Fixing The Mirror And Cleaning Up Mount Points

In the final step we attach the remaining oldrpool disk we just migrated data from using send and receive. The first thing to do is destroy the oldrpool and repartition it like the disk we just added. Instead of using format, we can use the shortcut with fmthard. After that, nothing more than attaching. ZFS will handle the rest and resilver the mirror.

root@panacea:~# zpool destroy oldrpool

root@panacea:~# prtvtoc /dev/dsk/c4t3d0s2 | fmthard -s - /dev/rdsk/c4t1d0s2

root@panacea:~# zpool attach xpool c4t3d0 c4t1d0

root@panacea:~# zpool status

pool: rpool

state: ONLINE

scrub: none requested

config:

NAME STATE READ WRITE CKSUM

rpool ONLINE 0 0 0

mirror ONLINE 0 0 0

c4t2d0s0 ONLINE 0 0 0

c4t0d0s0 ONLINE 0 0 0

errors: No known data errors

pool: xpool

state: ONLINE

scrub: resilver completed after 0h1m with 0 errors on Fri May 29 02:29:03 2009

config:

NAME STATE READ WRITE CKSUM

xpool ONLINE 0 0 0

mirror ONLINE 0 0 0

c4t2d0s1 ONLINE 0 0 0

c4t0d0s1 ONLINE 0 0 0

mirror ONLINE 0 0 0

c4t3d0 ONLINE 0 0 0 18.5M resilvered

c4t1d0 ONLINE 0 0 0 553M resilvered

errors: No known data errors

The last thing to do will be fix the mounts because everything is still mounted from the /oldrpool tree.

root@panacea:~# for fs in `zfs list | grep oldpanacea | awk '{print $1}' | sort -r `

> do

> zfs umount ${fs}

> done

root@panacea:~# zfs mount

rpool/ROOT/opensolaris /

rpool/export /export

rpool/export/home /export/home

rpool/export/home/mps /export/home/mps

rpool /rpool

xpool /xpool

root@panacea:~# for fs in `zfs list | grep oldpanacea | awk '{print $1}'`

> do

> zfs set mountpoint=/${fs} ${fs}

> done

root@panacea:~# zfs mount -a

root@panacea:~# zfs mount

rpool/ROOT/opensolaris /

rpool/export /export

rpool/export/home /export/home

rpool/export/home/mps /export/home/mps

rpool /rpool

xpool /xpool

xpool/oldpanacea /xpool/oldpanacea

xpool/oldpanacea/ROOT /xpool/oldpanacea/ROOT

xpool/oldpanacea/ROOT/opensolaris /xpool/oldpanacea/ROOT/opensolaris

xpool/oldpanacea/ROOT/opensolaris/opt /xpool/oldpanacea/ROOT/opensolaris/opt

xpool/oldpanacea/export /xpool/oldpanacea/export

xpool/oldpanacea/export/backups /xpool/oldpanacea/export/backups

xpool/oldpanacea/export/home /xpool/oldpanacea/export/home

We’re now done migrating the data from an existing rpool to a new pool called xpool. The disks now look like this:

Additional storage can now be added to the pool xpool as we fill up the pool.

If I were in a datacenter with plenty of disks and SAN to burn, I would have done this an entirely different way. I would not have split up the new OS disks into a smaller rpool, but I’m working on this as a home server for media and backups. The migration from an existing rpool into a system which already has an rpool is extremely relevant to any type of situation where perhaps there is data on the rpool of a system being upgraded and it needs to be moved into a new system.

There are better ways of organizing file systems where rpool is pretty much dedicated to all things with the base OS installation. However, this is the subject of another entry coming soon.

Summary

TBD.

Tuesday, 19 May 2009

next >

< previous