Background...

Let’s face it pigs want space and pigs are fairly hard to move around when they don’t want to.  So as a result they tend to just sit there until something motivates them to move.  OpenSolaris on Intel changed the “rules” we all knew and loved about partitioning hard disks on Solaris.  ZFS had an even greater impact, so much so, that normal partitions are of little consequence anymore.  However, if one wants to optimize or tweak the partition tables in a stock OpenSolaris installation, this will tell you how it’s done.  The impatient can go to the summary at the end of this page.

This technique can also be employed if upgrading to larger hard disks on your machine.  For instance, you started with 20GB disks and decided that 500GB disks were more your liking.

If you have an enterprise class server and a SAN, then these techniques probably aren’t what you are looking for, however, if you are looking to maximize your storage bang-for-the-buck and increase its flexibility either at home or a small to medium sized business, then this is for you.

This will assume two things:

1. ‣You have more than one disk on your system and intend to mirror your boot drive.
   

2. ‣ OpenSolaris 2008.11, though this may work on previous versions.  If you don’t have it, you can get it here.
   

One key item of note here is that I’m using a bridged network interface in the virtual host, along with DNS and DHCP, I can ensure that NWAM will get the network configuration.

A “Free Hog” partition used to be the partition on the disk which contained all the remaining unallocated disk space when using the Solaris format utility.  As partitions were created or resized, it would take up all remaining disk space in this “free hog” partition.  More background may be found on Sun’s web site in the “Basic System Administration Guide: Dividing a Disk Into Slices.”

First a bit of information about partitioning...

On Intel machines, there are two layers of “partitioning” done for the OpenSolaris installation.  One happens at the BIOS level on the disk using a utility such as “fdisk”, and the other is within one of these four BIOS partitions.   This creates the traditional VTOC partitioning scheme of slices people familiar with previous versions of Solaris know.  I won’t go into too much detail here on this because there is a great blog entry on how Solaris disk device names work.  OpenSolaris can only use one of these fdisk partitions to store it’s partitioning table or VTOC.

Why does all this matter?  When OpenSolaris installs, it gives you a choice to use all of the disk or to create fdisk partitions on the disk so you could use the disk for a dual-boot configuration.  Or, if you have a disk already partitioned with fdisk, it will allow you to select which partition you wish to use for the Solaris installation.  OpenSolaris the places a default VTOC on that fdisk partition which is somewhat less than optimal.

The entire fdisk partition selected is used, but the OpenSolaris partitions are set up with two partitions.  One for the ZFS rpool, and a second partition for swap.  The default on these is to use a swap size equal to your physical memory and a slice for the ZFS partition made up of everything else.  NOTE - 13 May 2009: I did an install on my PHYSICAL server today and discovered that the OpenSolaris 2008.11 instal does not create a swap partition, but creates a ZFS zvol.  This is good news since previously I was going to use SVM to mirror the swap partition.  Previous versions of the ZFS best practices warned against using a ZFS Volume for a swap device, but I guess it’s better now though I still cannot see to find too much information on the subject.  I did go back and check the VirtualBox installation again and it does create a separate swap partition on the vdisks, not sure why there is an inconsistency here.

This is alright, except I might not want all of a 500GB disk used for rpool, especially because later, the rpool virtual device (vdev) cannot be extended.  The base operating system really only needs about 8GB for an install.  I can create a smaller base footprint on the disk for the rpool and then create a second zpool on the disk for other things like zones, virtual hosts, database storage, or really anything else I’d like to be maintained separately from the operating system.

Additionally, sizing swap equal to physical memory by default just doesn’t cut it sometimes.  Some software vendors also, not that it’s proper, check the swap size with their installers to make sure you have two times memory for swap.  These products will sometimes fail to install is they fail the “swap check.”  (Oracle comes to mind, but isn’t supported on OpenSolaris 2008.11 - yet...)

So, on to the Install...

In the configuration panel, I will select to partition the disk.  Then only create a single partition of 8GB for the OpenSolaris base install.  The remaining partitions will be unused.  We’ll re-partition this disk later on in the process using the format and fdisk command line utilities in OpenSolaris.

The installation will takes about an hour and a half in the VirtualBox for me.  This time includes installing the guest additions and rebooting.  Installing the guest additions are probably unnecessary for this purpose, but I do so only for completeness.

One note about my interaction with the machine.  In the following text, my commands will be in blue and the computer responses will be in green.  For example, I will be doing all this as root so in OpenSolaris, I will shell into the machine and use pfexec su - to get root privileges.  Here is what that would look like:

WARNING! - Important Safety Tips

1. ‣All the following commands I will use are done using the root user.
   

2. ‣If you are not comfortable using root directly, then DO NOT TRY THIS.
   

3. ‣If you are not 100% comfortable with what you are doing, DO NOT TRY THIS!
   

4. ‣Use common sense, HAVE A BACKUP/RECOVERY PLAN BEFORE ATTEMPTING THIS!
   

5. ‣Unless you really know what you are doing and are comfortable with everything here, DO NOT ATTEMPT THIS ON PRODUCTION SYSTEMS.
   

6. ‣LAST, I CLAIM NO RESPONSIBILITY FOR YOUR ACTIONS.  IF YOU SCREW UP, YOU ARE ON YOUR OWN, SORRY.
   

Initial System Configuration

After setting up the system, the partitions and zpool look like this:

Since I used the SATA controller and SATA virtual disks in the VirtualBox, we are using controller number 4.  The boot disk is now /dev/rdsk/c0t0d0p0 for the entire physical disk.  Partition number 1 is used by Open Solaris to create the disk slices.

Using fdisk, we can also see in the listing from fdisk that we are only using a small portion of the physical disk:

So now we can see that the actual partition is 1023 cylinders when there are really 13054 cylinders available on the disk.

OpenSolaris has “sliced” this disk into several partitions:

Of particular note here is slice(partition) 8 which is the MBR for Solaris.  Partitions with number greater than 7 are not usable by the OS.  Notice here, also, that our swap partition, slice 1 is 512MB (1044225 * 512 = 534643200).  These settings are all the defaults set up by the OpenSolaris install.

Typically, it would be at this point in your installation you would want to attach your second disk as a mirror, but since we have done a minimal install using only a small portion of the disk, we want to do a few extra steps.

Resizing/Creating the Partitions and Moving Solaris.

So, now we turn our attention to the second disk.  We’re going to partition this disk to the sizes we would like.  In this case I would like to have 2GB swap, a Solaris OS partition for the rpool of 20GB and the remaining disk (about 78GB) to be available for another zpool (there it is the “free hog”).  So, the layout I’d like to have will be:

1. ‣s0 20GB rpool/OS Slice
   

2. ‣s1  2GB swap
   

3. ‣s2  Entire disk
   

4. ‣s3 78GB naspool/”Free hog”
   

Ok, so now, how do we do it?  Back to the command line.  We’ll use the format utility, which also gives us access to fdisk as well.  The second disk has no partition table on it yet because it’s never been used or accessed.  If you purchase a disk at a retail outlet, you may have a factory installed partition table on it.  This can be easily cleared out and a new one put in place.  We’ll do that in a bit once we finish up with the second disk, or /dev/dsk/c4t1d0p0 .

So that’s all there is to it if we wanted to use this disk for a plain ZFS pool.  We’d simply add the whole disk c4t1d0 to a new zpool.  However, I need to make partitions so we’ll stay in the format utility for now.  Also, in format, you can abbreviate commands using the first few letters which make that command unique, like fd for fdisk, pa for partition and pr for print.

Here we will use the modify command to modify a predefined partition table.  This will give us the option to select  “All Free Hog.”  Pretty much go with the defaults for all the questions except for the size of partition s0 and s1.  There’s no need to calculate starting and number of cylinders because format takes care of this for us.  Pretty simple?

The prtvtoc command just verified everything was in place correctly.

Now we’re ready to move the OS partition over to c4t1d0s0.  To do this, we will use zpool replace to replace the smaller 6GB partition with the new 20GB partition

ZFS accepts this disk as a replacement and begin moving the data over.  It also reminds us to make this disk bootable, which we would have to do anyway to use it as an effective bootable mirror.  We can also now check the status and we see the 20GB disk was added as a replacement and is being re-silvered.

The re-silver to the new device only took 4 minutes, and very impressive.  The zpool command now reports the new zpool is 19.9GB in size.  Here it is in the completed state:

Perform the installgrub to make this disk bootable.

We now need to relocate our swap to the new swap device on /dev/dsk/c4t1d0s1 and remove the original swap device before we re-partition the original disk as a mirror.

Now the new swap is in place, but we need to add an entry to /etc/vfstab for it so it is used next time we boot.  Take a look at your /etc/vfstab and add an entry for the new device after the entry for the original device.  My new /etc/vfstab now looks like this:

Next we need to create an identical set of fdisk partitions on the original disk and lay down the OpenSolaris VTOC.  This is fairly simple to do using fdisk, prtvtoc, and fmthard.

So, we wrote out the fdisk partition information from the physical disk c4t1d0p0 and put it on the physical disk c4t1d0p0.  Note these are the physical devices and not slices, therefore they use “p” in the device name.

Now do the VTOC, but we don’t have to write to a temporary file.

Setup the newly partitioned c4t0d0s0 as a mirror for rpool.  This will also involve doing a new installgrub to make the disk bootable.  and then lastly check the status of the zpool.

This is going to take a few minutes, so now would be a good time to create the new zpool for naspool which will be our NAS pool which we can extend with concatenated mirrors.  We will create the zpool with c4t0d0s6 and attach c4t1d0s6 which were the “free hog” partitions we created earlier.  Then create a mount point for this zpool called /export/nas.

All done.  Next to reboot with two new virtual disks and add them as concatenated mirrors to the zpool naspool.

After the reboot, we will see both swap devices online and swap -l will confirm this:

I’ve added two new 100GB disks which will be used to extend the zpool naspool.  I did this inside VrtualBox by creating two new 100GB disks and connecting them to the SATA controller like I did with the original two disks.  The disks do not have any fdisk information on them, so we will need to put it on there.  Luckily fdisk on OpenSolaris has a “-B” option to fdisk the entire disk as a single Solaris2 partition.  The final step is to attach the two new disks c4t2d0 and c4t3d0 to as mirrored concatenated stripes to the zpool naspool.

Notice that when we did the zpool add, we didn’t have to specify a slice.  ZFS is smart enough to examine the device and partition it as required.  Also notice we’ve added 100GB to the zpool naspool.

The new zpool status (after both pools scrubbed for good measure) looks like this:

All done, and now we can add pairs of additional disks of varying sizes to this mirrored stripe.

Summary

So here’s a high level overview of the step-by-step process we just went through.

1. ‣Construct a “machine” in VirtualBox, it’s fun and it’s free.
   

2. •2x100GB SATA virtual hard disks (or possibly 4, will save a step or two later)
   

3. •512MB RAM
   

4. •Bridged ethernet adapter, though I use this only so I can ssh into the machine from a terminal session.
   

5. ‣Get the latest OpenSolaris media and mount it to the virtual machine CD-ROM.
   

6. ‣Install on a single disk using the minimum - 8GB partition (I will assume “target 0”).
   

7. ‣Log into the machine and using a terminal window become root using: pfexec su -
   

8. ‣You are now executing commands as root.  BE VERY CAREFUL FROM THIS POINT ON.
   

9. ‣At the command prompt type: format
   

10. •Using format select “target 1”
    

11. •Use fdisk within format to create a 100% Solaris2 partition.
    

12. •Partition the disk to your preference.  I used:
    

13. •Modify a partition table using: mod
    

14. -Select the “All Free Hog” option.
    

15. -Select which partition is to be the “Free Hog.”
    

16. -Setup all other partitions by size.  I used:
    

17. Partition ‘0’ - 20GB Base OS Install
    

18. Partition ‘1’ - 2GB Swap
    

19. -Name the partition table.
    

20. -Label the disk.
    

21. •Quit format utility.
    

22. ‣“Replace” the initial install device with your resized device: zpool replace -f rpool c4t0d0s0 c4t1d0s0
    

23. ‣Make the disk bootable: /sbin/installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c4t1d0s0
    

24. ‣Add the new swap device: swap -a /dev/dsk/c4t1d0s1
    

25. ‣Remove the old swap device: swap -d /dev/dsk/c4t0d0s1
    

26. ‣Redo the partitioning tables on the original disk.  (NOTE: When using fdisk the devices are not “slices,” the “p” in the device name is intentional)
    

27. •Create a partition file from the new disk: fdisk -W - /dev/rdsk/c4t1d0p0 > /tmp/fdisk.par
    

28. •Re-partition the original disk: fdisk -F /tmp/fdisk.par /dev/rdsk/c4t0d0p0
    

29. •OpenSolaris partition table (slices): prtvtoc /dev/rdsk/c4t1d0s2 | fmthard -s - /dev/rdsk/c4t0d0s2
    

30. ‣Attach the new mirror device from the original disk: zpool attach -f rpool c4t1d0s0 c4t0d0s0
    

31. ‣Make the new(original) disk bootable: /sbin/installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c4t1d0s0
    

32. ‣Make sure there are entries in /etc/vfstab for both swap devices: /dev/dsk/c4t1d0s1 and /dev/dsk/c4t1d0s1
    

33. ‣Create the new zpool for NAS storage: zpool create -f naspool c4t0d0s6
    

34. ‣Mirror it: zpool attach -f naspool c4t0d0s6 c4t1d0s6
    

35. ‣Create a ZFS export filesystem: zfs create naspool/export
    

36. ‣Set the mount point for the export: zfs set mountpoint=/export/nas naspool/export
    

37. ‣Add two new 100GB disks to the virtual host.
    

38. ‣Reboot OpenSolaris: init 6
    

39. ‣Use fdisk to setup the new disks as 100% Solaris2 partitions.
    

40. •fdisk -B /dev/rdsk/c4t2d0p0
    

41. •fdisk -B /dev/rdsk/c4t3d0p0
    

42. ‣Add the two new disks as a concatenated mirror to zpool naspool: zpool add naspool mirror c4t2d0 c4t3d0
    

You’re done.