Frame Dragging

I Went To The Meetings And All I Got Was This T-Shirt...

Sun, 25 Apr 2010 22:22:56 +0900

Chaos Dominates Life
As with the lat blog entry, life is in a rather strange state of flux right now but I think things will be settling down to a steady state in the next few weeks. As with the last blog entry, I’d like to continue to point out to anyone who might be following my web site by RSS feed that I have a new version of my resume out. Please have a look at it.

Not All Work, Some Play
Got out to the Tokyo Solaris User’s Group Meeting, well really for the Nomikai which was the important part. Kato-san made sure I finally got my OpenSolaris Group Leader shirt. He and I continued discussing how we could involve more of the non-Japanese Solaris users and enthusiasts in the meetings. The classroom study has been put on hold, but we still have that planned on the back burner once I figure out whether I will remain in Tokyo or leave for some other port of call. It’s been a busy year of giving presentations, going to the various other Tokyo User community events, social and professional networking. All the while trying to promote OpenSolaris.

We’d still like to get the to get more of the ex-pats who use OpenSolaris or Solaris involved in coming to the meetings to learn more about Solaris and OpenSolaris as well as exchange ideas, culture and form a tighter community. The ex-pats who use Solaris have a lot of valuable experience, as do the Japanese group members. In the spirit of fostering communication between the communities, we will continue with plans to add a third study group who’s primary language is English centered around the OpenSolaris Bible as the text for the class. The study session will be open to anyone who wishes to attend, and the last 45-60 minutes of the session will be open for anyone who wants to present or give a talk about something interesting they’ve done, make community announcements. After that, there will be more cultural exchange and communication an the nomikai, nijikai, sanji....

There are a lot of cool things going on in the Japanese user community, but it doesn’t necessarily get communicated outside of Japan, and there are things going on in the international community which don’t always get translated or communicated to the Japanese community such as new White Papers, research projects and other cool aspects of computing. Our goal is to open the communication more and try to have some fun while doing it.

Looking forward to the next meeting and hope to see you there....

Putting Out a Fire With One’s Head in the Clouds

Wed, 7 Apr 2010 13:56:04 +0900

Fire Fighting - Your Life is on the Line
This is just a brief announcement to anyone who might be following my web site by RSS feed that I have a new version of my resume out. Please have a look at it.

I’ve been woefully behind updating my blog, but really so there has been so much chaos to deal with lately that there just hasn’t been time. When life gets back to some sense of stability, I’ll try to catch up with a number of projects I’ve been working on.

Lately I’ve been working on Solaris/OpenSolaris based HPC Cluster and Cloud Computing platform designs and implementation plans using research I’ve done into best practices. There are a lot of committees and players in this arena, a veritable alphabet soup of groups. The market is fragmented right now, but slowly standards are being agreed upon and published.

There are a few surprises here as well. Cisco considers themselves a Cloud Computing company, as does EMC. Also, one of the biggest providers of Cloud services, Amazon, doesn’t participate on any of the standards committees. This really surprised me due to their offerings of S3, EC2, and EBS becoming de-facto standards which cloud hosting providers must provide compatibility for their API’s.

A lot of really cool work has been done on this and when I get a chance I’ll post some of what I’ve learned, read, and designed, and hopefully started to implement.

Translating the OpenSolaris Bible into Japanese

Fri, 29 Jan 2010 03:24:14 +0900

A Very Large Task for a Very Large Book
The OpenSolaris Bible is probably the definitive guide to OpenSolaris available now. This book is a very thorough reference, but for the OpenSolaris enthusiasts here in Japan it is a challenge. The operating system is very large and complex, but also having to use this book in a language which is not your native language is an even more difficult hurdle to overcome.

The book was written by three senior engineers at Sun who have all played a major role in the production of OpenSolaris. Their areas of expertise cover most every aspect of Sun’s operating system. Beginning with running the operating system on your desktop and hooking devices from printers to digital cameras as an end user, to system administration and development using OpenSolaris. The book covers some of the more advanced features of OpenSolaris like ZFS, DTrace, virtualization, web services, application development, and much more. This is no small task, and this is no small book which weighs in at close to 1000 pages.

Because OpenSolaris is the Open Source brother to its Data Center capable counterpart Solaris, with all of its features and more, OpenSolaris is an operating system of great interest to casual users, hobbyists, and professional system administrators. OpenSolaris can run in a virtual machine such as VirtualBox or can be installed on most Intel and now some Sparc systems. Anyone can download the operating system, install it and use it for learning more about Solaris and OpenSolaris, or developing real applications on an enterprise class operating system for no charge at all. The source code is even freely available and if you are adventurous enough, you can even build and package your own distribution as many people have. This makes OpenSolaris similar to Linux in this way and there’s even a large community of developers and contributors both internal to Sun and outside independent developers. It is very different and distinct from Linux in that it is the first proprietary operating system to make the transition to Open Source and it is an operating system which is completely descended from Unix SVR4.

The challenge the OpenSolaris Bible Translation Project faces is to translate the material into Japanese, review it with all levels of Japanese users and engineers, and ensure it is correct before going to publication.

The Approach
The book is being translated into Japanese right now. That’s the first step. The next is to have the translation reviewed by engineers and users who are bi-lingual in English and Japanese. Then once the material is reviewed, it should then go for final review and printing.

The approach to reviewing the translated text is a bit unique. The material will be taught as a class using the English text as the course textbook. The Japanese translation will be provided to the classroom attendees to review and make notes. The material will be presented in a classroom format with the presentation materials in English and Japanese. In addition to the classroom presentation, there will also be hands-on labs in the class and take-away exercises.

While the project leader, Ken Okubo, is working coordinating and translating the English text into Japanese, I am working on building the classroom portion. I am writing a classroom presentation for each chapter, in English, which will also be available in Japanese. In addition to the classroom presentation, I am also developing the in class hands-on exercises and labs as well as the take-away assignments.

The classes will not be restricted to Japanese fluent individuals, but will be open to users, system administrators, developers, and anyone who is interested knowing more about or learning about OpenSolaris. This is a free open event for anyone who wishes to attend, and even if you aren’t Japanese fluent, the classroom lecture will be done in English. So I would encourage anyone who has an interest in learning more about OpenSolaris to come and participate. It’s also a good way to get to know each other and build a community around OpenSolaris and help support the Tokyo Open Solaris Users Group.

The Requirements
Every student participating in the class will be provided an English copy of the OpenSolaris Bible, the classroom presentation in English and Japanese, and the raw unformatted text translation for review. Each student is required to have the following items or abilities:

• Make the commitment to attend all classroom sessions
• Be able to read and write Japanese fluently (only of you are one of the translation project participants)
• Be able to read and comprehend spoken English
• Have a laptop computer capable of running VirtualBox in the following configuration:
• VirtualBox 3.1.2
• 1 GB RAM for virtual OpenSolaris host
• 2 sparse virtual disks of 10GB minimum for the OpenSolaris Installation
• 5 additional sparse virtual disks of 10GB for ZFS exercises
• WiFi networking or other supported wireless networking capability for classroom exercises

This class will assume you have basic computer skills, but will not assume advanced knowledge of OpenSolaris. We’re going to start at the basics and work our way through. You should also posses a willingness to learn and the desire to learn more about OpenSolaris and how it works and what it is capable of.

Are You Interested In Participating?
If you are interested in participating in this project, you can express your interest by using the Tokyo OpenSolaris User Group Mailing list. You can do so by signing up and sending mail to the list on the UG-TSUG mailing list sign-up form. We will also be using this list to make announcements regarding the classroom sessions as well as any questions or discussions regarding OpenSolaris as it pertains to this course and review process.

The classes are planned to start on the last Saturday of the month in either February or March. We would like to get at least 15-20 participants.

There will be no charge for this class, it will be completely free. All you need to do is show up.

This will take a substantial amount of time for all participants involved in producing the materials as well as the classroom participants and reviewers. Please do not sign up if you do not think you can make the commitment.

Additional Resources
• OpenSolaris Community
• OpenSolaris Download Page (Initially the class will use 2009.06 or snv_111b)
• Tokyo OpenSolaris Community
• Tokyo OpenSolaris Community Mailing List Signup
• VirtualBox

ZFS and NAS Presentation for Tokyo OpenSolaris Users Group

Mon, 29 Jun 2009 16:25:10 +0900

Saturday, I gave a presentation at the Tokyo OpenSolaris Users Group. My presentation was an overview of storage types, ZFS and my method for best setting it up for expansion and portability between system to avoid problems later. The presentation also included a live demonstration of setting up ZFS and connecting an OpenSolaris server to NFS, CIFS and iSCSI clients. The slides are available for download below.

For my demonstration and presentation I used a Mac Mini with 2GB RAM. I used two VirtualBox guests, one was the OpenSolaris “Server” and the other was a Windows XP client. Additionally the presentation was given in Keynote on the same machine. The machine held up fairly well under the load.

Jim Grisanzio was there and also a big help in organizing things as well as several of the other people at the Sun Microsystems office in Yoga, Tokyo.

Jim, an excellent photographer, even captured the moment in photos, available here.

The presentation is here in large (higher quality) and small PDF format.

TSUG-OSol-NAS-20090627.pdf 15.9MB
TSUG-OSol-NAS-20090627-small.pdf 3.3MB

The photo above is an iPod vending machine taken very quickly in the Memphis, TN airport. I’ve never seen a machine like this in Tokyo or any other part of Japan which is the home of all sorts of odd vending machines.

OpenSolaris Boot Environments - Freedom and Flexibility

Thu, 18 Jun 2009 16:44:06 +0900

Air Bags For Your System
Along with all the other cool things OpenSolaris brings along with ZFS and its snapshot ability, boot environments or be’s are a really nice feature built on top of the ZFS ability to snapshot and clone a filesystem. It is the ultimate airbag for those “oops” moments when something goes wrong with a patch or you wish to try something new out with the operating system and want to get the system back to a known state. They are also your best friend when trying to put together a demo and want to work out how to make it smooth, then roll-back the changes easily so you can do a fresh demo. Another use if for developers, packagers, and others who work at the system level and need to test out installation processes, system updates, or other system level functions. Boot Environments give everyone the ability to get the system back to a known point and the confidence to concentrate on the work necessary to deliver results rather than worrying about complicated back-out procedures.

In putting together a presentation on file systems, I built a virtual OpenSolaris machine in VirtualBox. I want to demonstrate the changes required for various type of file sharing such as NFS, SMB and iSCSI. To do this I also want to demonstrate the process of loading the required software packages. So, a little bit of practice and the ability to roll back to a known clean state is a very nice thing.

The first thing I did was to get a fresh install of the OS which is my core. The next thing I did was create anything I needed as a baseline configuration. Items like creating users, setting up a static IP address, filesystems, changing the state of any services in SMF, and any other housekeeping items to have ready at the beginning of the demo.

Next I create a “Baseline Configuration” boot environment which is a snapshot of the system at this point. It is now my master point which I can roll back to in order to get a clean system in a known state.

At this point, I check the /rpool/boot/grub/menu.lst file and make any changes I want to there. This could be turning on or off the verbose boot mode of the kernel, or adding the graphic splash screen to watch during boot time. I prefer not having the graphic splash screen while booting as it may mask startup problems I want to know about.

I’m a bit superstitious, so after creating the new boot environment, I reboot the machine to test it. Things should be just fine, but I like to verify there are no problems.

After this boot, I create another boot environment which will be where I actually do the work. I call this one “undo-demo”. It is in this be where I install all additional packages, make system configuration changes and build the final system for the demo.

Once I’m finished doing anything for testing or demo purposes, I can simply boot from my baseline configuration (be00) and destroy the demonstration configuration boot environment (undo-demo).

The same idea can be applied to systems being used for testing in a lab to make changes, test procedures, install software or patches before going into production. This method of creating a live boot environment and a boot environment used as a rollback point can give you a great deal of flexibility in an operational capacity in that you don’t have to worry about rollback procedures as much and can actually concentrate on moving forward.

In a production environment, I’d use the same technique, but also rotate my be’s. In other words, when my new environment is stable and configured how I want it, I can create a new be which is my active environment. Then before I begin making changes to the system and depending on how much time has passed since I last created my live be, i would create a new be as a starting point and another as my rollback point.

Once things are how I want them, I can rename the working configuration to be my live or rollback configuration using the beadm rename command.

Putting It Into Practice
So, let’s take a look quickly at how this can be put into practice. The first thing I want to do is to get the baseline be and then a second be which will be my live working configuration. Of course, as always, the usual disclaimers apply...

root@spirit:/# beadm create -d “Rollback Configuration (be00)” be00
root@spirit:~# beadm list
BE Active Mountpoint Space Policy Created
-- ------ ---------- ----- ------ -------
be00 - - 18.27M static 2009-06-17 11:41
opensolaris NR / 3.16G static 2009-06-16 21:47
root@spirit:~# bootadm list-menu
the location for the active GRUB menu is: /rpool/boot/grub/menu.lst
default 2
timeout 2
0 OpenSolaris 2009.06
1 OpenSolaris 2009.06 - Text only
2 Rollback Configuration (be00)
root@spirit:~# bootadm set-menu default=2
root@spirit:~# init 6

I used the bootadm command to set the default boot menu selection to be number 2. After rebooting, the system will start using the newly created be.

Immediately after start, I create another be on top of this one. The command sequence is pretty much identical.

root@spirit:/# beadm create -d “Working Configuration (wk00)” wk00
root@spirit:~# beadm list
BE Active Mountpoint Space Policy Created
-- ------ ---------- ----- ------ -------
be00 - - 18.27M static 2009-06-17 11:41
opensolaris NR / 3.16G static 2009-06-16 21:47
wk00 - - 77.0K static 2009-06-18 22:36
root@spirit:~# bootadm list-menu
the location for the active GRUB menu is: /rpool/boot/grub/menu.lst
default 2
timeout 2
0 OpenSolaris 2009.06
1 OpenSolaris 2009.06 - Text only
2 Rollback Configuration (be00)
3 Working Configuration (wk00)
root@spirit:~# bootadm set-menu default=3
root@spirit:~# init 6

Now my working configuration is where I will install my software, and is my working copy. Once I’m happy, I can then use beadm to destroy the “rollback configuration” and rename my working configuration to be my new baseline configuration.

If anything goes horribly wrong in the process, I can simply boot the rollback configuration, destroy the working configuration and repeat the above process to create a new working be.

A Smooth Upgrade - OpenSolaris 2009.06 snv_111b

Fri, 5 Jun 2009 18:58:43 +0900

So earlier this week the new OpenSolaris 2009.06 was released in conjunction with CommunityOne in San Francisco. The new stable release of OpenSolaris snv_111b is hot off the presses and is getting good reviews. It was actually selected as a release candidate the week before, but I’m sure Sun wanted to promote it during the CommunityOne festivities. I’m a two days late with this entry, but better late than never.

Upgrade The Server
My home NAS server I’ve been using for quite a while which I just migrated to OpenSolaris 2008.11 was doing just fine, but I wanted to have the latest build as I intend to try building OpenSolaris from scratch in the next few weeks and wanted to be up-to-date. After all it took me a year to get migrated to the latest version a few weeks ago, so I figured, why wait this time.

The upgrade was a snap. However, the usual disclaimers apply when doing this as when doing anything as root. Usual disclaimers apply, please read “the fine print.”

I went to Sun’s web site and found the section on upgrading an existing install to OpenSolaris install 2009.06 and it really couldn’t have been simpler. Before trying this out on my live server, even though be’s (boot environments) and zfs rollback works like a charm, I built a virtual machine in VirtualBox and installed OpenSoalris 2008.11 just to be safe. I did the following from the command line:

‣ pfexec su -
‣ pkg install -v SUNWipkg
‣ pkg image-update -v
‣ init 6

That’s it and I’m now up and running the latest version of OpenSolaris 2009.06.

I have to say it did take a little bit of time due to the extra load the Sun servers were having to deal with due to the enthusiasm of people wanting to upgrade and download. I did add the “-v” option for verbose because I wanted to see what it was doing while it was doing it and it gave me something to watch during the upgrade. In all, it took about 30-45 minutes due to slow response times and load on the servers.

One side note, I’ve only found one problem with the upgrade, but it was simple enough to fix and didn’t notice it until today.

root@spirit:/# man pkg
windex entry incorrect: pkg(1t) not found.
root@spirit:/root# catman -w &

This is easily fixed by issuing a catman -w to rebuild the whatis database.

UPDATE: 10 June 2009 - Did find one more things which needs to be done after upgrade. Any disk pools you have will most likely be on a lower version of ZFS. I only noticed this because I wanted to scrub the disks after a less than graceful shutdown and checked the status of the pools. Very easy to take care of using the zfs upgrade command.

root@spirit:~# zpool status rpool
pool: rpool
state: ONLINE
status: The pool is formatted using an older on-disk format. The pool can
still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'. Once this is done, the
pool will no longer be accessible on older software versions.
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c5d0s0 ONLINE 0 0 0
c4d0s0 ONLINE 0 0 0

errors: No known data errors
root@spirit:~# zpool upgrade
This system is currently running ZFS pool version 14.

The following pools are out of date, and can be upgraded. After being
upgraded, these pools will no longer be accessible by older software versions.

VER POOL
--- ------------
13 rpool
13 xpool

Use 'zpool upgrade -v' for a list of available versions and their associated
features.
root@spirit:~# zpool upgrade -v
This system is currently running ZFS pool version 14.

The following versions are supported:

VER DESCRIPTION
--- --------------------------------------------------------
1 Initial ZFS version
2 Ditto blocks (replicated metadata)
3 Hot spares and double parity RAID-Z
4 zpool history
5 Compression using the gzip algorithm
6 bootfs pool property
7 Separate intent log devices
8 Delegated administration
9 refquota and refreservation properties
10 Cache devices
11 Improved scrub performance
12 Snapshot properties
13 snapused property
14 passthrough-x aclinherit support
For more information on a particular version, including supported releases, see:

http://www.opensolaris.org/os/community/zfs/version/N

Where 'N' is the version number.
root@spirit:~# zpool upgrade rpool
This system is currently running ZFS pool version 14.

Successfully upgraded 'rpool' from version 13 to version 14

Pools are then updated. So simply:

• Check the pools on the machine: zpool upgrade
• Check the ZFS versions available in your installation: zpool upgrade -v
• Upgrade and repeat for each pool: zpool upgrade poolname

A Couple Of Extras
I’m very much a command line person and have been for a long time. I’m not particularly fond of GUI’s, I don’t like watching the little orange bar moving across my screen while the system is booting. Bad things sometimes happen during this process and you can be stuck staring at the graphical boot for a very long time. So, when I’m booting, this is what my /rpool/boot/grub/menu.lst file looks like. My changes highlighted in blue:

splashimage /boot/grub/splash.xpm.gz
background 215ECA
timeout 5
default 3
#---------- ADDED BY BOOTADM - DO NOT EDIT ----------
title OpenSolaris 2008.11 snv_101b_rc2 X86
findroot (pool_rpool,0,a)
splashimage /boot/solaris.xpm
foreground d25f00
background 115d93
bootfs rpool/ROOT/opensolaris
kernel$ /platform/i86pc/kernel/$ISADIR/unix -B $ZFS-BOOTFS,console=graphics
module$ /platform/i86pc/$ISADIR/boot_archive
#---------------------END BOOTADM--------------------

title OpenSolaris 2008.11 snv_101b_rc2 X86 text boot
findroot (pool_rpool,0,a)
bootfs rpool/ROOT/opensolaris
kernel$ /platform/i86pc/kernel/$ISADIR/unix -v -B $ZFS-BOOTFS
module$ /platform/i86pc/$ISADIR/boot_archive

title opensolaris-1
findroot (pool_rpool,0,a)
splashimage /boot/solaris.xpm
foreground d25f00
background 115d93
bootfs rpool/ROOT/opensolaris-1
kernel$ /platform/i86pc/kernel/$ISADIR/unix -B $ZFS-BOOTFS,console=graphics
module$ /platform/i86pc/$ISADIR/boot_archive
#============ End of LIBBE entry =============

title OpenSolaris 2009.06 snv_111b X86 text boot
findroot (pool_rpool,0,a)
bootfs rpool/ROOT/opensolaris-1
kernel$ /platform/i86pc/kernel/$ISADIR/unix -v -B $ZFS-BOOTFS
module$ /platform/i86pc/$ISADIR/boot_archive

Notice, I’m using the 4th boot entry (indicated by “default 3”, the first is “0”) and I also don’t like waiting so I set the timeout on the GRUB screen to 5 seconds which is plenty of time to select something else if I need to. Also, I’ve added the “-v” verbose option to the kernel so I have something to look at while it’s booting. It just gives me a warm feeling to see the machine “doing something” at boot and this is a “headless” server most of the time anyway.

In all, I’m very pleased with how smoothly the upgrade went. Many thanks go out to everyone who helped build, test and package this great operating system.

Get OpenSolaris 2009.06snv_111b now.

Tokyo OpenSolaris User’s Group Meeting - 30 May 2009

Sun, 31 May 2009 22:23:00 +0900

Yesterday I went to the Tokyo OpenSolaris Users Group meeting at Sun’s office in Yoga. The room was packed with many people all enthusiastic to learn more about OpenSolaris. It was an all day affair lasting well into the evening as the meeting moved on to “nijikai” down the street. Good food, good people, and nice exchange of ideas.

Had to get this gratuitous shot of our host from Sun Microsystems, Jim Grisanzio’s laptop covered with Kitty-chan stickers courtesy of his daughter. Jim has a large collection of laptop photos from various meetings like barcamp, and other meetings.

I was invited to give a presentation on ZFS, Home NAS, and a bit about using ZFS for backing up OS X using iSCSI.

It was very nice to see this much interest in OpenSolaris. The room was full, hopefully we’ll see as many or more at the next meeting. I am really looking forward to attending the next meeting, hopefully we’ll need a larger room.

Future plans, for meetings would be to have the Tokyo Linux and BSD communities meet for some sort of bake-off, or joint presentation. The Sun facilities are nicely suited for this sort of meeting. Several people are working on coordinating this. I’ll post more information here as I find out more, but am really looking forward to this as well.

More photos of the event may be found in my gallery, and on Jim’s OpenSolaris blog, as well as his Flickr pages.

Enjoy the photos!

ZFS File System Layout - Just A Couple Of Suggestions...

Fri, 29 May 2009 00:14:42 +0900

So Much Disk, How To Organize?
As I began mapping out my storage for my home NAS, I started trying to think about flexible file system configurations. During my upgrade and migration efforts, I started to run into some of the limitations of ZFS, in particular the handling of rpool and its relative inflexibility, to the rest of what ZFS offers as a whole. I wanted to come up with a way of making things as flexible as possible, eliminate potential conflicts created by moving storage from one system to another when the pool names might collide as they did when I migrated an existing rpool to a fresh installation.

I also wanted to incorporate other things into the mix as well. Things like backup areas implemented with CIFS, NFS, or iSCSI. I wanted traditional shared NAS for user files as well as large data sets like audio and video. Zones were also a consideration since I want to isolate the host operating system from the functions of any special type servers such as web, application, and infrastructure services (NTP, DNS, LDAP, etc.). Also, these special servers shouldn’t be tied to a particular fixed domain like the host operating system. So, my goals were fairly clear-cut and defined at this point.

This is a 10,000’ overview and I won’t even begin to touch on mount point for this discussion...

Configuration Goals
‣ Reduce The Impact Of rpool Configuration Inflexibility
• Must be no more complex than a “simple” mirror
• Cannot be increased in size
• Placing other file systems in rpool make those other filesystems harder to move at the pool level
‣ Core/Base Host OS Isolation
‣ Special Purpose Sections (zvol)
• swap
• iSCSI
• Raw for Database
‣ Traditional NAS
• User-based File Storage
• Group FIle Storage
• Common Shared Storage
‣ Structure and “Control Points” for File System Properties
• “Artificial” File Systems
• Property Setting Inheritance
‣ Flexibility for Zones and Other Virtual Hosts
‣ Importation of Potentially Conflicting zpools

This diagram lays out what I think to be a reasonable best practices template for ZFS file system organization. One item missing here is an area specifically for databases, either raw or “cooked”. These could easily be worked into this structure however, and I may add to this later, moving the “data” control point to the same level as “export”. Beneath that point we could then put database file systems directly under that point, or place two additional control points for raw zvol type database data, and another for file system based database files.

OpenSolaris ZFS iSCSI Time Machine in 20 Minutes

Mon, 25 May 2009 22:30:58 +0900

The Truth Is Out There...
I’ve seen many write-ups about using OS X Time Machine with NAS, and originally I was using my OpenSolaris machine for doing rsnapshot backups of our laptops, but this is just not a good long-term solution and can’t be managed very well. It relies on NFS or CIFS to mount the snapshot directory, and in general is sort of a Rube Goldberg contraption. Something had to be done, and I was wondering how to put iSCSI to use with ZFS. I wanted to have my own home-brew Time Capsule in effect.

While walking to the post office I was thinking about the alternatives for serving up the filesystem OS X Time Machine and whether to use NFS or CIFS. I’ve never had a problem with NFS in a Unix environment, mixing it with Macs has always been fairly easy and getting easier. However, most people use CIFS, and using either protocol is somewhat slow. The I started thinking, this might be just right for iSCSI.

A brief search on the Internet turned up this link: ZFS iSCSI pool integration which was close to what I wanted, but not quite complete. It did point me to this handy, free iSCSI initiator for OS X.

By the way, this is not at all supported by Apple and could just stop working with any OS X upgrade. So you are at your own risk here. If it does break at some point in time due to the overriding of preferences, with an undocumented preference, I’m sure there will be a work-around. But, as unlikely as it may be, you should use this with caution. Standard disclaimers apply.

What You Need
Here’s a list of everything I used:

‣ Mac OSX 10.5.7 and a Mac
‣ OpenSolaris 2008.11
‣ Studio Network Solutions globalSAN iSCSI initiator for OS X
‣ Sun iSCSI Target - SUNWiscsitgt
‣ Of course some hardware to run OpenSolaris on with a good bit of disk.
‣ About 20 minutes

Of course the hardware is extra. I am using the same NAS machine I’ve been setting up in the last posts for hosting the iSCSI volume.

‣ Download the iSCSI initiator and install. This will require a reboot.
‣ Log into the OpenSolaris server and do the following:
• Install the Sun iSCSI Target package
- pkg install SUNWiscsitgt
- svcadm enable /system/iscsitgt
• Create a ZFS zvol. I’m going to put multiple zvols for time machine so I created a ZFS file system for them and set the “sharescsi” attribute so any zvol created under that location will inherit this property. I created a sparse volume of 250GB. The host name is “acrylic.” Since it’s backups, it’s probably OK to set compression on.
- zfs create xpool/export/tm
- zfs set shareiscsi=on xpool/export/tm
- zfs set compression=on xpool/export/tm
- zfs create -s -V 250g xpool/export/tm/acrylic
‣ Back on the Mac...
• In a Terminal window type the following (this is the real magic):
- defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1
• Go to System Preferences
- Select globalSAN iSCSI
- Select the “Portals” tab and add a new portal for the iSCSI host
- Select the “Targets” tab
- Select the iSCSI target you created. This can be identified on the OpenSolaris host using:
iscsitadm list target
• Go into Disk Utility and you will see the iSCSI disk
- Select the disk and the “Partition” tab
- Create a single Mac OS Extended (Journaled) partition
‣ At this point Time Machine will see the volume and ask if you want to use it as a backup disk, tell it yes.

You’re done. It was that simple.

I wanted to get this up as soon as possible. I’ll have some screen but they are really big and boring.

Everything in Its Place - Moving and Reorganizing ZFS Storage

Tue, 19 May 2009 16:55:49 +0900

Order is good. Organizing, cataloging and putting everything in its place should also look nice. Everything should be easy to find, and fall into place naturally. My plan of migrating my OpenSolaris installation from one set of disks to another by doing a fresh install was a bit more complicated than I first planned. I followed my previous plan fairly closely, but found some stumbling points along the way. In the end, everything is back together, I have more space, and know a lot more about ZFS.

My biggest challenge was namespace collisions and also beadm (Boot Environment administration utility) got a bit confused, probably due to some metadata written in the old ZFS rpool or some other location causing it to think the old imported rpool was still usable as a boot environment. This prevented me from being able to get recursive snapshots, but I managed to find a work-around. Additionally, ZFS handles mount points more-or-less in an automatic manner which only exaggerated the namespace issues.

I also, in my opinion discovered a minor inconvenience in ZFS when dealing with zvols for swap and dump areas. This is related to taking recursive snapshots on the zpool which results in taking a snapshot of swap and dump. It is questionable whether there is any value in having a snapshot of the dump area, but there is no value I can see in having a snapshot of swap - ever.

The Plan
This entry will cover the following:

‣ Moving swap from partitions on disk to a ZFS zvol
‣ Importing a foreign rpool into an existing rpool
‣ Resolving namespace colisions
‣ Growing storage on a mirror
‣ ZFS send/receive to migrate off the foreign rpool onto a new pool
‣ Finishing up the mirroring process

That’s a lot...

The Environment
In order to recreate this experience, I’ll go back to a simulation using a VM in VirtualBox. The configuration I’ll use 1/10th the size for all the disk drives, but it will work just the same. Additionally, I have some file systems on the actual hardware I will not duplicate here. The system layout was done like in my previous post regarding disk repartitioning and other than swap now situated in a ZFS zvol, it’s pretty much the same.

This is the initial configuration of the virtual machine I will use for this demonstration. The main difference is that the two 100GB disks are on SATA port 0 and 1 and the two 50GB disks are on SATA port 2 and 3. These have the original OpenSolaris 2008.05 installation on them. I went on to rearrange the disks to look much closer to the physical hardware configuration for the migration and the final system. The physical hardware I’m using has dual SATA controllers with two ports each. I put one mirror on the first controller and the other mirror on the second controller. The next snapshot reflects this configuration.

I’m using the first 50GB disk as a boot disk and the second as it’s mirror. They will be on ports 0 and 2, while the 100GB disks which contain the existing rpool mirror reside on ports 1 and 3. Again, these disks are sized proportionately to the actual hardware I’m using a pair of 500GB disks and a pair of 1TB disks.

Let’s Do It!
So after shuffling the hardware around, we have the following configuration. The pool oldrpool has OpenSolaris 2008.05 installed and some additional directories we will need to move into the xpool storage hierarchy which I will do using zfs send/receive. However, before we can access the file systems on oldrpool, we will need to take care of some namespace collisions which occur due to mount points.

The first step will be importing the pool by using the zpool import and importing it as its new name oldrpool. Another thing we want to do at this point is also upgrade the ZFS version of oldrpool.

First the standard disclaimer as these procedure are done as root:

WARNING! - Important Safety Tips
‣ All the following commands I will use are done using the root user.
‣ If you are not comfortable using root directly, then DO NOT TRY THIS.
‣ If you are not 100% comfortable with what you are doing, DO NOT TRY THIS!
‣ Use common sense, HAVE A BACKUP/RECOVERY PLAN BEFORE ATTEMPTING THIS!
‣ Unless you really know what you are doing and are comfortable with everything here, DO NOT ATTEMPT THIS ON PRODUCTION SYSTEMS.
‣ LAST, I CLAIM NO RESPONSIBILITY FOR YOUR ACTIONS. IF YOU SCREW UP, YOU ARE ON YOUR OWN, SORRY.

First, lets move swap around. This is fairly simple to do.

root@panacea:~# swap -l
swapfile dev swaplo blocks free
/dev/dsk/c4t0d0s1 50,1 8 1044216 1044216

Swap only resides on /dev/dsk/c4t0d0s1 and we will use that partition when we build the xpool I described in my earlier post.

We need to create a new swap partition, add it and delete the old one. As a last step, we must add the swap device to /etc/vfstab.

root@panacea:~# zfs create -b 4096 -V 512m rpool/swap
root@panacea:~# swap -a /dev/zvol/dsk/rpool/swap
root@panacea:~# swap -l
swapfile dev swaplo blocks free
/dev/dsk/c4t0d0s1 50,1 8 1044216 1044216
/dev/zvol/dsk/rpool/swap 182,2 8 1048568 1048568
root@panacea:~# swap -d /dev/dsk/c4t0d0s1
root@panacea:~# swap -l
swapfile dev swaplo blocks free
/dev/zvol/dsk/rpool/swap 182,2 8 1048568 1048568

This is pretty straightforward system administration, so I really won’t go into details here. What is important is we created a 512MB zvol vdev /dev/zvol/dsk/rpool/swap for swap which will be mirrored using the zfs create command. After this is done, we want to make sure to replace the original swap device in /etc/vfstab with the new one.

Now, let’s take a look at how the configuration is actually reported.

root@panacea:~# zpool status
pool: rpool
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c4t2d0s0 ONLINE 0 0 0
c4t0d0s0 ONLINE 0 0 0

errors: No known data errors

pool: xpool
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
xpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c4t2d0s1 ONLINE 0 0 0
c4t0d0s1 ONLINE 0 0 0

root@panacea:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 3.14G 11.6G 72K /rpool
rpool/ROOT 2.39G 11.6G 18K legacy
rpool/ROOT/opensolaris 2.39G 11.6G 2.26G /
rpool/dump 256M 11.6G 256M -
rpool/export 613K 11.6G 19K /export
rpool/export/home 594K 11.6G 19K /export/home
rpool/export/home/mps 575K 11.6G 575K /export/home/mps
rpool/swap 512M 12.1G 37.2M -
xpool 69K 34.2G 18K /xpool
root@panacea:~# zpool import
pool: rpool
id: 2921547533073523785
state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier and
the '-f' flag.
see: http://www.sun.com/msg/ZFS-8000-EY
config:

rpool ONLINE
mirror ONLINE
c4t3d0s0 ONLINE
c4t1d0s0 ONLINE

The command zfs import shows the old rpool and even gives us a warning about it. Notice in the zfs list, I have attached the new mirrored pool, xpool, and I have the OpenSolaris 2008.11 installed in the attached rpool. By simply importing the 2008.05 rpool, we will end up with namespace collisions. This is because some of the metadata stored in the zfs file systems have conflicting mount points, even though we will import the old pool as a different name, these mount point still exist from when it was rpool.

To import the zpool we simply provide another pool name for ZSF like this:

root@panacea:~# zpool import rpool oldrpool
cannot import 'rpool': pool may be in use from other system, it was last accessed by panacea (hostid: 0x235a1388) on Wed May 20 13:30:08 2009
uroot@panacea:~# zpool import -f rpool oldrpool
cannot mount '/export': directory is not empty
cannot mount '/export/home': directory is not empty

Note I had to use the “-f” flag when importing because ZSF detects that the rpool was in use by another system. It even includes the hostid of the machine so, if we have a cross reference of hostid’s we can locate the system the pool came out of. Additionally, if this were a single disk of the mirror I’m importing, ZFS would import the pool and let me know that it is degraded and missing a mirror when I do a zpool status. So here’s what the system looks like now:

root@panacea:~# zpool status
pool: oldrpool
state: ONLINE
status: The pool is formatted using an older on-disk format. The pool can
still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'. Once this is done, the
pool will no longer be accessible on older software versions.
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
oldrpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c4t3d0s0 ONLINE 0 0 0
c4t1d0s0 ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c4t2d0s0 ONLINE 0 0 0
c4t0d0s0 ONLINE 0 0 0

errors: No known data errors

pool: xpool
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
xpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c4t2d0s1 ONLINE 0 0 0
c4t0d0s1 ONLINE 0 0 0

errors: No known data errors
root@panacea:~# zpool upgrade oldrpool
This system is currently running ZFS pool version 13.

Successfully upgraded 'oldrpool' from version 10 to version 13

Migration Time
The zpool command shows the pool is now sucessfully imported and it gives additional information that we need to upgrade the version of ZFS on oldrpool. This was painless, but now there is no turning back. These disks will likely no longer boot OpenSolaris 2008.05. At this point the system looks like the diagram here.

But, there are bigger problems, we have namespace collisions all over the ZFS devices. Before we can continue we must eliminate the conflicts. This will involve a number of ZFS properties to be adjusted. Here’s the output of zfs list with the newly imported oldrpool.

root@panacea:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
oldrpool 2.29G 93.7G 55K /oldrpool
oldrpool/ROOT 2.29G 93.7G 18K /oldrpool/ROOT
oldrpool/ROOT/opensolaris 2.29G 93.7G 2.23G legacy
oldrpool/ROOT/opensolaris/opt 3.61M 93.7G 3.61M /opt
oldrpool/export 484K 93.7G 21K /export
oldrpool/export/backups 18K 93.7G 18K /export/backups
oldrpool/export/home 428K 93.7G 410K /export/home
rpool 3.14G 11.6G 72K /rpool
rpool/ROOT 2.39G 11.6G 18K legacy
rpool/ROOT/opensolaris 2.39G 11.6G 2.26G /
rpool/dump 256M 11.6G 256M -
rpool/export 615K 11.6G 21K /export
rpool/export/home 594K 11.6G 19K /export/home
rpool/export/home/mps 575K 11.6G 575K /export/home/mps
rpool/swap 512M 12.1G 37.2M -
xpool 69K 34.2G 18K /xpool

Namespace Collisions and Resolution
Above when I imported the oldrpool (which was rpool in a previous life), ZFS informed us some of the file systems were already mounted. These are mounted automatically when we import because the properties are set to mount. In fact, if the system were to reboot at this point with the oldrpool imported, it would fail to start back up because the mount services will fail. The mount points for /ROOT/opensolaris, /export and /export/home are all in conflict between rpool and oldrpool. Due to these conflicts, the mount services fail and the machine will not boot cleanly. Manual intervention is needed. However, we can fix this now.

There are some items of interest between the 2008.05 and the 2008.11 installation. The file system for /opt is gone in 2008.11 and an additional file system for my user ID was added in 2008.11. I assume the file system for each user is so the user can setup and administer their own snapshots independently of the entire /export/home file system. This would also make sense given the new “Time-Slider” functionality in 2008.11.

The mount point listed as “legacy” is managed by means other than ZFS. This could be in /etc/vfstab or other location. In the case of /ROOT/opensolaris, it is the root file system mounted by the kernel.

We can find out which mount points are actually active using the zfs mount command.

root@panacea:~# zfs mount
rpool/ROOT/opensolaris /
rpool/export /export
rpool/export/home /export/home
rpool/export/home/mps /export/home/mps
rpool /rpool
xpool /xpool
oldrpool/export/backups /export/backups
oldrpool /oldrpool
oldrpool/ROOT /oldrpool/ROOT

By this I can see that the mount points which didn’t conflict with existing mount points are mounted.

In order to get rid of the conflicts and give the ZFS file systems on oldrpool a place to mount we must go change some of the properties. We can get a complete list of the properties set on an individual file system and it’s settings using the command zfs get all shown here.

root@panacea:~# zfs get all oldrpool
NAME PROPERTY VALUE SOURCE
oldrpool type filesystem -
oldrpool creation Wed May 20 10:32 2009 -
oldrpool used 2.29G -
oldrpool available 93.7G -
oldrpool referenced 57K -
oldrpool compressratio 1.00x -
oldrpool mounted yes -
oldrpool quota none default
oldrpool reservation none default
oldrpool recordsize 128K default
oldrpool mountpoint /oldrpool local
oldrpool sharenfs off default
oldrpool checksum on default
oldrpool compression off default
oldrpool atime on default
oldrpool devices on default
oldrpool exec on default
oldrpool setuid on default
oldrpool readonly off default
oldrpool zoned off default
oldrpool snapdir hidden default
oldrpool aclmode groupmask default
oldrpool aclinherit restricted default
oldrpool canmount on default
oldrpool shareiscsi off default
oldrpool xattr on default
oldrpool copies 1 default
oldrpool version 3 -
oldrpool utf8only off -
oldrpool normalization none -
oldrpool casesensitivity sensitive -
oldrpool vscan off default
oldrpool nbmand off default
oldrpool sharesmb off default
oldrpool refquota none default
oldrpool refreservation none default
oldrpool primarycache all default
oldrpool secondarycache all default

The two primary properties we are concerned about are “mountpoint” and “canmount.” On the oldrpool/ROOT/opensolaris the “canmount” property is set to “noauto” which will prevent them from being auto mounted when the pool is imported or we issue a zfs mount -a command. Since we are migrating these over to the new system, there may be files or settings in files we wish to have in the new Solaris install. Examples of this might be items in /etc and possibly crontabs, or items installed in /opt there may be other items depending on your particular system.

So here’s very quickly how to fix the namespace problems and get all the file systems mounted under /oldrpool.

root@panacea:~# zfs mount
rpool/ROOT/opensolaris /
rpool/export /export
rpool/export/home /export/home
rpool/export/home/mps /export/home/mps
rpool /rpool
xpool /xpool
oldrpool/export/backups /export/backups
oldrpool /oldrpool
oldrpool/ROOT /oldrpool/ROOT
root@panacea:~# for fs in `zfs mount | grep oldrpool | sort -r | awk '{print $1}'`
> do
> zfs umount ${fs}
> done
root@panacea:~# zfs mount
rpool/ROOT/opensolaris /
rpool/export /export
rpool/export/home /export/home
rpool/export/home/mps /export/home/mps
rpool /rpool
xpool /xpool
root@panacea:~# for fs in `zfs list | grep oldrpool | awk '{print $1}'`
> do
> zfs set mountpoint=/${fs} ${fs}
> done
root@panacea:~# zfs mount -a
root@panacea:~# zfs mount
rpool/ROOT/opensolaris /
rpool/export /export
rpool/export/home /export/home
rpool/export/home/mps /export/home/mps
rpool /rpool
xpool /xpool
oldrpool /oldrpool
oldrpool/ROOT /oldrpool/ROOT
oldrpool/export /oldrpool/export
oldrpool/export/backups /oldrpool/export/backups
oldrpool/export/home /oldrpool/export/home
root@panacea:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
oldrpool 2.29G 93.7G 55K /oldrpool
oldrpool/ROOT 2.29G 93.7G 18K /oldrpool/ROOT
oldrpool/ROOT/opensolaris 2.29G 93.7G 2.23G /oldrpool/ROOT/opensolaris
oldrpool/ROOT/opensolaris/opt 3.61M 93.7G 3.61M /oldrpool/ROOT/opensolaris/opt
oldrpool/export 484K 93.7G 21K /oldrpool/export
oldrpool/export/backups 18K 93.7G 18K /oldrpool/export/backups
oldrpool/export/home 428K 93.7G 410K /oldrpool/export/home
rpool 3.14G 11.6G 72K /rpool
rpool/ROOT 2.39G 11.6G 18K legacy
rpool/ROOT/opensolaris 2.39G 11.6G 2.26G /
rpool/dump 256M 11.6G 256M -
rpool/export 619K 11.6G 21K /export
rpool/export/home 598K 11.6G 19K /export/home
rpool/export/home/mps 579K 11.6G 579K /export/home/mps
rpool/swap 512M 12.1G 37.2M -
xpool 69K 34.2G 18K /xpool
root@panacea:~# zfs set canmount=on oldrpool/ROOT/opensolaris
root@panacea:~# zfs set canmount=on oldrpool/ROOT/opensolaris/opt
root@panacea:~# zfs mount -a
root@panacea:~# zfs mount
rpool/ROOT/opensolaris /
rpool/export /export
rpool/export/home /export/home
rpool/export/home/mps /export/home/mps
rpool /rpool
xpool /xpool
oldrpool /oldrpool
oldrpool/ROOT /oldrpool/ROOT
oldrpool/export /oldrpool/export
oldrpool/export/backups /oldrpool/export/backups
oldrpool/export/home /oldrpool/export/home
oldrpool/ROOT/opensolaris /oldrpool/ROOT/opensolaris
oldrpool/ROOT/opensolaris/opt /oldrpool/ROOT/opensolaris/opt

There a lot going on here. Let’s review. First, we unmount all the file systems which come from oldrpool. This will avoid mount points being used or other conflicts when we change the mount points and remount them. This is done as a simple for loop. Next I verify that there are no more oldrpool file systems mounted. Then we change all the mount points to be mounted from /oldrpool by the name of the zfs file system. Also accomplished by a simple for loop on the command line.

The zfs set commands reset the “canmount” property from “noauto” to “on.” Previously, these mounts were done as legacy mounts and not handled by ZFS. Once these are set to “on” we can issue another mount -a and see everything is mounted correctly.

Take a Snapshot, Split and Grow...
So, now that everything is in a separate namespace, we need to split the old mirror and add one of its disks to the new pool - xpool.

There are several things which must be done at this point. We need to make a snapshot before using send and receive for transferring the ZFS filesystem. The snapshot will also be useful in case we hose up any of the original data in the imported filesystem as we re-organize it later.

Splitting the mirror which makes up oldrpool will leave us a bit exposed as there will only be one copy of the data. If some of the data or file systems can be destroyed or deleted, including old snapshots to reduce the required migration to less than the available 400GB, then this is ideal. In my actual physical migration, this was fortunately the case.

Also it is at this point where I discovered an issue with be’s (boot environments). For some reason, the new Solaris installation sees the boot environments on the imported oldrpool. This caused me problems in creating snapshots for some reason. I haven’t been able to find any documentation on this issue, but have been able to replicate it easily in a virtual machine. This prevents me from issuing a recursive snapshot on the ROOT on the oldrpool, but I was able to perform the snapshots manually.

The issue with boot environments is illustrated here.

root@panacea:~# beadm list -a
BE/Dataset/Snapshot Active Mountpoint Space Policy Created
------------------- ------ ---------- ----- ------ -------
opensolaris
rpool/ROOT/opensolaris NR / 2.39G static 2009-05-20 14:26
rpool/ROOT/opensolaris@install - - 133.17M static 2009-05-20 15:09
root@panacea:~# zpool import oldrpool
root@panacea:~# beadm list -a
BE/Dataset/Snapshot Active Mountpoint Space Policy Created
------------------- ------ ---------- ----- ------ -------
opensolaris
oldrpool/ROOT/opensolaris N /oldrpool/ROOT/opensolaris 2.29G static 2009-05-20 10:32
oldrpool/ROOT/opensolaris/opt - /oldrpool/ROOT/opensolaris/opt 3.61M static 2009-05-20 10:32
oldrpool/ROOT/opensolaris/opt@install - - 0 static 2009-05-20 11:16
oldrpool/ROOT/opensolaris@install - - 56.88M static 2009-05-20 11:16
rpool/ROOT/opensolaris NR / 2.39G static 2009-05-20 14:26
rpool/ROOT/opensolaris@install - - 133.17M static 2009-05-20 15:09

Since they are both contained under the OpenSolaris be, I’m a bit at a loss as to how to delete it. However, when I did the send/receive, it was gone. This may be some of my own confusion with how ZFS and be’s interact.

So we’ll do snapshots of both pools using zfs snapshot.

root@panacea:~# zfs snapshot -r rpool@migration
root@panacea:~# zfs snapshot -r oldrpool@migration
root@panacea:~# zfs list -t all
NAME USED AVAIL REFER MOUNTPOINT
oldrpool 2.23G 93.7G 57K /oldrpool
oldrpool@migration 0 - 57K -
oldrpool/ROOT 2.23G 93.7G 19K /oldrpool/ROOT
oldrpool/ROOT@migration 0 - 19K -
oldrpool/ROOT/opensolaris 2.23G 93.7G 2.23G /oldrpool/ROOT/opensolaris
oldrpool/ROOT/opensolaris@migration 0 - 2.23G -
oldrpool/ROOT/opensolaris/opt 3.61M 93.7G 3.61M /oldrpool/ROOT/opensolaris/opt
oldrpool/ROOT/opensolaris/opt@migration 0 - 3.61M -
oldrpool/export 450K 93.7G 21K /oldrpool/export
oldrpool/export@migration 0 - 21K -
oldrpool/export/backups 18K 93.7G 18K /oldrpool/export/backups
oldrpool/export/backups@migration 0 - 18K -
oldrpool/export/home 410K 93.7G 410K /oldrpool/export/home
oldrpool/export/home@migration 0 - 410K -
rpool 3.18G 11.6G 72K /rpool
rpool@migration 0 - 72K -
rpool/ROOT 2.39G 11.6G 18K legacy
rpool/ROOT@migration 0 - 18K -
rpool/ROOT/opensolaris 2.39G 11.6G 2.26G /
rpool/ROOT/opensolaris@install 133M - 2.21G -
rpool/ROOT/opensolaris@migration 450K - 2.26G -
rpool/dump 256M 11.6G 256M -
rpool/dump@migration 0 - 256M -
rpool/export 619K 11.6G 21K /export
rpool/export@migration 0 - 21K -
rpool/export/home 598K 11.6G 19K /export/home
rpool/export/home@migration 0 - 19K -
rpool/export/home/mps 579K 11.6G 579K /export/home/mps
rpool/export/home/mps@migration 0 - 579K -
rpool/swap 549M 12.1G 37.2M -
rpool/swap@migration 0 - 37.2M -
xpool 69K 34.2G 18K /xpool

Now to split one of the mirrors off and add it to the new zpool “xpool”. Before I do this, I will scrub the zpool just to make sure there are no problems with either mirror.

root@panacea:~# zpool scrub oldrpool
root@panacea:~# zpool status oldrpool
pool: oldrpool
state: ONLINE
scrub: scrub completed after 0h6m with 0 errors on Thu May 28 22:59:10 2009
config:

NAME STATE READ WRITE CKSUM
oldrpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c4t3d0s0 ONLINE 0 0 0
c4t1d0s0 ONLINE 0 0 0

errors: No known data errors

Detach c4t3d0s0 from oldrpool and repartition it since it once had a swap partition from the original OpenSolaris 2008.05 installation.

From this point on, we are not protected in the event of disk failure, or any other potential data loss because we only have a single disk in the old pool. It would be really nice to add wither a disk to the mirror, or a temporary disk to the new pool for the storage migration. In that case, the new storage pool would have some minimal exposure but our original data would still reside in the original pool and be mirrored. The new pool could easily be reconstructed in the event of any problem.

root@panacea:~# zpool detach oldrpool c4t3d0s0
root@panacea:~# format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
0. c4t0d0 <DEFAULT cyl 6524 alt 2 hd 255 sec 63>
/pci@0,0/pci8086,2829@d/disk@0,0
1. c4t1d0 <DEFAULT cyl 13051 alt 2 hd 255 sec 63>
/pci@0,0/pci8086,2829@d/disk@1,0
2. c4t2d0 <ATA-VBOXHARDDISK-1.0 cyl 6524 alt 2 hd 255 sec 63>
/pci@0,0/pci8086,2829@d/disk@2,0
3. c4t3d0 <DEFAULT cyl 13051 alt 2 hd 255 sec 63>
/pci@0,0/pci8086,2829@d/disk@3,0
Specify disk (enter its number): 3
selecting c4t3d0
[disk formatted]

FORMAT MENU:
disk - select a disk
type - select (define) a disk type
partition - select (define) a partition table
current - describe the current disk
format - format and analyze the disk
fdisk - run the fdisk program
repair - repair a defective sector
label - write label to the disk
analyze - surface analysis
defect - defect list management
backup - search for backup labels
verify - read and display labels
save - save new disk/partition definitions
inquiry - show vendor, product and revision
volname - set 8-character volume name
!<cmd> - execute <cmd>, then return
quit
format> pa

PARTITION MENU:
0 - change `0' partition
1 - change `1' partition
2 - change `2' partition
3 - change `3' partition
4 - change `4' partition
5 - change `5' partition
6 - change `6' partition
7 - change `7' partition
select - select a predefined table
modify - modify a predefined partition table
name - name the current table
print - display the current table
label - write partition map and label to the disk
!<cmd> - execute <cmd>, then return
quit
partition> pr
Current partition table (original):
Total disk cylinders available: 13051 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks
0 root wm 262 - 13050 97.97GB (12789/0/0) 205455285
1 swap wu 1 - 261 2.00GB (261/0/0) 4192965
2 backup wu 0 - 13050 99.98GB (13051/0/0) 209664315
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 0 0 (0/0/0) 0
6 unassigned wm 0 0 (0/0/0) 0
7 unassigned wm 0 0 (0/0/0) 0
8 boot wu 0 - 0 7.84MB (1/0/0) 16065
9 unassigned wm 0 0 (0/0/0) 0

partition> 0
Part Tag Flag Cylinders Size Blocks
0 root wm 262 - 13050 97.97GB (12789/0/0) 205455285

Enter partition id tag[root]: unassigned
Enter partition permission flags[wm]:
Enter new starting cyl[262]: 0
Enter partition size[205455285b, 12789c, 12788e, 100319.96mb, 97.97gb]: 0
partition> 1
Part Tag Flag Cylinders Size Blocks
1 swap wu 1 - 261 2.00GB (261/0/0) 4192965

Enter partition id tag[swap]: unassigned
Enter partition permission flags[wu]: wm
Enter new starting cyl[1]: 0
Enter partition size[4192965b, 261c, 260e, 2047.35mb, 2.00gb]: 0
partition> label
Ready to label disk, continue? yes

partition> quit

FORMAT MENU:
disk - select a disk
type - select (define) a disk type
partition - select (define) a partition table
current - describe the current disk
format - format and analyze the disk
fdisk - run the fdisk program
repair - repair a defective sector
label - write label to the disk
analyze - surface analysis
defect - defect list management
backup - search for backup labels
verify - read and display labels
save - save new disk/partition definitions
inquiry - show vendor, product and revision
volname - set 8-character volume name
!<cmd> - execute <cmd>, then return
quit
format> quit

Next we will add the entire disk to “xpool”. This will append the disk to the pool extending it in size. This must be forced since xpool is mirrored and we are just appending a single disk to the pool. Remember, this is done using the zfs add command which is different from zfs attach.

root@panacea:~# zpool add -f xpool c4t3d0
root@panacea:~# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
oldrpool 97.5G 2.23G 95.3G 2% ONLINE -
rpool 15G 2.69G 12.3G 17% ONLINE -
xpool 134G 76.5K 134G 0% ONLINE -

With the disk added, we can now see it has grown by 100G, the size of oldrpool and the bit of swap space that was on the original disk.

Moving The Data With send/receive
Now that we have the new enlarged zpool, we can move the ZFS file systems from the oldrpool to the new. This is easily done using zfs send and receive. It works on the snapshot level, and is good for moving data not just across pools, but can move data across the network through a pipe and ssh, and even perform differential updates between any two snapshots as an incremental. This ability to do differential/incremental snapshot replication makes it an ideal tool for backups or replicating a machine for some basic level of disaster recovery.

Since we are moving an already existing pool and it’s file systems, that means the metadata will also be moved as well. This means the properties which control mounting the original pool will be transferred to the new file systems on the receiving end. Since they are mounted, receive will try to mount them here and fail. So, first all of oldrpool’s file systems must be unmounted.

root@panacea:~# for fs in `zfs list | grep oldrpool | awk '{print $1}' | sort -r`
> do
> zfs umount ${fs}
> done
root@panacea:~# zfs create xpool/oldpanacea
root@panacea:~# zfs send -R oldrpool@migration | zfs receive -F -d xpool/oldpanacea

After unmounting, I create a new file system to receive the base pool.

The send and receive is simple, the only thing here is that when dealing with an entire pool it must be forced into a new location.

Final Steps, Fixing The Mirror And Cleaning Up Mount Points
In the final step we attach the remaining oldrpool disk we just migrated data from using send and receive. The first thing to do is destroy the oldrpool and repartition it like the disk we just added. Instead of using format, we can use the shortcut with fmthard. After that, nothing more than attaching. ZFS will handle the rest and resilver the mirror.

root@panacea:~# zpool destroy oldrpool
root@panacea:~# prtvtoc /dev/dsk/c4t3d0s2 | fmthard -s - /dev/rdsk/c4t1d0s2
root@panacea:~# zpool attach xpool c4t3d0 c4t1d0
root@panacea:~# zpool status
pool: rpool
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c4t2d0s0 ONLINE 0 0 0
c4t0d0s0 ONLINE 0 0 0

errors: No known data errors

pool: xpool
state: ONLINE
scrub: resilver completed after 0h1m with 0 errors on Fri May 29 02:29:03 2009
config:

NAME STATE READ WRITE CKSUM
xpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c4t2d0s1 ONLINE 0 0 0
c4t0d0s1 ONLINE 0 0 0
mirror ONLINE 0 0 0
c4t3d0 ONLINE 0 0 0 18.5M resilvered
c4t1d0 ONLINE 0 0 0 553M resilvered

errors: No known data errors

The last thing to do will be fix the mounts because everything is still mounted from the /oldrpool tree.

root@panacea:~# for fs in `zfs list | grep oldpanacea | awk '{print $1}' | sort -r `
> do
> zfs umount ${fs}
> done
root@panacea:~# zfs mount
rpool/ROOT/opensolaris /
rpool/export /export
rpool/export/home /export/home
rpool/export/home/mps /export/home/mps
rpool /rpool
xpool /xpool
root@panacea:~# for fs in `zfs list | grep oldpanacea | awk '{print $1}'`
> do
> zfs set mountpoint=/${fs} ${fs}
> done
root@panacea:~# zfs mount -a
root@panacea:~# zfs mount
rpool/ROOT/opensolaris /
rpool/export /export
rpool/export/home /export/home
rpool/export/home/mps /export/home/mps
rpool /rpool
xpool /xpool
xpool/oldpanacea /xpool/oldpanacea
xpool/oldpanacea/ROOT /xpool/oldpanacea/ROOT
xpool/oldpanacea/ROOT/opensolaris /xpool/oldpanacea/ROOT/opensolaris
xpool/oldpanacea/ROOT/opensolaris/opt /xpool/oldpanacea/ROOT/opensolaris/opt
xpool/oldpanacea/export /xpool/oldpanacea/export
xpool/oldpanacea/export/backups /xpool/oldpanacea/export/backups
xpool/oldpanacea/export/home /xpool/oldpanacea/export/home

We’re now done migrating the data from an existing rpool to a new pool called xpool. The disks now look like this:

Additional storage can now be added to the pool xpool as we fill up the pool.

If I were in a datacenter with plenty of disks and SAN to burn, I would have done this an entirely different way. I would not have split up the new OS disks into a smaller rpool, but I’m working on this as a home server for media and backups. The migration from an existing rpool into a system which already has an rpool is extremely relevant to any type of situation where perhaps there is data on the rpool of a system being upgraded and it needs to be moved into a new system.

There are better ways of organizing file systems where rpool is pretty much dedicated to all things with the base OS installation. However, this is the subject of another entry coming soon.

Summary
TBD.