Building an OpenIndiana based ZFS File Server – part 2

OpenIndiana Installation

Getting OpenIndiana onto our file-server hardware was a pretty simple affair, download the memory stick image and dd it onto a fresh memory stick and install … actually we did struggle with this to start with – we were planning on testing it on an old PC in my office (an 2007 generation 965 based Core2 Duo PC – can’t believe these are machines coming out of desktop service!)

Basically follow the text based installer – its quite reminiscent of pre-Jumpstart installs! We didn’t enable any additional users as we were planning on integrating with our Active Directory for authentication. If you do, then the installer will disable root logins by default.

The installer creates a default ZFS rpool for the root file-system, as we want that to be a mirror, we needed to add a second device once it had booted. This blog entry has some instructions on this. As its been a while since I’ve done Solaris, I’d actually approached it via a different manner (used format and fdisk to manipulate the disk partitions by hand, but prtvtoc | fmthard is of course the way we used to do it with Solstace DiskSuite, and is quicker and easier.

If we hadn’t already installed Linux onto the disk to test initially, we probably wouldn’t have got the error

cannot label 'c3t1d0': EFI labeled devices are not supported
    on root pools

when trying to add the disk directly when running

zpool attach rpool c3t0d0s0 c3t1d0s0

Clearing and reinitialising the partition table resolved that problem though. Don’t forget to use the installgrub command on the second disk to install the boot loader onto the disk, otherwise you won’t actually be able to boot off it in the event of a primary disk failure!

installgrub -m /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c3t1d0s0

Ensure that the disks have finished resilvering before you reboot the system! (zpool status will tell you!).

The resilver process in ZFS is different to those familiar with mirrors in DiskSuite or linux software RAID as it doesn’t create an exact block for block mirror of the disks, it only copies data blocks, so the data layout on the disks is likely to be different. It also means that if you have a 100GB drive with only 8GB data, you only need to mirror the 8GB of data, not all the empty file-system blocks as well …

Updating the OS

Since OpenIndiana 151a was released, there’s a bunch of updates been made available, so you want to upgrade to the latest image. First off, update the pkg package:

pkg install package/pkg

You can then do a trial run image update:

pkg image-update -nv

Run it again without the “n” flag to actually do the update. This will create a new boot environment, you can list using the command:

root@bigstore:~# beadm list
BE            Active Mountpoint Space Policy Created
openindiana   -      -          8.17M static 2012-05-18 15:11
openindiana-1 NR     /          3.95G static 2012-05-18 15:54
openindiana-2 -      -          98.0K static 2012-05-24 11:58

That’s it, you’re installed and all up to date!

In addition to the base Operating System repositories, we also add some extra repos:

pkg set-publisher -p http://pkg.openindiana.org/sfe
pkg set-publisher -p http://pkg.openindiana.org/sfe-encumbered

Enabling Serial Console

All of our servers are connected to Cyclades serial console servers, this lets us connect to the serial ports via ssh from remote locations, which is great when bits of the system go down. To enable to serial console, you need to edit the grub config file. As we have ZFS pool zpool, its located at /rpool/boot/grub/menu.lst. You need to comment out the splashimage line if present (as you can’t display the XPM down a serial port!), then add a couple more lines to enable serial access for grub.

 ###splashimage /boot/grub/splash.xpm.gz
 serial --unit=0 --speed=9600
 terminal --timeout=5 serial

You also need to find the kernel line and append some data to that:

-B console=ttya,ttya-mode="9600,8,n,1,-"

As there was already a -B flag in use, our resulting kernel line looked like this:

kernel$ /platform/i86pc/kernel/$ISADIR/unix -B $ZFS-BOOTFS,console=ttya,ttya-mode="9600,8,n,1,-"

You might want to tweak the line-speed, parity etc for your environment.

Static IP address

For our servers, we prefer to use static IP addressing rather than have then DHCP. As we selected auto-configure during the installer, we need to swap to static IP address. Note that if you don’t have access (serial console or real console, you’ll likely get disconnected)!

First off, we need to disable the DHCP client. OpenIndiana dhcpclient is part of the nwam service:

svcadm disable network/physical:nwam

Boom! Down goes the network connection. So make sure you have access via another method! I’ve been adminning Solaris for a long time (since 2001), so I’m going to configure networking in the traditional (pre Solaris 10) manner … with some files, but you can do this step with ipadm command.

/etc/hostname.ixgbe0
    bigstore
/etc/hosts
    147.188.203.45 bigstore bigstore.cs.bham.ac.uk bigstore.local
    ::1 bigstore bigstore.local localhost loghost
/etc/defaultrouter
    147.188.203.6
/etc/netmasks
    147.188.0.0     255.255.255.0
/etc/resolv.conf
    domain  cs.bham.ac.uk
    search cs.bham.ac.uk
    nameserver  147.188.192.4
    nameserver  147.188.192.8
cp /etc/nsswitch.dns /etc/nsswitch.conf

Finally we need to enable the static IP configuration service:

svcadm enable network/physical:default

part 1 – Hardware and Basic information
part 2 – Base Operating System Installation
part 3 – Configuring Active Directory authentication integration
part 4 – Configuring the storage pool and auto snapshots
part 5 – NFS Server config & In-kernel CIFS

Comments are closed.