Building an OpenIndiana based ZFS File Server

We’ve recently been looking at options to replace one of our ageing file-servers which stores research data. The data is currently sat on an out-of-warranty Infortrend RAID array which we purchased back in July 2005. Basically its got 3TB of storage, so was a reasonable size when we bought it. Its attached to a Sun V20z box running Linux, serving data via NFS and Samba.

Our home directory storage is hung off a NetApp FAS3140 filer head, but we just couldn’t afford to purchase additional storage on there for the huge volumes of research data people want to keep.

So we looked around. Spoke to some people. Then came up with a plan. We’ve decided to home-brew a file-server based on ZFS as the underlying file system. ZFS was developed by Sun for Solaris and is designed to cope with huge file-systems, and provides a number of key features for modern file-systems, including snapshotting, deduplication, cloning and management of disk storage in pools. You can create and destroy file systems on the fly within a ZFS pool (a file-system looks like a directory in userland, you just create it with a simple zfs create command).

Since OpenSolaris, there’ve been ports of ZFS into other operating systems, FreeBSD has an implementation, and there’s event ZFS on Linux.

Hardware spec

As we were pondering an Illumos based OS for the system, we wanted kit we’d be fairly happy was supported. There’s no real compatibility list for Illumos, but digging around we found some stuff we’d be fairly happy would work.

One pair of 40GB SSDs is for the Operating System mirrored pair, one pair of mirrored SSDs for the ZFS intent log (ZIL), and the 2 160GB SSDs are to provide an L2ARC cache.

Once the kit arrived and was racked up, a bit of testing was done, but I was out of the office for a few weeks so didn’t inspect it properly until later. In short, the External SAS module (based on LSI 1604e chipset) was causing us some problems – the 3TB drives were being truncated to 2TB. After several calls to Viglen technical support, it transpires that the external SAS module doesn’t support 3TB drives. They’ve agreed to ship out a new LSI 9205-8e HBA. We believe this should work fine with OpenIndiana.

Intially we were offered a MegaRAID card, but there’s no mention of a Solaris 10 driver on the LSI site, so we steered clear of it.

And for an operating system?

We started off with one of our standard Scientific Linux 6 server installs to test out the hardware. We also built up ZFS on Linux to try it out. It works great (and we later even imported a pool created under Linux into a SmartOS instance). Our main issues with this approach are:

  • We have to build ZFS for the kernel, and kernel updates would break our ZFS access – we’d have to rebuild for each update.
  • ZFS on Linux with the POSIX layer isn’t as mature as ZFS on Illumos
  • We’d have to use Samba to share to our Windows clients
  • ZFS snapshots are inaccessible from NFS clients

We were always planning on looking at an Illumos based build (we did consider Oracle Solaris on non-Oracle hardware, but the cost is quite high). So we looked around at our options.

SmartOS

Hey wouldn’t it be neat if we could run SmartOS on it – this an Illumos based build which boots of a memory stick, and then you run KVM/Zones virtual machines inside to provide services. The outer SmartOS is transient, but the inner VMs are persistent on the ZFS pool. The big problem with this though is that you can only run an NFS server in the global zone as its tightly hooked into the kernel. This basically scuppers the plan to use SmartOS as our option here.

OpenIndiana

There’s numerous other Illumos based distributions about, but OpenIndiana is probably the closest to how OpenSolaris was when it was “closed”, and possibly the most stable of the distributions.

Installing OI is easy from USB memory stick or CD. Getting it fully integrated into our system was another matter! We had a bunch of issues with Kerberos authentication (we authenticate Linux against our AD), autofs and our AD/LDAP maps, and then getting the in-kernel CIFS service working as we have a disjoint DNS namespace (our Windows DNS name is different from our normal working cs.bham.ac.uk) – it broke the Kerberos authentication we’d already got working!

I’ll post some more details on getting things working in another article!

part 1 – Hardware and Basic information
part 2 – Base Operating System Installation
part 3 – Configuring Active Directory authentication integration
part 4 – Configuring the storage pool and auto snapshots
part 5 – NFS Server config & In-kernel CIFS

2 thoughts on “Building an OpenIndiana based ZFS File Server

    • Hi Chris,

      Not in anger – we’re partly restricted by what our vendor will supply with a 5 year warranty. They’re also more expensive that the 320s we bought, price being another big factor in what we were able to purchase.

      Though yes, they are SLC so would probably be better.

      http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#Separate_Log_Devices

      Has some bits on ZIL sizing … “The maximum size of a log device should be approximately 1/2 the size of physical memory because that is the maximum amount of potential in-play data that can be stored. For example, if a system has 16 GB of physical memory, consider a maximum log device size of 8 GB”

      We’ve only got 12GB of RAM in the system, and the 710s start at 100GB …

      Simon