Re-mounting a broken NFS share on AIX

Recently I was called by a customer who was having trouble mounting an NFS share after a network outage. They forced unmounted the directory however couldn’t remount it because the NFS server had already lost communication with the client and still thinks the share is mounted remotely.

They performed the following:

umount /share
umount: 16 error while unmounting nfsserver:/share - Device busy

umount -f /share
forced unmount of /share

mount /share
mount: giving up on:
        nfsserver:/share
vmount: Connection timed out
NFS fsinfo failed for server nfsshare: error 3 (RPC: 1832-006 Unable to send)

We getting this error because if we look at the output of “showmount -a”, the share is still listed as mounted by the client.

showmount -a nfsserver

Now usually one would just restart the NFS service on the server and call it a day.

stopsrc -g nfs
startsrc -g nfs

However in this instance the NFS shares were still active to many other clients, and it was critical to production systems. This is how to fix it without taking an outage to the NFS service.

On the Client, unmount the broken nfs share if you haven’t already done it.

umount -f /share

On the NFS server, confirm nfs sessions that haven’t been released.

showmount -a localhost

Edit and remove offending entries in /etc/rmtab

Send a hangup signal to rpm.mountd process.

kill -HUP $(ps -ef | grep rpc.mountd | grep -v grep |  awk '{print $2}')

On the Client, remount the share.

mount /share

Rpmplus 0.4 Release

Rpmplus 0.4 for AIX now supports package bundles.

A package bundle is a list of independent packages that is grouped together for install. It is useful for conveniently installing commonly used packages, such as your favourite tools after a fresh AIX install.

Another example is installing prerequisite packages for software applications not packaged in the repositories. Note that the bundle doesn’t install the application software itself, just all the prerequisite RPM packages it requires.

Rpmplus uses a bundle file to store the definitions. The format can be seen below:

 cat /repos/oss4aix/bundles
lpar2rrd: httpd expat perl-TimeDate perl-XML-Simple rrdtool freetype2 libart_lgpl libpng zlib
lpar2rrd-nohttpd: perl-TimeDate perl-XML-Simple rrdtool freetype2 libart_lgpl libpng zlib

At the moment the list is pretty small but in the long run, and hopefully with some collaboration it will grow to be a more comprehensive one.

To get the most up-to-date bundles file (Note. there is nothing stopping you from maintaining your own private file).

 wget https://sourceforge.net/projects/rpmplus/files/bundles/download -O /repos/oss4aix/bundles

We have to define the bundle file as the $RPMPLUSBUNDLE environment variable. You can put it in root .profile.

# grep RPM .profile
export RPMPLUSREPO=/repos/oss4aix/aix71/current/.snapshot/220414
export RPMPLUSBUNDLE=/repos/oss4aix/bundles
# echo $RPMPLUSBUNDLE
/repos/oss4aix/bundles

Below is an example of installing the lpar2rrd bundle on a clean install of AIX. As you can see Rpmplus takes care of not only installing the packages in the bundle definition, but also all of the package dependencies.

# rpmplus -rBUvh lpar2rrd
Group "apache" does not exist.
User "apache" does not exist.
httpd                       ##################################################
expat                       ##################################################
perl-TimeDate               ##################################################
perl-XML-Simple             ##################################################
rrdtool                     ##################################################
freetype2                   ##################################################
libart_lgpl                 ##################################################
libpng                      ##################################################
zlib                        ##################################################
apr                         ##################################################
apr-util                    ##################################################
apr-util-ldap               ##################################################
db4                         ##################################################
warning: /var/ssl/openssl.cnf saved as /var/ssl/openssl.cnf.rpmorig
openssl                     ##################################################
pcre                        ##################################################
perl                        ##################################################
perl-XML-Parser             ##################################################
libxcb                      ##################################################
lzo                         ##################################################
pixman                      ##################################################
cairo                       ##################################################
gettext                     ##################################################
libgcc                      ##################################################
libffi                      ##################################################
libiconv                    ##################################################
glib2                       ##################################################
libdbi                      ##################################################
libxml2                     ##################################################
pango                       ##################################################
dejavu-sans-mono-fonts      ##################################################
dejavu-lgc-sans-mono-fonts  ##################################################
bzip2                       ##################################################
bash                        ##################################################
openldap                    ##################################################
readline                    ##################################################
gdbm                        ##################################################
perl-URI                    ##################################################
fontconfig                  ##################################################
librsvg2                    ##################################################
libXrender                  ##################################################
xz-libs                     ##################################################
libthai                     ##################################################
warning: /opt/freeware/info/dir created as /opt/freeware/info/dir.rpmnew
info                        ##################################################
Please check that /etc/info-dir does exist.
You might have to rename it from /etc/info-dir.rpmsave to /etc/info-dir.
atk                         ##################################################
libjpeg                     ##################################################
jbigkit-libs                ##################################################
libtiff                     ##################################################
jasper                      ##################################################
gtk2                        ##################################################
libcroco                    ##################################################
libdatrie                   ##################################################

Note. You have to go to the Lpar2rrd website for instructions on configuring Lpar2rrd.

Feel free to send me any bundle definitions to add to the master bundle file.

FreeBSD Root on ZFS: Handling Disk Failure

Recently I had a failed disk in a mirrored Root on ZFS configuration which I have been running since FreeBSD 8.0.

Here is a quick run through of the steps to replace a disk, which is not covered in the link above.

# zpool list
NAME       SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
zroot      692G  17.3G   675G     2%  1.00x  DEGRADED  -
zstorage  3.62T  3.03T   609G    83%  1.00x  ONLINE  -
# zpool status zroot
  pool: zroot
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: resilvered 419K in 0h0m with 0 errors on Fri Feb 28 21:01:25 2014
config:

        NAME                     STATE     READ WRITE CKSUM
        zroot                    DEGRADED     0     0     0
          mirror-0               DEGRADED     0     0     0
            gpt/disk0            ONLINE       0     0     0
            9549061384354502670  UNAVAIL      0     0     0  was /dev/gpt/disk1

errors: No known data errors

FreeBSD does a wonderful job of numbering disk drives based on the slot number it is plugged into. Therefor if you plug a drive into sata1 on your motherboard it will always be /dev/ada1. I prefer this over other approaches, because I physically label the SATA cable and disk with a permanent marker to match the SATA port number on the motherboard. I also GPT labeled the drive during partitioning and reference the label when I created the pool, so there is never confusion identifying and pulling out a failed disk.

The replacement disk as a result is also ada1 (aka /dev/gpt/disk1 after labeling).

Create a GPT disk.

# gpart create -s gpt ada1
ada1 created
# gpart list ada1
Geom name: ada1
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 1465147021
first: 34
entries: 128
scheme: GPT
Consumers:
1. Name: ada1
   Mediasize: 750155292160 (698G)
   Sectorsize: 512
   Mode: r0w0e0

There is actually a little more work with prepping root pool disks over data pool disks. This is because we need a boot and swap partition. If I was replacing a storage disk I would just allocate the whole drive as a single device.

Define the partition scheme.

# gpart add -b 34 -s 64K -t freebsd-boot ada1
ada1p1 added
gpart add -s 4G -t freebsd-swap -l swap1 ada1
ada1p2 added
gpart add -t freebsd-zfs -l disk1 ada1
ada1p3 added
# ls /dev/gpt
disk0   disk1   swap0   swap1

Replace the failed disk.

# zpool replace zroot /dev/gpt/disk1
Make sure to wait until resilver is done before rebooting.

If you boot from pool 'zroot', you may need to update
boot code on newly attached disk '/dev/gpt/disk1'.

Assuming you use GPT partitioning and 'da0' is your new boot disk
you may use the following command:

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

Write boot code to the new disk boot partition.

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
bootcode written to ada1

Grab some popcorn and watch the re-silver finish, the great thing about ZFS is it will only re-silver what is required. Last year when I shipped my server from Australia to NZ, one of the sata cables came undone. When I booted into FreeBSD the array showed as degraded, but after re-seating the cable the system came back up and re-silvered only a few KB of out-of-sync data. With MD raid on Linux, unless you using write-intent bitmap, which will take a performance hit, you have to re-sync the whole array.

root@blueprint:/dev/gpt # zpool status
pool: zroot
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Thu Mar 27 15:50:51 2014
9.82G scanned out of 17.3G at 15.3M/s, 0h8m to go
9.82G resilvered, 56.70% done
config:

NAME STATE READ WRITE CKSUM
zroot DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
gpt/disk0 ONLINE 0 0 0
replacing-1 UNAVAIL 0 0 0
9549061384354502670 UNAVAIL 0 0 0 was /dev/gpt/disk1/old
gpt/disk1 ONLINE 0 0 905 (resilvering)

Conclusion

So this has got to be one of the worlds most boring blog posts: rebuilding an array. However what I can report is ZFS seems to be pretty reliable on FreeBSD on two occasions for me in the last 5 years. The interesting thing is, I put the failed disk into a Windows machine and formatted it and the disk was completely usable. Thinking ZFS misdiagnosed the failure, I manually failed the new disk and put the old one back and repeated the steps above to rebuild the array. Within minutes ZFS detected once again that the drive was bad and failed it again. This disk is a great candidate for causing silent corruption in older generation file systems.

Managing RPM repositories in AIX

Introduction

AIX has “Linux affinity” for Open Source software, but unfortunately there is no out of the box tools for managing binary package (RPM) repositories, nor is there automatic dependency resolution by the RPM package manager.

This article will guide you through the steps for maintaining RPM software repositories. For package dependency resolution, I would like to introduce to you a rpm wrapper script I wrote, called RPMPLUS. It does automatic dependency resolution, and allows you to easily update from one repository version to another.

Why do I need static repositories?

Simply to keep everything in sync, you wouldn’t want to install some insignificant tool on your server, only to find it has updated package dependencies of core application services.

Whether you use the “AIX Toolbox for Linux Applications” shipped by IBM or download more up-to-date RPMs from www.perzl.org, thanks to the amazing work by Michael Perzl. The usual trend for storing a repository of RPM files is extracting them into individual directories with some kind of date description.

Generally it will look something like this:

baz@nimserver:/repos/software/rpms  $ ls
AIX_toolbox_07112012  AIX_toolbox_21032013 AIX_toolbox_15102013

Essentially this is a crude form of version control and while it works, the problem is there is a lot of wasted space by duplicate files, especially if some packages haven’t had newer versions released over the years. Also you have the administration overhead of manually downloading and extracting each new archive, tagging a new directory and remembering which client LPAR uses which repository. After a while this becomes all too hard, that we end up keeping one repository of RPM packages and every client LPAR uses that version for the remainder of its life. This leads to software never being updated, and as a result a lack of functionality and security that goes with running outdated software.

It would be nice if we can automate repository updates and choose when to tag a revision, then sync the client LPARs to a specific version during a software maintenance window.

JFS2 Snapshots

A former colleague of mine, Chris Gibson, wrote an interesting article on JFS2 snapshots and it got me thinking how I can utilise it for various administration tasks on AIX. By utilising JFS2 snapshots we can easily maintain repository versions, while only storing the changes in files that have been updated.

Read on and I will walk you through the steps to do this.

Overview

The idea is we create a snapshot capable filesystem for each version of AIX in our environment, we call this current and it will contain bleeding edge packages. We then schedule cron to automatically keep this directory up-to-date using the rsync tool to the public OSS4AIX repository online. When it is time to cut a release, we then tag the current tree by taking a snapshot, this now becomes a static version of our repository. Client LPARs can then be configured individually to use different repository versions. For example, you may want to test newer versions of software on a Dev/Test server before pointing your Prod servers to the new repository.

Note. Since IBM doesn’t maintain the “AIX Toolbox for Linux” often enough, I generally always use RPMs from www.perzl.org in my own environment, this is what I’ll be using for this guide.

Initial configuration

It is logical to use your NIM server to host the RPM repositories, but you can use any AIX server in your environment for this task.

First we create snapshot capable filesystems used for the repositories. We create a separate filesystem for each version of AIX in our environment.

# mklv -t jfs2 -y oss4aix71lv nimvg 1
# crfs -v jfs2 -d oss4aix71lv -m /repos/oss4aix/aix71/current -A yes -a isnapshot=yes
# mount /repos/oss4aix/aix71/current
# chfs -a size=2G /repos/oss4aix/aix71/current
#
# mklv -t jfs2 -y oss4aix53lv nimvg 1
# crfs -v jfs2 -d oss4aix53lv -m /repos/oss4aix/aix53/current -A yes -a isnapshot=yes
# mount /repos/oss4aix/aix53/current
# chfs -a size=2G /repos/oss4aix/aix53/current

Next we populate the filesystem with RPM packages from the oss4aix public repository, we only do this once with ftp. I’m using ftp because I’m assuming this is a fresh install of AIX and we don’t have any opensource tools installed on our server yet. At the time of writing you can expect to download over 1GB to get the full repository.

# cd /repos/oss4aix/aix71/current
# ftp -i www.oss4aix.org      # use anonymous login
ftp> bin
ftp> cd /compatible/aix71
ftp> mget *
ftp> quit

Now is a good time to get a coffee…

We only do this for the latest version of AIX (ie. the version we running on our NIM server). To populate the AIX 5.3 directory we can copy most of the common binaries, which can amount to at least a GB of space. If you have created an AIX 6.1 repository, you must repeat the steps below for the “aix61/current” destination directory as well.

# cd /repos/oss4aix/aix71/current
# find . -name '*aix5.1*' -exec cp {} /repos/oss4aix/aix53/current \;
# find . -name '*aix5.2*' -exec cp {} /repos/oss4aix/aix53/current \;

Next we install rsync, which is the tool we use to keep our current directory bleeding edge. This is the only time we need to install a package from current. Future packages are installed from our snapshots.

# rpm -ivh /repos/oss4aix/aix71/current/rsync-[0-9]*
or
# rpmplus -ivh /repos/oss4aix/aix71/current/rsync

Rsync the current repos, we won’t see much change to our AIX 7.1 current directory, but this step will populate the uncommon binaries for the remaining versions of AIX.

# export RSYNC_PASSWORD=public
# cd /repos/oss4aix/aix71/current
# rsync -aPv --delete rsync://public@www.oss4aix.org/public/compatible/aix71/ .
#
# cd /repos/oss4aix/aix53/current
# rsync -aPv --delete rsync://public@www.oss4aix.org/public/compatible/aix53/ .

Now we tag our first static repository.

# snapshot -o snapfrom=/repos/oss4aix/aix71/current -n $(date +"%d%m%y")
# snapshot -o snapfrom=/repos/oss4aix/aix53/current -n $(date +"%d%m%y")

List snapshot

# snapshot -q /repos/oss4aix/aix71/current
Snapshots for /repos/oss4aix/aix71/current
Current Name Time
* 181213 Wed Dec 18 08:42:56 NZDT 2013

Since we have a static repo that will never change, we can now keep the current directory up-to-date so that we may tag a new repository in the future. The following commands can be scheduled in cron, or you can run it manually prior to taking the next snapshot.

export RSYNC_PASSWORD=public
cd /repos/oss4aix/aix71/current && \
rsync -aPv --delete rsync://public@www.oss4aix.org/public/compatible/aix71/ .
cd /repos/oss4aix/aix53/current && \
rsync -aPv --delete rsync://public@www.oss4aix.org/public/compatible/aix53/ .

If you no longer need a repository, we just delete the snapshot.

# snapshot -q /repos/oss4aix/aix71/current
# snapshot -d -n 181213 /repos/oss4aix/aix71/current

Before I configure my client LPARs to use the new repository we have created. I configure the parent directory to share via NFS. As these are just binary packages I share it as a public directory with read-only access.

# print "/repos/oss4aix/aix71/current -vers=3,public,sec=sys,ro" >> /etc/exports
# print "/repos/oss4aix/aix53/current -vers=3,public,sec=sys,ro" >> /etc/exports
# exportfs -va

Working with repositories

Now that I have a repository in place I can configure my client LPARs to use it. If you have DSH configured you can use this to configure all your clients, just remember to single quote any commands below to prevent variable or command expansion by the NIM server’s local shell.

On each client machine, we configure NFS mounts to our NIM server, you likely would want to use autofs in your environment.

client # mkdir -p /repos/oss4aix/aix$(oslevel -r | cut -c 1-2)/current
client # mount nimserver:/repos/oss4aix/aix$(oslevel -r | cut -c 1-2)/current /repos/oss4aix/aix$(oslevel -r | cut -c 1-2)/current

Configure RPMPLUSREPO environment variable for the root user.

client # print "export RPMPLUSREPO=/repos/oss4aix/aix$(oslevel -r | cut -c 1-2)/current/.snapshot/181213" >> $HOME/.profile

DSH example of the command above, with correct quoting.

dshserver # dsh 'print "export RPMPLUSREPO=/repos/oss4aix/aix$(oslevel -r | cut -c 1-2)/current/.snapshot/181213" >> $HOME/.profile'

Once you refresh your .profile by logging back in. Check the environment variable is correct.

client # print $RPMPLUSREPO
/repos/oss4aix/aix1/current/.snapshot/181213

You can now use rpmplus with the -r option to install packages.

client # rpmplus -rUvh wget

To sync all packages installed on a client with a later repository. Simply update $RPMPLUSREPO with the newer snapshot number.

client # perl -p -i -e "s/181213/270514/g" $HOME/.profile

Rpmplus is designed to strip the version numbers from the output of “rpm -qa” and will update each package to a later version if available.

client # rpmplus -rUvh $(rpm -qa)

Summary

The initial configuration may seem like significantly more work than just extracting files the old way. However once this is done, the daily administration components can easily be scripted. In the long run I hope that this will make managing open source software much easier on AIX.