Place Ghantoos

I just upgraded my server from FreeBSD 7.2-RELEASE to 8.0-RELEASE.

The upgrade steps are very clearly stated in the FreeBSD 8.0-RELEASE Announcement.

Everything went as planned, until the penultimate step, where I was supposed to follow Colin Percival’s instructions and rebuild all my ports before executing the final: freebsd-update install.

Instead of rebuilding my ports, I just executed the last step (# freebsd-update install), which rendered *all* of my third party application unusable. This of course includes bash(1)! I was now unable to start a new bash session, hence unable to start new ssh connections, as my user’s default shell is /usr/bin/local/bash.

As the title says, I should have had something like /bin/sh (or /bin/csh) as my root’s shell (so you don’t rely on your ports); but no, it was set to /usr/local/bin/bash. I still remember iMil telling me that root should *always* have a shell that’s part of the base system. He was clearly right, again.

I found this thread, that did not help me sort out my problem (but could may be give ideas).

So here I was left, with 2 SSH connections on my server, unable to login as root because bash(1) was missing libraries!

The goal now, was to get root access, to reinstall portupgrade, in order to reinstall all my ports.

Here are some of the things I tried to do (I am pasting some of the errors here as this could help someone who searching these strings using his/her favorite search engine):

heliopolis:~$ sudo -s
/libexec/ld-elf.so.1: Shared object "libutil.so.7" not found, required by "sudo"
heliopolis:~$ screen  -ls
/libexec/ld-elf.so.1: Shared object "libncurses.so.7" not found, required by "screen"
heliopolis:~$ bash
/libexec/ld-elf.so.1: Shared object "libncurses.so.7" not found, required by "bash"
heliopolis:~$ su -
Password:
/libexec/ld-elf.so.1: Shared object "libncurses.so.7" not found, required by "bash"

At some point, I even tried the FreeBSD local r00t zeroday without success.

Then, talking on IRC with iMil, he asked if I tried changing my shell using chsh. !@#!@# I had no idea a user could change his shell! Everything got clearer in a fraction of second.
I had to change my user’s shell to /bin/sh then su(1) using the “-m” in order preserve the current environment, hence using the current user’s shell information: /bin/sh.

heliopolis:~$ chsh -s /bin/sh
Password:
chsh: user information updated
heliopolis:~$ su -m
Password:
heliopolis:/usr/home/ghantoos# whoami
root
heliopolis:/usr/home/ghantoos# 

Once root access was recovered, and before rebooting the machine, I reinstalled (the order is important): lang/perl, lang/ruby18, databases/ruby-bdb, ports-mgmt/portupgrade and sysutils/screen. I could now execute the step I missed, that resulted in all this stress:

# portupgrade -afcP

Once this was done, I rebooted my machine:

heliopolis:~$ uname -a
FreeBSD heliopolis.ghantoos.org 8.0-RELEASE FreeBSD 8.0-RELEASE #0: Sat Nov 21 15:48:17 UTC 2009
root@almeida.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  i386

La morale de l’histoire est la suivante:
1- NEVER LET YOUR ROOT’S SHELL USE A PORT INSTALLED SHELL, USE /BINSH OR /BIN/CSH!
2- RTFM CAREFULLY BEFORE EXECUTING STUFF (especially when you don’t have physical access to the machine).

I hope this will help someone someday.

Cheers,
Ignace M -ghantoos-