instant root shell to change that pesky forgotten root password via grub

forget knoppix. just do it natively. oh, this only works if grub's not passworded. usually isn't.

* press esc at the grub prompt
* highlight the line that begins with kernel XXX , press e
* go to the very end of the line, add rw init=/bin/bash
note:  that is if bash is in /bin ; if not try /usr/bin or /usr/local/bin
* press enter, then press b to boot system
note:  system should now boot as root shell sans password.
* type passwd username.  set password.
* type reboot

using ssh to restrict commands by key

as we all know, if you have ssh access to a box, you can run arbitrary commands via one liners using the -C switch. neat. or not.
but! you can also have the sysadminly joy of restricting what an ssh client connection can do if you do not want said connection to have unfettered shell access.

anyway. here was the probem:
one day i was getting sort of tired of renaming pems and moving them around between systems. believe me, it was a drag. i wanted a one-liner, so i made one.

i plopped an ssh key for the robot account (robot@mordor) via which i wanted to do all these mundane tasks and then prepended its definition in authorized_keys2 on the client box with the a specific command string.

here's a snippet of authorized_keys2:

command="/usr/local/bin/secretcommands $SSH_ORIGINAL_COMMAND" ssh-rsa == robot@mordor

what this does is restrict robot@mordor to only be able to issue that specific command. no interactive shell, no nothing, just the command. but. in my case, i really needed that robot to be able to do more than one thing. so! i made my command a menu of choices - secretcommands. now, robot@mordor can do only what is on the menu of choices - like look at messages or rename things.

to allow robot@mordor to scp things, i needed to also explicitly place the client system's scp command in the same authorized_keys2 file:

command="/root/bin/scp-wrapper" ssh-rsa == robot@mordor

here's secretcommands:

#!/bin/sh
; secretcommands
case "$1" in
 messages)
  /bin/cat /var/log/messages
  ;;
 delicious)
  /usr/local/bin/rename.sh /etc/secretplaceforsecretthings/crl.pem
  ;;
 help)
  echo 'messages and delicious' 1>&2
  exit 1
 ;;
 *)
  echo 'please do not do that' 1>&2
  exit 1
  ;;
esac

here's the scp command per the snail book:

#!/usr/bin/env perl
# from http://www.snailbook.com/faq/restricted-scp.auto.html

# location of the server-side scp we want to run
$scp_server = "/usr/bin/scp";

sub fail {
    my ($msg) = @_;
    print STDERR "$0: ", $msg, "\n";
    exit 1;
}

# This just makes me feel better.

$TRUE  = (0 == 0);
$FALSE = (0 == 1);

# Since this script is called as a forced command, need to get the
# original scp command given by the client.

($command = $ENV{SSH_ORIGINAL_COMMAND})
    || fail "environment variable SSH_ORIGINAL_COMMAND not set";

# Split the command string to make an argument list, and remove the first
# element (the command name; we'll supply our own);

@scp_argv = split /[ \t]+/, $command;

# Complain if the command is not "scp".

fail "account restricted: only scp allowed (\"$scp_argv[0]\")"
    unless $scp_argv[0] eq "scp";

# Wipe the environment as a security precaution.  This might conceivably
# break something, but if it does you can filter the environment more
# selectively here.

%ENV = ();

# Ensure that either -t or -f is on the command line, to enforce running
# scp in server mode.

$ok = $FALSE;
foreach $arg (@scp_argv) {
    if ($arg eq '-t' || $arg eq '-f') {
        $ok = $TRUE;
        last;
    }
}

fail "Restricted; only server mode allowed."
    unless $ok;

# if we're OK, run our desired "scp" with arguments.

shift(@scp_argv);
exec($scp_server, @scp_argv);

the rename script in secretcommands:

#!/bin/sh
# rename.sh
for file in ${@}; do
    if [ -e $file ]; then
        timestamp=`ls -l --time-style=+%Y%M%d-%k%M%S $file |cut -d' ' -f6`
        basename=`echo $file|cut -d'.' -f1`
        ext=`echo $file | cut  -d'.' -f2-`
        mv -v $file $basename.$ext.$timestamp
    fi
done

here's what it would look like in a one liner:

mordor$ /usr/bin/ssh -i /home/robot/.ssh/id_rsa_nothere.secretcommand -l root nothere delicious

delicious.

multiple pub keys

i usually have a myriad of keys for different automated processes. i really like backing up stuff using unattended rsync scripts; and since you can wrap rsync in ssh, having multiple keys is a good idea - plus the security aspect is cool, too.

my nomenclature for keys is the following:
id_typeofencryption_clientboxname.application

here's how to create an additional key and place it on the target (client) box:

# ssh-keygen -t rsa -b 2048 -f id_rsa_client.rsync
# ssh user@client <- it is a good idea to see if you can connect to the client before the next step
# cat id_rsa_client.rsync.pub | ssh user@client 'sh -c "cat >> .ssh/authorized_keys2"'
# ssh -l user -i id_rsa_client.rsync client

got it?

this is particularly useful if you want to run a specific command when a client connects; say, update files, move things around. neato.

who the heck has been goofing with my nfs exported data?

if you work in a big linux shop, you'll probably find yourself wondering who is altering and maybe deleting nfs data. nfs, as a general rule, does not have logging. if you daemonize the below, you'll get all the logs you want, and more.

#!/usr/bin/perl

$PIDFILE = "/var/run/nfs-remove-monitor.pid";
$LOGFILE_BASE = "/var/log/nfs-remove-monitor";
$EXIT = 0;
$SIG{CHLD} = IGNORE;

if ( -e $PIDFILE ) {
        $PID = `cat $PIDFILE`;
        `kill -HUP $PID`;
        $DATE=`date +%F`;
        chomp $DATE;
        unlink "$LOGFILE_BASE.$DATE.log";
        unlink "$LOGFILE_BASE.$DATE.log.bz2";
        rename "$LOGFILE_BASE.log", "$LOGFILE_BASE.$DATE.log";
        unless (fork()) {
                sleep 5;
                `bzip2 -9 $LOGFILE_BASE.$DATE.log`;
                exit;
        }
}

open PID, ">$PIDFILE";
print PID $$;
close PID;

open LOG, ">$LOGFILE_BASE.log";
$STDOUT = select LOG;
$|=1;
select $STDOUT;

open TCPDUMP, "tcpdump -vvvvvv -l -i any -s 0 tcp 2>/dev/null |";
$STDOUT = select TCPDUMP;
$|=1;
select $STDOUT;

$SIG{HUP} = sub { $EXIT = 1; };

while ($line = ) {
        if ($line =~ /remove/) {
                print LOG $line;
        }
        last if $EXIT;
}

close TCPDUMP;
close LOG;

don't forget to rotate your logs...

who have been my nfs clients?

sure, you can plod through /var/log/messages to maybe see who has connected to your nfs server. why not just look at rmtab?

#!/usr/bin/perl

$|=1;

use strict;

use Net::DNS;
use Data::Dumper;

use constant DEBUG=>0;

sub logmsg
{
    print STDERR scalar(localtime), " $0 pid $$ ", @_, "\n";
}

sub dnsLookup($)
{
    chomp();
    my $queryname = shift;
    logmsg("Checking queryname $queryname...") if DEBUG==1;
    my $res   = Net::DNS::Resolver->new;
    my $query = $res->search($queryname);
    if ($query) {
        foreach my $rr ($query->answer) {
            if ($rr->type eq "A" ) {
              return $rr->address;
            } 
            if ($rr->type eq "PTR") {
              return $rr->ptrdname;
            }
        }
    } else {
        return $res->errorstring;
    }
}

open fRMTAB, "/var/lib/nfs/rmtab" || die "Unable to open rmtab for reading: $!";
while() {
        my($host,$mount,$count) = split /:/, $_;
        if ( $host =~ m/\d+\.\d+\.\d+\.\d+/ ) {
                print dnsLookup($host), "\n";
        }
}
close fRMTAB;

aix, nfs insecurity and vmount

if you're on an aix box and see this when you're trying to do a mount from your nice linux server:
"vmount: Operation not permitted."

all is not lost.

"but, i set that export as insecure because that pesky firewall sometimes munges
things," you might say.
that setting is: insecure (within the parens in exports)

whatever. ibm says:

AIX versions 4.x and 5.x
Sometimes Linux NFS servers will do port checking and require that 
the NFS client use a reserved port.

     nfso -o nfs_use_reserved_ports=1

If the mount is going to be permanent, then the change needs to survive across a
 reboot. The nfs option must be changed permanently. On AIX 4.x and 5.1, the 
command above should be added to the startup scripts (possibly /etc/rc.nfs). On AIX 
5.2 and above, the change can be made permanent by adding the -p flag.

     nfso -p -o nfs_use_reserved_ports=1

yeah, that works.

tcpping and smokeping

sometimes you just need to measure network latency to specific services between sites. a common thing to do is to do an icmp (ping) test between systems. however, oftentimes your ping tests are dropped. this is what firewalls do from time to time.
a better thing to do is to initiate a half-open scan - just like what nmap does to map services. tcptraceroute or hping3 are terrific tools in this regard.
however (there's always a however) graphs and warnings and the like would be good things™ to have. smokeping with tcpping offers both the latter and former, respectively.

getting them to work together is another kettle of fish, entirely.

tcpping is nothing more than a wrapper script written to have tcptraceroute feed its data to smokeping. the wrapper script is below - don't forget to chmod +x it. if you're on an ubuntu or debian system, you'll need both tcptraceroute and bc. apt-get them. i like to put tccping in /usr/local/bin (because that's where i put all me-made things).

however, smokeping (if it is a really old version) doesn't always have TCPPing.pm (as you may find) thus enabling you to use this coolio probe. you can check this out by going here:

/usr/share/perl5/smokeping/Smokeping/probes/

ls the directory. if TCPPing.pm isn't there, drop in the below TCPPing.pm . Then issue:

smokeping -makepod Smokeping::probes::TCPPing

just remember that you'll need all the modules in the pm prior to doing the make pod and running smokeping and averting a myriad of errors. you can grab each of them (if you haven't them already) by issuing:

perl -MCPAN -e "install IPC::Open3" <- for instance.

now, configure smokeping appropriately. under Probes, add:

+TCPPing
binary = /usr/bin/local/tcpping
forks = 5
offset = 50%
step = 300
timeout = 15

and, as an example here's a stanza for checking both icmp and samba

++ fileservers

menu = fileservers
title = fileservers connectivity

+++ fileservold

menu = fileservers - fileserver10 (icmp)
title = fileservers - fileserver10 (icmp) - 10.0.0.10
host = 10.0.0.10


+++ fileservoldsmb

menu = fileservers - fileservold10 (smb)
title = fileservers - fileserver10 (smb) - 10.0.0.10
probe = TCPPing
host = 10.0.0.10
port = 139

tcpping

#!/bin/sh
#
# tcpping: test response times using TCP SYN packets
#          URL: http://www.vdberg.org/~richard/tcpping.html
#
# uses tcptraceroute from http://michael.toren.net/code/tcptraceroute/
#
# (c) 2002-2005 Richard van den Berg  under the GPL
#               http://www.gnu.org/copyleft/gpl.html
#
# 2002/12/20 v1.0 initial version
# 2003/01/25 v1.1 added -c and -r options
#                 now accepting all other tcptraceroute options
# 2003/01/30 v1.2 removed double quotes around backquotes
# 2003/03/25 v1.3 added -x option, courtesy of Alvin Austin 
# 2005/03/31 v1.4 added -C option, courtesy of Norman Rasmussen 
# 2007/01/11 v1.5 catch bad destination addresses
# 2007/01/19 v1.6 catch non-root tcptraceroute


ver="v1.6"
format="%Y%m%d%H%M%S"
d="no"
c="no"
C="no"
ttl=255
seq=0
q=1
r=1
w=3
topts=""

usage () {
 name=`basename $0`
 echo "tcpping $ver Richard van den Berg "
 echo
 echo "Usage: $name [-d] [-c] [-C] [-w sec] [-q num] [-x count] ipaddress [port]"
 echo
 echo "        -d   print timestamp before every result"
 echo "        -c   print a columned result line"
 echo "        -C   print in the same format as fping's -C option"
 echo "        -w   wait time in seconds (defaults to 3)"
 echo "        -r   repeat every n seconds (defaults to 1)"
 echo "        -x   repeat n times (defaults to unlimited)"
 echo
 echo "See also: man tcptraceroute"
 echo
}

_checksite() {
 ttr=`tcptraceroute -f ${ttl} -m ${ttl} -q ${q} -w ${w} $* 2>&1`
 if echo "${ttr}" | egrep -i "(bad destination|got roo)" >/dev/null 2>&1; then
  echo "${ttr}"
  exit
 fi
}
 
_testsite() {
 myseq="${1}"
 shift
 [ "${c}" = "yes" ] && nows=`date +${format}`
 [ "${d}" = "yes" ] && nowd=`date`
 ttr=`tcptraceroute -f ${ttl} -m ${ttl} -q ${q} -w ${w} $* 2>/dev/null`
 host=`echo "${ttr}" | awk '{print $2 " " $3}'`
 if echo "${ttr}" | egrep "\[(open|closed)\]" >/dev/null 2>&1; then
  rtt=`echo "${ttr}" | awk '{print $5}'`
 else
  rtt=`echo "${ttr}" | awk '{print $4}'`
 fi
 not=`echo "${rtt}" | tr -d ".0123456789"`
 [ "${d}" = "yes" ] && echo "$nowd"
 if [ "${c}" = "yes" ]; then
  if [ "x${rtt}" != "x" -a "x${not}" = "x" ]; then
   echo "$myseq $nows $rtt $host"
  else
   echo "$myseq $nows $max $host"
  fi
 elif [ "${C}" = "yes" ]; then
  if [ "$myseq" = "0" ]; then
   echo -n "$1 :"
  fi
  if [ "x${rtt}" != "x" -a "x${not}" = "x" ]; then
   echo -n " $rtt"
  else
   echo -n " -"
  fi
  if [ "$x" = "1" ]; then
   echo
  fi
 else
  echo "${ttr}" | sed -e "s/^.*\*.*$/seq $myseq: no response (timeout)/" -e "s/^$ttl /seq $myseq: tcp response from/"
 fi
#       echo "${ttr}"
}

while getopts dhq:w:cr:nNFSAEi:f:l:m:p:s:x:C opt ; do
 case "$opt" in
  d|c|C) eval $opt="yes" ;;
  q|w|r|x) eval $opt="$OPTARG" ;;
  n|N|F|S|A|E) topt="$topt -$opt" ;;
  i|l|p|s) topt="$topt -$opt $OPTARG" ;;
  f|m) ttl="$OPTARG" ;;
  ?) usage; exit ;;
 esac
done

shift `expr $OPTIND - 1`

if [ "x$1" = "x" ]; then
 usage
 exit
fi

max=`echo "${w} * 1000" | bc`

if [ `date +%s` != "%s" ]; then
 format="%s"
fi

_checksite ${topt} $*

if [ "$x" = "" ]; then
 while [ 1 ] ; do
  _testsite ${seq} ${topt} $* &
  pid=$!
  if [ "${C}" = "yes" ]; then
   wait $pid
  fi
  seq=`expr $seq + 1`
  sleep ${r}
 done
else
 while [ "$x" -gt 0 ] ; do
  _testsite ${seq} ${topt} $* &
  pid=$!
  if [ "${C}" = "yes" ]; then
   wait $pid
  fi
  seq=`expr $seq + 1`
  x=`expr $x - 1`
  if [ "$x" -gt 0 ]; then
   sleep ${r}
  fi
 done
fi

exit

TCPPing.pm

package Smokeping::probes::TCPPing;

=head1 301 Moved Permanently

This is a Smokeping probe module. Please use the command 

C

to view the documentation or the command

C

to generate the POD document.

=cut

use strict;
use base qw(Smokeping::probes::basefork);
use IPC::Open3;
use Symbol;
use Carp;

sub pod_hash {
      return {
              name => <<'DOC',
Smokeping::probes::TCPPing - TCPPing Probe for SmokePing
DOC
              description => <<'DOC',
Integrates TCPPing as a probe into smokeping. The variable B must
point to your copy of the TCPPing program. If it is not installed on
your system yet, you can get it from http://www.vdberg.org/~richard/tcpping.
You can also get it from http://www.darkskies.za.net/~norman/scripts/tcpping.

The (optional) port option lets you configure the port for the pings sent.
The TCPPing manpage has the following to say on this topic:

The problem is that with the widespread use of firewalls on the modern Internet,
many of the packets that traceroute(8) sends out end up being filtered, 
making it impossible to completely trace the path to the destination. 
However, in many cases, these firewalls will permit inbound TCP packets to specific 
ports that hosts sitting behind the firewall are listening for connections on. 
By sending out TCP SYN packets instead of UDP or ICMP ECHO packets, 
tcptraceroute is able to bypass the most common firewall filters.

It is worth noting that tcptraceroute never completely establishes a TCP connection 
with the destination host. If the host is not listening for incoming connections, 
it will respond with an RST indicating that the port is closed. If the host instead 
responds with a SYN|ACK, the port is known to be open, and an RST is sent by 
the kernel tcptraceroute is running on to tear down the connection without completing 
three-way handshake. This is the same half-open scanning technique that nmap(1) uses 
when passed the -sS flag.
DOC
                authors => <<'DOC',
Norman Rasmussen 
Patched for Smokeping 2.x compatibility by Anton Chernev 
DOC
        }
}


sub new($$$)
{
    my $proto = shift;
    my $class = ref($proto) || $proto;
    my $self = $class->SUPER::new(@_);

    # no need for this if we run as a cgi
    unless ( $ENV{SERVER_SOFTWARE} ) {
        my $return = `$self->{properties}{binary} -C -x 1 localhost 2>&1`;
        if ($return =~ m/bytes, ([0-9.]+)\sms\s+.*\n.*\n.*:\s+([0-9.]+)/ and $1 > 0){
            $self->{pingfactor} = 1000 * $2/$1;
            print "### tcpping seems to report in ", $1/$2, " milliseconds\n";
        } else {
            $self->{pingfactor} = 1000; # Gives us a good-guess default
            print "### assuming you are using an tcpping copy reporting in milliseconds\n";
        }
    };

    return $self;
}

sub ProbeDesc($){
    my $self = shift;
    return "TCP Pings";
}

sub probevars {
 my $class = shift;
 return $class->_makevars($class->SUPER::probevars, {
  _mandatory => [ 'binary' ],
  binary => { 
   _doc => "The location of your TCPPing script.",
   _example => '/usr/bin/tcpping',
   _sub => sub { 
    my $val = shift;

           return "ERROR: TCPPing 'binary' does not point to an executable"
                unless -f $val and -x _;

    my $return = `$val -C -x 1 localhost 2>&1`;
    return "ERROR: TCPPing must be installed setuid root or it will not work\n"
     if $return =~ m/only.+root/;

    return undef;
   },
  },
 });
}

sub targetvars {
 my $class = shift;
 return $class->_makevars($class->SUPER::targetvars, {
  port => {
   _doc => "The TCP port the probe should measure.",
   _example => '80',
   _sub => sub {
    my $val = shift;

    return "ERROR: TCPPing port must be between 0 and 65535"
     if $val and ( $val < 0 or $val > 65535 ); 

    return undef;
   },
  },
 });
}

sub pingone ($){
    my $self = shift;
    my $target = shift;
    # do NOT call superclass ... the ping method MUST be overwriten
    my $inh = gensym;
    my $outh = gensym;
    my $errh = gensym;

    my @times; # Result times

    my @port = () ;
    push @port, $target->{vars}{port} if $target->{vars}{port};

    my @cmd = (
                    $self->{properties}{binary},
                    '-C', '-x', $self->pings($target), 
                    $target->{addr}, @port);
    $self->do_debug("Executing @cmd");
    my $pid = open3($inh,$outh,$errh, @cmd);
    while (<$outh>){
        chomp;
        next unless /^\S+\s+:\s+[\d\.]/; #filter out error messages from tcpping
        @times = split /\s+/;
        my $ip = shift @times;
        next unless ':' eq shift @times; #drop the colon

        @times = map {sprintf "%.10e", $_ / $self->{pingfactor}} sort {$a <=> $b} grep /^\d/, @times;
    }
    waitpid $pid,0;
    close $inh;
    close $outh;
    close $errh;

    return @times;
}

1;