what to do when your ipv4 forwarder has a scrambled arp cache

I have various network segments doing all kinds of things. As opposed to routing all of them, I place them being boxes that shape traffic and the like. ipv4 forwarding is my friend, mostly. Except when its cascading switches go weird.

So, to figure out where a specific host is connected, I issue:

arp -a | grep ethX

That's nice. If I see incomplete entries, that means that the MAC address of said machine is in a weird state. If I know that the system is sitting on a specific interface and not the reported one, it does me good to flush the entire arp cache. But! There's no command in Linux to flush the cache itself. You have to do it quick and dirty via a script or the command line using the -d switch. Plop this in or on either on your boxes that use /proc/net/arp:

for i in `awk -F ' ' '{ if ( $1 ~ /[0-9{1,3}].[0-9{1,3}].[0-9{1,3}].[0-9{1,3}]/ ) \
 print $1 }' /proc/net/arp` ; do arp -d $i ; done

And now the cache is clear and life goes on.

wget is my friend

sometimes you need yank a site and edit it and really don't feel like doing it live. one way to get the content is cp'ing the directory and do whatever to it. other times wget is a quick and dirty if you're particularly lazy or if your content is all over the place. here's the command line i typically use for wget:

wget --random-wait -r -p -e robots=off -U mozilla http://your.site.org <- yanks *
wget --random-wait -r --no-parent -p http://your.site.org/content/dir <- grabs a dir

some useful swtiches per man:

-p ; include all files, including images.
-e robots=off ; do not obey server-side robots.txt 
-U mozilla ; browser identity.
--random-wait ; number of seconds to wait, thus avoiding server black list.
--limit-rate=15k ; throttle the download rate.
-b ; continue application in background.
-o ; output log (as opposed to scrolling on screen).

reassociate vmware images

so i got the grand idea to scp some vmware images from a dead nas to another living nas. since the dead one was dead, there really wasn't a good way to move the images in the usual vmware way.

so, i proceeded to set up another iscsi device on that nas, got it mounted right with all the correct data. neat.
my vmware esx server still reported that its images were "Unknown Host" of course. but blast it, what were the names of these systems before? my vsphere console server was down. crud.

easy, i went here:

esxserver:/etc/vmware/hostd/vmInventory.xml

and there they were.

all i needed to do to reassociate them was to change the /vmfs/volume/path to the correct one. i rebooted the vmware esx server and all was cool.

instant root shell to change that pesky forgotten root password via grub

forget knoppix. just do it natively. oh, this only works if grub's not passworded. usually isn't.

* press esc at the grub prompt
* highlight the line that begins with kernel XXX , press e
* go to the very end of the line, add rw init=/bin/bash
note:  that is if bash is in /bin ; if not try /usr/bin or /usr/local/bin
* press enter, then press b to boot system
note:  system should now boot as root shell sans password.
* type passwd username.  set password.
* type reboot

using ssh to restrict commands by key

as we all know, if you have ssh access to a box, you can run arbitrary commands via one liners using the -C switch. neat. or not.
but! you can also have the sysadminly joy of restricting what an ssh client connection can do if you do not want said connection to have unfettered shell access.

anyway. here was the probem:
one day i was getting sort of tired of renaming pems and moving them around between systems. believe me, it was a drag. i wanted a one-liner, so i made one.

i plopped an ssh key for the robot account (robot@mordor) via which i wanted to do all these mundane tasks and then prepended its definition in authorized_keys2 on the client box with the a specific command string.

here's a snippet of authorized_keys2:

command="/usr/local/bin/secretcommands $SSH_ORIGINAL_COMMAND" ssh-rsa == robot@mordor

what this does is restrict robot@mordor to only be able to issue that specific command. no interactive shell, no nothing, just the command. but. in my case, i really needed that robot to be able to do more than one thing. so! i made my command a menu of choices - secretcommands. now, robot@mordor can do only what is on the menu of choices - like look at messages or rename things.

to allow robot@mordor to scp things, i needed to also explicitly place the client system's scp command in the same authorized_keys2 file:

command="/root/bin/scp-wrapper" ssh-rsa == robot@mordor

here's secretcommands:

#!/bin/sh
; secretcommands
case "$1" in
 messages)
  /bin/cat /var/log/messages
  ;;
 delicious)
  /usr/local/bin/rename.sh /etc/secretplaceforsecretthings/crl.pem
  ;;
 help)
  echo 'messages and delicious' 1>&2
  exit 1
 ;;
 *)
  echo 'please do not do that' 1>&2
  exit 1
  ;;
esac

here's the scp command per the snail book:

#!/usr/bin/env perl
# from http://www.snailbook.com/faq/restricted-scp.auto.html

# location of the server-side scp we want to run
$scp_server = "/usr/bin/scp";

sub fail {
    my ($msg) = @_;
    print STDERR "$0: ", $msg, "\n";
    exit 1;
}

# This just makes me feel better.

$TRUE  = (0 == 0);
$FALSE = (0 == 1);

# Since this script is called as a forced command, need to get the
# original scp command given by the client.

($command = $ENV{SSH_ORIGINAL_COMMAND})
    || fail "environment variable SSH_ORIGINAL_COMMAND not set";

# Split the command string to make an argument list, and remove the first
# element (the command name; we'll supply our own);

@scp_argv = split /[ \t]+/, $command;

# Complain if the command is not "scp".

fail "account restricted: only scp allowed (\"$scp_argv[0]\")"
    unless $scp_argv[0] eq "scp";

# Wipe the environment as a security precaution.  This might conceivably
# break something, but if it does you can filter the environment more
# selectively here.

%ENV = ();

# Ensure that either -t or -f is on the command line, to enforce running
# scp in server mode.

$ok = $FALSE;
foreach $arg (@scp_argv) {
    if ($arg eq '-t' || $arg eq '-f') {
        $ok = $TRUE;
        last;
    }
}

fail "Restricted; only server mode allowed."
    unless $ok;

# if we're OK, run our desired "scp" with arguments.

shift(@scp_argv);
exec($scp_server, @scp_argv);

the rename script in secretcommands:

#!/bin/sh
# rename.sh
for file in ${@}; do
    if [ -e $file ]; then
        timestamp=`ls -l --time-style=+%Y%M%d-%k%M%S $file |cut -d' ' -f6`
        basename=`echo $file|cut -d'.' -f1`
        ext=`echo $file | cut  -d'.' -f2-`
        mv -v $file $basename.$ext.$timestamp
    fi
done

here's what it would look like in a one liner:

mordor$ /usr/bin/ssh -i /home/robot/.ssh/id_rsa_nothere.secretcommand -l root nothere delicious

delicious.

multiple pub keys

i usually have a myriad of keys for different automated processes. i really like backing up stuff using unattended rsync scripts; and since you can wrap rsync in ssh, having multiple keys is a good idea - plus the security aspect is cool, too.

my nomenclature for keys is the following:
id_typeofencryption_clientboxname.application

here's how to create an additional key and place it on the target (client) box:

# ssh-keygen -t rsa -b 2048 -f id_rsa_client.rsync
# ssh user@client <- it is a good idea to see if you can connect to the client before the next step
# cat id_rsa_client.rsync.pub | ssh user@client 'sh -c "cat >> .ssh/authorized_keys2"'
# ssh -l user -i id_rsa_client.rsync client

got it?

this is particularly useful if you want to run a specific command when a client connects; say, update files, move things around. neato.

who the heck has been goofing with my nfs exported data?

if you work in a big linux shop, you'll probably find yourself wondering who is altering and maybe deleting nfs data. nfs, as a general rule, does not have logging. if you daemonize the below, you'll get all the logs you want, and more.

#!/usr/bin/perl

$PIDFILE = "/var/run/nfs-remove-monitor.pid";
$LOGFILE_BASE = "/var/log/nfs-remove-monitor";
$EXIT = 0;
$SIG{CHLD} = IGNORE;

if ( -e $PIDFILE ) {
        $PID = `cat $PIDFILE`;
        `kill -HUP $PID`;
        $DATE=`date +%F`;
        chomp $DATE;
        unlink "$LOGFILE_BASE.$DATE.log";
        unlink "$LOGFILE_BASE.$DATE.log.bz2";
        rename "$LOGFILE_BASE.log", "$LOGFILE_BASE.$DATE.log";
        unless (fork()) {
                sleep 5;
                `bzip2 -9 $LOGFILE_BASE.$DATE.log`;
                exit;
        }
}

open PID, ">$PIDFILE";
print PID $$;
close PID;

open LOG, ">$LOGFILE_BASE.log";
$STDOUT = select LOG;
$|=1;
select $STDOUT;

open TCPDUMP, "tcpdump -vvvvvv -l -i any -s 0 tcp 2>/dev/null |";
$STDOUT = select TCPDUMP;
$|=1;
select $STDOUT;

$SIG{HUP} = sub { $EXIT = 1; };

while ($line = ) {
        if ($line =~ /remove/) {
                print LOG $line;
        }
        last if $EXIT;
}

close TCPDUMP;
close LOG;

don't forget to rotate your logs...