Friday, July 22, 2011

who the heck has been goofing with my nfs exported data?

if you work in a big linux shop, you'll probably find yourself wondering who is altering and maybe deleting nfs data. nfs, as a general rule, does not have logging. if you daemonize the below, you'll get all the logs you want, and more.

#!/usr/bin/perl

$PIDFILE = "/var/run/nfs-remove-monitor.pid";
$LOGFILE_BASE = "/var/log/nfs-remove-monitor";
$EXIT = 0;
$SIG{CHLD} = IGNORE;

if ( -e $PIDFILE ) {
        $PID = `cat $PIDFILE`;
        `kill -HUP $PID`;
        $DATE=`date +%F`;
        chomp $DATE;
        unlink "$LOGFILE_BASE.$DATE.log";
        unlink "$LOGFILE_BASE.$DATE.log.bz2";
        rename "$LOGFILE_BASE.log", "$LOGFILE_BASE.$DATE.log";
        unless (fork()) {
                sleep 5;
                `bzip2 -9 $LOGFILE_BASE.$DATE.log`;
                exit;
        }
}

open PID, ">$PIDFILE";
print PID $$;
close PID;

open LOG, ">$LOGFILE_BASE.log";
$STDOUT = select LOG;
$|=1;
select $STDOUT;

open TCPDUMP, "tcpdump -vvvvvv -l -i any -s 0 tcp 2>/dev/null |";
$STDOUT = select TCPDUMP;
$|=1;
select $STDOUT;

$SIG{HUP} = sub { $EXIT = 1; };

while ($line = ) {
        if ($line =~ /remove/) {
                print LOG $line;
        }
        last if $EXIT;
}

close TCPDUMP;
close LOG;

don't forget to rotate your logs...

who have been my nfs clients?

sure, you can plod through /var/log/messages to maybe see who has connected to your nfs server. why not just look at rmtab?

#!/usr/bin/perl

$|=1;

use strict;

use Net::DNS;
use Data::Dumper;

use constant DEBUG=>0;

sub logmsg
{
    print STDERR scalar(localtime), " $0 pid $$ ", @_, "\n";
}

sub dnsLookup($)
{
    chomp();
    my $queryname = shift;
    logmsg("Checking queryname $queryname...") if DEBUG==1;
    my $res   = Net::DNS::Resolver->new;
    my $query = $res->search($queryname);
    if ($query) {
        foreach my $rr ($query->answer) {
            if ($rr->type eq "A" ) {
              return $rr->address;
            } 
            if ($rr->type eq "PTR") {
              return $rr->ptrdname;
            }
        }
    } else {
        return $res->errorstring;
    }
}

open fRMTAB, "/var/lib/nfs/rmtab" || die "Unable to open rmtab for reading: $!";
while() {
        my($host,$mount,$count) = split /:/, $_;
        if ( $host =~ m/\d+\.\d+\.\d+\.\d+/ ) {
                print dnsLookup($host), "\n";
        }
}
close fRMTAB;

Friday, July 15, 2011

aix, nfs insecurity and vmount

if you're on an aix box and see this when you're trying to do a mount from your nice linux server:
"vmount: Operation not permitted."

all is not lost.

"but, i set that export as insecure because that pesky firewall sometimes munges
things," you might say.
that setting is: insecure (within the parens in exports)

whatever. ibm says:

AIX versions 4.x and 5.x
Sometimes Linux NFS servers will do port checking and require that 
the NFS client use a reserved port.

     nfso -o nfs_use_reserved_ports=1

If the mount is going to be permanent, then the change needs to survive across a
 reboot. The nfs option must be changed permanently. On AIX 4.x and 5.1, the 
command above should be added to the startup scripts (possibly /etc/rc.nfs). On AIX 
5.2 and above, the change can be made permanent by adding the -p flag.

     nfso -p -o nfs_use_reserved_ports=1

yeah, that works.

Tuesday, July 12, 2011

tcpping and smokeping

sometimes you just need to measure network latency to specific services between sites. a common thing to do is to do an icmp (ping) test between systems. however, oftentimes your ping tests are dropped. this is what firewalls do from time to time.
a better thing to do is to initiate a half-open scan - just like what nmap does to map services. tcptraceroute or hping3 are terrific tools in this regard.
however (there's always a however) graphs and warnings and the like would be good things™ to have. smokeping with tcpping offers both the latter and former, respectively.

getting them to work together is another kettle of fish, entirely.

tcpping is nothing more than a wrapper script written to have tcptraceroute feed its data to smokeping. the wrapper script is below - don't forget to chmod +x it. if you're on an ubuntu or debian system, you'll need both tcptraceroute and bc. apt-get them. i like to put tccping in /usr/local/bin (because that's where i put all me-made things).

however, smokeping (if it is a really old version) doesn't always have TCPPing.pm (as you may find) thus enabling you to use this coolio probe. you can check this out by going here:
/usr/share/perl5/smokeping/Smokeping/probes/
ls the directory. if TCPPing.pm isn't there, drop in the below TCPPing.pm . Then issue:
smokeping -makepod Smokeping::probes::TCPPing
just remember that you'll need all the modules in the pm prior to doing the make pod and running smokeping and averting a myriad of errors. you can grab each of them (if you haven't them already) by issuing:
perl -MCPAN -e "install IPC::Open3" <- for instance.
now, configure smokeping appropriately. under Probes, add:
+TCPPing
binary = /usr/bin/local/tcpping
forks = 5
offset = 50%
step = 300
timeout = 15
and, as an example here's a stanza for checking both icmp and samba
++ fileservers

menu = fileservers
title = fileservers connectivity

+++ fileservold

menu = fileservers - fileserver10 (icmp)
title = fileservers - fileserver10 (icmp) - 10.0.0.10
host = 10.0.0.10


+++ fileservoldsmb

menu = fileservers - fileservold10 (smb)
title = fileservers - fileserver10 (smb) - 10.0.0.10
probe = TCPPing
host = 10.0.0.10
port = 139
tcpping
#!/bin/sh
#
# tcpping: test response times using TCP SYN packets
#          URL: http://www.vdberg.org/~richard/tcpping.html
#
# uses tcptraceroute from http://michael.toren.net/code/tcptraceroute/
#
# (c) 2002-2005 Richard van den Berg  under the GPL
#               http://www.gnu.org/copyleft/gpl.html
#
# 2002/12/20 v1.0 initial version
# 2003/01/25 v1.1 added -c and -r options
#                 now accepting all other tcptraceroute options
# 2003/01/30 v1.2 removed double quotes around backquotes
# 2003/03/25 v1.3 added -x option, courtesy of Alvin Austin 
# 2005/03/31 v1.4 added -C option, courtesy of Norman Rasmussen 
# 2007/01/11 v1.5 catch bad destination addresses
# 2007/01/19 v1.6 catch non-root tcptraceroute


ver="v1.6"
format="%Y%m%d%H%M%S"
d="no"
c="no"
C="no"
ttl=255
seq=0
q=1
r=1
w=3
topts=""

usage () {
 name=`basename $0`
 echo "tcpping $ver Richard van den Berg "
 echo
 echo "Usage: $name [-d] [-c] [-C] [-w sec] [-q num] [-x count] ipaddress [port]"
 echo
 echo "        -d   print timestamp before every result"
 echo "        -c   print a columned result line"
 echo "        -C   print in the same format as fping's -C option"
 echo "        -w   wait time in seconds (defaults to 3)"
 echo "        -r   repeat every n seconds (defaults to 1)"
 echo "        -x   repeat n times (defaults to unlimited)"
 echo
 echo "See also: man tcptraceroute"
 echo
}

_checksite() {
 ttr=`tcptraceroute -f ${ttl} -m ${ttl} -q ${q} -w ${w} $* 2>&1`
 if echo "${ttr}" | egrep -i "(bad destination|got roo)" >/dev/null 2>&1; then
  echo "${ttr}"
  exit
 fi
}
 
_testsite() {
 myseq="${1}"
 shift
 [ "${c}" = "yes" ] && nows=`date +${format}`
 [ "${d}" = "yes" ] && nowd=`date`
 ttr=`tcptraceroute -f ${ttl} -m ${ttl} -q ${q} -w ${w} $* 2>/dev/null`
 host=`echo "${ttr}" | awk '{print $2 " " $3}'`
 if echo "${ttr}" | egrep "\[(open|closed)\]" >/dev/null 2>&1; then
  rtt=`echo "${ttr}" | awk '{print $5}'`
 else
  rtt=`echo "${ttr}" | awk '{print $4}'`
 fi
 not=`echo "${rtt}" | tr -d ".0123456789"`
 [ "${d}" = "yes" ] && echo "$nowd"
 if [ "${c}" = "yes" ]; then
  if [ "x${rtt}" != "x" -a "x${not}" = "x" ]; then
   echo "$myseq $nows $rtt $host"
  else
   echo "$myseq $nows $max $host"
  fi
 elif [ "${C}" = "yes" ]; then
  if [ "$myseq" = "0" ]; then
   echo -n "$1 :"
  fi
  if [ "x${rtt}" != "x" -a "x${not}" = "x" ]; then
   echo -n " $rtt"
  else
   echo -n " -"
  fi
  if [ "$x" = "1" ]; then
   echo
  fi
 else
  echo "${ttr}" | sed -e "s/^.*\*.*$/seq $myseq: no response (timeout)/" -e "s/^$ttl /seq $myseq: tcp response from/"
 fi
#       echo "${ttr}"
}

while getopts dhq:w:cr:nNFSAEi:f:l:m:p:s:x:C opt ; do
 case "$opt" in
  d|c|C) eval $opt="yes" ;;
  q|w|r|x) eval $opt="$OPTARG" ;;
  n|N|F|S|A|E) topt="$topt -$opt" ;;
  i|l|p|s) topt="$topt -$opt $OPTARG" ;;
  f|m) ttl="$OPTARG" ;;
  ?) usage; exit ;;
 esac
done

shift `expr $OPTIND - 1`

if [ "x$1" = "x" ]; then
 usage
 exit
fi

max=`echo "${w} * 1000" | bc`

if [ `date +%s` != "%s" ]; then
 format="%s"
fi

_checksite ${topt} $*

if [ "$x" = "" ]; then
 while [ 1 ] ; do
  _testsite ${seq} ${topt} $* &
  pid=$!
  if [ "${C}" = "yes" ]; then
   wait $pid
  fi
  seq=`expr $seq + 1`
  sleep ${r}
 done
else
 while [ "$x" -gt 0 ] ; do
  _testsite ${seq} ${topt} $* &
  pid=$!
  if [ "${C}" = "yes" ]; then
   wait $pid
  fi
  seq=`expr $seq + 1`
  x=`expr $x - 1`
  if [ "$x" -gt 0 ]; then
   sleep ${r}
  fi
 done
fi

exit
TCPPing.pm
package Smokeping::probes::TCPPing;

=head1 301 Moved Permanently

This is a Smokeping probe module. Please use the command 

C

to view the documentation or the command

C

to generate the POD document.

=cut

use strict;
use base qw(Smokeping::probes::basefork);
use IPC::Open3;
use Symbol;
use Carp;

sub pod_hash {
      return {
              name => <<'DOC',
Smokeping::probes::TCPPing - TCPPing Probe for SmokePing
DOC
              description => <<'DOC',
Integrates TCPPing as a probe into smokeping. The variable B must
point to your copy of the TCPPing program. If it is not installed on
your system yet, you can get it from http://www.vdberg.org/~richard/tcpping.
You can also get it from http://www.darkskies.za.net/~norman/scripts/tcpping.

The (optional) port option lets you configure the port for the pings sent.
The TCPPing manpage has the following to say on this topic:

The problem is that with the widespread use of firewalls on the modern Internet,
many of the packets that traceroute(8) sends out end up being filtered, 
making it impossible to completely trace the path to the destination. 
However, in many cases, these firewalls will permit inbound TCP packets to specific 
ports that hosts sitting behind the firewall are listening for connections on. 
By sending out TCP SYN packets instead of UDP or ICMP ECHO packets, 
tcptraceroute is able to bypass the most common firewall filters.

It is worth noting that tcptraceroute never completely establishes a TCP connection 
with the destination host. If the host is not listening for incoming connections, 
it will respond with an RST indicating that the port is closed. If the host instead 
responds with a SYN|ACK, the port is known to be open, and an RST is sent by 
the kernel tcptraceroute is running on to tear down the connection without completing 
three-way handshake. This is the same half-open scanning technique that nmap(1) uses 
when passed the -sS flag.
DOC
                authors => <<'DOC',
Norman Rasmussen 
Patched for Smokeping 2.x compatibility by Anton Chernev 
DOC
        }
}


sub new($$$)
{
    my $proto = shift;
    my $class = ref($proto) || $proto;
    my $self = $class->SUPER::new(@_);

    # no need for this if we run as a cgi
    unless ( $ENV{SERVER_SOFTWARE} ) {
        my $return = `$self->{properties}{binary} -C -x 1 localhost 2>&1`;
        if ($return =~ m/bytes, ([0-9.]+)\sms\s+.*\n.*\n.*:\s+([0-9.]+)/ and $1 > 0){
            $self->{pingfactor} = 1000 * $2/$1;
            print "### tcpping seems to report in ", $1/$2, " milliseconds\n";
        } else {
            $self->{pingfactor} = 1000; # Gives us a good-guess default
            print "### assuming you are using an tcpping copy reporting in milliseconds\n";
        }
    };

    return $self;
}

sub ProbeDesc($){
    my $self = shift;
    return "TCP Pings";
}

sub probevars {
 my $class = shift;
 return $class->_makevars($class->SUPER::probevars, {
  _mandatory => [ 'binary' ],
  binary => { 
   _doc => "The location of your TCPPing script.",
   _example => '/usr/bin/tcpping',
   _sub => sub { 
    my $val = shift;

           return "ERROR: TCPPing 'binary' does not point to an executable"
                unless -f $val and -x _;

    my $return = `$val -C -x 1 localhost 2>&1`;
    return "ERROR: TCPPing must be installed setuid root or it will not work\n"
     if $return =~ m/only.+root/;

    return undef;
   },
  },
 });
}

sub targetvars {
 my $class = shift;
 return $class->_makevars($class->SUPER::targetvars, {
  port => {
   _doc => "The TCP port the probe should measure.",
   _example => '80',
   _sub => sub {
    my $val = shift;

    return "ERROR: TCPPing port must be between 0 and 65535"
     if $val and ( $val < 0 or $val > 65535 ); 

    return undef;
   },
  },
 });
}

sub pingone ($){
    my $self = shift;
    my $target = shift;
    # do NOT call superclass ... the ping method MUST be overwriten
    my $inh = gensym;
    my $outh = gensym;
    my $errh = gensym;

    my @times; # Result times

    my @port = () ;
    push @port, $target->{vars}{port} if $target->{vars}{port};

    my @cmd = (
                    $self->{properties}{binary},
                    '-C', '-x', $self->pings($target), 
                    $target->{addr}, @port);
    $self->do_debug("Executing @cmd");
    my $pid = open3($inh,$outh,$errh, @cmd);
    while (<$outh>){
        chomp;
        next unless /^\S+\s+:\s+[\d\.]/; #filter out error messages from tcpping
        @times = split /\s+/;
        my $ip = shift @times;
        next unless ':' eq shift @times; #drop the colon

        @times = map {sprintf "%.10e", $_ / $self->{pingfactor}} sort {$a <=> $b} grep /^\d/, @times;
    }
    waitpid $pid,0;
    close $inh;
    close $outh;
    close $errh;

    return @times;
}

1;

Friday, May 13, 2011

i like to see what others type + syslog-ng

Once you've gone through the trouble of patching bash to send output to local5, you might find that you're not using syslog, as assumed in a previous post. Instead, you're using syslog-ng. That's cool.

In your syslog-ng.conf file, you'll need to edit some stanzas, filters and destinations.

Set local5 (bash output) destination, if you want it to go to a file.
# bash destination 
destination d_local5 { file("/var/log/local5"); }; 

# bash filters 
filter f_local5 { facility(local5); }; 

and in messages filter, add local5
filter f_messages {
        level(info,notice,warn)
            and not facility(auth,authpriv,cron,daemon,mail,news,local5);
};
and finally, set the log destination:
# local5
log {
        source(s_all); 
        filter(f_local5);
        destination(d_bash);
};

If you have a remote syslog daemon or logger such as loggly or splunk set up, drop their destination definitions in the log stanza for "local5". e.g.:

# loggy
#
destination d_loggly { tcp("logs.loggly.com" port(XXXXXX)); };

# local5
log {
        source(s_all); 
        filter(f_local5);
        destination(d_local5);
        destination(d_loggly); 
};

On loggly (if you've allowed the destination in your remote device list) or splunk, you should see something akin to:
2011 May 13 16:09:19.000 s_all@host1 bash-ub610: history: [pid:5379 uid:0] exit
2011 May 13 16:09:50.000 s_all@host1 bash-ub610: history: [pid:5584 uid:0] ls -la
2011 May 13 16:09:52.000 s_all@host1 bash-ub610: history: [pid:5584 uid:0] cd /opt/
In your old /etc/syslog.conf or /etc/rsyslog.d/50-default.conf add the following:
auth,authpriv.*;local5.*        @syslogserver
auth,authpriv.*;local5.*        @logs.loggly.com:yourportno

Thursday, May 12, 2011

microsoft dhcp and me

I like to use dhcpd on a Linux box because I can set all kinds of options, like giving search suffixes to my DHCP clients. Microsoft Windows Server iterations have no such option - or so you're told. By default, yes, this isn't an option, but at least in Server 2008, you can add new DHCP options to include the DHCP-supplied option. To do as such:

1.  Open the DHCP mmc
2.  Expand DHCP, select DHCP server name.
3.  Right Click IPv4
4.  Select "Set Predefined Options"
5.  Click Add.

A new window appears

6.  Enter the following:
Name: "Domain suffix search order" (without quotation marks)
Data Type: String
Code: "135" (without the quotation marks)
Description: "List of domain suffixes in order" (without the quotation marks)
String: Enter search suffixes separated by comma with no spaces
 
7.  Click OK.
8.  Close DHCP MMC and restart DHCP Server Service.

Now, re-open the DHCP mmc, scroll to the end of the DHCP options, and the newly created option will appear.

Monday, March 7, 2011

apache & openldap group authentication

For Apache 2.2, check your mod-enabled and mods-available directory. Make certain your ls in mods-available have the following symlinked from mods-enabled; e.g.:

 alias.load -> ../mods-available/alias.load
 auth_basic.load -> ../mods-available/auth_basic.load
 authnz_ldap.load -> /etc/apache2/mods-available/authnz_ldap.load
 authz_default.load -> ../mods-available/authz_default.load
 authz_user.load -> ../mods-available/authz_user.load
 ldap.load -> ../mods-available/ldap.load

In your site-available file, load these two loaded mods, with the following:

 LoadModule ldap_module           /usr/lib/apache2/modules/mod_ldap.so
 LoadModule authnz_ldap_module    /usr/lib/apache2/modules/mod_authnz_ldap.so

In the directory structure where you'd like to have LDAP authentication to take place, add the following stanza:

 AuthBasicProvider ldap
 AuthType Basic
 AuthzLDAPAuthoritative on
 AuthName "restricted site access"
 AuthLDAPURL ldap://www.xxx.yyy.zzz/ou=users,dc=your,dc=com?uid
 AuthLDAPGroupAttribute memberUid
 AuthLDAPGroupAttributeIsDN off
 Require ldap-group cn=agroup,ou=groups,dc=your,dc=com
 Require ldap-user adude anotherdude
 Satisfy any

If you have a round-robin LDAP setup, place the FQDN of your OpenLDAP server in the AuthLDAPURL section. The uid condition means that your authentication control is via uid. AuthLDAPGroupAttribute and its allied Require ldap-group, states that you're checking for membership in a specific group "agroup", and those members have the attribute "memberUid". You can tack on an individual user (or users on the same line), by specifying "Require ldap-user". And, To allow for both groups and users, have the "Satisfy any" directive set; otherwise no one will be able to log on and use your web-resource.