stop polluting syslog

so yeah, local6 is pretty awesome. it will write and write and write to the local system. but, i like to have it reserved for super secret stuff, you know? and all that writing takes up space. here's a clue. to not log local6 to the local system and to forward it along, edit two lines:

*.*;auth,authpriv.none,local6.none       -/var/log/syslog
auth,authpriv.*;local5.*;local6.*        @remotesyslog

the deal with the dash in an rsyslog file is that after writing, the file does not sync.
and the .none means no no do not log no no. as opposed to asterisk which means yes please all.
fun for me. fun for you.

what can grep do for you?

a lot actually. suppose you want to see what lines are the same in a file. here's how to do it: fgrep -xf file1 file2 > file3

file3

is what is the same. and now you know.

get splunk 3.4 off the system and put it back on

 remove splunk 3.4 and reinstall  
   
because it feels good to do. but let's not forget your 
awesome hacks.  
   
~# mkdir splunk.add ; cd splunk.add  
~/splunk.add# cp /opt/splunk/lib/python2.5/site-packages/splunk/ \
     appserver/oxiclean/SearchService.py .  
~/splunk.add# cp /opt/splunk/etc/system/default/web.conf .  
~# dpkg -l |grep 'splunk'  
   
ii splunk              3.4-44873             Splunk   
   
~# dpkg -P splunk  
   
(Reading database ... 58972 files and directories currently installed.)  
Removing splunk ...  
   
~# rm -rf /opt/splunk  
~# dpkg -i splunk-3.4-44873-linux-2.6-amd64.deb  
~# cp splunk.add/SearchService.py /opt/splunk/lib/python2.5/ \
     site-ackages/splunk/appserver/oxiclean/  
~# cp splunk.add/web.conf /opt/splunk/etc/system/default/  
~# ./opt/splunk/bin/splunk start  
   

ms dns debug logs to splunk

 
 yeah.  so you want to send dns debug logs to splunk so you can figure out who is 
 checking out redpepper.  that's a hot site! 
 
 well.  you can really forward them to any syslog server using the below, but splunk is my syslog 
 aggregator of choice.  plus it uses nice fonts and has pretty graphs and runs on ubuntu.
   
 follow the instructions here:  
 http://stratumsecurity.com/2012/07/03/splunk-security/  
   
 Your DNS logs will undoubledly go here:  
 C:\WINDOWS\system32\dns  
   
 And no worries about your logs rotating or any sillines, the max byte size is default 500M; after that the 
 log resets itself.  
   
 ...  
   
 if you're using splunk free, like me, you'll eventually figure out that you cannot use the splunk forwarder 
 to forward to your  splunk server.  you'll try to enable splunktcp:9997 in inputs.conf on your splunk server. 
 but the daemon won't listen. you'll try another port only to be met with frustration.  
   
 you might even try forwarding to syslog, but you'll see:  
 _linebreaker\x00\x00"  
   
 that means that data is raw splunk data.  
 http://answers.splunk.com/answers/10346/splunk-is-adding-weird-strings-like-linebreaker-x00-x00-to-my-events-what-is-going-on.html  
   
 that's a super raw deal. but, you're smart and motivated and everyone at work knows you're plain 
 awesome. yeah. right.
    
 so, your awesome self uses this:  
 http://www.syslogserver.com/syslogagent.html  
   
 the datagram syslog agent is fairly straightforward.  
 * udp transport, syslog server and port (conf'd for splunk)  
 * check enable forwarding of appl logs.  
 * edit application  
 - application name: dns debug  
 - specific file (static): C:\WINDOWS\system32\dns\dns.log  
 - syslog protocol conformity:   
 - parse date/time  
 - parse host  
 - use name "dns debug"  
   
 ...  
   
 head to your splunk console and check for  
 "dns debug"  
   
 Jan 12 14:11:58 dns.server Jan 12 14:12:03 valeriano dns debug[info] 20150112 8D58 PACKET 0229A720 
 UDP Rcv 10.10.10.10  6ea9 Q [0001  D  NOERROR] A   (4)redpepper(0)   
  
 here's looking at you 10.10.10.10.
 

knock know who's there?

sometimes you just want to set up a virtual system to see if there are any scans going on your network.

 
on an ubuntu 14.04 lts server...
   
 # apt-get install build-essential mail-utils perl

   
run cpan get the following cpan dependencies:  
   
 # cpan  
 > install Net::DNS  
 > install Net::Server  
 > install YAML  
 > install MIME::Lite::TT  
 > install Data::Dumper  
 > install Getopt::Long  
 > install Net::IP::Match::Regexp  
 > exit  
   
place perl script and conf in /usr/local/bin/honey or wherever   
do a little ln action.  
   
 # ln -s /usr/local/honey/honey.conf /etc/honey.conf  
   
set background to 1 in honey.conf to make it turn into a background process.  
   
and... when putting in your ports, if any ports happen to be open when 
honey.pl is called, honey.pl will die.  
 figure them out?  
   
 # netstat -tpln  
   
speaking of which look at the honey.conf below, we're going to be opening 
a scad of ports. tune the system:  
   
 # ulimit -n 70000  
 # echo "32768 65535" >/proc/sys/net/ipv4/ip_local_port_range  
   
test it  
   
 # perl -c honey.pl should return 'OK'  
   
start with  
 # perl /usr/local/honey.pl --config /etc/honey.conf  
   
if running via /etc/rc.local :  
 # chmod +x /etc/rc.local  
   
add the line before exit 0:  
   
perl /usr/local/honey.pl --config /etc/honey.conf  
   
 ##########  
 # honey.pl  
   
 #!/usr/bin/perl  
 # version 1.3  
 package Honey;  
 use MIME::Lite::TT;  
 use Net::DNS;  
 use strict;  
 use warnings;  
 use Data::Dumper;  
 use base qw(Net::Server::PreForkSimple);  
 use YAML;  
 use Getopt::Long;  
 use Net::IP::Match::Regexp qw( create_iprange_regexp match_ip );  
 sub logger($);  
   
 my $configfile = "./honey.conf";  
 GetOptions ("config=s" => \$configfile);  
 my ($hashref, $arrayref, $string) = YAML::LoadFile( $configfile );  
   
 print Dumper($hashref);  
 my %config = %$hashref;  
   
   
 #############################  
 ### CONFIG   
 #############################  
 my $from_email = $config{'from_email'};  
 my $subject   = $config{'subject'};  
 my $to_email  = $config{'to_email'};  
 my $mailserver = $config{'mailserver'};  
 my $mail_thres = $config{'mail_thres'};  
 my $tempfolder = $config{'tempfolder'};  
 my $lp_ref   = $config{'listenports'};  
 my $background = $config{'background'};  
 my $ih_ref   = $config{'ignorehosts'};  
 my $logpath   = $config{'logpath'};  
   
 unless ( 1 &&  
     defined($from_email) &&   
     defined($subject) &&   
     defined($to_email) &&   
     defined($mailserver) &&   
     defined($mail_thres) &&   
     defined($tempfolder) &&   
     defined($lp_ref) &&   
     defined($background) &&   
     defined($ih_ref) &&   
     1 ) {  
     die "invalid configuration\n";  
 }  
   
 unless (defined($logpath)) {  
     print STDERR "No logpath given, will default to /var/log/honey.log";    
     $logpath = "/var/log/honey.log";  
 }  
   
 logger("honey init");  
   
 #############################  
 ### VARS  
 #############################  
 my ($i);  
 my @lports   = @$lp_ref;  
 my @ih     = split(",",$ih_ref);  
 my $ignorehosts = create_iprange_regexp(@ih);  
 my $tempcache  = $tempfolder . "honeycache";  
 my $tempports  = $tempfolder . "honeyports";  
   
   
 my $template = <<TEMPLATE;  
 Unauthorized connection noted \n\r  
 Connection details: [% connection_string %] \n\r  
 Source details: [% srcip %] ([% srcip_dns %]) \n\r  
 Timestamp: [% timestamp %]  
 TEMPLATE  
   
   
 ############################################  
   
   
 sub post_accept {  
     #print STDERR "post accept in $$\n";  
 }   
   
 sub process_request {  
     my $self = shift;  
   
     #print STDERR "process request in $$\n";  
   
     my $connection_info = $self->{'server'}->{'peeraddr'} . ":" . $self->{'server'}->{'peerport'};  
     $connection_info = $connection_info . " --> " . $self->{'server'}->{'sockaddr'} . ":" . $self->{'server'}->{'sockport'};  
     connection_identified( $self->{'server'}->{'peeraddr'}, $connection_info);  
       
 }  
   
   
 sub connection_identified ($$) {  
     my $srcIP        = $_[0];  
     my $connection_info   = $_[1];  
   
     logger("ok we got a connection from $srcIP");  
   
     # RESTORE HASH  
     my $ref = restore_hash();  
     my %last_email_timestamp = %$ref;  
     my $skip_email = 0;  
   
     # check if we should ignore this IP  
     if (match_ip($srcIP, $ignorehosts)) {  
         logger("ignoring $srcIP!");  
         $skip_email = 1;  
         return 0;  
     }  
   
     # email max every $mail_thres secs   
     my $current_timestamp = time();  
     if (defined($last_email_timestamp{$srcIP})) {  
         logger("We have already seen a connection from the host before");  
       
         my $diff = $current_timestamp - $last_email_timestamp{$srcIP};  
         if ( $diff < $mail_thres) {  
             logger("OK so we saw a connection less than $mail_thres secs ago .. skippin email");  
             $skip_email = 1;  
         } else {  
             logger("but it was a long time ago, diff is $diff");  
         }  
     } else {  
         logger("we have not seen this IP before");  
     }  
     $last_email_timestamp{$srcIP} = $current_timestamp;  
     save_hash(\%last_email_timestamp);  
   
     if ($skip_email) { return; };  
     logger("ok gonna send an email using $mailserver, timeout 60");  
   
     # reverse dns  
     my $srcip_dns = PTR_lookup($srcIP);  
   
     # timestamp  
     my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)=localtime(time);  
     my $timestamp = sprintf("%4d-%02d-%02d %02d:%02d:%02d\n",$year+1900,$mon+1,$mday,$hour,$min,$sec);  
   
     # SEND EMAIL?  
     my %params;  
     $params{connection_string}  = $connection_info;  
     $params{srcip_dns}   = $srcip_dns;  
     $params{srcip}     = $srcIP;  
     $params{timestamp}   = $timestamp;  
   
     my $msg = MIME::Lite::TT->new(  
       From     => $from_email,  
       To      => $to_email,  
       Subject   => $subject,  
       Template   => \$template,  
       TmplParams  => \%params,  
     );  
   
     $msg->send('smtp', $mailserver, Timeout => 60 );  
     logger("email sent using $mailserver");  
 }  
   
 sub PTR_lookup {  
     my $tname = shift;  
     my $type = "Reverse (PTR)";  
   
     my $rr;  
   
     my $res = new Net::DNS::Resolver;  
     my $query = $res->query("$tname","PTR");  
   
     if ($query) {  
         foreach $rr ($query->answer) {  
             next unless $rr->type eq "PTR";  
             my $ip = $rr->ptrdname;  
             return ($ip);  
         }  
     } else {  
         my $logstring = "Reverse lookup query failed for $tname : " . $res->errorstring . "\n";  
         logger($logstring);  
         return ($tname);  
     }  
 }   
   
 sub restore_hash() {  
   
     my %thash = ();  
     if (-e $tempcache) {  
         open(FOO,"<$tempcache") or die;  
         foreach my $line (<FOO>) {  
             chomp($line);  
             my ($ip,$time) = split(":",$line);  
             $thash{$ip} = $time;  
         }  
         close(FOO);  
     }  
   
     return \%thash;  
 }  
   
 sub save_hash($) {  
     my $hash_ref = $_[0];  
   
     my %thash = %$hash_ref;  
   
     open(FOO,">$tempcache") or die;  
     foreach my $key (keys %thash) {  
         if (defined($thash{$key})) {  
             print FOO $key . ":" . $thash{$key} . "\n";  
         }  
     }  
     close(FOO);  
   
 }  
   
 sub logger($) {  
     my $message = $_[0];  
   
     # LOG  
     open(POO,">>$logpath") or die;  
     print POO get_date() . ": $message\n";  
     close(POO);   
   
     # STDERR  
     print STDERR get_date() . ": $message\n";  
   
 }  
   
 sub get_date () {  
     my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)=localtime(time);  
     my $timestamp = sprintf("%4d-%02d-%02d %02d:%02d:%02d\n",$year+1900,$mon+1,$mday,$hour,$min,$sec);  
     chop($timestamp);  
     return $timestamp;  
 }  
   
   
 open(FOO,">$tempports") or die;  
 foreach my $lport (@lports) {  
       
     if ($lport =~ /(\d+)-(\d+)/) {  
         my $startport = $1;  
         my $stopport = $2;  
         if ($stopport < $startport) { die "Invalid config, check the range\n"; };  
         for ($i=$startport;$i<=$stopport;$i++) {  
             print FOO "port $i\n";  
         }  
     } elsif ($lport =~ /^(\d+)/) {  
         print FOO "port $lport\n";  
     } else {  
         logger("Invalid port $lport specified");  
     }  
 }  
 close(FOO);  
   
 if ($background) {  
     Honey->run(   background => 1,  
         conf_file => "$tempports");  
 } else {  
     Honey->run(conf_file => "$tempports");  
 }  
   
 ##########  
 # honey.conf  
   
 from_email: me@here  
 to_email: you@there  
 mailserver: smtpserver  
 ignorehosts: 10.10.10.10, 0.0.0.0  
 mail_thres: 60  
 tempfolder: /tmp/  
 subject: Unauthorized connection to knockknock  
 background: 1  
 logpath: /var/log/honeylog.log  
 listenports:  
   - 1-21  
   - 23-24  
   - 26-65500  
   

apache logs to syslog

the other day i flipped out. well, flipped out in my own quiet way. i heard about some apache access issues and it gave me a slight headache. the super cool thing about linux boxes is that the "truth is in the logs". and i heart logs and log aggregation.

the cool thing about apache is that if configured correctly it will log all access and all errors. sadly, apache, by default, writes its logs on the local system and not via syslog processes. crap.
i really really don't want to go to each of my boxes and grep through /var/log/apache/blah.txt, how do i throw the logs access the network?

well, most linux boxes have tee and logger. tee is a nice program that basically say, do this and this. logger can send arbitrary data to syslog. yay.

in my enabled site, i changed my ErrorLog and CustomLog sections from this:

ErrorLog /var/log/apache2/error.log
CustomLog /var/log/apache2/access.log combined

to this:

ErrorLog "|/usr/bin/tee -a /var/log/apache2/error.log | /usr/bin/logger -thttpd -plocal6.err"
CustomLog "|/usr/bin/tee -a /var/log/apache2/access.log | /usr/bin/logger -thttpd -plocal6.notice" combined

i'm calling local6 and sending it off to syslog in httpd format. neat.

since i'm using rsyslogd, i edited my /etc/rsyslog.d/50-default conf to pipe off my logs to my remote syslog server:

auth,authpriv.*;local6.* @remotesyslogserver

if you'd rather not log to /var/log/syslog, add: local6.none to the -/var/log/messages stanza.

reload your daemons and voila.

openvas v7 create a new user

well. there are so many ways to create a user. this allows for integration with greenbone security desktop; a scanner user.

root@openvas:~#  openvasmd --create-user younotme
User created with password 'd25d4c66-5f7a-4156-84ee-f3ee101381fa'.

root@openvas:~# openvasmd --user=younotme --new-password=notcreatedhere

that was easy.