ataraxia: February 2012

Monday, February 27, 2012

installation of hpacucli on Ubuntu 10.04.4 LTS (Lucid) x86_64

it all began with a simple query:

do we have a write cache?

and down the rabbit hole I went...

# cat /proc/driver/cciss/cciss*

cciss0: HP Smart Array P410i Controller
Board ID: 0x3245103c
Firmware Version: 5.14
IRQ: 63
Logical drives: 1
Current Q depth: 0
Current # commands on controller: 3
Max Q depth since init: 9
Max # commands on controller since init: 318
Max SG entries since init: 31
Sequential access devices: 0

cciss/c0d0:     1799.79GB       RAID 5

yay. i guess. to administer this, i can either take the system offline and mess around on the controller. or! i can install the hp tool HP Array Configuration Utility CLI for Linux (hpacucli). it has the added bonus of being able to be called by nagios... but i'm getting ahead of myself. it works with the following controllers:

Smart Array 5312 Controller
Smart Array 5302 Controller
Smart Array 5304 Controller
Smart Array 532 Controller
Smart Array 5i Controller
Smart Array 641 Controller
Smart Array 642 Controller
Smart Array 6400 Controller
Smart Array 6400 EM Controller
Smart Array 6i Controller
Smart Array P600 Controller
Smart Array P400 Controller
Smart Array P400i Controller
Smart Array E200 Controller
Smart Array E200i Controller
Smart Array P800 Controller
Smart Array E500 Controller
Smart Array P700m Contoller
Smart Array P410i Controller
Smart Array P411 Controller
Smart Array P212 Controller
Smart Array P712m Contoller
Smart Array B110i SATA RAID
Smart Array P812 Controller
MSA500 Controller
MSA500 G2 Controller
MSA1000 Controller
MSA1500 CS Controller
MSA20 Controller

the tool is supplied on HP Support Pack CDs, if you've got them; but you can download a newer version from HP here; this links to hpacucli-8.50-6.0.noarch.rpm.
after downloading, we need to convert the rpm into a format that we can work with. alien does this for us in ubuntu... other tools are rpm2cpio & rpm2tgz. i like alien. apt-get it.

# alien --to-tgz hpacucli-8.50-6.0.noarch.rpm

alien will report some errors and warnings; in your source directory, you'll see hpacucli-8.50.tgz.

# tar -xzf hpacucli-8.50.tgz

Move the unpacked files to corresponding locations:

# mv opt/compaq /opt/
# mv usr/sbin/* /usr/sbin/

since i'm running an x86_64 box, i need to:

# apt-get install ia32-libs

hpacucli should run. it does.

# hpacucli
=> ctrl all show      

Smart Array P410i in Slot 0 (Embedded)    (sn: 5001438017EA3640)

we have a RAID controller in Slot 0. Good to know.

=> ctrl all show detail      

Smart Array P410i in Slot 0 (Embedded)
   Bus Interface: PCI
   Slot: 0
   Serial Number: xxxxxxxxxxxxxxxxx
   Cache Serial Number: xxxxxxxxxxxxxxxxx
   RAID 6 (ADG) Status: Disabled
   Controller Status: OK
   Chassis Slot: 
   Hardware Revision: Rev C
   Firmware Version: 5.14
   Rebuild Priority: Medium
   Expand Priority: Medium
   Surface Scan Delay: 3 secs
   Queue Depth: Automatic
   Monitor and Performance Delay: 60 min
   Elevator Sort: Enabled
   Degraded Performance Optimization: Disabled
   Inconsistency Repair Policy: Disabled
   Wait for Cache Room: Disabled
   Surface Analysis Inconsistency Notification: Disabled
   Post Prompt Timeout: 15 secs
   Cache Board Present: True
   Cache Status: Not Configured
   Accelerator Ratio: 100% Read / 0% Write
   Read Cache Size: 0 MB
   Write Cache Size: 0 MB
   Drive Write Cache: Disabled
   Total Cache Size: 912 MB
   No-Battery Write Cache: Disabled
   Cache Backup Power Source: Capacitors
   Battery/Capacitor Count: 1
   Battery/Capacitor Status: OK
   SATA NCQ Supported: True

   Array: A
      Interface Type: SAS
      Unused Space: 0 MB
      Status: OK

      Logical Drive: 1
         Size: 1.6 TB
         Fault Tolerance: RAID 5
         Heads: 255
         Sectors Per Track: 63
         Cylinders: 65535
         Stripe Size: 256 KB
         Status: OK
         Array Accelerator: Not Configured
         Parity Initialization Status: Initialization Completed
         Unique Identifier: 600508B1001C5D95C9C5A46D895F6036
         Disk Name: /dev/cciss/c0d0
         Mount Points: /boot 243 MB
         OS Status: LOCKED
         Logical Drive Label: AE8582015001438017EA36402B33

      physicaldrive 1I:1:1
         Port: 1I
         Box: 1
         Bay: 1
         Status: OK
         Drive Type: Data Drive
         Interface Type: SAS
         Size: 300 GB
         Rotational Speed: 10000
         Firmware Revision: HPD4
         Serial Number: xxxxxxxxxxxxxxxxx
         Model: HP      EG0300FBDSP     
         PHY Count: 2
         PHY Transfer Rate: 6.0GBPS, Unknown

well, it looks like write caching is not enabled. great.

=> ctrl slot=0 modify dwc=enable
=> ctrl slot=0 modify cacheratio=25/75
=> ctrl slot=0 logicaldrive 1 modify aa=enable
=> ctrl all show config detail

Smart Array P410i in Slot 0 (Embedded)
   Bus Interface: PCI
   Slot: 0
   Serial Number: xxxxxxxxxxxxxxxxx
   Cache Serial Number: xxxxxxxxxxxxxxxxx
   RAID 6 (ADG) Status: Disabled
   Controller Status: OK
   Chassis Slot: 
   Hardware Revision: Rev C
   Firmware Version: 5.14
   Rebuild Priority: Medium
   Expand Priority: Medium
   Surface Scan Delay: 3 secs
   Queue Depth: Automatic
   Monitor and Performance Delay: 60 min
   Elevator Sort: Enabled
   Degraded Performance Optimization: Disabled
   Inconsistency Repair Policy: Disabled
   Wait for Cache Room: Disabled
   Surface Analysis Inconsistency Notification: Disabled
   Post Prompt Timeout: 15 secs
   Cache Board Present: True
   Cache Status: Not Configured
   Accelerator Ratio: 100% Read / 0% Write
   Read Cache Size: 0 MB
   Write Cache Size: 0 MB
   Drive Write Cache: Enabled
   Total Cache Size: 912 MB
   No-Battery Write Cache: Disabled
   Cache Backup Power Source: Capacitors
   Battery/Capacitor Count: 1
   Battery/Capacitor Status: OK
   SATA NCQ Supported: True

   Array: A
      Interface Type: SAS
      Unused Space: 0 MB
      Status: OK

      Logical Drive: 1
         Size: 1.6 TB
         Fault Tolerance: RAID 5
         Heads: 255
         Sectors Per Track: 63
         Cylinders: 65535
         Stripe Size: 256 KB
         Status: OK
         Array Accelerator: Not Configured
         Parity Initialization Status: Initialization Completed
         Unique Identifier: 600508B1001C5D95C9C5A46D895F6036
         Disk Name: /dev/cciss/c0d0
         Mount Points: /boot 243 MB
         OS Status: LOCKED
         Logical Drive Label: AE8582015001438017EA36402B33

      physicaldrive 1I:1:1
         Port: 1I
         Box: 1
         Bay: 1
         Status: OK
         Drive Type: Data Drive
         Interface Type: SAS
         Size: 300 GB
         Rotational Speed: 10000
         Firmware Revision: HPD4
         Serial Number: xxxxxxxxxxxxxxxxx
         Model: HP      EG0300FBDSP     
         PHY Count: 2
         PHY Transfer Rate: 6.0GBPS, Unknown

crap. it didn't update. or did it?

=> exit

apparently this is a bug. or a feature. we need to exit the utility, and then start it up again for the changes to be reflected. of course.

# hpacucli
=> ctrl all show config detail

Smart Array P410i in Slot 0 (Embedded)
   Bus Interface: PCI
   Slot: 0
   Serial Number: xxxxxxxxxxxxxxxxx
   Cache Serial Number: xxxxxxxxxxxxxxxxx
   RAID 6 (ADG) Status: Disabled
   Controller Status: OK
   Chassis Slot: 
   Hardware Revision: Rev C
   Firmware Version: 5.14
   Rebuild Priority: Medium
   Expand Priority: Medium
   Surface Scan Delay: 3 secs
   Queue Depth: Automatic
   Monitor and Performance Delay: 60 min
   Elevator Sort: Enabled
   Degraded Performance Optimization: Disabled
   Inconsistency Repair Policy: Disabled
   Wait for Cache Room: Disabled
   Surface Analysis Inconsistency Notification: Disabled
   Post Prompt Timeout: 15 secs
   Cache Board Present: True
   Cache Status: OK
   Accelerator Ratio: 25% Read / 75% Write
   Drive Write Cache: Enabled
   Total Cache Size: 1024 MB
   No-Battery Write Cache: Disabled
   Cache Backup Power Source: Capacitors
   Battery/Capacitor Count: 1
   Battery/Capacitor Status: OK
   SATA NCQ Supported: True

   Array: A
      Interface Type: SAS
      Unused Space: 0 MB
      Status: OK

      Logical Drive: 1
         Size: 1.6 TB
         Fault Tolerance: RAID 5
         Heads: 255
         Sectors Per Track: 63
         Cylinders: 65535
         Stripe Size: 256 KB
         Status: OK
         Array Accelerator: Enabled
         Parity Initialization Status: Initialization Completed
         Unique Identifier: 600508B1001C5D95C9C5A46D895F6036
         Disk Name: /dev/cciss/c0d0
         Mount Points: /boot 243 MB
         OS Status: LOCKED
         Logical Drive Label: AE8582015001438017EA36402B33

and of course, we want to see that our once dismal performance is not so. download and run iozone. and then test throughput:

# iozone -t4 -I

note:
everyone says, BBWC is a must have. but, as seen on this capacitor-backed up cache, all is cool. we're running with FBWC.
FBWC is a flash based cache module that does not have the battery limitation of how long it can retain what is written to the module.

addendum:
i got ahead of myself with nagios. use the plugin here to monitor the state of the array.

and...
if someone else is doing something funny and manages to crash your new friend... you'll need to clean up...

Error: Another instance of ACU is already running (possibly a service). Please
terminate the ACU application before running the ACU CLI. Press ENTER to
exit.

delete the shared IPC that hpacucli left when it died.

# ipcs

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      

------ Semaphore Arrays --------
key        semid      owner      perms      nsems     
0xffffffff 56890      root       0          1         

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages    
Then use ipcrm to remove the array with the semid you want:
# ipcrm -s 56890

and try to start hpacucli again.

postgresql and tmp

"Because PostgreSQL writes the write-ahead log to disk on every transaction commit using fsync(), and waits for that write to complete, users will see a huge performance boost if a write cache is used. Therefore, for performance and reliability, it is ideal if PostgreSQL can use a battery-backed write cache."

Moreover, Postgres recommends that if you are using RAID5, you should mount your /tmp dir on a spare drive if you have one.

But what do you do if you don't have a spare drive? And you're using everything for your RAID5 array?

mount /tmp to a 2G ram disk. of course.

Let's do it!

With any install, /tmp is usually always there. Usually. And since we're dealing with a DB, we want the data to be around, like just in case.

# mkdir /tmp <---- if it isn't there already.

Check and see if anyone is using /tmp ; if these are crucial daemons; I'd suggest stopping them.

Add this line to /etc/fstab in to mount the drive at boot-time:

tmpfs           /tmp tmpfs      defaults,size=2048M 0 0

tmpfs, by virtue of being tmpfs doesn't allocate all of that space in one go; only as needed. tmpfs is alright using up to half of your available RAM; use free -m to figure it out. I guess it is also worth mentioning that you do not need to recreate tmpfs each time the system is rebooted; it will auto-create between boots due to it being tmpfs.

That being said, mount the new filesystem after adding its entry in /etc/fstab.

# mount /tmp

Check to see that it's mounted

# mount
# df -h

You should see the following in mount and df -h output:

tmpfs on /tmp type tmpfs (rw,relatime,size=2097152k)
tmpfs         2.0G  0.0G  2.0G   0% /tmp

Next we need to create a directory to store the backup copies of whatever we've got in /tmp. /var is as good a place as any.

# mkdir /var/tmp-bak

Create script /etc/init.d/tmp-bak:

#! /bin/sh 
# /etc/init.d/tmp-bak
#
 
case "$1" in
  start)
    echo "copying files to tmp-bak"
    rsync -av /var/tmp-bak/ /tmp/
    echo [`date +"%Y-%m-%d %H:%M"`] tmp synched >> /var/log/tmp-bak_sync.log
    ;;
  sync)
    echo "synching files from tmp to tmp-bak"
    echo [`date +"%Y-%m-%d %H:%M"`] tmp synched to tmp-bak >> /var/log/tmp-bak_sync.log
    rsync -av --delete --recursive --force /tmp/ /var/tmp-bak/
    ;;
  stop)
    echo "synching files from tmp to tmp-bak"
    echo [`date +"%Y-%m-%d %H:%M"`] tmp synched to tmp-bak >> /var/log/ramdisk_sync.log
    rsync -av --delete --recursive --force /tmp/ /var/tmp-bak/
    ;;
  *)
    echo "Usage: /etc/init.d/tmp-bak {start|stop|sync}"
    exit 1
    ;;
esac

exit 0

Now set tmp-bak to run at startup:

# update-rc.d tmp-bak defaults 00 99

As a good rule of thumb, place the sync process in /etc/crontab:

5 * * * * root        /etc/init.d/tmp-bak sync >> /dev/null 2>&1

Friday, February 24, 2012

cidr cheetsheet

sometimes you need to know a cidr mask. sometimes.

Netmask              Netmask (binary)                 CIDR     Notes
_____________________________________________________________________________
255.255.255.255  11111111.11111111.11111111.11111111  /32  Host (single addr)
255.255.255.254  11111111.11111111.11111111.11111110  /31  Unuseable
255.255.255.252  11111111.11111111.11111111.11111100  /30    2  useable
255.255.255.248  11111111.11111111.11111111.11111000  /29    6  useable
255.255.255.240  11111111.11111111.11111111.11110000  /28   14  useable
255.255.255.224  11111111.11111111.11111111.11100000  /27   30  useable
255.255.255.192  11111111.11111111.11111111.11000000  /26   62  useable
255.255.255.128  11111111.11111111.11111111.10000000  /25  126  useable
255.255.255.0    11111111.11111111.11111111.00000000  /24 "Class C" 254 useable

255.255.254.0    11111111.11111111.11111110.00000000  /23    2  Class C's
255.255.253.0                                                3  Class C's
255.255.252.0    11111111.11111111.11111100.00000000  /22    4  Class C's
255.255.251.0                                                5  Class C's
255.255.250.0                                                6  Class C's
255.255.249.0                                                7  Class C's
255.255.248.0    11111111.11111111.11111000.00000000  /21    8  Class C's
255.255.240.0    11111111.11111111.11110000.00000000  /20   16  Class C's
255.255.224.0    11111111.11111111.11100000.00000000  /19   32  Class C's
255.255.192.0    11111111.11111111.11000000.00000000  /18   64  Class C's
255.255.128.0    11111111.11111111.10000000.00000000  /17  128  Class C's
255.255.0.0      11111111.11111111.00000000.00000000  /16  "Class B"

255.254.0.0      11111111.11111110.00000000.00000000  /15    2  Class B's
255.252.0.0      11111111.11111100.00000000.00000000  /14    4  Class B's
255.248.0.0      11111111.11111000.00000000.00000000  /13    8  Class B's
255.240.0.0      11111111.11110000.00000000.00000000  /12   16  Class B's
255.224.0.0      11111111.11100000.00000000.00000000  /11   32  Class B's
255.192.0.0      11111111.11000000.00000000.00000000  /10   64  Class B's
255.128.0.0      11111111.10000000.00000000.00000000  /9   128  Class B's
255.0.0.0        11111111.00000000.00000000.00000000  /8   "Class A"

254.0.0.0        11111110.00000000.00000000.00000000  /7
252.0.0.0        11111100.00000000.00000000.00000000  /6
248.0.0.0        11111000.00000000.00000000.00000000  /5
240.0.0.0        11110000.00000000.00000000.00000000  /4
224.0.0.0        11100000.00000000.00000000.00000000  /3
192.0.0.0        11000000.00000000.00000000.00000000  /2
128.0.0.0        10000000.00000000.00000000.00000000  /1
0.0.0.0          00000000.00000000.00000000.00000000  /0   IP space

                                   Net     Host    Total
Net      Addr                      Addr    Addr    Number
Class   Range      NetMask         Bits    Bits   of hosts
----------------------------------------------------------
A        0-127    255.0.0.0         8      24     16777216   (i.e. 114.0.0.0)
B      128-191    255.255.0.0      16      16        65536   (i.e. 150.0.0.0)
C      192-254    255.255.255.0    24       8          256   (i.e. 199.0.0.0)
D      224-239    (multicast)
E      240-255    (reserved)
F      208-215    255.255.255.240  28       4           16
G      216/8      ARIN - North America
G      217/8      RIPE NCC - Europe
G      218-219/8  APNIC
H      220-221    255.255.255.248  29       3            8   (reserved)
K      222-223    255.255.255.254  31       1            2   (reserved)

ref: RFC1375 & http://www.iana.org/assignments/ipv4-address-space
               http://www.iana.org/numbers.htm

----------------------------------------------------------

The current list of special use prefixes:
 0.0.0.0/8
 127.0.0.0/8
 192.0.2.0/24
 10.0.0.0/8
 172.16.0.0/12
 192.168.0.0/16
 169.254.0.0/16
 all D/E space

ref: RFC1918 http://www.rfc-editor.org/rfc/rfc1918.txt
       or     ftp://ftp.isi.edu/in-notes/rfc1918.txt
rfc search:   http://www.rfc-editor.org/rfcsearch.html
              http://www.ietf.org/ietf/1id-abstracts.txt
              http://www.ietf.org/shadow.html

Thursday, February 23, 2012

how's nfs' latency today?

let's check it out with tshark & iostat.

$ tshark -q -z rpc,rtt,100003,3,'nfs.nfsstat3!=70'

or... put something is a pcap file.

$ tshark -nlr nfs.pcap -R "rpc.time>0.5"

or... you can use iostat.

# iostat -x -n

Tuesday, February 7, 2012

my teeth chattr

need i say more?

#!/bin/sh

# changes ext2 or ext3 file attributes

for file in resolv.conf passwd shadow group motd hosts hostname
do
    if [ "$1" == "" ]    ; then lsattr    /etc/$file ; fi
    if [ "$1" == "on" ]  ; then chattr +i /etc/$file ; fi
    if [ "$1" == "off" ] ; then chattr -i /etc/$file ; fi
done

Thursday, February 2, 2012

macos 10.7.2 dmg to iso

sigh. you need to disk made from a dmg that you can ferry around, say to a xen box. turn it into an iso and away you go.

# hdiutil convert your.dmg -format UDTO -o your.iso
# mv your.iso.cdr your.iso

you could do this through Disk Utility, but the command line is always better.

Wednesday, February 1, 2012

solaris 9 notes

you see:
snmpXdmid: Error in Adding Row for Subscription Table Entry

Disable it...

   cd /etc/rc3.d
   ./S76snmpdx stop
   ./S77dmi stop
   mv S76snmpdx s76snmpdx
   mv S77dmi s77dmi