Thursday, August 30, 2007

WMI and "Permission denied"

Today we were trying to use vbscript and WMI to audit the software installed on PCs connected to our network. The PC's were running Windows XP Pro, and were members of an Active Directory Domain.

The script worked fine in most cases, but we occasionally got errors like these:

Error 462 on GetObject()
"The remote server machine does not exist or is unavailable"

Error 70 on GetObject()
"Permission denied"


The line of code in question was:
Set objReg = GetObject("winmgmts:{impersonationLevel=impersonate,authenticationLevel=Pkt}!//" & strComputer & sDNS & "/root/default:StdRegProv")


To cut a long story short, we used WireShark to capture the conversation between the source and target PC's.

Here is a RPC packet sent from the source to the target PC.
The target PC is "PC-ONE-LAP".

DCE RPC Bind, Fragment: Single, FragLen: 1440, Call: 2
Version: 5
Version (minor): 0
Packet type: Bind (11)
Packet Flags: 0x03
Data Representation: 10000000
Frag Length: 1440
Auth Length: 1360
Call ID: 2
Max Xmit Frag: 5840
Max Recv Frag: 5840
Assoc Group: 0x0000b254
Num Ctx Items: 1
Ctx Item[1]: ID:1
Context ID: 1
Num Trans Items: 1
Abstract Syntax: ISystemActivator V0.0
Interface: ISystemActivator UUID: 000001a0-0000-0000-c000-000000000046
Interface Ver: 0
Interface Ver Minor: 0
Transfer Syntax[1]: 8a885d04-1ceb-11c9-9fe8-08002b104860 V2
Auth type: SPNEGO (9)
Auth level: Connect (2)
Auth pad len: 0
Auth Rsrvd: 0
Auth Context ID: 1277840
GSS-API Generic Security Service Application Program Interface
OID: 1.3.6.1.5.5.2 (SPNEGO - Simple Protected Negotiation)
SPNEGO
negTokenInit
mechTypes: 3 items
Item: 1.2.840.48018.1.2.2 (MS KRB5 - Microsoft Kerberos 5)
Item: 1.2.840.113554.1.2.2 (KRB5 - Kerberos 5)
Item: 1.3.6.1.4.1.311.2.2.10 (NTLMSSP - Microsoft NTLM Security Support Provider)
mechToken: 6E82050A30820506A003020105A10302010EA20703050020...
krb5_blob: 6E82050A30820506A003020105A10302010EA20703050020...
Kerberos AP-REQ
Pvno: 5
MSG Type: AP-REQ (14)
Padding: 0
APOptions: 20000000 (Mutual required)
Ticket
Tkt-vno: 5
Realm: COMPANY.NET
Server Name (Service and Instance): RPCSS/PC-ONE-LAP.company.net
enc-part rc4-hmac
Authenticator rc4-hmac


You can see it is using kerberos to authenticate.
Here is the response from the target PC:

DCE RPC Bind_ack, Fragment: Single, FragLen: 199, Call: 2
Version: 5
Version (minor): 0
Packet type: Bind_ack (12)
Packet Flags: 0x03
Data Representation: 10000000
Frag Length: 199
Auth Length: 131
Call ID: 2
Max Xmit Frag: 5840
Max Recv Frag: 5840
Assoc Group: 0x0000b254
Scndry Addr len: 4
Scndry Addr: 135
Num results: 1
Context ID[1]
Auth type: SPNEGO (9)
Auth level: Connect (2)
Auth pad len: 0
Auth Rsrvd: 0
Auth Context ID: 1277840
GSS-API Generic Security Service Application Program Interface
SPNEGO
negTokenTarg
negResult: accept-incomplete (1)
supportedMech: 1.2.840.48018.1.2.2 (MS KRB5 - Microsoft Kerberos 5)
responseToken: 606606092A864886F71201020203007E573055A003020105...
krb5_blob: 606606092A864886F71201020203007E573055A003020105...
KRB5 OID: 1.2.840.113554.1.2.2 (KRB5 - Kerberos 5)
krb5_tok_id: KRB5_ERROR (0x0003)
Kerberos KRB-ERROR
Pvno: 5
MSG Type: KRB-ERROR (30)
stime: 2007-08-30 10:19:06 (Z)
susec: 851504
error_code: KRB5KRB_AP_ERR_MODIFIED (41)
Realm: COMPANY.NET
Server Name (Principal): PC-TWO-LAP$

The response show an Kerberos error "KRB5KRB_AP_ERR_MODIFIED", as it is basically saying - hold on, my name is "PC-TWO-LAP" and not "PC-ONE-LAP".

In the System Event Log for the source PC, you will see the following:
Event Type: Error
Event Source: DCOM
Event Category: None
Event ID: 10009
Date: 30/08/2007
Time: 10:22:52
User: COMPANY\USER
Computer: SOURCEPC
Description:
DCOM was unable to communicate with the computer PC-ONE-LAP.company.net
using any of the configured protocols.

Event Type: Error
Event Source: Kerberos
Event Category: None
Event ID: 4
Date: 30/08/2007
Time: 10:22:52
User: N/A
Computer: SOURCEPC
Description:
The kerberos client received a KRB_AP_ERR_MODIFIED error from the
server PC-TWO-LAP$. This indicates that the password used to encrypt
the kerberos service ticket is different than that on the target server.
Commonly, this is due to identically named machine accounts in the
target realm (COMPANY.NET), and the client realm.

The explanation is that the DNS server was returning the same IP address for both "PC-ONE-LAP" and "PC-TWO-LAP". Only "PC-TWO-LAP" was actually connected to the network, but when we tried "PC-ONE-LAP", it was actually "PC-TWO-LAP" that responded.

Kerberos authenticates both the User and the Computer, so that is why we got the error. If you get this error, then check your DNS records.

For other similar problems you may find this link useful:
Failed to get remote resources: Remote server is unavailable. The RPC server is unavailable.

Monday, August 27, 2007

A problem with wget on OpenSolaris

Ok, here is a stupid little problem.I waisted an hour trying to figure this one out. Maybe this will help someone to avoid the same mistake!

I was using 'wget' to try to download a file from a http server onto my OpenSolaris PC. Here is what I did:
# uname -a
SunOS solaris 5.11 snv_60 i86pc i386 i86pc
# wget -V
GNU Wget 1.10.2
# cd /home
# /usr/sfw/bin/wget http://www.nwsmith.net/index.htm
--18:31:13-- http://www.nwsmith.net/index.htm
=> `index.htm'
Resolving www.nwsmith.net... NNN.NNN.NNN.NNN
Connecting to www.nwsmith.net|NNN.NNN.NNN.NNN|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 11,774 (11K) [text/html]
index.htm: Operation not applicable
Cannot write to `index.htm' (Operation not applicable).

Did you spot my mistake?
I must have thought I was using a Linux PC, because then the '/home' directory would have been fine. But on Solaris...
# ls -ld /home
dr-xr-xr-x 1 root root 1 Apr 18 00:03 /home

...you cannot write to that directory.
Choose a directory that is writeable, and then wget will work without error.

Thursday, August 16, 2007

fdisk, sfdisk, mdadm and SCSI hard drive geometry

At work, one of our servers, uses Linux software RAIDand we have two mirrored hard drives setup as RAID1.
Smartmontools reported that one of the hard drives was starting to fail:
# smartctl -a /dev/sda
SMART Health Status: LOGICAL UNIT FAILURE PREDICTION THRESHOLD EXCEEDED [asc=5d,ascq=2]
# smartctl -l selftest /dev/sda
SMART Self-test log
Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ]
Description number (hours)
# 1 Background long Failed in segment --> 2 10572 0x 22f0e6c [0x4 0x40 0x85]
# 2 Background long Failed in segment --> 2 10404 0x 22f0e6c [0x3 0x11 0x0]
# 3 Background long Failed in segment --> 2 10236 0x 22f0e63 [0x4 0x40 0x85]

So we executed the following commands:
mdadm --manage /dev/md0 --set-faulty  /dev/sda1
mdadm --manage /dev/md0 --remove /dev/sda1
mdadm --manage /dev/md1 --set-faulty /dev/sda2
mdadm --manage /dev/md1 --remove /dev/sda2
mdadm --manage /dev/md2 --set-faulty /dev/sda3
mdadm --manage /dev/md2 --remove /dev/sda3
mdadm --manage /dev/md3 --set-faulty /dev/sda5
mdadm --manage /dev/md3 --remove /dev/sda5

And then hot un-plugged the drive.
When the replacement drive arrived, although it was an identical model
to the original, and had an identical total size, the new drive had a different geometry.
[root@ifsclstr02 ~]# smartctl -i /dev/sda
Device: IBM-ESXS BBD036C3ESTT0ZFN Version: JP86
Device type: disk
Transport protocol: Parallel SCSI (SPI-4)

[root@ifsclstr02 ~]# smartctl -i /dev/sdb
Device: IBM-ESXS BBD036C3ESTT0ZFN Version: JP85
Device type: disk
Transport protocol: Parallel SCSI (SPI-4)

# fdisk -l /dev/sda
Disk /dev/sda: 36.4 GB, 36401479680 bytes
64 heads, 32 sectors/track, 34715 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

# fdisk -l /dev/sdb
Disk /dev/sdb: 36.4 GB, 36401479680 bytes
255 heads, 63 sectors/track, 4425 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

The potential problem was that if we had to specify the start and end of the partition in terms of cylinders, then we would not be able to get an exact match in the size of the partitions between the new disk and the existing working half of the mirror.
After some googling, we concluded that having to align the partition boundaries with the cylinders was a DOS legacy issue, and was not something that would cause a problem for Linux.
So to copy the partitions from the working disk to the new disk we used the following:
# sfdisk -d /dev/sdb | sfdisk --Linux /dev/sda
Checking that no-one is using this disk right now ...
OK
Disk /dev/sda: 34715 cylinders, 64 heads, 32 sectors/track
Old situation:
Units = cylinders of 1048576 bytes, blocks of 1024 bytes, counting from 0

Device Boot Start End #cyls #blocks Id System
/dev/sda1 0 - 0 0 0 Empty
/dev/sda2 0 - 0 0 0 Empty
/dev/sda3 0 - 0 0 0 Empty
/dev/sda4 0 - 0 0 0 Empty
New situation:
Units = sectors of 512 bytes, counting from 0

Device Boot Start End #sectors Id System
/dev/sda1 * 63 208844 208782 fd Linux raid autodetect
/dev/sda2 208845 16980704 16771860 fd Linux raid autodetect
/dev/sda3 16980705 25366634 8385930 fd Linux raid autodetect
/dev/sda4 25366635 71087624 45720990 5 Extended
/dev/sda5 25366698 71087624 45720927 fd Linux raid autodetect
Warning: partition 1 does not end at a cylinder boundary
Warning: partition 2 does not start at a cylinder boundary
Warning: partition 2 does not end at a cylinder boundary
Warning: partition 3 does not start at a cylinder boundary
Warning: partition 3 does not end at a cylinder boundary
Warning: partition 4 does not start at a cylinder boundary
Warning: partition 4 does not end at a cylinder boundary
Warning: partition 5 does not end at a cylinder boundary
Successfully wrote the new partition table
Re-reading the partition table ...
# mdadm --manage /dev/md0 --add /dev/sda1
# mdadm --manage /dev/md1 --add /dev/sda2
# mdadm --manage /dev/md2 --add /dev/sda3
# mdadm --manage /dev/md3 --add /dev/sda5
# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sda2[0] sdb2[1]
8385856 blocks [2/2] [UU]

md2 : active raid1 sda3[2] sdb3[1]
4192896 blocks [2/1] [_U]
resync=DELAYED
md3 : active raid1 sda5[2] sdb5[1]
22860352 blocks [2/1] [_U]
[============>........] recovery = 61.8% (14148928/22860352) finish=2.1min speed=67707K/sec
md0 : active raid1 sda1[0] sdb1[1]
104320 blocks [2/2] [UU]

unused devices:

Using the sfdisk command, you can specify the unit of measure when listing the partition table. Use '-uS' for Sectors and '-uC' for Cylinders:
# sfdisk -l -uS /dev/sda
Disk /dev/sda: 34715 cylinders, 64 heads, 32 sectors/track
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Units = sectors of 512 bytes, counting from 0

Device Boot Start End #sectors Id System
/dev/sda1 * 63 208844 208782 fd Linux raid autodetect
/dev/sda2 208845 16980704 16771860 fd Linux raid autodetect
/dev/sda3 16980705 25366634 8385930 fd Linux raid autodetect
/dev/sda4 25366635 71087624 45720990 5 Extended
/dev/sda5 25366698 71087624 45720927 fd Linux raid autodetect

# sfdisk -l -uS /dev/sdb
Disk /dev/sdb: 4425 cylinders, 255 heads, 63 sectors/track
Units = sectors of 512 bytes, counting from 0

Device Boot Start End #sectors Id System
/dev/sdb1 * 63 208844 208782 fd Linux raid autodetect
/dev/sdb2 208845 16980704 16771860 fd Linux raid autodetect
/dev/sdb3 16980705 25366634 8385930 fd Linux raid autodetect
/dev/sdb4 25366635 71087624 45720990 5 Extended
/dev/sdb5 25366698 71087624 45720927 fd Linux raid autodetect

It's only when you think in terms of cylinders, that there appears to be a problem:
# sfdisk -l -uC /dev/sda
Disk /dev/sda: 34715 cylinders, 64 heads, 32 sectors/track
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Units = cylinders of 1048576 bytes, blocks of 1024 bytes, counting from 0

Device Boot Start End #cyls #blocks Id System
/dev/sda1 * 0+ 101- 102- 104391 fd Linux raid autodetect
/dev/sda2 101+ 8291- 8190- 8385930 fd Linux raid autodetect
/dev/sda3 8291+ 12386- 4095- 4192965 fd Linux raid autodetect
/dev/sda4 12386+ 34710- 22325- 22860495 5 Extended
/dev/sda5 12386+ 34710- 22325- 22860463+ fd Linux raid autodetect

# sfdisk -l -uC /dev/sdb
Disk /dev/sdb: 4425 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

Device Boot Start End #cyls #blocks Id System
/dev/sdb1 * 0+ 12 13- 104391 fd Linux raid autodetect
/dev/sdb2 13 1056 1044 8385930 fd Linux raid autodetect
/dev/sdb3 1057 1578 522 4192965 fd Linux raid autodetect
/dev/sdb4 1579 4424 2846 22860495 5 Extended
/dev/sdb5 1579+ 4424 2846- 22860463+ fd Linux raid autodetect

The pluses and minuses, just mean that the numbers are not exact and are rounded up or down.
For comparison, here is what fdisk reports:
# fdisk -l /dev/sda
Disk /dev/sda: 36.4 GB, 36401479680 bytes
64 heads, 32 sectors/track, 34715 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 102 104391 fd Linux raid autodetect
Partition 1 does not end on cylinder boundary.
/dev/sda2 102 8292 8385930 fd Linux raid autodetect
/dev/sda3 8292 12387 4192965 fd Linux raid autodetect
/dev/sda4 12387 34711 22860495 5 Extended
/dev/sda5 12387 34711 22860463+ fd Linux raid autodetect

# fdisk -l /dev/sdb
Disk /dev/sdb: 36.4 GB, 36401479680 bytes
255 heads, 63 sectors/track, 4425 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 * 1 13 104391 fd Linux raid autodetect
/dev/sdb2 14 1057 8385930 fd Linux raid autodetect
/dev/sdb3 1058 1579 4192965 fd Linux raid autodetect
/dev/sdb4 1580 4425 22860495 5 Extended
/dev/sdb5 1580 4425 22860463+ fd Linux raid autodetect

Wednesday, August 08, 2007

Smartmontools and fixing Unreadable Disk Sectors

Smartmontools was showing some problems on the disk.
At least two bad LBAs:
# smartctl -l selftest /dev/hda
smartctl version 5.36 [i686-pc-linux-gnu] Copyright (C) 2002-6 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed: read failure 20% 1596 44724966
# 2 Extended offline Completed: read failure 40% 1519 12622427

# smartctl -A /dev/hda | egrep 'Reallocated|Pending|Uncorrectable'
5 Reallocated_Sector_Ct 0x0033 253 253 063 Pre-fail Always - 2
196 Reallocated_Event_Count 0x0008 252 252 000 Old_age Offline - 1
197 Current_Pending_Sector 0x0008 253 253 000 Old_age Offline - 2
198 Offline_Uncorrectable 0x0008 252 252 000 Old_age Offline - 1

I found a document:
"SHOWS HOW TO IDENTIFY THE FILE ASSOCIATED
WITH AN UNREADABLE DISK SECTOR, AND HOW TO
FORCE THAT SECTOR TO REALLOCATE."

and followed the procedure.

Note the LBA values are given as decimal values. The document seems to refer
to an older version of smarctl that gives the LBA as a hexadecimal number.

Lets look at the partition sizes to see where this LBA drops in.
# fdisk -lu /dev/hda

Disk /dev/hda: 255 heads, 63 sectors, 3738 cylinders
Units = sectors of 1 * 512 bytes

Device Boot Start End Blocks Id System
/dev/hda1 * 63 160649 80293+ 83 Linux
/dev/hda2 160650 1204874 522112+ 82 Linux swap
/dev/hda3 1204875 53737424 26266275 83 Linux
/dev/hda4 53737425 60050969 3156772+ f Win95 Ext'd (LBA)
/dev/hda5 53737488 55841939 1052226 83 Linux
/dev/hda6 55842003 57946454 1052226 83 Linux
/dev/hda7 57946518 60050969 1052226 83 Linux

Ok, so the problem is in '/dev/hda3 '.
What's mounted there? As the partitions are labeled, we need to use:
# grep `e2label /dev/hda3` /etc/fstab
LABEL=/var /var ext3 defaults 1 2

Ok, so the problem is in '/var'.
# tune2fs -l /dev/hda3 | grep Block
Block count: 6566568
Block size: 4096

Let's do the maths:
LBA 12622427 - 1204875 and multiply by (512/4096) equals 1427194.
LBA 44724966 - 1204875 and multiply by (512/4096) equals 5440011.375

Ok, now let's use 'debugfs':
# debugfs
debugfs 1.27 (8-Mar-2002)
debugfs: open /dev/hda3
debugfs: icheck 1427194
Block Inode number
1427194 526482
debugfs: ncheck 526482
Inode Pathname
526482 /log/ntp/peers.20070717
debugfs: icheck 5440011
icheck: Can't read next inode while doing inode scan
debugfs: quit

So that means LBA 12622427 is in file "/var/log/ntp/peers.20070717".
And it looks like LBA 44724966 is in currently unused space on the disk.

As this file is not critical, I will just overwrite part of it
to force it to be reallocated:
# dd if=/dev/zero of=/dev/hda3 bs=4096 count=1 seek=1427194
1+0 records in
1+0 records out
# sync
# smartctl -A /dev/hda | egrep 'Reallocated|Pending|Uncorrectable'
5 Reallocated_Sector_Ct 0x0033 253 253 063 Pre-fail Always - 1
196 Reallocated_Event_Count 0x0008 252 252 000 Old_age Offline - 1
197 Current_Pending_Sector 0x0008 253 253 000 Old_age Offline - 1
198 Offline_Uncorrectable 0x0008 252 252 000 Old_age Offline - 1

Ok, that seems to have made that error go away for the time being.

Then while googling I found this perl script to help with
automation of badblocks on Linux:

"smartfixdisk - assistant that helps to repair bad LBAs detected by Smartmontools"

developed by the "IT-Support-Group" (ISG.EE),
which is a service organisation of the
"Department of Information Technology and Electrical Engineering" (D-ITET)
of the "Swiss Federal Institute of Technology", Zurich.

I wanted to use it on a old RedHat 9 server.
The script immediately fell over on this line:
open(DISKEND,"</sys/block/$diskname/size") or die "$!";

Not too surprising, as the '/sys' does not exist on my old server!
It seems to be a feature of newer kernels.
On a Centos-5 box I tried this:
# cat /proc/ide/hda/capacity
78165360
# cat /sys/block/hda/size
78165360
# cat /proc/ide/hda/geometry
physical 16383/16/63
logical 65535/16/63

So DISKEND seems to be related to the number of sectors on the hard drive.
The '/proc' version was available on the old Redhat 9 server,
so I change the perl code line like this:
open(DISKEND,"</proc/ide/$diskname/capacity") or die "$!";

..and it was happy.
To figure out what the script is doing, it's useful to add in a few
'print' commands into the script, or just uncomment the ones that
already in place. Here's what the script told me on this server:
# ./smartfixdisk.pl --noaction /dev/hda
Block size = 4096, factor = 0.125
Searching for inode... this may take a while...

LBA 12622427
Partition and partition type: /dev/hda3 Linux_Ext2
Status: used
Comment: EXT2/3: File found at inode 526482: /log/ntp/peers.20070717

LBA 44724966
Partition and partition type: /dev/hda3 Linux_Ext2
Status: free
Comment: block not used in filesystem
dd if=/dev/zero of=/dev/hda seek=5590620 bs=4096 count=1 conv=sync

Looks good.
Ok, so let's finish off:
# dd if=/dev/zero of=/dev/hda seek=5590620 bs=4096 count=1 conv=sync
1+0 records in
1+0 records out
# sync
# smartctl -A /dev/hda | egrep 'Reallocated|Pending|Uncorrectable'
5 Reallocated_Sector_Ct 0x0033 253 253 063 Pre-fail Always - 1
196 Reallocated_Event_Count 0x0008 252 252 000 Old_age Offline - 1
197 Current_Pending_Sector 0x0008 253 253 000 Old_age Offline - 0
198 Offline_Uncorrectable 0x0008 252 252 000 Old_age Offline - 1

# smartctl -t long /dev/hda

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Extended self-test routine immediately in off-line mode".
Drive command "Execute SMART Extended self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 17 minutes for test to complete.
Test will complete after Wed Aug 8 16:38:15 2007

Use smartctl -X to abort test.
# smartctl -l selftest /dev/hda
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 1599 -
# 2 Extended offline Completed: read failure 20% 1596 44724966
# 3 Extended offline Completed: read failure 40% 1519 12622427

# smartctl -A /dev/hda | egrep 'Reallocated|Pending|Uncorrectable'
5 Reallocated_Sector_Ct 0x0033 253 253 063 Pre-fail Always - 1
196 Reallocated_Event_Count 0x0008 252 252 000 Old_age Offline - 1
197 Current_Pending_Sector 0x0008 253 253 000 Old_age Offline - 0
198 Offline_Uncorrectable 0x0008 253 252 000 Old_age Offline - 0

Ok, that looks to have cleared the errors for the time being.
But I'm going to keep a careful eye on that disk, using smartd and logwatch.

Sunday, March 25, 2007

Using 'prtpci' to list PCI devices

On Solaris, if you want to check the details of the hardware installed
in your PC, you can use the command 'prtconf -pv'. However, the output is
very long, and gives too great a level of detail. Sun's Dan Mick has written
a perl program, called 'prtpci' which gives a more useful summary.
Here is how I installed the program:

bash-3.00# wget ftp://playground.sun.com/pub/dmick/prtpci.tar.Z
--00:08:09-- ftp://playground.sun.com/pub/dmick/prtpci.tar.Z
=> `prtpci.tar.Z'
Resolving playground.sun.com... 192.9.5.5
Connecting to playground.sun.com|192.9.5.5|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD /pub/dmick ... done.
==> PASV ... done. ==> RETR prtpci.tar.Z ... done.
Length: 163,050 (159K) (unauthoritative)

100%[====================================>] 163,050 57.69K/s

00:08:16 (57.54 KB/s) - `prtpci.tar.Z' saved [163050]

bash-3.00# gunzip < prtpci.tar.Z | tar xvf -
x prtpci, 5954 bytes, 12 tape blocks
x pciids/pci.ids, 349313 bytes, 683 tape blocks
x pciids/class.ids, 3445 bytes, 7 tape blocks
x pciids/getnew.pci.ids, 81 bytes, 1 tape blocks

bash-3.00# chown -R root:root prtpci pciids/

bash-3.00# ls -l
total 350
drwxr-xr-x 2 root root 512 Mar 25 00:10 pciids
-r-xr-xr-x 1 root root 5954 Jul 12 2005 prtpci
-rw-r--r-- 1 root root 163050 Mar 25 00:08 prtpci.tar.Z

bash-3.00# ls -l pciids/
total 714
-rw-rw-r-- 1 root root 3445 Mar 24 2002 class.ids
-rwxrwxr-x 1 root root 81 Mar 18 2005 getnew.pci.ids
-rw-rw-r-- 1 root root 349313 May 27 2005 pci.ids

And here is the (edited) output of what it shows
on my Dell Precision 380:

bash-3.00# ./prtpci
0/0x0/0 8086,2774 (1028,1a8) rev 0x0
Intel Corporation 955X Memory Controller Hub
class 6/0/0: Bridge/Host bridge

0/0x1/0 8086,2775 0 rev 0x0
Intel Corporation 955X PCI Express Graphics Port
class 6/4/0: Bridge/PCI bridge

1/0x0/0 1002,71d2 (1002,3b02) rev 0x0
ATI Technologies Inc
class 3/0/0: Display controller/VGA compatible controller
BAR[0]: prefetchable 64-bit memory 0xe0000000 0x10000000
BAR[2]: 64-bit memory 0xfe9e0000 0x10000
BAR[4]: I/O 0xdc00 0x100
legacy reg #3: aliased I/O 0x3b0 0xc
legacy reg #4: aliased I/O 0x3c0 0x20
legacy reg #5: 32-bit memory 0xa0000 0x20000

1/0x0/1 1002,71f2 (1002,3b03) rev 0x0
ATI Technologies Inc
class 3/80/0: Display controller/Display controller
BAR[0]: 64-bit memory 0xfe9f0000 0x10000

0/0x1c/0 8086,27d0 1 rev 0x0
Intel Corporation 82801G (ICH7 Family) PCI Express Port 1
class 6/4/0: Bridge/PCI bridge

4/0x0/0 14e4,1677 (1028,1a8) rev 0x1
Broadcom Corporation NetXtreme BCM5751 Gigabit Ethernet PCI Express
class 2/0/0: Network controller/Ethernet controller
BAR[0]: 64-bit memory 0xfe7f0000 0x10000

0/0x1f/0 8086,27b8 1 rev 0x0
Intel Corporation 82801GB/GR (ICH7 Family) LPC Interface Bridge
class 6/1/0: Bridge/ISA bridge

0/0x1f/1 8086,27df (1028,1a8) rev 0x1
Intel Corporation 82801G (ICH7 Family) IDE Controller
class 1/1/8a: Mass storage controller/IDE interface
BAR[0]: I/O 0x1f0 0x8
BAR[1]: I/O 0x3f6 0x1
BAR[2]: I/O 0x170 0x8
BAR[3]: I/O 0x376 0x1
BAR[4]: I/O 0xffa0 0x10

0/0x1f/2 8086,27c1 (1028,1a8) rev 0x1
Intel Corporation 82801GR/GH (ICH7 Family) Serial ATA Storage Controllers cc=AHCI
class 1/6/1: Mass storage controller/
BAR[0]: I/O 0xfe00 0x8
BAR[1]: I/O 0xfe10 0x4
BAR[2]: I/O 0xfe20 0x8
BAR[3]: I/O 0xfe30 0x4
BAR[4]: I/O 0xfea0 0x10
BAR[5]: 32-bit memory 0xfebfbc00 0x400

0/0x1f/3 8086,27da (1028,1a8) rev 0x1
Intel Corporation 82801G (ICH7 Family) SMBus Controller
class c/5/0: Serial bus controller/SMBus
BAR[4]: I/O 0xece0 0x20

The above clearly shows the ATI graphics card, the Broadcom Gigabit Ethernet card, and the ICH7 SATA card (using AHCI).

Here's the link to the post on Dan Mick's blog.

Saturday, March 24, 2007

Using wget, gcc & gmake on Solaris, by fixing the PATH

Solaris has a number of useful commands, like gcc and wget, hidden away in
directory '/usr/sfw/bin'. But, by default, for user root, Solaris does not add this
directory to the path. So assuming you are using Bash as your shell,
try the following:

bash-3.00# echo $PATH
/usr/sbin:/usr/bin
bash-3.00# export PATH=$PATH:/usr/sfw/bin
bash-3.00# echo $PATH
/usr/sbin:/usr/bin:/usr/sfw/bin
bash-3.00# wget -V
GNU Wget 1.10.2
bash-3.00# gcc -v
gcc version 3.4.3 (csl-sol210-3_4-20050802)
bash-3.00# gmake -v
GNU Make 3.80
bash-3.00# uname -a
SunOS solaris 5.11 snv_57 i86pc i386 i86pc

Friday, February 23, 2007

How to waste half a day with EDI

Yes, thats EDI or Electronic Data Interchange.

The problem for me, is that one of our big customers wants to stop sending us Odette DELINS messages, and instead start using Edifact DELFOR messages. It's a problem for me, because I need to work out how to do it!

So let's start by seeing if we can translate a simple DELFOR file into the in-house format to feed into our ERP system. Let's created the test file, by copying & pasting the sample provided in the pdf of the specification. Simple... No...it does not work. Why? Every thing looks ok!

After hours of double checking & going around in circles. Finally the answer.

In an EDI file, each line is terminated with a ' character - thats ASCII code 0x27. But I finally realised that the lines in my file were terminated with character code 0x92, which looks very similar in the character set I was using.

When you use the correct terminating character, it all works a lot better!
So todays lesson is - be careful when you copy & paste text from a pdf!

Wednesday, February 21, 2007

Checking Solaris iScsi Performance with Dtrace

I've just discovered David Weibel's Blog and he has a couple of interesting posts regarding using dtrace(1M) to check performance issues with iScsi on Solaris.

David worked on the Solaris iScsi initiator code when he was at Sun Microsystems, Inc.

Wednesday, January 17, 2007

Using the NetBSD iScsi Target code on Solaris

The NetBSD iScsi target will, quite happily, compile and run on Solaris. The code was developed by Alistair Crooks, and is based on code released by Intel, under the BSD licence.

To try it out, I used the Belenix LiveCD distribution of OpenSolaris.
(root)# uname -a
SunOS belenix 5.11 BeleniX0.4.3 i86pc i386 i86pc
(root)# gcc -v
Reading specs from /usr/foss/lib/gcc/i386-pc-solaris2.10/3.4.3/specs
Thread model: posix
gcc version 3.4.3 (csl-sol210-3_4-branch+sol_rpath)
(root)# cd tmp
(root)# curl -O http://www.alistaircrooks.co.uk/src/netbsd-iscsi-20060526.tar.gz
(root)# ls -l netbsd-iscsi-20060526.tar.gz
-rw------- 1 root root 239964 May 27 22:21 netbsd-iscsi-20060526.tar.gz
(root)# gunzip < netbsd-iscsi-20060526.tar.gz | tar xvf -
(root)# cd iscsi
(root)# cd src
(root)# ./configure
(root)# make
(root)# cd ../bin
(root)# ls -l
-rwx------ 1 root root 226580 May 27 22:24 iscsi-harness
-rwx------ 1 root root 219028 May 27 22:24 iscsi-target

I just wanted to use the iScsi target to confirm if the Microsoft iScsi initiator would work ok with it, and take some Ethereal traces of the initial login sequence. So I just used the following simple configuration file, which exports out a 100 mega byte file.
(root)# cat /etc/iscsi/targets
extent0 /tmp/iscsi-target0 0 100MB

target0 rw extent0 0.0.0.0/0
(root)# ./iscsi-target &
(root)# ls -l /tmp/iscsi-target0
-rw------- 1 root root 104857601 May 27 23:17 /tmp/iscsi-target0

I tried the target with v2.03 of the Microdoft iScsi initiator, and and I can report that every thing I tried worked ok.

You can browse the source code of the iScsi target at the NetBSD CVS server. To get the latest version of the code, it may be best to get it from the CVS server.

I've just noted that Alistair make some further changes to the code at the start of 2007, which are noted at this digest.

Sunday, January 14, 2007

Linux iScsi Target and the Inquiry LUN Response

Ok, last time we completed compiling and installing the Linux iScsi target. Now we need to configure it and start it. All I wanted to do, at this stage, was a simple test. So I just did the minimal amount of configuration. I was going to be running the target on Centos-4, running under VmWare, so it was easy to add a second virtual hard drive to the environment, in this case '/dev/sdb', to export out from the target.
# cat /etc/ietd.conf
#IncomingUser nwsmith secretsecret
Target iqn.2007-01.net.nwsmith:test01
Lun 0 /dev/sdb fileio
Alias iscsitest01

# service iscsi-target start
Starting iSCSI target service: [ OK ]

# netstat -ntlp | grep 3260
tcp 0 0 0.0.0.0:3260 0.0.0.0:* LISTEN 3160/ietd

# tail /var/log/messages
Jan 14 15:59:02 localhost kernel: iSCSI Enterprise Target Software - version 0.4.5
Jan 14 15:59:02 localhost kernel: iotype_init(91) register fileio
Jan 14 15:59:03 localhost kernel: target_param(109) d 1 8192 262144 65536 2 20 8 0
Jan 14 15:59:03 localhost iscsi-target: ietd startup succeeded

The Linux iScsi target seems to work fine with the Microsoft iScsi initiator (v2.03).

Ok, now we have the target running, one of the thing I wanted to look at was how the Linux iScsi target handled the initial login and connection from the Microsoft initiator. I was looking for how it handles the "Inquiry LUN" packet. I used Ethereal to capture and analyse the packets - filter on port 3260.

I had a look at the source code, and found what I was looking for in file "target_disk.c" and in function:
static int build_inquiry_response(struct iscsi_cmnd *cmnd)

By editing the values in this function, you can adjust the following fields in the response packet:

Version, Flags, Vendor Id:, Product Id:, Product Revision Level:

I tried changing some of these fields in the code, recompiling, restarting the target, and then used Ethereal to check that the response (from the target) to the (initiators) 'Inquiry' LUN packet changed as expected. And it did what I expected. (To change the Version returned, edit the line "data[2] = 3;").

The Version field indicates which version of SCSI is supported by the target. You can check which values are valid by reading the specification. The specification is called "SCSI Primary Commands - 3" or SPC-3 and you can download it from www.t10.org.

In the case of the version field, the valid values are shown in table 84:
0x00 - The device does not claim conformance to any standard.
0x02 - Obsolete.
0x03 - The device complies to ANSI INCITS 301-1997 (SPC).
0x04 - The device complies to ANSI INCITS 351-2001 (SPC-2).
0x05 - The device complies to T10/1416-D (SPC-3).

Ok, I think that's enough detail for this post.

Saturday, January 13, 2007

Centos 4 and the Linux iScsi Target

I wanted to do some tests with the Microsoft iScsi initiator and the Linux iScsi target, but first I had to install and configure the target. I had Centos 4.4 (a RHEL4 Linux clone) setup in VmWare, so i thought that using that would be the easy way to do the test.

I had read in PcPro magazine, last year, issue 137 (March 2006) page 197, that the Linux iScsi target would compile & install on Centos-4, but they did not say exactly which version they used.

Ok, so I download the latest version "iscsitarget-0.4.14.tar.gz".

I knew there was a potential problem, as the README file said it needed kernel version of 2.6.14 or newer, and the CONFIG_CRYPTO_CRC32C option had to be enabled. Centos 4.4 uses kernel 2.6.9, but with many additional patches from latter kernels, and the PCpro people said it would be ok.

Ok, so lets try it:

# cat /etc/redhat-release
CentOS release 4.4 (Final)
# uname -a
Linux localhost.localdomain 2.6.9-42.0.3.EL #1 Fri Oct 6 05:59:54 CDT 2006 i686 i686 i386 GNU/Linux
# cat /usr/src/kernels/2.6.9-42.0.3.EL-i686/crypto/Kconfig | grep -A 6 CRC
config CRYPTO_CRC32C
tristate "CRC32c CRC algorithm"
depends on CRYPTO
select LIBCRC32C
help
Castagnoli, et al Cyclic Redundancy-Check Algorithm. Used
by iSCSI for header and data digests and by others.
See Castagnoli93. This implementation uses lib/libcrc32c.
Module will be crc32c.

# cd /home
# tar xfz iscsitarget-0.4.14.tar.gz
# chown -R root:root iscsitarget-0.4.14
# ll
drwxr-xr-x 7 root root 4096 Oct 19 10:48 iscsitarget-0.4.14
-rw-r--r-- 1 root root 92608 Mar 14 2006 iscsitarget-0.4.14.tar.gz
# cd iscsitarget-0.4.14
# export KERNELSRC=/usr/src/kernels/2.6.9-42.0.3.EL-i686/
# make
make -C usr
make[1]: Entering directory `/home/iscsitarget-0.4.14/usr'
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o ietd.o ietd.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o iscsid.o iscsid.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o conn.o conn.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o session.o session.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o target.o target.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o message.o message.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o ctldev.o ctldev.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o log.o log.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o chap.o chap.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o event.o event.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o param.o param.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o plain.o plain.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o isns.o isns.c
cc ietd.o iscsid.o conn.o session.o target.o message.o ctldev.o log.o chap.o event.o param.o plain.o isns.o -o ietd -lcrypto
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o ietadm.o ietadm.c
cc ietadm.o param.o -o ietadm
make[1]: Leaving directory `/home/iscsitarget-0.4.14/usr'
make -C /lib/modules/2.6.9-42.0.3.EL/build SUBDIRS=/home/iscsitarget-0.4.14/kernel modules
make[1]: Entering directory `/usr/src/kernels/2.6.9-42.0.3.EL-i686'
CC [M] /home/iscsitarget-0.4.14/kernel/tio.o
CC [M] /home/iscsitarget-0.4.14/kernel/iscsi.o
CC [M] /home/iscsitarget-0.4.14/kernel/nthread.o
CC [M] /home/iscsitarget-0.4.14/kernel/wthread.o
CC [M] /home/iscsitarget-0.4.14/kernel/config.o
/home/iscsitarget-0.4.14/kernel/config.c:312: error: unknown field `unlocked_ioctl' specified in initializer
/home/iscsitarget-0.4.14/kernel/config.c:312: warning: initialization from incompatible pointer type
/home/iscsitarget-0.4.14/kernel/config.c:313: error: unknown field `compat_ioctl' specified in initializer
/home/iscsitarget-0.4.14/kernel/config.c:313: warning: initialization from incompatible pointer type
make[2]: *** [/home/iscsitarget-0.4.14/kernel/config.o] Error 1
make[1]: *** [_module_/home/iscsitarget-0.4.14/kernel] Error 2
make[1]: Leaving directory `/usr/src/kernels/2.6.9-42.0.3.EL-i686'
make: *** [mods] Error 2

# cat /home/iscsitarget-0.4.14/kernel/config.c | grep -n ioctl
219:static long ioctl(struct file *file, unsigned int cmd, unsigned long arg)
312: .unlocked_ioctl = ioctl,
313: .compat_ioctl = ioctl,

Oops! It failed to compile.
Ok, let's google around and see who else has this problem...

It seems other people are getting the same sort of problem with the target versions 4.13 and 4.12 and the older Centos kernel 2.6.9-22.EL

I found this link, where you can download a patch for the target version 0.4.12, which should allow it to compile on kernel 2.6.9, but I did not try it, as it's a "quick hack - Use at your own risk". Maybe someone else can try this patch and give a report.

Ok, so should I dump Centos-4 for now and install Fedora 6, which has an up-to-date kernel. Or should I try and work out which version of the target it was that the Pcpro guy's were using? Ok, I choose the latter.

I went back and downloaded the older versions of the target. And to cut a long story short, I discovered that this version "iscsitarget-0.4.5.tar.gz" will compile on Centos-4. (If you look at the README for that version of the target, it says it is happy with kernel 2.6.10)
Here is the proof that it compiles:

# tar xfz iscsitarget-0.4.5.tar.gz
# chown -R root:root iscsitarget-0.4.5
# cd iscsitarget-0.4.5
# make
make -C usr
make[1]: Entering directory `/home/iscsitarget-0.4.5/usr'
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o ietd.o ietd.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o iscsid.o iscsid.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o conn.o conn.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o session.o session.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o target.o target.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o message.o message.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o ctldev.o ctldev.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o log.o log.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o md5.o md5.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o isns.o isns.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o sha1.o sha1.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o chap.o chap.c
cc ietd.o iscsid.o conn.o session.o target.o message.o ctldev.o log.o md5.o isns.o sha1.o chap.o -o ietd
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o ietadm.o ietadm.c
cc ietadm.o ctldev.o -o ietadm
make[1]: Leaving directory `/home/iscsitarget-0.4.5/usr'
make -C /usr/src/kernels/2.6.9-42.0.3.EL-i686/ SUBDIRS=/home/iscsitarget-0.4.5/kernel modules
make[1]: Entering directory `/usr/src/kernels/2.6.9-42.0.3.EL-i686'
CC [M] /home/iscsitarget-0.4.5/kernel/tio.o
CC [M] /home/iscsitarget-0.4.5/kernel/iscsi.o
CC [M] /home/iscsitarget-0.4.5/kernel/nthread.o
CC [M] /home/iscsitarget-0.4.5/kernel/wthread.o
CC [M] /home/iscsitarget-0.4.5/kernel/config.o
CC [M] /home/iscsitarget-0.4.5/kernel/digest.o
CC [M] /home/iscsitarget-0.4.5/kernel/conn.o
CC [M] /home/iscsitarget-0.4.5/kernel/session.o
CC [M] /home/iscsitarget-0.4.5/kernel/target.o
CC [M] /home/iscsitarget-0.4.5/kernel/volume.o
CC [M] /home/iscsitarget-0.4.5/kernel/iotype.o
CC [M] /home/iscsitarget-0.4.5/kernel/file-io.o
CC [M] /home/iscsitarget-0.4.5/kernel/target_disk.o
LD [M] /home/iscsitarget-0.4.5/kernel/iscsi_trgt.o
Building modules, stage 2.
MODPOST
CC /home/iscsitarget-0.4.5/kernel/iscsi_trgt.mod.o
LD [M] /home/iscsitarget-0.4.5/kernel/iscsi_trgt.ko
make[1]: Leaving directory `/usr/src/kernels/2.6.9-42.0.3.EL-i686'

# make install
`usr/ietd' -> `/usr/sbin/ietd'
`usr/ietadm' -> `/usr/sbin/ietadm'
if [ -f /etc/debian_version ]; then \
install -v -m 755 etc/initd/initd.debian /etc/init.d/iscsi-target; \
elif [ -f /etc/redhat-release ]; then \
install -v -m 755 etc/initd/initd.redhat /etc/init.d/iscsi-target; \
elif [ -f /etc/slackware-version ]; then \
install -v -m 755 etc/initd/initd /etc/rc.d/iscsi-target; \
else \
install -v -m 755 etc/initd/initd /etc/init.d/iscsi-target; \
fi
`etc/initd/initd.redhat' -> `/etc/init.d/iscsi-target'
install: creating directory `/lib/modules/2.6.9-42.0.3.EL/kernel/iscsi'
`kernel/iscsi_trgt.ko' -> `/lib/modules/2.6.9-42.0.3.EL/kernel/iscsi/iscsi_trgt.ko'
depmod -aq
#

Version 0.4.5 was released on 2005-02-21, so that PcPro article must have been on the editors desk for a year!

Ok, I still need to see if it will work, but that something for another post!