# Hardware Monitoring on IBM xSeries

## tomgan

Hi!

Is it possible to monitor temperature/stuff on the IBM eServer xSeries?

ACPI seems to be empty... The i2c_piix4 loads but does

not find any devices. lm_sensors' sensors-detect shows some

lm92-units attached to the piix4, however.

This together with the lm_sensors <-> Thinkpad problems led me to think

someone must have had this problem before. Certainly, a server must be

monitorable...

/Tommie

----------

## tomatopi

I have a number of xSeries servers (5 different ones) and haven't figured it out yet, either. It deffinately would be nice to know how to get all the monitoring and management features working.

If I'm correct, the ASM processor collects all the data and you'd need to query it rather than the sensors directly, but I've never figured out how. I've got the ibmasm module running and the filesystem is listed in proc/filesystems, but that's as far as I ever got. IBM documentation seems to point that the ibmasm modules allows mouting a filesystem which could be queried for the stats, but I just get an empty directory when I try and mount it.

IBM workstations and laptops without the ASM seem to be detectable using lm_sensors.

----------

## tomgan

I wonder if this is something IBM wants us to use Director for, or if it's just

hard to find info on.

----------

## djay

 *tomgan wrote:*   

> Hi!
> 
> Is it possible to monitor temperature/stuff on the IBM eServer xSeries?
> 
> ACPI seems to be empty... The i2c_piix4 loads but does
> ...

 

Hi tommie

We use xSeries in our product and recently started to develop Hardware Monitoring system, so I had to design the hardware part.

There are few tools which allow to monitor temperature/stuff on IBM machines.

It is possible to use both lmsensors and IPMI. 

I tried lmsensors, but support for it wasn't good enough in 2.4 kernels (that what we use at the moment). 

IPMI works ok (also because we have other SuperMicro machines, and IPMI works on them as well).

So, 1. Set BMC controller from BIOS (or using CD from IBM"s site, depends on your xServer model) to use IPMI and send SNMP traps to some other IP, not its own (although its possible).

2. From linux, you can use something like this: 

```
ipmitool I lan H <ipmi_IP> U ADMIN P ADMIN sdr
```

when ipmi_IP is an IP of the machine you're connected to (say, you have two machines monitoring each other), not local. 

It will return full monitorable devices list, with their current values, for example on my x3650 it will return:

```

Ambient Temp     | 23 degrees C      | ok

Fan 1 Tach       | disabled          | ns

Fan 2 Tach       | disabled          | ns

Fan 3 Tach       | 4800 RPM          | ok

Fan 4 Tach       | 5250 RPM          | ok

Fan 5 Tach       | disabled          | ns

Fan 6 Tach       | disabled          | ns

Fan 7 Tach       | disabled          | ns

Fan 8 Tach       | 5475 RPM          | ok

Fan 9 Tach       | 4575 RPM          | ok

Fan 10 Tach      | 4425 RPM          | ok

CPU 1 Temp       | disabled          | ns

Fan 1 Present    | Not Readable      | ns

Fan 2 Present    | Not Readable      | ns

Fan 3 Present    | 0x00              | ok

Fan 4 Present    | 0x00              | ok

Fan 5 Present    | Not Readable      | ns

Fan 6 Present    | Not Readable      | ns

Fan 7 Present    | Not Readable      | ns

Fan 8 Present    | 0x00              | ok

Fan 9 Present    | 0x00              | ok

Fan 10 Present   | 0x00              | ok

CPU 2 Temp       | disabled          | ns

Planar 5V        | 5.16 Volts        | ok

Planar 12V       | 12.14 Volts       | ok

Planar VBAT      | 3.05 Volts        | ok

Riser Detect     | 0x00              | ok

RSA II Detect    | 0x01              | ok

FP Cable Detect  | 0x00              | ok

PS 2 Status      | Not Readable      | ns

SCSI BP Detect   | 0x00              | ok

Planar 3.3V      | 3.31 Volts        | ok

CPU 1 Status     | 0x80              | ok

ABR Status       | 0x00              | ok

CPU 2 Status     | 0x80              | ok

USB Cable Detect | 0x00              | ok

NMI State        | 0x00              | ok

Power 12V Fault  | 0x00              | ok

Power 5V Fault   | 0x00              | ok

PS 1 Fan Fault   | 0x00              | ok

PS 2 Fan Fault   | Not Readable      | ns

PS 1 Status      | 0x01              | ok

Drive 1 Status   | 0x01              | ok

Drive 2 Status   | 0x01              | ok

Drive 3 Status   | Not Readable      | ns

Drive 4 Status   | Not Readable      | ns

Drive 5 Status   | Not Readable      | ns

Drive 6 Status   | Not Readable      | ns

Drive 7 Status   | Not Readable      | ns

Drive 8 Status   | Not Readable      | ns

RAID Error       | 0x00              | ok

CPU Fault Reboot | 0x00              | ok

SEL Fullness     | 16 messages       | ok

Sys pwr monitor  | 0x00              | ok

Watchdog         | 0x00              | ok

VRD Status       | 0x00              | ok

VRM Status       | 0x01              | ok

Pwr Planar Fault | 0x00              | ok

Critical Int     | 0x00              | ok

CPU 1 OverTemp   | 0x00              | ok

CPU 2 OverTemp   | 0x00              | ok

```

There are other options for IPMI which allow you to monitor other things.

Hope it helps.

Alex.

----------

## tomgan

Thanks!

I'll check it out.

/Tommie

----------

## tomatopi

IPMI works on some xSeries, but not all. IBM's have a number of different management options. RSA (Remote Supervisor Adapers), BMC (Baseboard Management Controller), and ASM (Advanced System Management) are the most common and some servers can have more than one. 

Most of mine are ASM's which seem to trap lm_sensors and they don't all support IPMI and have no real BIOS control and instead are setup via software. I have used IBM Director on Windows to get the ASM working, but under Gentoo I have not been successful as of yet.

None of my servers have BMC. Darn.

----------

## djay

 *tomatopi wrote:*   

> IPMI works on some xSeries, but not all. IBM's have a number of different management options. RSA (Remote Supervisor Adapers), BMC (Baseboard Management Controller), and ASM (Advanced System Management) are the most common and some servers can have more than one. 
> 
> Most of mine are ASM's which seem to trap lm_sensors and they don't all support IPMI and have no real BIOS control and instead are setup via software. I have used IBM Director on Windows to get the ASM working, but under Gentoo I have not been successful as of yet.
> 
> None of my servers have BMC. Darn.

 

That is correct.

I just assumed that all would come with BMC while it is an option on purchase.

As I saw it, BMC and RSA (or RSA II for that matter) allow the best options for monitoring. In any case, this is the following order options-wise:

ASM -> BMC -> RSA -> RSA II

----------

## tomatopi

I would get BMC, but all my servers are old and used (though still overpowered for most of my applications). As such, I'm battling to get ASM working since that's all I have in most servers. 

There's little to no info available on the ibmasm module which I think it the key pivot point. I've even tried the ibmasm drive from IBM's site, but it's nearly identical to the code that ships with the kernel and it didn't work. I'm not enough of a programmer to be able to trace and see what's not working.

----------

## djay

 *tomatopi wrote:*   

> I would get BMC, but all my servers are old and used (though still overpowered for most of my applications). As such, I'm battling to get ASM working since that's all I have in most servers. 
> 
> There's little to no info available on the ibmasm module which I think it the key pivot point. I've even tried the ibmasm drive from IBM's site, but it's nearly identical to the code that ships with the kernel and it didn't work. I'm not enough of a programmer to be able to trace and see what's not working.

 

I'd think lmsensors would be solution for you as it doesn't depend on management card but on MB components which all recent motherboards have.

----------

## tomatopi

I have tried lm_sensors without success. From what I understand, the ASM processor interrupts lm_sensor readings so they can't be probed directly and instead need to be checked via the ASM. That's the part I got stuck with. I believe newer units with IPMI can be probed to an extent, but not older ones.

----------

## tomgan

The ibmasm module seem to supply a virtual file system, which I can mount with

```
mount -t ibmasmfs none /mnt/floppy
```

However, the ASM seems to be a PCI device, and nothing shows up on my lspci.

Also, the mounted file system is empty, with no error messages with dmesg.

So, in my case it seems I don't have ASM.

The source for the ibmasm module discusses commands which seem to be like SNMP (OIDs).

Maybe IBM has some MIB on this?

Loading the IPMI drivers doesn't seem to find anything, but I will try to apply the

OpenIPMI patches too...

The hunt continues.  :Smile: 

/Tommie

----------

## tomatopi

I had roughly the same experiences. Mouting the ibmasm filesystem was empty and I know I have ASM processors since Windows can probe them (and it's physically visible on the mainboard as a PowerPC processor). It doesn't show up under lspci or lshw, either, but it is physically on the system.

IPMI detected something on my x335's, but I didn't play with it too much. Nothing on my x330's or x340/x342's since they don't support IPMI.

IBM's documentation is quite vague on the whole thing and they simply say to "use a supported Linux" and that's it. Not much help.

What IBM system are you working with?

----------

## tomgan

Its a Netfinity 4500R (is this the same as eServer xSeries 340?)

I haven't investigated the hardware, since its located at my parents house,

but I will in a couple of days.

BTW, I checked the IBM site, and after much searching, I found two

interesting files:

MPCLI - Management Processor Command Line Interface 5.10

Director 5.10 for Linux

However, MPCLI fails to "logonlocal", and I haven't tried the Director yet.

According to manual, MPCLI is able to fetch health information and more.

I stopped trying until I can take a look at BIOS settings, though. I bought the server

with Win2003 installed, but I never looked for monitoring hardware (as it

is in no way a priority on a home-server). So it could be shut down in BIOS,

or something.

The most weird thing about MPCLI is that it doesn't say it needs the "ibmasm"

driver in the README, so how it communicates with hardware is a mystery. Written

in Java with a lot of native code.

Also, there is a program called xCat, a cluster administration toolkit from IBM.

I believe it's open-source and uses various interfaces for reading the ASM (http://xcat.org/).

Not investigated it further.

Anyhow, both MPCLI and Director are supposed to support the 4500.

/Tommie

----------

## tomatopi

The 4500R is a dual-P3 system with the ServerWorks chipset and ASM. Some have RS-424 headers for management and some don't. x340 is the same after they renamed their product line to eServer xSeries and x342 is the socket version and often adds an extra 3 hot-swap bays. The x330 is the same basic architecture but in a 1U system with integrated KVM for chaining but no native connections for video or keyboard. x335 is the P4 version of the x330 but adds IPMI support.

I recall using the MPCLI on a 4500R under Windows but never tried it under Linux. Do you have link where to download it? It's really hard to find on their site. I'd also like to test it on my x330's.

I never installed Director under Linux because it needs X and I really don't want X on my server. Also, if Director for Linux is anything like Director for Windows, it uses major memory and cpu resources just to monitor stuff. I read someplace that using command-line utilities from cron is far less resource intensive but less real-time. I have successfully done that with IBM ServeRAID controllers and the ipssend utility. I tried the ServeRAID management utilities under Linux but it couldn't detect the controller for some reason, even if ipssend did - I'm guessing it's trying to use the ASM as well.

I would guess there would be plenty of trail-and-error kernel settings to get MPCLI working. It took forever to find the right combination to get ipssend to work. I will experiment a little on my servers and see if I can get anywhere. If I knew what bus and/or settings are required in the kernel to get ASM to work, it would make for far less reboots while testing.

----------

## tomgan

I guess I have one x340 and one x342, then.

Fourth hit on Google for "mpcli", http://www-1.ibm.com/support/docview.wss?uid=psg1MIGR-54216.

It's an RPM, and includes a JRE. I did

```

rpm2cpio mpcli.rpm | cpio -di

cd tmp

tar xf asmcli.tgz

cd asmcli

```

then edit MPCLI.sh to make better sense... (Removed JRE references, mostly)

Seems to work (aside from a faulty reference to asmcli.xml).

I strace'd mpcli while doing "logonlocal", but it showed up with nothing (maybe it forks, or something?).

On Director, isn't there a server/agent part?

/Tommie

----------

## tomatopi

There is a server/agent part to Director. But you would need a server and viewer to see what agent is doing and you need X for that. I have tried the agent without success as well, again something to do with ASM. Director can still collect basic Linux info without the agent, but none of the environmental sensors which makes it rather pointless.

I will try the CLI on my machines as well to see if anything works.

----------

## tomatopi

I tried the MPCLI on my x330 and got the same results failing to logon. I tried it both with and without the ibmasm module running. I also tried compiling I2C modules and loading them for the heck of it, still failing to logon.

I'm stumped.

----------

## silverchris

I have been playing with the ibmasm module and there seems to be some utilities for it called ibmasm-utils once got it to make some entries in /dev but haven't been able to recreate them since. I would really like to get this going so any help is apriecated.

----------

