ciss and mpt disk devices numbering priority on a DL385 G2

[Kai Gallasch]

Hello,

on a DL385 G2 I needed some JBOD disks and built a SAS controller into
the server, because the builtin Compaq P400 controller does not support
JBOD.

Luckily the 8 x SAS 2.5" drive cage of the server has two SAS connectors
on the back, with one addressing the first four SAS disks (A) and the
other one disks five to eight (B)

So I had the P400 connected to A and the LSI SAS Controller 3442E-R to
connector B of the drive cage.

The problem now is, that with 7.2 Release the SCSI disks of the LSI SAS
controller show up as da0, da1, etc. What I wanted was that disk devices
of the ciss controller take precedence over disk devices of the LSI SAS
controller (mpt driver) so I have da0 as boot disk on the P400

So I have right now..

mpt0: da0
mpt0: da1
ciss0: da3
ciss0: da4

and not as wanted:

ciss0: da0
ciss0: da1
ciss0: da2
mpt0: da3
mpt0: da4

What I already tried:

- Putting the 3442E-R into two different PCIe slots

What options do I have to get the ciss controlled da0-da3 show up before
the mpt devices?

Any hint appreciated,

Cheers,
Kai.

----------------- devices in dmesg -------------

mpt0: <LSILogic SAS/SATA Adapter> port 0x4000-0x40ff mem
0xfddf0000-0xfddf3fff,0xfdde0000-0xfddeffff irq 36 at device 0.0 on pci67
mpt0: [ITHREAD]
mpt0: MPI Version=1.5.19.0
mpt0: Capabilities: ( RAID-0 RAID-1E RAID-1 )
mpt0: 0 Active Volumes (2 Max)
mpt0: 0 Hidden Drive Members (14 Max)

ciss0: <HP Smart Array P400> port 0x5000-0x50ff mem
0xfdf00000-0xfdffffff,0xfdef0000-0xfdef0fff irq 32 at device 0.0 on pci70
ciss0: [ITHREAD]

da0 at mpt0 bus 0 target 8 lun 0
da0: <HP DG146ABAB4 HPDC> Fixed Direct Access SCSI-5 device
da0: 300.000MB/s transfers
da0: Command Queueing Enabled
da0: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C)

da2 at ciss0 bus 0 target 0 lun 0
da2: <COMPAQ RAID 5  VOLUME OK> Fixed Direct Access SCSI-5 device
da2: 135.168MB/s transfers
da2: Command Queueing Enabled
da2: 26109MB (53472480 512 byte sectors: 255H 32S/T 6553C)

da3 at ciss0 bus 0 target 1 lun 0
da3: <COMPAQ RAID 5  VOLUME OK> Fixed Direct Access SCSI-5 device
da3: 135.168MB/s transfers
da3: Command Queueing Enabled
da3: 143357MB (293596800 512 byte sectors: 255H 32S/T 35980C)

da1 at mpt0 bus 0 target 9 lun 0
da1: <HP DG146ABAB4 HPDC> Fixed Direct Access SCSI-5 device
da1: 300.000MB/s transfers
da1: Command Queueing Enabled
da1: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C)

da4 at ciss0 bus 0 target 2 lun 0
da4: <COMPAQ RAID 5  VOLUME OK> Fixed Direct Access SCSI-5 device
da4: 135.168MB/s transfers
da4: Command Queueing Enabled

_______________________________________________
freebsd-proliant@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-proliant
To unsubscribe, send any mail to "freebsd-proliant-unsubscribe@..."

[John Cagle]

Hello Kai,

Have you looked at the options in the BIOS setup utility (RBSU)?  There
should be an option to select the boot controller (which should be the P400
in your case).

If that is already set to the P400, then the problem may be one of PCI
enumeration.  If FreeBSD loads the mpt driver first, then it may grab da0.
Maybe there's a way to force it to load the ciss driver first?

John

On Wed, Jun 17, 2009 at 4:43 AM, Kai Gallasch <gallasch@...> wrote:

> Hello,
>
> on a DL385 G2 I needed some JBOD disks and built a SAS controller into
> the server, because the builtin Compaq P400 controller does not support
> JBOD.
>
> Luckily the 8 x SAS 2.5" drive cage of the server has two SAS connectors
> on the back, with one addressing the first four SAS disks (A) and the
> other one disks five to eight (B)
>
> So I had the P400 connected to A and the LSI SAS Controller 3442E-R to
> connector B of the drive cage.
>
> The problem now is, that with 7.2 Release the SCSI disks of the LSI SAS
> controller show up as da0, da1, etc. What I wanted was that disk devices
> of the ciss controller take precedence over disk devices of the LSI SAS
> controller (mpt driver) so I have da0 as boot disk on the P400
>
> So I have right now..
>
> mpt0: da0
> mpt0: da1
> ciss0: da3
> ciss0: da4
>
> and not as wanted:
>
> ciss0: da0
> ciss0: da1
> ciss0: da2
> mpt0: da3
> mpt0: da4
>
> What I already tried:
>
> - Putting the 3442E-R into two different PCIe slots
>
> What options do I have to get the ciss controlled da0-da3 show up before
> the mpt devices?
>
> Any hint appreciated,
>
> Cheers,
> Kai.
>
> ----------------- devices in dmesg -------------
>
> mpt0: <LSILogic SAS/SATA Adapter> port 0x4000-0x40ff mem
> 0xfddf0000-0xfddf3fff,0xfdde0000-0xfddeffff irq 36 at device 0.0 on pci67
> mpt0: [ITHREAD]
> mpt0: MPI Version=1.5.19.0
> mpt0: Capabilities: ( RAID-0 RAID-1E RAID-1 )
> mpt0: 0 Active Volumes (2 Max)
> mpt0: 0 Hidden Drive Members (14 Max)
>
> ciss0: <HP Smart Array P400> port 0x5000-0x50ff mem
> 0xfdf00000-0xfdffffff,0xfdef0000-0xfdef0fff irq 32 at device 0.0 on pci70
> ciss0: [ITHREAD]
>
> da0 at mpt0 bus 0 target 8 lun 0
> da0: <HP DG146ABAB4 HPDC> Fixed Direct Access SCSI-5 device
> da0: 300.000MB/s transfers
> da0: Command Queueing Enabled
> da0: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C)
>
> da2 at ciss0 bus 0 target 0 lun 0
> da2: <COMPAQ RAID 5  VOLUME OK> Fixed Direct Access SCSI-5 device
> da2: 135.168MB/s transfers
> da2: Command Queueing Enabled
> da2: 26109MB (53472480 512 byte sectors: 255H 32S/T 6553C)
>
> da3 at ciss0 bus 0 target 1 lun 0
> da3: <COMPAQ RAID 5  VOLUME OK> Fixed Direct Access SCSI-5 device
> da3: 135.168MB/s transfers
> da3: Command Queueing Enabled
> da3: 143357MB (293596800 512 byte sectors: 255H 32S/T 35980C)
>
> da1 at mpt0 bus 0 target 9 lun 0
> da1: <HP DG146ABAB4 HPDC> Fixed Direct Access SCSI-5 device
> da1: 300.000MB/s transfers
> da1: Command Queueing Enabled
> da1: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C)
>
> da4 at ciss0 bus 0 target 2 lun 0
> da4: <COMPAQ RAID 5  VOLUME OK> Fixed Direct Access SCSI-5 device
> da4: 135.168MB/s transfers
> da4: Command Queueing Enabled
>
> _______________________________________________
> freebsd-proliant@... mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-proliant
> To unsubscribe, send any mail to "freebsd-proliant-unsubscribe@...
> "
>

_______________________________________________
freebsd-proliant@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-proliant
To unsubscribe, send any mail to "freebsd-proliant-unsubscribe@..."

[Kai Gallasch]

John Cagle wrote:

> Have you looked at the options in the BIOS setup utility (RBSU)?  There
> should be an option to select the boot controller (which should be the
> P400 in your case).
>
> If that is already set to the P400, then the problem may be one of PCI
> enumeration.  If FreeBSD loads the mpt driver first, then it may grab
> da0.  Maybe there's a way to force it to load the ciss driver first?

Hi John.

In the end I came up with some "solution". I built a kernel without
support for the LSI HBA (mpt driver).

When the server has booted I do a "kldload mpt" and "camcontrol rescan"
to bring up the mpt scsi devices. The scsi devices of the mpt controller
then show up after those of the smart array (ciss).

In my case I need the LSI controller to give me some JBOD devices, so I
can go and play with zfs on the server. (hp/compaq smart array offers no
jbod)

It is easy to patch /etc/rc.d/zfs to load the mpt kernel module and do
the rescan of the scsi bus:

Index: zfs
===================================================================
RCS file: /common/config/server/ROOT/etc/rc.d/zfs,v
retrieving revision 1.1
retrieving revision 1.2
diff -r1.1 -r1.2
15c15
< required_modules="zfs"
---
> required_modules="mpt:zfs"
25a26,29
> # required module mpt is on bus 2. do a rescan, so the
> # devices are found and become available.
> camcontrol rescan 2

So when zfs starts, the rest happens automatically.

--Kai.
_______________________________________________
freebsd-proliant@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-proliant
To unsubscribe, send any mail to "freebsd-proliant-unsubscribe@..."