hbj11 - the Hackerspace Blade JBOD adapter

Replacement cards for M610 blades at the hackerspace.

Usually, Dell M610 blades have the following disk controller setup:

             “Mezzanine connector”
             .'-------'-------'.
.-----------------.       .----------------.
| Motherboard     |       | Dell CERC card |
|-----------------| PCIe  |----------------|
|                 |-------| PCIe-- | LSI   |
|                 | SAS   |     .--| Crap  |
|           .-----|-------| SAS-'  '-------|
|           |     |       '----------------'
|           |     |
|           |     | SAS   .----------------.
|           '-----|-------| Disk Backplane |
'-----------------'       '----------------'
                              |       |
                          .-----.  .-----.
                          | HDD |  | HDD |
                          |     |  |     |
                          |     |  |     |
                          '-----'  '-----'

However, the Dell CERCs are hot garbage. Put them in writeback mode, and the non-battery-backed cache will lose data on power loss (and block the boot process after reboot to let you know it lost data). Put it in write-through mode, and lose most of your IOPS.

Turns out, the connector for the Dell CERC mezzanine cards carries not just PCIe to the CPU’s root complex and SAS to the disks, it also carries two SATA lanes (non-SAS) to the built-in AHCI controller on the motherboard. Thus, we can fabricate a ‘jumper’ card like this:

             “Mezzanine connector”
             .'-------'-------'.
.-----------------.       .----------------.
| Motherboard     |       | HBJ11          |
|-----------------| PCIe  |----------------|
|                 |-------| N/C            |
|                 | SATA  |                |
|            AHCI |-------|--.             |
|                 | SAS   |  |             |
|           .-----|-------|--'             |
|           |     |       '----------------'
|           |     |
|           |     | SAS   .----------------.
|           '-----|-------| Disk Backplane |
'-----------------'       '----------------'
                              |       |
                          .-----.  .-----.
                          | HDD |  | HDD |
                          |     |  |     |
                          |     |  |     |
                          '-----'  '-----'

This allows us to use the drives via the built in AHCI controller on the motherboard! Of course, only one lane of each SAS lane is connected, so you can only use SATA drives. But who cares about SAS, anyway. We just put SSDs in these machines anyway.

There actually is a commercially available card from Dell that does almost this, but it uses some expensive SATA redrivers. Turns out, you don’t need them - and we’ve tested this quite extensively.

There’s one more thing on the connector to the card: an I2C flash that identifies the card according to the IPMI FRU Information Storage Definition. We need to place these on our custom cards and flash them. We decided to over-engineer that, find that below.

Hardware

A JLCSMT-compatible design for a four-layer ‘HBJ11’ jumper card is available in hardware.

EEPROM & flasher

See flasher for a WebUSB/STM32 based flasher for the cards.