Difference between revisions of "NAS Hardware Selection (2012)"

Revision as of 22:35, 27 July 2020

Requirements

In 2012, my current off-the-shelf storage server (NAS) was 75% full, so I wanted to buy a replacement NAS. In addition to my NAS I have 4(!) external disk drives connected to my main desktop, and wanted to reduce that number. Also, I now often decommission smaller size disks which would still be useful in a server. So my first requirement was a system with at least four drive bays, but preferably more.

I first considered (in order from low price to high quality) a LaCie 5big network 2, Synology DS412+, Drobo FS, or IX Systems' FreeNAS mini and QNAP TS-559 Pro II. All these systems have four or five 3.5" disk bays, and range from €300 to €750 without disks. Add about €100 for each Terabyte disk.

Two requirements prompted my to build my own solution:

Services

The server should run the SMB and AFP protocols, as well as allow direct streaming of media, all over IPv6. A custom server gives full control over the software and is more flexible in the long run (though streaming media is still hard due to the DRM protections nowadays)

Price

Building your own solution costs about the same for the bare system, but is cheaper per terabyte.

So I decided to build a custom FreeBSD (or FreeNAS) system. Not only would that give flexibility, it also allows me to play with ZFS, my favourite file system.

Choice of chassis

I mostly considered Fractal Design and Li Lian chassis, which offer a good wide choice in products. My first decision is what size I wanted to case to be. There are a few cases of less than 10 litre which allow 4 disks, like the [Lian Li PC-Q12]. However, that would limit myself to 2.5" disk bays. I wanted at least six 3.5" bays, and it was possible to achieve this in a mini-tower chassis of 20-25 litre. For comparison, a regular tower PC typically is 30-35 litre in size. Good choices in this size where the Fractal Design Array R2 NAS Chassis (with six 3.5" and one 2.5" disk bays) and the Lian Li PC-Q25 (with seven 3.5" and one 2.5" disk bays, where five of the 3.5" bays are hot swappable). Both these cases support mini-ITX and mini-DTX motherboard (the difference is that a mini-ITX has one expansion slot, and the mini-DTX has two expansion slots).

I picked the Lian Li PC-Q25 after reading a few favourable reviews. For those interested: the specification of the disk bays is somewhat confusing. The PC-Q25 can support eight disks at most. Five of these are in a hot swappable disk bay, the other three on a fixed metal plated. The middle location on the metal plate must be a 2.5" disk; the other locations can be either 2.5" or 3.5" disks.

Choice of CPU

My biggest hurdle was the choice between a low power CPU soldered on the motherboard or a regular pluggable CPU with some more power. CPU World certainly helped me compare the different CPUs, especially in combination with the PassMark Benchmark. I naïvely thought that "a Atom CPU would do", not realising that there are many types of Atom CPUs, AMD alternatives, and also other juicy CPUs that only consume 30-35 Watt of power.

For starters, I looked up the benchmark of CPUs that I used previously. My 3-year old MacBook Pro benchmarks at 1501, while a consumer laptop my wife just bought now benchmarks at 2668. Those are the high-ends for me. The low end is my Soekris net6501 router which also runs FreeBSD. While it's load is near-zero for the switching routing and VPN it does now, it is sluggish when I want to compile a new tool. So that's my low-end. A colleague with his own NAS had a Sempron 140 at first, but recently upgrade to a Athlon II x2 250e so he could use it as a more spicy server.

I was extremely impressed with the benchmark of the Intel Atom D2700 compared to its power usage. Since Atoms, like AMD E350s are only used as soldered onboard a motherboard, its availability is limited by the motherboards available. While making my purchase, no suitable motherboard was available with a Atom N2800 or D2700. I briefly considered a low power Intel core i3 variant, like the i3-2100T. After some consideration (including the $200 price increase for motherboard, cooling and separate CPU), I decided that my NAS wouldn't need that power, and decided that any low-power CPU with benchmark over 700 would do. That's comparable to an AMD Sempron, and good enough for basic file I/O and an occasional compile of a new kernel.

Choice of Motherboard

Mini-ITX has one expansion slot; mini-DTX has two expansion slots. So it's important to determine how many slots are needed. The main requirement for my motherboard was clearly the number of SATA ports. I needed as much as eight(!) SATA ports, and no motherboard had this many available. So I needed one expansion slot for a SATA card. Some (but not all) Jetway motherboards could use a SATA daughter board, so that would free the regular expansion slot. Unfortunately, the Jetways that support the SATA daughter board all have an old PCI instead of a PCI-express expansion slot, so that turned out to be a pig in a poke. I decided to rather buy a SATA adapter for a PCI-express slot. All modern CPUs (all AMD, Intel Atom, Pentium and Core listed above) have an internal graphics chips (GPU) on-board, so I didn't need a separate graphics card (it's a server after all). So my requirement was just one PCI-express slot, so I could use either mini-ITX or mini-DTX motherboard. In practice, a mini-DTX is harder to get by, so I ended up with mini-ITX.

Here are the issues I paid attentions at:

Brand

I was surprised to see that the better brands (like Intel and Supermicro) where not the best choices. Not only where they more expensive, they often didn't have a low-power product. So I mostly looked at Asus, ASRock, and to lesser extend Gigabyte, Jetway, Zotac and Sapphire.

SATA II or SATA III

At the time of purchase, SATA III (aka SATA-600) was getting more common. SATA-300 is certainly good enough for a regular hard disk drive, though SATA-600 might be useful for a solid state drive.

Expansion slots

Nearly all motherboards had a PCI-expression version 2.0 x16 (sixteen lane) slot, but only x4 (four lanes) are wired. Despite the difference in advertisement as either "PCI-e x4" or "PCI-e 2.0 x16", there was no difference in practice. Attention was necessary: some motherboards still featured old PCI slots instead of PCI-express, while others supported the newer PCI-e 3.0 standard.

USB 2 or USB 3

At the time of purchase, USB 3 was getting more common. (Edit: using USB3 gave some problems at the time. I could not use USB3 ports to boot from a USB stick when making the initial install, and the USB FreeBSD 8 at the time crashed with a hardware exception when trying to reboot the device. This was only fixed over a year later. The lesson learned is to be a bit more conservative in hardware choices, or at least be prepared for issues).

Memory

All models support DDR3, although some use regular 240-pin DIMM slots and others the "laptop" 204-pin SO-DIMM slots. The amount of memory that can be installed, and the speed of the DIMMs did differ considerably,

Network

All models provided one gigabit Ethernet slot. Some motherboards has on-board wifi, but the later was a non-issue for me: wifi is way too slow (both in bandwidth and in latency) for a NAS.

mini PCI-e/mSATA

A few motherboards had a mini PCI-e slot. Such a slot has the same form factor as a mSATA slot, but the latter is not standardised and in practice most mini PCI-e can only be used to connect a wifi circuit, but not a mini SATA disk. If you want to connect a disk, check for mSATA support.

RAID

Since I want to connect multiple disk with ZFS (sometime referred to as RaidZ), I don't want hardware RAID. From experiments in 2010 on a very high end system, we found that software RAID was as fast as hardware RAID. All models support both (legacy) IDE and AHCI modes. AHCI is slightly faster than IDE, but IDE is still the default setting in most BIOS, so be sure to turn it on before installing the Operating System.

Video connector

I have seen any combination of VGA, DVI and HDMI on the motherboard. Most build-in graphics processors even support two (but not three) concurrent displays.

Serial port

I would liked a serial port. Most motherboard still offer an on-board serial connector, but few still provide an external RS-232 port. Those that do tend to be the more expensive non low-power motherboards. In the end, I decided on an (expensive) Marmitek HD transmitter that transmits USB + HDMI over UTP. I was lucky to have picked a motherboard with HDMI (surprisingly, many motherboards still come with DVI or even VGA!)

FreeBSD support

It may be worth while to check support for certain hardware by the OS before purchase. This is particular true for expansion cards which usually feature a specific chipset.

The table below lists some of the better options. Despite that Intel has some pretty good south bridges, I wasn't able to find some low-power motherboards with enough SATA-3 ports. On the other hand, despite that the AMD A50M Hudson controller does not support USB3, the vendors en-masse added another chip to include USB3 on the motherboards. Note that the list does not contain maximum memory size and bus speeds; finding that information proved to be too unreliable.

My first choice was the Asus E35M1-I, but I learned that it was no longer available just when I wanted to make the purchase. The E35M1-I deluxe and E45M1-I deluxe were a lot more expensive, and had one fewer SATA-600 port (it was replaced with eSATA). My second choice was the Sapphire PURE Mini E350, but I decided against by lack of proper online documentation. The next choice was ASRock E350M1/USB3, which I bought.

Choice of SSD and HDD

As usual, I plotted some disk sizes and price per Terabyte. The optimum was at 2 or 3 TByte, so I bought two low power 3 TByte disks. I considered the Seagate Barracuda and Western Digital Caviar Green. In the end, I decided on the Western Digital, but both seem a good choice at the time. (Edit: in retrospect, I should have bought a WD Red instead of WD Green for a NAS. Also, I later got the recommendation to buy two brands of disks, as it is less likely that those would fail at the same time.)

Finding the right solid state disk (SSD) was more work. I was very pleased by the improved write speeds. Around 2010, the numbers were 250 MByte/s read and 70 MByte/s write for the (cheaper) multi-level cell SSDs, and 250 MByte/s read and 170 MByte/s write for the (more expensive) single-level cell SSDs. Since the SSD would mainly function as a ZIL (ZFS Intent Log), it would mostly need a fast write speed, so I went out looking for a single level cell (SLC) SSD. None was too be found. It turned out that multi-level cells (MLC) are now as fast as SLC with typical values of 500 MByte/s read and 500 MByte/s write speed. For my application (a ZIL), sustained write speed is important. For a database, the read and write IOPS (I/O operations per second) would probably a more important factor.

It seems that the caching of the SandForce 22xx series controller is the cause of the improved read and write performance. Given that experts such as Zsolt Kerekes are already writing about successors to the SandForce (including controllers from FusionIO and STEC), I expect more dramatic improvements in reliability and price in the coming years.

Getting reliable data on access times, reliability, and endurance is not possible, although all are important. Displaying a table with all models of Corsair, Crucial, Intel, Mushkin, OCZ and others would not be very useful (since it's outdated by the time you read this), I just listed the four Intel models with SATA 3 (6 GByte/s) interface. It does give you a hint that the differences in read and write speed and IOPS can be significant.

(Edit: For reliability, the Backblaze Hard Drive Data and Stats may be a good starting point.)

Choice of Other Hardware

Additional SATA Controller

I was not able to find a SATA controller with 4 SATA-600 ports, so I opted for the Digitus DS-30104 which has 4 SATA-300 ports. This model seems to be sold under different names, including Exsys EX-3505, Rosewill RC-218, and Dawicontrol DC 324e. There were other options available, but this controller has a Marvell 88SX7042 chipset which is certainly supported under FreeBSD.

Replacement Fans

The chassis comes with two fans: a 14cm for the air inlet at the front and a 12cm fan for the air outlet at the top. I was not very happy with the fans, especially because they would both run at full speed all the time: neither has a 4-pin PWM (pulse width modulation) connector, meaning that the motherboard does not control their speed.

In addition, the CPU was still fan cooled, and the PSU also has a build-in fan.

I decided to buy replacement fans, hoping to reduce the noise. The noise of a fan is proportional to the fifth power of the rotational velocity. So a fan at 2000 rpm generates 2⁵ = 32 times as much noise as the same fan at 1000 rpm. Sound is usually measured in decibel (dB), which is a log-scale of the energy. So a factor of 32 in energy equals to a offset of (10×¹⁰log 32 =) 15 dB. That's a big difference. Here is some stats for noise:

There's a second thing to consider: a 1200 rpm fan makes 20 rotations per second. So I suspect a large part of the noise to be around 20 Hz. That just happens to be the threshold for the human ear: we can't hear lower frequencies. Again, lower frequency means less noise.

Since the chassis I choose does not have a 5.25" bay, I could not install a third party fan controller, and had to use the controller on-board the motherboard. I decided that the 12cm outlet would be controlled with a PWM connector, while the 14cm inlet would run at full speed, so it could cool the disks, regardless of the motherboard temperature. (The motherboard only had one PWM connector, and one non-PWM connector, but a few manufacturers offer an option to share the same PWM pin with multiple fans.)

The result was remarkable, which prompted to also buy a replacement CPU fan after all. I tried to do some measurements, but lacking professional equipment, I only could use my iPhone for some crude measurements. It later turns out that the iPhone can not register sound levels lower than roughly 30 dB(SPL).

While a low-noise fan does make a lot of difference, it certainly is highly non-trivial to measure the cooling capacity versus the sound level of a fan. The whole discussion triggered me to write an article on sound levels. Without going into details, here are the take aways:

• low speed (rpm) means much less noise. Either get a PWM-enabled fan (which runs at low speed if it's not required), or get a fan that always runs at low velocity. 1000 rpm is often enough. Don't go above 1300 rpm.
• Having a bit of cooling is better than no cooling. Anything above 40 cfm (cubic feet per minute) only displaces more air (and makes more noise), but does not actually cool better.
• Be careful with 14cm fans in a 12cm form factor, or 3.5 cm (instead of 2.5 cm) thick fans. They may not fit in your chassis. However, if they fit, they may offer a better cooling at the same or lower velocity.

Here is a table with the product with the best specs listed first:

I bought the Noctua NF-P12 PWM and Sharkoon Silent Eagle 800 for the chassis cooling. Noctua's products are generally expensive and are considered the best in the market. Sharkoon should be complimented as the only manufacturer who publishes a table of airflow and noise levels for various PWM percentages (as opposed to only the best figure).

For the CPU fan, I first bought a Fractal Design Silent Fan FD-FAN-40, but it started to become noisy after only a few days. I must have gotten a rotten one. I replaced it with a Noiseblocker BlackSilentFan XM1.

Artic Cooling should be critized for publishing the sound levels in sone, which is a rather subjective measurement. dB(A) doesn't have perfect equal loudness contour, but at least is is exact (provided that the distance and position between fan and noise meter is known). My recommendation is to convert Sone to phons, and use that value as a rough estimate of the number of dB(A). Perhaps add up to 10 dB(A) penalty because the dB(A) equal loudness contour is closer to the 0 dB point that the ISO 226:2003 equal loudness contours for low noise levels at low frequencies.

Power Supply

Four factors influenced my choice for the power supply unit (PSU): physical size, power output, available connectors, and efficiency.

The Lian Li PC-Q25 only supports a small power supply unit (PSU). In fact, the company changed the specs from "maximum 180mm" to "maximum 140mm" after reviewers complained about it.

I expected the PC to run at 100 Watt power even with all disks installed, and most PSU are well above 300 Watt. With such a margin, this was of no criterium to me.

I only needed a 24-pin power socket for the motherboard, three 4-pin molex connectors for the five disk bays (some bays share the connector), and three SATA power cables. I didn't expect to use any more connectors for fans (the motherboard has two pins) or PCI-express cards. All PSU that I considered offered this, so this was also not a big criterium to me.

Power efficiency was important to me. Most PSU have an efficiency of 80%, thus 20 Watt of each 100 Watts input would be spoiled as heat in the PSU. I instead opted for a Chieftec A-85 Series CTB-350S, which has a 85% efficiency.

The Result

I'm rather satisfied with the result. It seems to bring flexibility and lots of disk bays with a moderately sized chassis and low power and low noise.

The power consumption appeared to be about 60 Watts, with a minimum of about 50 Watt for idle, and 80 Watt during high load and lots of disk activity.

The noise is low enough that it can be used in a workplace, especially after replacing the CPU fan to the BlackSilentFan. It is not silent though, and the noise level is roughly as annoying as the noise of a desktop PC, certainly less silent than a laptop. The noise is there, and low enough for me not to be able to pinpoint the source, but loud enough to be noticeable. I can work with it, but your mileage may vary. It is too bad that I wasn't able to buy the Asus E35M1-I which not only had more SATA-600 ports, but also had fan-less CPU cooling.

I have not yet done extensive IOPS testing, but suspect that it's good enough for a file server.