Choosing the hardware

Just for the record, before building my own server I used a D-Link DNS-323 NAS. This was actually an interesting 2 bay device, with web management, multiple hard drive configurations (including mixed configuration, for example you can choose to have half of your drive space as a RAID 1 array and the rest in a JBOD configuration), user management with drive quotas, email alerts, power management, ftp access, UPnP server, download manager, USB print server port, backup software. Unfortunately the performance level is rather limited, topping at around 15 MB/sec on a gigabyte connection.

Of course you can do all of this (and much more) and with a better performance with a self built server, but you will have to work a little.
Choosing the components is probably the longest activity when building your own server. After establishing your requirements you will have to find the components that suit your needs and your budget and probably read reviews on your most trusted sites. Keep in mind that user reviews on big sites like Amazon should be always taken with a grain of salt, because not all users know what to do inside a computer. If you are good with hardware (or have a friend with good knowledge) you have an advantage. For a home server you will most certainly use the same components as you will use in a desktop computer, not those of a big business server.
My choice of components was highly influenced by the fact that I wanted a small form factor case but with support for at least 4 hard drives beside the system disk.
So I’ve gone for a (beautiful) silver Lian Li PC-Q25 Mini ITX case. It’s a small case, with 2 relatively quiet system fans (140mm front and 120mm top), 5 hot swap 3.5-inch bays and support for another not hot swap two 3.5 drives or three 2.5 drives. There may be two problems with these case – you must use a maximum 140mm length power supply and you have absolutely no front panel connectors or optical drive bays (just a power button with a led). The drive bays support is exceptional for such a small case and cooling potential good. This case will be 129.99 USD on newegg today. You can go smaller if you don’t need so many drive bays and you certainly can go cheaper than this (but you may have to chose a bigger case).
Unfortunately that limit of 140mm for PSU didn’t allowed me to use a modular power supply, so the cable management was tricky. I have chosen a Seasonic S12II-430W, reliable, quiet and efficient. This has plenty of power and lot of connectors (maybe event to many for such a small case). 59.99 USD on newegg. If you don't have the size constraint, I would recommend a modular PSU for better cable management. You can go for a case with included PSU, but this should be good quality (like those from Antec) and have enough power and connectors for your needs.
Of course with a Mini ITX case I will use a Mini ITX motherboard. In this case an Asus P8H77-I, with 6 SATA connectors, 2 USB 3.0 on the back panel and another 2 on-board. This allows me to use one SATA connection for the system disk and connect all those 5 hot swap drive for future use. The missing part – no eSATA connectors. For reference now this costs 109.99 USD. This board is complemented by an Intel Core i3 2120 processor (dual core with hyperthreading) with box cooler for 124.99 USD and 4GB RAM Geil 1333Mhz Value Plus at 22.99 USD. The number of Mini ITX boards on the market is much lower than than mATX or normal ATX. You can also go for a board with included processor (Intel Atom or AMD APU), but you will probably not get that many SATA ports and will have lower performance (be very careful especially if you want good support for virtual machines or you want to do transcoding); also you can go for a lower performance processor and still have plenty of performance for most tasks (a dual core Celeron G530 should suffice and the price difference is considerable). 4GB RAM is enough (maybe even 2GB for lower needs), and the speed or the producer is not that important. As this will not be used for games, I don’t need a graphic card, and the Intel processor has all the needed graphic “power”.
For the noise of the case fans I got a fan controller. As the case has no front bay, I got one which can go on the back of the case. It controls both fans with a single knob (not the best choice, but for me it will do). It’s an Xigmatek XT-MONOCOOL which can control 3 fans, at 10.99 USD. Some cases already have a fan controller included.
I decided to use a separate system disk, and to get the maximum performance if I also put one or more virtual machines on it, I’ve gone for a SSD. An Intel 330 series 60GB offers plenty of space and performance at around 89.99 USD. This will probably do even if you decide to use Windows (7 or 2008 R2 Server). For Linux, a standard install will be very far from using so much space (my install is now, with no virtual machines, under 3 GB; I also have space reserved for a swap partition). So for Linux you can go with enough peace of mind with a smaller disk, or even an USB thumb drive (for FreeNAS this is actually recommended by the producers); or, why not, you can chose a smaller size mechanical drive, even a 2.5-inch version.
So, the grand total, before adding the mass storage (which is why you actually build this system) is at almost 550 USD, delivery or rebate cards not included. This is expensive, but you can go much lower with different choices, if you don’t need some of this bells and whistles. Just by choosing the Celeron G530 you will save a massive 75 USD, a smaller or lower performance SSD can add another 30-40 USD savings, and a few other small savings or rebates can easy get you over 25% less, WITHOUT loosing performance in most day to day tasks; virtual machines, transcoding or a lot of multitasking maybe affected, but if you don’t need this you can save good money. So 400 USD will give you high quality and performance you can’t actually compare with the ready-made NAS on the market (at least with those you can buy). And don’t forget you can actually UPGRADE this build, something you cannot do with a commercial NAS. For example a 4 bay Synology DS411 will cost 450 USD and won’t give you the same level of performance or features. And keep in mind that my build is a FIVE bay server, and a 5 bays NAS from Synology or QNAP, with similar levels of speed as the cheaper build will take you above 800 USD so double the money (this will come with Atom processors and 1 GB RAM, but will give you eSATA ports and dual LAN connections).
Actually in my country prices are around 20% higher than this, but as this is valid for all prices, the comparison is still valid.
For mass storage I chose to reuse for now my two 500GB Seagate Constellation ES.2 in RAID 1 from my old NAS (for those all important pictures or backups of my projects) and get a 2.5TB Western Digital Caviar Green for media files (this may transform into a RAID 1 or 5 when HDD prices drop again).