The nut's job is to hold the strings, properly spaced, as they pass from the tuners along the fretboard. The string-slots set the height of the strings (at that end of the neck) and the nut essentially makes up one 'anchor-point' of a string's 'sounding length' (with the other being the bridge/saddle).
If we just consider 'conventional' nuts (leaving locking and roller nuts out of the picture for now), the main thing to focus on is what it's made of.
Through most of the 20th, bone (generally cattle bone) was the most common nut material used on guitars and similar stringed instruments. Ivory nuts were sometimes used too but, for obvious reasons, these become much less common as we get closer to the present-day (not that ivory was widely used anyway—it was a little bit of a luxury item).
More modern guitars see a much wider range of materials being pressed into service for nuts. Budget instruments will frequently be fitted with a moulded, plastic nut. In many cases these are pretty nasty and don't tend to wear well. Cheaper plastics don't tend to sound very good either.
The next step up the plastics chain is the harder, composite materials. These wear and sound a little better and are relatively serviceable. There are a bundle of different materials that fall into this category and, these days, all but the cheapest instruments will generally have a reasonably hard material used for its nut.
Micarta, Corian and Tusq
Synthetic materials with brand names you'll probably know (Micarta, Corian, Tusq) are becoming much more popular and widespread. Corian (used by Gibson for pretty much everything these days) and Micarta are actually products developed for use as kitchen counters.
These are pretty good sounding materials and certainly a hefty step up from the usual plastics. I find that they tend to wear a little more quickly than a bone nut and, on a purely personal note, I find them a pain to work with (not that you should worry about that).
On the aesthetic side, these materials 'age' differently too. Gibson still finish most instruments with a nitrocellulose lacquer which, as it gets older, fades and yellows. It's a great look in an older instrument. Corian nuts don't fade or yellow. They stay WHITE. A Corian nut on an ageing guitar will look like a septuagenarian Hollywood actor with a brand new set of false-teeth.
Tusq, on the other hand, has the opposite problem. Tusq darkens pretty quickly and keeps going. I've had guitars where I'd have trouble telling the Tusq nut from the rosewood fingerboard. That said, I find Tusq to have the best tone of the synthetic nuts (mainly as it's the closest-sounding to bone).
Graphite, PTFE, and Lubricated Nuts
Getting all high-tech now. There have been a few different types of 'lubricated' nuts. These are materials similar to those mentioned above which have been impregnated with a lubricant like graphite or Teflon/PTFE. A material, marketed as Slipstone (actually Delrin), has also been used as guitar nuts although this is harder to find in guitar-friendly forms at the moment. There are other materials too but these are the more common you'll hear about.
The idea of a lubricated nut is obviously to cut down friction so you'll often find them installed or retro-fitted on trem-equipped instruments to help with the tuning.
Bear in mind, though, that the nut's setup has a massive impact on tuning stability. A poorly fitted and slotted graphite nut will do little to improve tuning just as a well cut and installed non-lubed nut can do wonders for tuning stability.
Ivory was once considered a snazzy and great-sounding alternative to a common bone nut. Wouldn't it be great if we could get this little piece of luxury without having to go around hacking the tusks off elephants?
Enter fossilised ivory. This stuff died thousands of years ago. Apparently.
You can buy nuts (and saddles) of fossilised walrus ivory. If you're more adventurous, you can buy fossilised mammoth ivory too.
Now, the sceptic in me wonders at the number of places on the internet that sell mammoth ivory and tries to imagine how many mammoth tusks have been found and put aside for the sole use of the guitar parts trade. I imagine museums, desperate for money, reluctantly sawing up tusks to be shipped to AllParts.
Still, if nothing else, it's certainly a talking point to say you've a chunk of wooly mammoth in your guitar. As far as looks and tone goes, it tends to be quite similar to bone—maybe a little harder and more ivory coloured (obviously).
Mother of Pearl
Still in the luxury category with this one. To be honest, I'm mentioning this for the sake of completeness as it's not something that you'll often see. Mother of pearl was occasionally used in super-high-end instruments, mainly from the early end of the 20th century. It's pretty damn rare to see a MOP nut but it's so damned hard that the 100-year old originals are probably wearing just fine.
I blame the seventies. That's when the brass nut craze really got going. Not to say it's gone—I still get an occasional request for a brass nut. They're absolute hell to cut and slot but they can look well when properly polished up and some swear by their tone.
Not me, though. I prefer:
Yes, I'm a traditionalist. I like to have a hunk of cow's leg sitting at the end of my neck. Pound for pound, I think bone tends to be one of the best looking, best sounding and best wearing materials for a nut.
As it's a natural material, there can be inconsistencies—a softer spot, for instance—but, if you're careful you can usually tell that before you start working. And, talking of that, it's much easier to work with than pretty much all of the materials already mentioned.
Guitar/Bass Nuts and Tone
Here's the thing: All of these materials will have subtly different tones. The key word is subtle. While you'll certainly see an improvement moving from a budget, plastic nut to a nicely fitted bone nut, the difference between, say, a Tusq nut and a Corian or a bone nut is much less marked.
Here's the other thing: Any discussion about a nut's tonal benefits applies only to the open string. Once you fret a note, that's the nut out of the picture.
The Bottom Line: Is It Worth Upgrading?
Yep. Especially if you've got an instrument with a plastic nut.
A well-cut nut of suitable material can improve tuning stability and can make for a better tone from your open strings.
What sort of material? Take your pick, really. While there's tons of choice, it's probably worth remembering that, unless you have a particular necessity or deep desire for some other material, you unlikely to go far wrong with a bone nut.