Most nuts have a pretty simple design. The strings sit in slots and 'take off' from the front edge of the nut. This marks what is effectively the 'zero-fret' position and each string 'starts' at that same zero location.
It doesn't have to, though.
It's possible to move the starting point for each string slightly back towards the headstock or forwards towards the bridge.
Why would you do such a thing?
Well, in a similar way to that in which your bridge saddles are compensated to intonate the guitar and help provide some tuning consistency all over the neck, the nut can also be compensated to help a guitar play more in tune.
I should note that, when talking about playing in tune or tuning consistency here, I'm not discussing the way a guitar slip out of tune. I'm talking about the way each note (open and fretted) should be neither too sharp or too flat compared to what it should be. For instance, if I play fret the third fret of the fifth string, I want to hear an in-tune C note.
Compensation is the adjustment of a string's length so that it plays as closely as possible in tune at each fret position. Generally, on a guitar, we compensate (or intonate) by moving the position of the bridge saddles. On an acoustic guitar, you'll have noticed the saddle is generally installed at a slight angle—so the sixth string is a little longer than the first. It's possible to further compensate the acoustic saddle itself.
Even the best setup guitar can still have some fret positions where a sensitive ear can hear some inconsistencies. In many cases, this hangs around the first few positions.
How to address this?
The Compensated Nut
A compensated nut on your guitar or bass moves the 'take-off' point for the string back or forwards in an effort to improve overall tuning consistency. It's essentially the same process as for the bridge.
Some manufacturers, such as Music Man, fit compensated nuts to their instruments as standard. Each string slot has a shifted 'edge' or take-off point.
It's also possible to have a compensated nut retro-fitted to your guitar or bass.
The big players on the retro-fit scene are mostly Buzz Feiten and Earvana. These take slightly different approaches to address similar problems. Earvana's nut is similar to the Music Man nuts pictured above (and, in fact, I seem to recall a lawsuit between those two companies on the subject). Earvana adjusts each string by a different amount and the instrument is then set-up and tuned as normal.
The Buzz Feiten system uses a 'shelf' nut that moves the string's take-off point closer to the bridge by a particular distance (calculated based on a number of factors including string-gauge, scale-length and fret-width). Each string is moved by the same amount but the instrument is intonated differently to 'sweeten' the tunings. When tuning the guitar, you tune to the 'E' note on each string rather than tuning the open strings (or just simply use a Feiten-approved tuner).
Both approaches can make for very good results. Personally, I prefer the Buzz Feiten as I think it looks and sounds better (it's a bone nut) and more 'musically correct'. Full disclosure: I do have a dog in this race, though, as I'm an authorised Buzz Feiten fitter.
The Bottom Line: Is It Worth Upgrading?
It depends. Most of us are happily hammering away on our guitars without being bugged too badly by intonation and tuning issues and, even with a standard nut, a good setup can go a long way towards addressing most problems for most people.
However, if you're constantly on edge that some of your chords just refuse to sound right; if you're playing with a pianist/keyboardist and are plagued by tuning issues on certain notes, it might be worth considering.
Don't be surprised, though, if a good tech advises you to consider a fret level/re-crown at the same time though. Worn frets can bring their own intonation issues and it's generally a good idea to address these at the same time so you'll get the full benefit.
Also, remember that these aren't magic. While a compensated nut can improve things, there are limitations inherent in any fretted instrument with equal temperament. That's probably a discussion for another day but keep in mind that we can never achieve perfection (we can get close, though).