The audio carried by HDMI comes in two general flavors: PCM and bitstream. PCM is just regular, unencoded sound in a format similar to a CD. It has certain properties like depth (16bit vs 24bit), resolution (32 to 192kHz) and comes in a number of channels from 2 for stereo to 8 for fulll 7.1 surround. The digital coax/optical connection can only pass 2-channel PCM, whereas HDMI can do 8. The 4000-series Radeon cards support 8-channel PCM via HDMI (3000-series only did 2-channel PCM).
Then, bitstreams. Those are encoded compression formats designed to pass more sound over less bandwidth. There is a number of Dolby and DTS formats designed to work over 2-channel pipe that coax/optical connection provides. Most popular probably is Dolby Digital 5.1 which passes 6 channells of sound, and is widely used on DVDs and HDTV broadcasts. Then, you got lossless 7.1 HD formats like Dolby TrueHD and DTS-Master designed to work only via HDMI and usually found on Blu-Rays. In order to play a bitstream format, a device must be capable of decoding it..
If you use digital coax or optical connection that does not 'talk back', it is up to user to assure that destination can handle the source. User can set source device to any kind of output, but if destination device can't handle it, you get silence. With HDMI, destination device provides source with an EDID which lists supported formats - and in most cases, source will restrict user to those formats only.
A monitor with HDMI input is likely to report only PCM-2 in its EDID - which means that is all your PC will send it. It is still worth it to go into your audio devices manager, selecting ATI HDMI Audio, clicking 'Configure' and choosing 'Supported Formats' tab. (AFAIK, this is only doable in Vista/Win7, not in XP). For all you know, Dolby Digital support might be there, and all you need to do is check the box - but chances of that are slim. Here is a writeup (which I myself did not try) on how one might override monitor EDID for advanced audio support.