The AC97/HD Audio setting is for front audio ports, and has nothing to do with optical. But if your mobo has HDMI, there should be an HDMI - S/PDIF setting in the build-in devices menu, be sure to set to S/PDIF.
Be sure to get latest drivers. You don't say what OS you run, and not sure if XP has it, but under Vista/7 you should get Dolby Digital Live support. Go into Realtek control panel (looks approximately like this: http://img363.imageshack.us/img363/2728/ddlive0fw.jpg ) and set it to Dolby Digital 5.1. Also, go into settings for optical and play with sample rate. Older receivers do not support 192kHz - you may need to bump it down to 96.