Took me a while but I figured it out. Just in case someone else has this problem:
It's not the browser encoding settings, it's the Java applet runtime settings. My browser was already set for utf-8 encoding when I had this problem.
The applet here shows you what your default Java runtime encoding scheme is. For me it was Cp1252. By adding "-Dfile.encoding=UTF-8" to the runtime parameters, and restarting your browser (making sure the Java process restarts as well), you can make your Java runtime display it in utf-8. (I rechecked that this was the reason by changing it back to Cp1252 and restarting my browser again).
In Windows (haven't tried it in *nix), when you run Java there's a little icon that you can right click to get to the Java Control Panel. Once you open that, go to the Java tab, then click on the "View" button under Java Applet Runtime Settings, from there you should see a window similar to the attached image.
At the row with the version of Java that is currently enabled ('Enable' checkbox checked), add the following under the "Java Runtime Parameters" field, add "-Dfile.encoding=UTF-8".