You can either select the maximum supported resolution of your set (and the PS3 will assume everything listed above your choice is also supported):
Or, if you select Custom, it simply allows you to manually select which resolutions are supported from the list:
Note that these are with a Component/D-Video cable. I don't know if the options are different for HDMI.