I have been thinking for the longest time that R could only handle one graphics windows, I should have looked a little deeper. It turns out that it is very easy to do.
To create a new window, simply type: windows() or X11(). This will open a new graphics device and give it the next available number.
To make window k the active window, the command is windows(k).
To close window k, dev.off(k) is the way to go.
For more commands and information, see help(dev.cur) in Rgui.