Figured out how to do a file upload in a cfwindow using cfform. If you try to receive a file upload in a cfwindow, regardless of the enctype specified in the cfform it will incorrectly receive the form as of type "application/x-www-form-urlencoded", instead of the more-better "multipart/form-data. My guess is that this has something to do with the ajax used to submit the form. This error happens even if the originating form is in the cfwindow.
Easy work around? Embed an iframe in the cfwindow, put your form in the iframe, and have the form submit within the iframe using a simple submit button (rather than using ColdFusion.navigate). If you submit the form using a submit button in an iframe, the form's enctype will submit properly, and ColdFusion will correctly receive your file and pass it onto cffile.
This technique will also work with putting a cfpresentation tag in a cfwindow.