Just curious - what is it that you consider "kernel bloat" ?
If you copy things to your home directory, you should make sure you own all the files (make sure you set the permissions to 'me:me' on all the files, assuming 'me' is your username). And you only need root to do the install, not the compile.
It is possible, but perhaps you are making a mistake somewhere?
Follow the link listed below and from there see if you still have problems:
http://fedoraproject.org/wiki/Docs/CustomKernel
Also, I hope you know that you do not have to always install the nvidia driver by hand. You can use 'akmod' which will dynamically recreate the nvidia driver (kmod-nvidia) everytime a new kernel RPM is installed:
# yum install akmod-nvidia
Also, there is a small possibility that if you deviate too much from the Fedora kernel that the kmod-nvidia you have may not work. Since it is compiled for the Fedora default kernel, not *your* modified kernel. Although you can always recompile the kmod-nvidia from source against your new kernel as well.