Never had any problem with a few unpatched kernels.
What's really interesting for desktop machines is the support for more hardware, like DVB or motherboards,..
apt-get source linux-image-2.6xxx
Then look at the diff.gz file if you want to know the patch - easy
There's a webpage available also, where is the question
You can keep your .config that's important. make oldconfig or make directly(default option if you don't make make config, it looks for an old one in /boot). Takes about 15mn otherwise to redo it no?