SunOS 5.6 = Solaris 2.6
There is nowhere to get 2.6 anymore. It hasn't been used on production systems for several years, but you are correct to want to keep it. I find that it's much faster on older hardware (I own a dual proc SS1000 - Ebay ~120). Speaking of EBay - there are usually boxed software sets for 2.6 there. However, 2.6 and Java /barely/ get along - so alas, I'm taking the plunge, and upgrading to Solaris 8 (free download from sun.com).
Unknown as to the origin of your Bad Magic number - although - check your hardware support. Sol9 dropped my SS1000, so make sure your hardware is supported by the newer Solaris.
Linux will install on a large amount of Sun Hardware, but you may have to search around for decent tips and tricks. Check out the Solaris forum at
www.tek-tips.com, it's been around longer than the Solaris forum here, and has some back-threads about Linux on Sun Hardware.