I want to have my home machine be accessible from the Internet through VNC.

If I run a x11vnc server, connect to the machine remotely through ssh -L, and then connect with VNC client, I still need to have SSL enabled on x11vnc. And if I enable SSL on x11vnc - why can't I just securely(?) connect straight to the machine through VNC client that supports SSL?

I'm not talking about portforwardings, just from security perspective (or in general), where does the difference lie when I connect

vncviewer IP:5900

versus

ssh -L 5900:localhost:5900 username@IP
vncviewer localhost:5900

From a user experience, I have to enter two passwords on the second scenario, however isn't it a bit more dangerous too to have SSH open to the world?

P.S. I know not to use standard ports, and I can't create strict rules on my router limiting IPs that can connect.

What I would do is to set up a cryptographically-secured OpenVPN server on your home machine, with tls-auth protection as I describe in my blog here. Then, restrict everything so that it only listens to ports that represent successfully-connected OpenVPN clients, and use firewalls to guarantee that there is no other avenue in or out.

Now, you simply use your client – which contains your tls-auth credential and your non-revoked, one-of-a-kind, 4096-bit certificate – to connect to your tunnel. From there, you can simply use VNC, or anything else you please.

Anyone who "port-scans" your system will find ... nothing. Anyone who suspects that you have an OpenVPN server will be unable to detect it. The number of unauthorized access attempts will be zero.

And yet, you won't be inconvenienced at all. You can reach your home system as though it were sitting on a nearby subnet accessed through a router – because OpenVPN is "a secure software-implemented router." You don't have to use any application-specific security or cumbersome tunneling, or bear the risk that something might not actually be secured.

Disclaimer: I never use vnc, but I do tunnel other protocols through ssh. As far as I can tell, the vnc ssl solution is encrypted, but not authenticated in any way. A vnc expert can correct me if I'm wrong. This means that just about anyone can connect to your login page. Hope your system is up to date.

For the ssh tunnel, if you're entering your password twice, you're doing it wrong. You shouldn't expose ssh with passwords enabled -- you should always use key-based authentication. In my opinion, I trust ssh more than vnc or even openvpn, and every one of my servers has ssh port open. In my experience, the annoying password guessers filling your logs are eliminated by moving ssh to the right port number.