I used to do this many years ago, but for some reason I am not able to anymore.
I downloaded this to /boot, decompressed it and renamed it to 'memtest'.
I added this to /etc/lilo.conf:
When I select it in lilo boot menu, the screen goes dark for about 3 seconds and then the POST messages appear.
I tried this with the previous version (4.20) and the result is the same.
What am I doing wrong?

is memtest installed?
on my system it's "memtest86+"

edit:
oops i just noticed you did it manually.
on my distro it is possible to install it via repos, tried that?

or maybe it's just not executable? (sudo chmod +x memtest)

But I don't intend to boot linux and then run memtest86+ as an application. I want to boot directly the memtest binary. Does it still have to be executable for lilo to read it and run?

memtest does not run "as an application".
i don't know about the executable bit.
but why guess at things, when your packet manager can do it for you, properly?
you haven't answered all my questions.

Well, a package manager implies an installed OS, and while I do have two installed, to run memtest you don't really need one, do you? You just need to run (even from a live CD/USB) lilo to put the necessary code in the MBR of a HDD to point to where the memtest binary is located, isn't it?
Now I have lilo working and able to select at boot time to start windows or linux, but when I tried to add memtest, for some reason it doesn't work.
So my question really is: does memtest need anything special to be configured in lilo.conf other than what I already have mentioned above?