ldd falsely generates "not found" for valid library

selfprogrammed · 08-03-2023, 12:21 PM

Thanks for all the responses.

I went to try that bit from the Wiki about finding the rpath, but there is a problem.
The deps program I am writing only knows the library that it found. It does not know what program (like Firefox) that might be trying to use it.
It might even be used by different programs, using different RPATH settings !

I think the false positives problem is going to have to recognize that a library in a non-standard location may NOT be fully checked using ldd.
There will probably need to be a check for dependent libraries among ALL the libraries found on the system anywhere.
If such are found in a non-standard location, it would warrant a special message to the user that the directory was found, but is in a program dependent location.
Finding the dependent library in the SAME directory as the library being checked, it would probably be safe to assume that all is OK and no message would be necessary.

How, does that sound.

Is anyone making these dependency checkers available in any location where the general user could actually find them.
They should be shipped with the Slackware distribution. This is a common enough problem.

This is probably as solved as it is going to get, but I will leave it for a while, as I am not done with it yet. Got too many of these programs going at the same time, and I am likely to forget.

selfprogrammed · 08-03-2023, 12:31 PM

Testing the logic of the Wiki about RPATH.

>> ldd /usr/lib/seamonkey/libmozavcodec.so
libmozavutil.so not found

>> readelf -d /usr/lib/seamonkey/libmozavcodec.so | grep PATH
gives nothing

>> readelf -d /usr/bin/seamonkey | grep PATH
gives nothing

Looking at the elf of seamonkey does show the standard libraries it uses, but not any of the libraries from /usr/lib/seamonkey.
I think that RPATH is not going to be of any use.

Petri Kaukasoina · 08-03-2023, 02:05 PM

Yes, seamonkey and firefox don't use RPATH. And the problem can't be connected to RPATH, anyway.

Code:

$ objdump -p /usr/lib64/NetworkManager/1.42.8/libnm-device-plugin-bluetooth.so|grep RPATH
  RPATH                /usr/lib64/NetworkManager/1.42.8
$ ldd /usr/lib64/NetworkManager/1.42.8/libnm-device-plugin-bluetooth.so
        linux-vdso.so.1 (0x00007ffea452d000)
        libbluetooth.so.3 => /usr/lib64/libbluetooth.so.3 (0x00007fbfa3cb3000)
        libnm-wwan.so => /usr/lib64/NetworkManager/1.42.8/libnm-wwan.so (0x00007fbfa3c9b000)
        libmm-glib.so.0 => /usr/lib64/libmm-glib.so.0 (0x00007fbfa3b7d000)
        libgio-2.0.so.0 => /usr/lib64/libgio-2.0.so.0 (0x00007fbfa399d000)
        libgobject-2.0.so.0 => /usr/lib64/libgobject-2.0.so.0 (0x00007fbfa393d000)
        libgmodule-2.0.so.0 => /usr/lib64/libgmodule-2.0.so.0 (0x00007fbfa3934000)
        libglib-2.0.so.0 => /usr/lib64/libglib-2.0.so.0 (0x00007fbfa37f2000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fbfa35fb000)
        libz.so.1 => /lib64/libz.so.1 (0x00007fbfa35e1000)
        libmount.so.1 => /lib64/libmount.so.1 (0x00007fbfa357a000)
        libffi.so.8 => /usr/lib64/libffi.so.8 (0x00007fbfa356e000)
        libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007fbfa34d1000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fbfa3d2d000)
        libblkid.so.1 => /lib64/libblkid.so.1 (0x00007fbfa3477000)
$ 
$ objdump -p /usr/lib64/gconv/EUC-CN.so|grep RPATH
  RPATH                $ORIGIN
$ ldd /usr/lib64/gconv/EUC-CN.so
        linux-vdso.so.1 (0x00007ffcb09f2000)
        libGB.so => /usr/lib64/gconv/libGB.so (0x00007fd6c24c0000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fd6c2299000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fd6c24dc000)
$

ldd finds libraries in those nonstandard places with the help of RPATH just as well as the dynamic linker were using them normally. It's understandable as ldd actually does that, with LD_TRACE_LOADED_OBJECTS set. Compare output of commands

Code:

/lib64/ld-linux-x86-64.so.2 /bin/ls

and

Code:

LD_TRACE_LOADED_OBJECTS= /lib64/ld-linux-x86-64.so.2  /bin/ls

The first one runs ls and the second one does the same as ldd...

In case of firefox, the parent process does not have LD_LIBRARY_PATH in its environment, but it sets it in all its children, as I showed in post #8. If you search for string LD_LIBRARY_PATH in files of firefox package, it's found only in libxul.so, which is loaded by firefox parent, as shown by

Code:

ls -l /proc/$(pgrep firefox)/map_files|grep libxul.so

How has firefox loaded libxul.so, as it's not listed by 'ldd /usr/lib64/firefox/firefox-bin' ? Probably loaded with dlopen, as firefox-bin links to dlopen. Actually there are 146 so files listed there in map_files, and ldd shows only half a dozen (none of them "not found").

Petri Kaukasoina · 08-03-2023, 03:48 PM

This reminds me of a faq some time ago, in Slackware 14.2. People were asking why firefox did not play videos. The answer was that at some version firefox started needing ffmpeg that was not included in the distro, but was provided by AlienBob. If you try to use ldd, you can't find that firefox depends on libavcodec of ffmpeg. A firefox library, libxul.so, loads a libavcodec library using dlopen, libavcodec with one of many possible SONAMES, see:

Code:

strings /usr/lib64/firefox/libxul.so|grep libavcodec

Petri Kaukasoina · 08-04-2023, 04:22 AM

Quote:

Originally Posted by selfprogrammed

Finding the dependent library in the SAME directory as the library being checked, it would probably be safe to assume that all is OK and no message would be necessary.

How, does that sound.

Yes, that assumption removes many false postitives. There is a little script below that goes through all the installed packages and always sets LD_LIBRARY_PATH to the directory where the program or library is. If ldd tells a library is 'not found', the script checks if it's really NEEDED according to the elf headers and not indirectly via some other library.

Code:

#!/bin/sh
#search for missing library dependencies
OBJDUMPOUT=$(mktemp)
LDDOUT=$(mktemp)
trap 'rm $OBJDUMPOUT $LDDOUT' EXIT
cd /var/adm/packages
for pkg in *; do
( cd /
  while read line; do
    [ "$line" = "FILE LIST:" ] && break
  done
  while read f; do
    if [ -x "$f" -a -f "$f" -a -r "$f" ]; then
      if LD_LIBRARY_PATH=$(dirname /"$f") ldd "$f" 2>/dev/null|grep -q 'not found'; then
         objdump -p "$f"|grep NEEDED|sed 's/ *NEEDED *\(l.*\)/\1/'>$OBJDUMPOUT
         LD_LIBRARY_PATH=$(dirname /"$f") ldd "$f"|grep 'not found'|sed 's/.*\(lib.*\) => not found/\1/'>$LDDOUT
         if grep -qf $OBJDUMPOUT $LDDOUT; then
           echo "$pkg":/"$f"
           grep -f $OBJDUMPOUT $LDDOUT
           echo
         fi
      fi
    fi
  done 
) < $pkg
done

This is the output from 15.0:

Code:

mozilla-firefox-115.0.3esr-x86_64-1_slack15.0:/usr/lib64/firefox/gmp-clearkey/0.1/libclearkey.so
libmozsqlite3.so

speech-dispatcher-0.10.2-x86_64-5:/usr/lib64/speech-dispatcher-modules/sd_baratinoo
libbaratinoo.so

speech-dispatcher-0.10.2-x86_64-5:/usr/lib64/speech-dispatcher-modules/sd_ibmtts
libibmeci.so

speech-dispatcher-0.10.2-x86_64-5:/usr/lib64/speech-dispatcher-modules/sd_kali
libKali.so
libKGlobal.so
libKTrans.so
libKParle.so
libKAnalyse.so

speech-dispatcher-0.10.2-x86_64-5:/usr/lib64/speech-dispatcher-modules/sd_voxin
libvoxin.so

Firefox (from testing) shows a false positive, because libclearkey.so is in /usr/lib64/firefox/gmp-clearkey/0.1 and not in /usr/lib64/firefox where libmozsqlite3.so is.

chrisretusn · 08-04-2023, 04:37 AM

Got to say this is an interesting thread. Thanks!

selfprogrammed · 08-04-2023, 10:56 PM

Anyway, I spent most of yesterday getting the thing done.
During the whole of testing yesterday, not once, did I see my machine catch fire, or any files outright disappear.
As it is new software, I cannot guarantee anything, nor offer much more than that there are no obvious file changing commands within it.
It will write a file called depchk.log.
That this gets run as root makes me nervous, so I gave it the ability to be run as a user too.
It is derived from orphans, but that is only barely recognizable now. A few comment lines remain and a couple of script functions will still prove that.

Due to it being run as root, it will need some brave volunteers to test, or examine it, and testify to its non-hostility.
It should be put into a secure distribution location. I do not have one.

You may copy it, or even include it in Slackware.
It is GPL, the same as orphans.

This is the depchk dependency checker.

selfprogrammed · 08-04-2023, 11:14 PM

Interesting that I (depchk) got the same hits on speech-dispatcher and libvoxin.
Is there something wrong with the installation of that thing ?

However, it does clear firefox and seamonkey.
I have hits on glportal a game that needs recompiling, but does not have a current slackbuild, and a few more like that.

I think at the moment that I will have to investigate the output that I have before I can come to any conclusion about including any more code to eliminate positives.
At the moment it is safer to allow the user to deal with them.
There is also a -verbose switch that outputs messages about suspicious finds and conclusions.

selfprogrammed · 08-04-2023, 11:59 PM

speech-dispatcher
This just goes to show that some dependency problems are not going to be figured out by any code checks I can invent.
Those libraries are missing from the distribution.
Now, how are the users supposed to know that.
How is the speech-dispatcher code actually working. I would think that it would refuse to load.

https://www.linuxquestions.org/quest...er-4175687609/

I had just put the Slackware 15.0 disc in and was going to search for whatever library that I must have missed during installation.
That would have taken a long time.

Didier Spaier · 08-05-2023, 02:04 AM

Patrick builds all modules for speech-dispatcher ('configure' enable them by default), but some require third party TTS engines, not shipped in Slackware (baratinoo, ibmtts, kali and voxin). However this does not prevent speech-dispatcher to work with all included TTS (in case of Slackware, only espeak-ng) and associated voices, so does not hurt.

henca · 08-05-2023, 04:21 AM

Looking with ldd on speech-dispatcher you will find that it is able to find all dynamically linked libraries on a full Slackware install. However, some speech-dispatcher modules might try dlopen in attempts to open dynamic libraries which may or may not be on the system. Those libraries could be considered optional dependencies and most of them are not included in a standard Slackware installation.

regards Henrik

Petri Kaukasoina · 08-05-2023, 06:16 AM

I polished the script of post #20 a little. It's here if someone is interested.

Code:

#!/bin/sh
#search for missing library dependencies
#there are false positives because programs set LD_LIBRARY_PATH to find libs
#pk 20230804 public domain
#don't care these are missing (speech-dispatcher modules):
BLACKLIST='libbaratinoo.so|libibmeci.so|libKali.so|libKGlobal.so|libKTrans.so|libKParle.so|libKAnalyse.so|libvoxin.so'
NOTFOUND=$(mktemp)
NEEDED=$(mktemp)
trap 'rm $NOTFOUND $NEEDED' EXIT
cd /var/adm/packages
for pkg in *; do
( cd /
  while read line; do
    [ "$line" = "FILE LIST:" ] && break
  done
  while read f; do
    if [ -x "$f" -a -f "$f" -a -r "$f" ]; then
      if LD_LIBRARY_PATH=$(dirname "$f") ldd "$f" 2>/dev/null|grep -q 'not found'; then
        LD_LIBRARY_PATH=$(dirname "$f") ldd "$f"|grep 'not found'>$NOTFOUND
        objdump -p "$f"|grep NEEDED|sed 's/ *NEEDED *\(l.*\)/\1/'>$NEEDED
        if grep -f $NEEDED $NOTFOUND|grep -qEv $BLACKLIST; then
          echo "$pkg"
          echo /"$f"
          grep -f $NEEDED $NOTFOUND|grep -Ev $BLACKLIST|sort|uniq
          echo
        fi
      fi
    fi
  done 
) < $pkg
done

Petri Kaukasoina · 08-05-2023, 08:13 AM

Quote:

Originally Posted by selfprogrammed

This is the depchk dependency checker.

When some library is upgraded and its SONAME changes, all packages linking to it must be rebuilt. Let's suppose some packages are missed. Your script could be of some help then. For example:

Code:

Sat Feb 11 20:36:32 UTC 2023
l/libvpx-1.13.0-x86_64-1.txz:  Upgraded.
  Shared library .so-version bump.

mplayer stopped working: https://www.linuxquestions.org/quest...px-4175721956/. Let's pretend it happens now, let's simulate libvpx is upgraded by removing the present package. Mplayer gives error message:

Code:

mplayer: error while loading shared libraries: libvpx.so.8: cannot open shared object file: No such file or directory

and your script tells this for 56 programs, including mplayer:

Code:

 Dependencies for: /usr/bin/mplayer
 ** MISSING: libvpx.so.8

But it's not mplayer or any other of those 56 programs which needs to be recompiled. It's package ffmpeg, which was rebuilt the next day. mplayer links to ffmpeg which links to libvpx. mplayer does not link directly to libvpx.

Code:

Mon Feb 13 01:09:36 UTC 2023
l/ffmpeg-5.1.2-x86_64-2.txz:  Rebuilt.
  Recompiled against libvpx-1.13.0.
l/gst-plugins-good-1.22.0-x86_64-2.txz:  Rebuilt.
  Recompiled against libvpx-1.13.0.
l/qt5-5.15.8_20230208_cae1723a-x86_64-1.txz:  Upgraded.
  Compiled against libvpx-1.13.0.
xap/xine-lib-1.2.13-x86_64-2.txz:  Rebuilt.
  Recompiled against libvpx-1.13.0.

Similar thing happened with exiv2. It for example broke gimp even though gimp did not link directly to exiv2: https://www.linuxquestions.org/quest...2/#post6431125

OK, that was motivation for the following. ldd prints not only direct dependencies but also dependencies of dependencies etc. I think you should filter out those indirect ones. You can look at the elf headers for NEEDED libraries.

In that example with upgraded libvpx above, your script gave 56 messages of missing libvpx.so.8 as dependencies of 56 programs, all in /usr/bin. But libvpx.so.8 is not a direct dependency of any of them. My script from post #27 finds missing libvpx dependencies for 4 libraries (libavcodec.so.59.37.100 of ffmpeg, libgstvpx.so of gst-plugins-good, libQt5WebEngineCore.so.5.15.10 of qt5, and xineplug_decode_libvpx.so of xine-lib), exactly the same as PV rebuilt. No programs at all.

chrisretusn · 08-07-2023, 01:31 AM

Good gravy, I ran both of these awesome tools with multilib installed. Yikes!

Since I'm not really using multilib right now (Steam), I removed it. Now I get sane results.

Petri's tool picked up a few "not found" on a couple of third party packages..

Code:

calibre-6.17.0-x86_64-1alien
/usr/lib64/calibre/bin/podofobox
        libpodofo.so.0.9.7 => not found

calibre-6.17.0-x86_64-1alien
/usr/lib64/calibre/bin/podofocolor
        libpodofo.so.0.9.7 => not found

calibre-6.17.0-x86_64-1alien
/usr/lib64/calibre/bin/podofocountpages
        libpodofo.so.0.9.7 => not found

calibre-6.17.0-x86_64-1alien
/usr/lib64/calibre/bin/podofocrop
        libpodofo.so.0.9.7 => not found

calibre-6.17.0-x86_64-1alien
/usr/lib64/calibre/bin/podofoencrypt
        libpodofo.so.0.9.7 => not found

calibre-6.17.0-x86_64-1alien
/usr/lib64/calibre/bin/podofogc
        libpodofo.so.0.9.7 => not found

calibre-6.17.0-x86_64-1alien
/usr/lib64/calibre/bin/podofoimg2pdf
        libpodofo.so.0.9.7 => not found

calibre-6.17.0-x86_64-1alien
/usr/lib64/calibre/bin/podofoimgextract
        libpodofo.so.0.9.7 => not found

calibre-6.17.0-x86_64-1alien
/usr/lib64/calibre/bin/podofoimpose
        libpodofo.so.0.9.7 => not found

calibre-6.17.0-x86_64-1alien
/usr/lib64/calibre/bin/podofoincrementalupdates
        libpodofo.so.0.9.7 => not found

calibre-6.17.0-x86_64-1alien
/usr/lib64/calibre/bin/podofomerge
        libpodofo.so.0.9.7 => not found

calibre-6.17.0-x86_64-1alien
/usr/lib64/calibre/bin/podofonooc
        libpodofo.so.0.9.7 => not found

calibre-6.17.0-x86_64-1alien
/usr/lib64/calibre/bin/podofopages
        libpodofo.so.0.9.7 => not found

calibre-6.17.0-x86_64-1alien
/usr/lib64/calibre/bin/podofopdfinfo
        libpodofo.so.0.9.7 => not found

calibre-6.17.0-x86_64-1alien
/usr/lib64/calibre/bin/podofosign
        libpodofo.so.0.9.7 => not found

calibre-6.17.0-x86_64-1alien
/usr/lib64/calibre/bin/podofotxt2pdf
        libpodofo.so.0.9.7 => not found

calibre-6.17.0-x86_64-1alien
/usr/lib64/calibre/bin/podofotxtextract
        libpodofo.so.0.9.7 => not found

calibre-6.17.0-x86_64-1alien
/usr/lib64/calibre/bin/podofouncompress
        libpodofo.so.0.9.7 => not found

calibre-6.17.0-x86_64-1alien
/usr/lib64/calibre/bin/podofoxmp
        libpodofo.so.0.9.7 => not found

dropbox-client-1.6.1-x86_64-1alien
/usr/lib64/nautilus/extensions-2.0/libnautilus-dropbox.so
        libnautilus-extension.so.1 => not found

tigervnc-1.13.1-x86_64-1
/usr/bin/vncviewer
        libfltk.so.1.3 => not found
        libfltk_images.so.1.3 => not found

vlc-3.0.18-x86_64-1alien
/usr/lib64/vlc/plugins/codec/libflac_plugin.so
        libFLAC.so.8 => not found

zoom-bin-5.15.2.4260-x86_64-1cgs
/usr/lib64/zoom/zoom
        libcef.so => not found
        libffmpeg.so => not found

calibre, libpodofo.so.0.9.7 is included in the package (/usr/lib64/calibre/lib/libpodofo.so.0.9.7). I have my own build of podofo installed (0.9.8).

dropbox-client, libnautilus-extension.so.1 is included in the package but are "/usr/lib64/nautilus/extensions-2.0/libnautilus-dropbox.so". Easily fixed in the SlackBuild.

tigervnc, a package from extra I installed quite a while ago. A quick look in the extra/tigervnc directory showed my why. The file called "the_fltk_package_must_also_be_installed" from 12-15-2015. Easy fix, installed fltk.

vlc, libFLAC.so.8 is not included in package. slackware64 has libFLAC.so.12 installed, guessing a rebuild might be in order, though vlc work okay, probably because I didn't try to using with a file that needed libFLAC.

zoom, a binary build, the two libraries missing are both in the package in /usr/lib64/zoom/cef/.

depchk.sh, found three broken links, two are my fault, the other I have no idea were it came from. The link is " /usr/lib64/libXvMCgallium.so" any ideas? I suspect possible NVIDIA drivers. At first I thought slackware package libXvMC, after looking at older version of this package, not from there. No matter rm'd broken link. This system is 13 years old and has never been fresh installed. The Slackware installation used was the back up the old computer.

depchk.sh, appears to only scan 32-bit directories not 64-bit. Of note --all means exactly that, let it run over night, still going in the morning, Ctrl-C'd out, it was busy working on my backup drive.

Code:

# bash depchk.sh -b

## depchk.sh Version 1.08 for Slackware.
## Executed: Sun Aug  6 20:09:46 PST 2023
## Mode: basic
## Output: /root/scripts/depchk.log
## Setup mode. 
## finding lib directories ... 
LIB32DIRS= /lib /lib/modules/6.1.41/kernel/lib /lib/modules/6.1.41/kernel/virt/lib /lib/modules/6.1.43/kernel/lib /lib/modules/6.1.43/kernel/virt/lib /lib/modules/6.1.42/kernel/lib /lib/modules/6.1.42/kernel/virt/lib /usr/lib /usr/local/lib /opt/VirtualBox/sdk/installer/build/lib /opt/VirtualBox/dtrace/lib
LIB64DIRS=
## finding bin directories ... 
BINDIRS= /bin /usr/bin /usr/local/bin
.
## Setup Done!
## Installed packages: 1770
## Check for broken symlinks that might indicate missing libs..
## Broken Links Done!
## Checking for missing dependencies of installed libs/binaries..
##
## depchk.sh Version 1.08 for Slackware.
## Executed: Sun Aug  6 20:09:46 PST 2023
## Mode: basic
## ##########################################

## Installed packages: 1770

## Check for broken symlinks that might indicate missing libs..
1: Broken link to: /usr/lib64/liblua5.2.so
1:> Possibly comes from package: Unknown.
1: Broken link to: /usr/lib64/libXvMCgallium.so
1:> Possibly comes from package: Unknown.
1: Broken link to: /usr/lib64/liblua5.4.so
1:> Possibly comes from package: Unknown.
## Broken Links Done!

## Checking for missing dependencies of installed libs/binaries..
## When there is a : at last line, it is LESS waiting for you to scroll the page !
# Checking files in: /bin
# Checking files in: /usr/bin
# Checking files in: /usr/local/bin
# Checking files in: /lib
# Checking files in: /lib/modules/6.1.41/kernel/lib
# Checking files in: /lib/modules/6.1.41/kernel/virt/lib
# Checking files in: /lib/modules/6.1.43/kernel/lib
# Checking files in: /lib/modules/6.1.43/kernel/virt/lib
# Checking files in: /lib/modules/6.1.42/kernel/lib
# Checking files in: /lib/modules/6.1.42/kernel/virt/lib
# Checking files in: /usr/lib
# Checking files in: /usr/local/lib
# Checking files in: /opt/VirtualBox/sdk/installer/build/lib
# Checking files in: /opt/VirtualBox/dtrace/lib
## Dependency Done!

Great tools

to you both.

Petri Kaukasoina · 08-07-2023, 05:32 AM

Quote:

Originally Posted by chrisretusn

vlc, libFLAC.so.8 is not included in package. slackware64 has libFLAC.so.12 installed, guessing a rebuild might be in order, though vlc work okay, probably because I didn't try to using with a file that needed libFLAC.

I seem to have extracted file libFLAC.so.8.3.0 from the last flac-1.3.X package of -current and put it in /usr/local/lib64. AlienBob builds packages on stable versions like 15.0, and they might miss some older library versions on -current. He deliveres 'compat' packages for some libraries where he collects older library versions, but I have done the same thing by copying to /usr/local/lib64.

Code:

wget https://slackware.uk/cumulative/slackware64-current/slackware64/ap/flac-1.3.4-x86_64-1.txz
tar xf flac-1.3.4-x86_64-1.txz usr/lib64/libFLAC.so.8.3.0 --strip-components=2 --one-top-level=/usr/local/lib64
ldconfig