Simple C: Beyond Bash Part 2 - Dependency lister for dir branch (recursive)
Tags compile, computer mad science, deb, dependencies, rpm
[Note to myself: Needs rewrite using libLQ.]
Simple C: Beyond Bash Part 2 - Dependency lister for dir branch (recursive)
CHANGELOG
* Sat Feb 18, 2012 -rs
find-deps version 1.2
In 'find_elf()' changed pipecmd to "find * -executable -o name *.so*" because some libs
may not have have their executable flag set, such as an unpacked from an rpm package that
doesn't have the perms set until install time. This change will set off error messages about the files that need corrected permissions.
Also quotes around the "*.so*" for the 'find' call. (It couldn't have worked without it, but somehow this was mis-posted). (corrected feb 21) -rs
Features:
Seems to work great when used "normally" (to read binary trees in package folders), and it's pretty fast. Here's part of a dump of my /bin deps.
The "external deps" listed are libs loaded at run time that are not resolved in or below the prefix branch (i.e., /bin in this case).
I have installed the code and executable in my sandbox so it's linked into my path and is easy to edit. (see early blog entries re. new.symlink, etc.). And I gave the folder a version number so I can easily see which one is current if this needs any changes later.
The slib.cpp and slib.h files are in Part 1. Put those in the same folder with the new ones. Don't worry, the compiler will tell you which are missing if you forget. :-)
There are three new files (five total). Here's what's new.
file: version.h
file: usage_str.dat
file: find-deps.cpp
Compile as you did with the toy at the bottom of Part 1, or if you like we can cheat and simply
in the deps-list.cpp file itself. This trick works sometimes. And it will this time.
But after this, having a bit of a taste of what's possible with C/C++, you might want to tool up and do it for real with
And here's my own bad idea if you want to check it out. It's easy to edit the templates, but I should have done that myself. "autobake new c-32" is ok out-of-box tho.
http://rainbowsally.org/rainbowsally...1-12-29.tar.gz
Also needs a type lister like 'new.image' has and some other improvements/reorganizations. It's easier to delete stuff than write stuff though, so this is on the back burner as far as my own needs are concerned.
But that's one bad idea. And it may give you some bad ideas of your own!
:-)
Now we have find-deps.
We'll put it to good use a bit later, if all continues to go well here in blog-land.
Some more interesting stuff you can do once you've got a way to compile this and run it.
Simple C: Beyond Bash Part 2 - Dependency lister for dir branch (recursive)
CHANGELOG
* Sat Feb 18, 2012 -rs
find-deps version 1.2
In 'find_elf()' changed pipecmd to "find * -executable -o name *.so*" because some libs
may not have have their executable flag set, such as an unpacked from an rpm package that
doesn't have the perms set until install time. This change will set off error messages about the files that need corrected permissions.
Also quotes around the "*.so*" for the 'find' call. (It couldn't have worked without it, but somehow this was mis-posted). (corrected feb 21) -rs
Features:
- Read a pipe (uses slist functions from Part 1).
- Commandline switch reordering (non-destructive).
- Non-regex parsing & translating in do{...}while(0) AND block (see 'elfpaths_only') ,
- Unsorted 'uniq' example in C
- -- A tool to read dependencies from all elf executables and libraries in a directory branch.
- -- Can be used to check make-based[tm] :-) installations or rpm and deb packages before packing up.
- bash
- make
- g++
- ldd
- find
Seems to work great when used "normally" (to read binary trees in package folders), and it's pretty fast. Here's part of a dump of my /bin deps.
Code:
[list of 45 unique deps deleted -rs] --------------------------------------------------- binaries found : 125 ldd pipe errors: 0 external deps : 45 --------------------------------------------------- real 0m1.667s -- less than two seconds (cmd: 'time find-deps /bin -i') user 0m0.727s sys 0m0.778s
I have installed the code and executable in my sandbox so it's linked into my path and is easy to edit. (see early blog entries re. new.symlink, etc.). And I gave the folder a version number so I can easily see which one is current if this needs any changes later.
Code:
HOME \-- bin \-- src // \-- find-deps-1.0 [everything goes here] \-- find-deps-1.2 ...
There are three new files (five total). Here's what's new.
- version.h - version and changelog in one
- usage_str.dat - easier to edit a text file, this is the output, the non-C type file extension avoids potential linker errors with tools such as AutoBake (makefile creator).
- find-deps.cpp - the program
file: version.h
Code:
// initial app created //#define VERSION "1.0" // #define VERSION "1.0" // #define VERSION "1.1" // added --unres switch and printouts for unresolved libs // #define VERSION "1.2" // #define VERSION "1.2" // added *.so* files to elf file search command #define VERSION "1.2"
Code:
/* usage.txt converted with txt2cstr */ const char* usage_str = "\n" "Usage: find-deps <branch>\n" "\n" " Primary usage is for discovering files needed by a binary package before\n" " installation/building, etc.\n" "\n" " Prints out a list of all libs used by executable files and libs \n" " found in the directory <branch> (recursive). \n" "\n" " Does not list files that are resolved by other files in the same \n" " branch.\n" "\n" " Does not look in any other branches for resolvers.\n" "\n" "Switches:\n" " -i | --info display info about files found\n" " -v | --version show version\n" "\n" "Notes: \n" "\n" " 1. Needs linux tools 'find' and 'ldd' to run.\n" "\n" " 2. If a symlink points to a non-existent file (as encountered in \n" " development packages) this will be still be counted as a self-resolved \n" " dependency.\n" "\n" " 3. The error message \"No elf binaries found\" sent to stderr could mean\n" " that there really aren't any or just that the path doesn't exist.\n" "\n" " 4. May not work for system dirs (e.g. /usr/bin, /usr/lib, /lib) due to \n" " ldd quirks but we can get 'info' on the number of elf files.\n" "\n" ;
file: find-deps.cpp
Code:
// find-deps.cpp // shotgun load some commonly used headers #include <stdio.h> // printf, sprintf, etc. #include <stdlib.h> // exit() #include <unistd.h> // tons of stuff #include <malloc.h> // memory allocation #include <string.h> // strings and block comparisons and moves // our own includes #include "slist.h" #include "version.h" void dbg(){} // a breakpoint for kdbg/gdb ////////////////////////////////////////////////////// // prototypes section // saved current directory path extern char* HERE; // true if we want to know how many files were processed extern int infoflg; // number of pipe to ldd errors extern int ldderrs; // name of currently running app extern char* appname; // reorder args so switches precede args and return // # of switches. int reorder_args(int argc, char** argv); // find and store names of elf executables in elflist, // returns 0 on success int find_elf(char*** pelflist, const char* prefix); // find and store dependency libs in depslist, not including files // in elflist, returns 0 on success int find_deps(char*** depslist, const char* prefix, char** elflist); // process recognized switches and return errcode if bad switch // value. int handle_switches(int n, char** argv); // usage note int usage(int errcode); ////////////////////////////////////////////////////// // the program // saved current directory path char* HERE; // true if we want to know how many files were processed int infoflg = 0; // number of pipe to ldd errors int ldderrs = 0; // name of currently running app char* appname; int main(int argc, char** argv) { dbg(); appname = basename(argv[0]); int err = 0; char** elflist = slist_new(); char** depslist = slist_new(); int offset = 0; // increases for each commandline switch // syntax checks, should return usage(1) if argc < 2, but... whatever :-) if((argc < 2) || (strcmp(argv[1], "--help") == 0)) return usage(0); // switches offset = reorder_args(argc, argv); err = handle_switches(offset, argv); if(err) return err; // setup const char* prefix = argv[1 + offset]; HERE = getenv("PWD"); // get list of all elf executables in prefix dir err = find_elf(&elflist, prefix); int num_elf = slist_count(elflist); if(num_elf == 0) { fprintf(stderr, "No elf binaries found.\n"); return 1; } // get list of deps in all elf files that are not in the // list already in the prefix dir err = find_deps(&depslist, prefix, elflist); int num_deps = slist_count(depslist); for(int i = 0; i < slist_count(depslist); i++) printf("%s\n", depslist[i]); if(infoflg) { const char* hline = "---------------------------------------------------"; printf("%s\n", hline); printf("binaries found : %d\n" "ldd pipe errors: %d\n" "external deps : %d\n", num_elf, ldderrs, num_deps); printf("%s\n", hline); } // cleanup and return slist_delete(depslist); slist_delete(elflist); return 0; } ////////////////////////////////////////////////////// // the library of functions and various utils // utils - misc // returns true (1) if filename is an elf file int is_elf(const char* fname) { FILE* fp = fopen(fname, "r"); if(! fp) return 0; // error, file not found // read the first few bytes of the file and look for the // signature of a valid elf binary. char buf[8] = {0}; fread(buf, 1, 4, fp); if(memcmp(buf, "\177ELF", 4) == 0) return 1; // true return 0; // false } int reorder_args(int argc, char** argv) { // make literal copy of argv list in temporary list char** tmp_args = (char**)malloc(argc * sizeof(char**)); memcpy(tmp_args, argv, argc * sizeof(char**)); int cnt = 1; // current copy back number, app path is at [0] (no change) int nswitches = 0; // the value to return // copy switches (anything starting with '-') back first for(int i = 1; i < argc; i++) if(tmp_args[i][0] == '-') argv[cnt++] = tmp_args[i]; nswitches = cnt - 1; // number of actual switches // copy non-switches back next for(int i = 1; i < argc; i++) if(tmp_args[i][0] != '-') argv[cnt++] = tmp_args[i]; free(tmp_args); return nswitches; } ///////////////////////////////////////////////////////// // find and store names of elf executables in elflist, // returns 0 on success int find_elf(char*** pelflist, const char* prefix) { // init the flag and the list which is an array of // pointers terminated by a zero and save current // directory so we can always return to it. int err = 0; char pipcmd[512]; char** elflist = *pelflist; /// remove after test char** tmp = slist_new(); // Create a non-looping do-while block we can leave // at the first error. Functionally equivalent to a // string of ORs using err as the flag. do { char pipecmd[512]; err = chdir(prefix); if(err) break; // get complete list of all executables, then // remove those not elf type. // sprintf(pipecmd, "find * -executable"); // added all *.so files, some libs in rpm packages may not be executable -rs sprintf(pipecmd, "find * -executable -o -name \"*.so*\""); err = slist_readPipe(&elflist, pipecmd); if(err) { slist_clear(elflist); break; } // remove non-elf files for(int i = 0; i < slist_count(elflist); i++) { if(is_elf(elflist[i])) slist_append(&tmp, elflist[i]); } }while(0); if(err) return err; // elflist = tmp; slist_delete(elflist); //elflist = tmp->list(); elflist = tmp; *pelflist = elflist; // delete tmp; chdir(HERE); return 0; } void elfpaths_only(char*** plist) { int state = 0; char** list = *plist; char** tmp = slist_new(); for(int i = 0; i < slist_count(list); i++) { // THANK YOU :-) to the svanah non-gnu BNF project for this trick char buf[256]; sprintf(buf, list[i]); char* ip = buf, *op = buf; do { // skip whitespaces while(*ip <= ' ') ip++; // don't want blank lines if(!*ip) break; // want lines starting with slash if(*ip == '/') break; // don't want lines not pointing to lib path ip = strstr(ip, " => "); if(!ip) break; ip += 4; // move to file path, should start with '/' // don't want lines not having a lib path if(*ip != '/') break; // MATCHED - now copy to first non text and terminate string while(*ip > ' ') *op++ = *ip++; *op = 0; state = 1; // signale that we got one }while(0); if(state) slist_append(&tmp, buf); } // save new result and return slist_delete(list); *plist = tmp; } // find and store dependency libs in depslist, not including files // in elflist, returns 0 on success int find_deps(char*** pdepslist, const char* prefix, char** elflist) { // init char** depslist = *pdepslist; int err = 0; char pipecmd[512]; err = chdir(prefix); if(err) return 1; int len = slist_count(elflist); for(int i = 0; i < len; i++) { // get list of ldd libs loaded by each elf file and add // to the depslist, temporarily before removing dups. char** sublist = slist_new(); sprintf(pipecmd, "/usr/bin/ldd %s", elflist[i]); err = slist_readPipe(&sublist, pipecmd); // count ldd errors if(err) ldderrs++; elfpaths_only(&sublist); // "you will be assimilated" said depslist to sublist.... slist_addList(&depslist, sublist); slist_delete(sublist); }while(0); int ip; int op; // remove duplicates if any items to copy ip = 0; for(ip = 0; ip < slist_count(depslist); ip++) { op = ip + 1; while(op < slist_count(depslist)) { // shrink list or advance pointer... if(strcmp(depslist[op], depslist[ip]) == 0) slist_remove(&depslist, op); else op++; } } // remove self-resolved and hope the paths are right // after installation because we won't (can't) check them at // this point without setting LD_LIBRARY_PATH and that can // be as error prone as this, so... here we go. If it // exists at all in our branch, we count is as 'resolved'. for(int elf = 0; elf < slist_count(elflist); elf++) { char* elffile = basename(elflist[elf]); for(int dep = 0; dep < slist_count(depslist); dep++) { char* depfile = basename(depslist[dep]); if(strcmp(elffile, depfile) == 0) slist_remove(&depslist, dep); } } *pdepslist = depslist; chdir(HERE); return 0; } int usage(int errcode) { #include "usage_str.dat" printf(usage_str); return errcode; } // process recognized switches and return errcode if bad switch // value. int handle_switches(int n, char** argv) { n++; // don't count argv[0] (app path) // we already know the first char is a '-' so we look for // substrings common to both long and short switches. for(int i = 1; i < n; i++) { // -v or --version if(strstr(argv[i], "-v")) { printf("%s version %s\n", appname, VERSION); exit(0); } // -i or --info else if(strstr(argv[i], "-i")) infoflg = 1; else { fprintf(stderr, "Unrecognized switch %s\n", argv[i]); return 1; // not a valid switch } } return 0; // ok } // if you must, here's a way to do this without linking explicitly. :-) // #include "slist.cpp" // uncomment this line. // COMPILE: g++ find-deps.cpp -o find-deps
Code:
#include "slist.cpp"
But after this, having a bit of a taste of what's possible with C/C++, you might want to tool up and do it for real with
- a good debugger. I HIGHLY recommend kdbg version 5.0 or greater. (insight genre.)
- a decent programming editor. With macro keys if you can get em. kdevelop3 is my pick. (Haven't tried Jen's programming editor with Wine yet -- that's also a good one.)
- and a way to create your own makefiles if you don't like the default. I like AutoBake.
And here's my own bad idea if you want to check it out. It's easy to edit the templates, but I should have done that myself. "autobake new c-32" is ok out-of-box tho.
http://rainbowsally.org/rainbowsally...1-12-29.tar.gz
Also needs a type lister like 'new.image' has and some other improvements/reorganizations. It's easier to delete stuff than write stuff though, so this is on the back burner as far as my own needs are concerned.
But that's one bad idea. And it may give you some bad ideas of your own!
:-)
Now we have find-deps.
We'll put it to good use a bit later, if all continues to go well here in blog-land.
Some more interesting stuff you can do once you've got a way to compile this and run it.
- Compile the app and run 'strip find-deps' and check the file size after it's stripped. Compare to the -s switch (sent to the linker).
- Add the -g3 switch to the compile and run in a good debugger. Find where the ldd pipe errors come from when checking /usr/lib if your system does the same thing as mine. (Same whether you chdir to the folder or not. You'll see what I mean.)
- Create a TREE folder containing the directory image of an installation you might want to generate. Probably in TREE/usr/local, for a somewhat safe place if you decide to actually install it later. And 'find-deps TREE'. :-)
Total Comments 0