shell getopts: opt w/ optional parameter is taking next opt as its parameter!

GrapefruiTgirl · 10-24-2010, 09:09 AM

This is the first time ever, I'm trying to use `getopt(s)` in a shell script (figured I should eventually, some day, learn how to use it). I'm only doing short (-a -b) options for now; once that works, I'll deal with --long options.

I'm using manpages for reference, as well as:
http://tldp.org/LDP/abs/html/internal.html#EX33
And tell you what, that manpage could use some examples!

Anyhow, while I question whether there's really anything to gain by using getopts over manually dealing with options (which has been working just fine, thanks), it's basically so far so good... Except: like the title says, I have one option which can take a (optional) parameter to it, and it's screwing up the getopts parsing.

First, my getopts code:

Code:

while getopts ":l.#vhAabcdkpPswzR:" OPT; do
echo "$OPT , $OPTARG"
 case $OPT in
          l) BASENAME="no" ;;
        '.') NAMEOPT="-name"; DOTOPT="*" ;;
        '#') SHEBANG="yes" ;;
          v) VERBOSE='1' ;;
          h) help; exit 0 ;;
          A) add_shell all ;;
          a) add_shell ash ;;
          b) add_shell bash ;;
          c) add_shell csh ;;
          d) add_shell dash ;;
          k) add_shell ksh ;;
          p) add_shell perl ;;
          P) add_shell python ;;
          s) add_shell sh ;;
          w) add_shell awk ;;
          z) add_shell zsh ;;
          R) [ "$OPTARG" -a $((OPTARG)) -ge 1 ] && MAXDEPTH="-maxdepth $OPTARG" || unset MAXDEPTH ;;
          *) echo "Invalid parameter." >&2; exit 1 ;;
 esac
done

So, the -R option is the problem. I have a debug line inserted above, in red. That line prints the $OPT and $OPTARG during getopts loop.
The (temporary) butchery code I have in the case/esac for R) was to see if an arg to -R was given, and if it seemed to be an integer. That code doesn't work and is crappy.

Here's some sample runs of the program, demonstrating the problem:

This is GOOD:

Code:

root@reactor: sl -R2 -lA# /etc
R , 2
l , 
A , 
# ,

This is GOOD:

Code:

root@reactor: sl -lA# -R2 /etc
l , 
A , 
# , 
R , 2

This is BAD:

Code:

root@reactor: sl -lA# -R /etc
l , 
A , 
# , 
R , /etc

This is BAD:

Code:

root@reactor: sl -RlA# /etc
R , lA#

This is BAD:

Code:

root@reactor: sl -R2lA# /etc
R , 2lA#

So hopefully that shows how -R is taking whatever comes after it as its parameter, which is messing up the rest of the program operation. It is the only opt that takes an optional arg; the arg should be an integer, (technically from 0 to infinity, but whatever, as long as it's an integer). If there's no integer given, that's fine.

So... I can manually sanity-check to see that, if -R was given, it has a proper (integer) arg to it, and if not, ignore it, HOWEVER, by that point, getopts has already wrongly sucked up the remaining opts passed to the program (those that followed -R), and so getopts's done processing opts as far as it's concerned, and the rest of the args are discarded..

What to do?

The 3 "BAD" examples given above, should be "GOOD" if possible.

For the record:
1) this is Bash shell at this time, but will be Dash ultimately.
2) /bin/getopt doesn't work like this one - I can't make it work at all. Does it ever work? Should I be using that instead?

Thanks for your insight, folks!

EDIT: For clarity (in case I've been confusing above):

Code:

root@reactor: sl -ls#R2 /etc/profile.d
l , 
s , 
# , 
R , 2

...and...

Code:

root@reactor: sl -lR2s# /etc/profile.d
l , 
R , 2s#

...should both be interpreted the same as if:

Code:

sl -l -R2 -s -# /etc/profile.d

...were given.

grail · 10-24-2010, 10:17 AM

I would say it is working exactly as it should

Any option that you tell to have optional arguments after will take whatever is next on the command line.
If you ran:

Code:

sl -lA# -R

This will slip the -R option through to your default:

Quote:

*) echo "Invalid parameter." >&2; exit 1 ;;

It is also designed to only pick up any continuous characters, so as your last example demonstrates, the space preserves /etc from being sucked in.

Not sure if that helps, but it appears whatever is next will always be drawn in.

GrapefruiTgirl · 10-24-2010, 10:46 AM

Hi grail,

thanks for the input; I don't think it helped though. To demonstrate that a [space] doesn't help here:

Code:

root@reactor: sl -l -R 2 -s /etc/profile.d
l , 
R , 2
s ,

Code:

root@reactor: sl -l -R -s /etc/profile.d
l , 
R , -s

So above, the first example works fine. R gets the value of 2.
In the second example, R should be happy with no value, because a value is optional; it should not take -s as the value of -R.

Basically what I'm looking for here in getopts, is: if the character(s) immediately following -R (possibly preceeded by one or more [space]'s) are digit(s) then they are the value for -R. But if non-numeric character(s) (possibly preceeded by 1+ [spaces]) follow -R, then give -R no value, and continue with parsing opts beginning with the character following R. Is this (more) clear?

Maybe what I should be doing, is pre-parsing my ${@} before the getopts, to isolate/separate the -R situation, and create a new string of opts/args and then use getopts on that new string (can I use getopts on a string rather than an array? I think so..) but this-all seems to defeat the purpose of using getopts to begin with.

crts · 10-24-2010, 11:31 AM

Hi GrapefruiTgirl,

one possible way to deal with optional parameters is to pass them like

Code:

script -R=-5 filename

So instead of allowing a space after -R the script has to expect a '=' sign. If there is a space after -R then we can safely assume, that the optional argument was omitted.
We can test for a space after -R by utilising $OPTIND. Example:

Code:

#!/bin/bash

while getopts ":1R:2" OPT; do
echo --------------------------------------------
echo "OPT: $OPT"
echo "OPTARG: $OPTARG"
echo "OPTIND: $OPTIND"
echo
 case $OPT in
          1) echo 'This is option -1';;
          2) echo 'This is option -2';;
          R) echo -n '"$"$((OPTIND-1))"": '
          eval echo "$"$((OPTIND-1))""
          echo
          echo 'This is option -R'
          echo
          if eval [[ "$"$((OPTIND-1))"" == "$OPTARG" ]];then
              echo '"$"$((OPTIND-1))"" has the same value as $OPTARG!'
              echo "That means tht the optional parameter is NOT present!"
              echo "The next option has been falsly interpreted"
              echo "as an argument to -R."
              echo "Decreasing OPTIND so it will be recognised as"
              echo "as option in the next loop."
              echo "OPTIND is now: $((--OPTIND))" # decrease OPTIND; corrected output!
          else
              echo '"$"$((OPTIND-1))"" does NOT have the same value as $OPTARG!'
              echo "The optional parameter is present!"
              echo 'The argument can be extracted with ${OPTARG:1}: '${OPTARG:1}
              echo 'OPTIND does not need to be manipulated.'
      fi;;
          *) echo "Invalid parameter." >&2; exit 1 ;;
 esac
echo --------------------------------------------
done

I chose -1 and -2 as options (as,e.g in ls -1). Otherwise one might be tempted to check if OPTARG begins with a '-' sign. This would backfire if the -R option could take negative integers as argument.
Let me know if this example clears things up a bit. If not I'll try to be more verbose.

[EDIT]
I changed

Code:

echo "OPTIND is now: $((OPTIND--))" # decrease OPTIND so that the

to

Code:

echo "OPTIND is now: $((--OPTIND))" # decrease OPTIND so that the

which now displays the correct information.

grail · 10-24-2010, 11:31 AM

I understand you perfectly, but I think I am not explaining very well

Both examples you just gave are demonstrating what I meant about the space. That is on the consecutive characters, be it the single 2 or both -s characters, are being used as
optional argument. So nothing else is being drawn in is what I meant. Hope that is clearer.

I think part of your issue is that although it says 'optional' argument, it actually means that you can but don't have to put anything after, but if you do put anything
after it it will be considered as the argument for -R.

I played with changing the OPTIND variable and had both success and failure. If you change your -R option to the following:

Code:

R) [ "$OPTARG" -a $((OPTARG)) -ge 1 ] && MAXDEPTH="-maxdepth $OPTARG" || {unset MAXDEPTH;((OPTIND--));} ;;

This will work for your last 2 examples.

The downside to this is that if you enter the following:

Code:

sl -RlA

This will cause an infinite loop

Also, on a side note, having a check for OPTARG has a value at all is not required as by default no argument would make -R slip to your default.

grail · 10-24-2010, 11:42 AM

Hmmm ... so crts, why do I get an infinite loop but yours seems to work ok??

grail · 10-24-2010, 11:56 AM

Also I just found that if I enter:

Code:

sl -R2

Using crts' code it does not yield the result I would expect, ie nothing as OPTARG for -R and -2 being evaluated.

On the plus side -2R works

crts · 10-24-2010, 12:18 PM

Quote:

Originally Posted by grail

Also I just found that if I enter:

Code:

sl -R2

Using crts' code it does not yield the result I would expect, ie nothing as OPTARG for -R and -2 being evaluated.

On the plus side -2R works

That would be because as I said, the script expects the argument to -R in the form of

Code:

script -R=option -2

The '=' is mandatory. Of course, you can rewrite it so that you don't have to use a '=', like

Code:

script -Roption -2

But this is less readable. Remember, a space after '-R' determines, if the optional argument has been passed.
So

Code:

script -R2

is interpreted correctly by getopts as option '-R' with argument '2'. But nothing gets printed because the correct way to pass it to the script would be

Code:

script -R=2

So, to catch the wrong way of passing you need to add some error checking, i.e. check if the argument starts with a '='. If not, exit with an error. That is the easy way.
Or you could try to handle that parameter in the '-R' branch.

crts · 10-24-2010, 12:25 PM

Quote:

Originally Posted by grail

Hmmm ... so crts, why do I get an infinite loop but yours seems to work ok??

That would be because you do not utilize OPTIND to check for the optional argument:

Code:

sl -Rla
R) [ "$OPTARG" -a $((OPTARG)) -ge 1 ]                        ... || {...; ((OPTIND--));};;
       ^ OPTARG is la -> ^ This evaluates to false           ------>        ^ results into decreasing OPTIND.
                                                                              In the next iteration OPTIND points again to
                                                                              the parameter '-Rla' and we are back at square one.

crts · 10-24-2010, 12:44 PM

minor change in post #4. Corrected false output!

grail · 10-24-2010, 08:46 PM

So I am running really vague here this morning

Why does your code decrement OPTIND and not cause an infinite loop and mine does?
I know the answer will be simple but it is eluding me

GGirl .. sorry to hijack this thread a little .. just trying to wrap my head around it

crts · 10-25-2010, 12:46 AM

Quote:

Originally Posted by grail

So I am running really vague here this morning

Why does your code decrement OPTIND and not cause an infinite loop and mine does?
I know the answer will be simple but it is eluding me

GGirl .. sorry to hijack this thread a little .. just trying to wrap my head around it

Well, it is not that simple. And sometimes I articulate myself badly. Knowing how something works and then explaining it are two different issues

Ok, I'll give another try.
Basically, getopts simply parses the positional parameters (PP); just like any script does.
It uses its internal OPTIND variable to keep track which PP has to be parsed next.
Let's have a look at this example:

Code:

while getopts ":ab:c:d" OPT; do
 case $OPT in
          a) echo 'This is option a';;
          b) echo 'This is option b; it takes an argument';;
          c) echo 'This is option c; it takes an argument';;
          d) echo 'This is option d';;
          *) echo "Invalid parameter." >&2; exit 1 ;;
 esac
done

Calling it as follows

Code:

script -a -b ArgumentToB -cArgumentToC -d filename

I) "-a"
First iteration OPTIND has the value 1. It gets the PP $1 by internally evaluating ${OPTIND}, which results to ${1} and finally to '-a'. It strips the '-' and stores 'OPT=a'. Since option 'a' was declared to take NO argument, getopts internally increases: OPTIND=OPTIND+1
Now OPTIND has the value 2. At this point our code inside the while-loop will be executed, with OPTIND not pointing to the current PP, but to the next one!

II) "-b ArgumentToB"
In the next iteration getopts will fetch PP $2. PP $2 is '-b' in our example. However, 'b' was declared to come with an argument. So getopts analyzes the length of PP $2. In this case the length of '-b' is 2. For getopts this means, that the argument to option 'b' must be stored in PP $3. So again it strips away the leading '-' of $2 and stores OPT=b. In addition it fetches $3, which is 'ArgumentToB', and stores it into OPTARG.
So in order to get the next option (which is stored in PP $4) in the next iteration of the while-loop OPTIND has to be increased: OPTIND=OPTIND+2
Our code is executed with OPTIND pointing to PP $4.

III) "-cArgumentToC"
Now in the next loop OPTIND has the value of 4. So it fetches PP $4. $4 is '-cArgumentToC' in our example. The option is the first character after the '-', 'c' in this case. This option also has been declared to have an argument. getopts analyzes the length '-cArgumentToC' and this time it results to something greater than 2. Now getopts 'knows' that the argument is also stored in the same PP and NOT in the next one, as it was previously with '-b'. getopts strips the '-', stores the first character into OPT and the rest of $4 into OPTARG.
Since option AND argument were stored in the same PP, getopts increases: OPTIND=OPTIND+1
OPTIND has now the value of 5 and in the next iteration it will get PP $5 and thus process '-d'.

Now, let us have a look at this

Code:

script -cad filename

This construct is analyzed in the same manner as in III). getopts sees that PP $1 starts with '-c'. An option that takes an argument. Analyzing the length it concludes that the argument must be stored in the same PP. Hence, it passes 'ad' to OPTARG. getopts does not 'know' optional arguments. An option either has an argument or it does not. That's it as far as getopts is concerned. So it does not care, that 'a' and 'd' also define options; in this case they are just one argument.

... to be continued

crts · 10-25-2010, 12:47 AM

... continuation

The headaches start when you want to handle options, which take an optional argument. Since getopts won't do it for us, we have to take care of it ourselves. So let us make 'c' have an optional argument. We want to call our script like

Code:

script -c -d -a filename

Now getopts will handle this as in II). 'c' gets assigned to OPT, '-d' is stored in OPTARG and finally OPTIND has the value 3. Nope, getopts does not care that '-d' is another option. It has already determined that the length of PP $1 is exactly 2 and therefore PP $3 must be its argument. PP $3 is not further analyzed by getopts.
So when we step into the loop, the situation is as follows:

Code:

OPT=c
OPTARG=-d
OPTIND=3

In our case/esac-construct we can intervene and correct this. Remember what I said in an earlier post? We have to make some limitations to the passing convention, if we want to be able to deal with optional arguments. Our convention is, that options that take optional arguments have to be passed as follows:

Code:

script -c[=optional] # the '=' sign is for readability!

Now, we are able to utilise OPTIND in order to determine if the optional argument is present. For this we compare the value of the PP at the location ((OPTIND-1)) to the value of $OPTARG. If those values are equal then the optional parameter was NOT passed.

Code:

if eval [[ "$"$((OPTIND-1))"" == "$OPTARG" ]];then    # I know, it's ugly
# evaluates to 
# if [[ "$2" == "$OPTARG" ]]
# resulting in true
# if [[ "-d" == "-d" ]]
    echo 'optional parameter is NOT present!'
fi

So the value in OPTARG is really the next option and NOT the argument for 'c'. In order for getopts to fetch PP $2 in the next loop we simply decrease OPTIND by one, hence pointing it back to the correct next PP $2:

Code:

while getopts ":ab:c:d" OPT; do
 case $OPT in
          a) echo 'This is option a';;
          b) echo 'This is option b; it takes an argument';;
          c) 
            if eval [[ "$"$((OPTIND-1))"" == "$OPTARG" ]];then    # I know, it's ugly
               echo 'optional parameter is NOT present!'
               ((--OPTIND))    # correcting OPTIND
            fi  ;;
          d) echo 'This is option d';;
          *) echo "Invalid parameter." >&2; exit 1 ;;
 esac
done

Let's have a look at the situation, if we were to pass the optional parameter according to our convention

Code:

script -c=ArgumentToC -d filename

This results into

Code:

OPT=c
OPTARG==ArgumentToC # yes, OPTARG has the value '=ArgumentToC'
OPTIND=2

Our comparison again

Code:

if eval [[ "$"$((OPTIND-1))"" == "$OPTARG" ]]
# evaluates to
# if [[ "$1" == "OPTARG" ]]
# which results to false
# if [[ "-c=ArgumentToC" == "=ArgumentToC" ]]

So by utilising OPTIND and applying a restrictive argument passing convention we gain the liberty of having optional arguments.

I hope this clears things up a bit. I know, I did not cover

Code:

script -ad filename   # this is valid

But I wanted to keep things simple. For the time be, let's just assume there is some magic in getopts

Now one might think to check if OPTARG start with a '-'. But then you would not be able to pass, e.g. negative integers or generally arguments that start with '-'. So it is not valid as a general procedure.

As for the infinite loop issue:
It is situation III) all over again.
Check what getopts does with PP $1, OPTIND, OPTARG and then what the code does with OPTIND etc. The condition can only evaluate to false.

grail · 10-25-2010, 03:10 AM

Quote:

script -ad filename # this is valid

Was this supposed to be:

Code:

script -cd filename   # this is valid

Because I think this is the one that loses me

Thank you very much for the explanation as it did clear up some things for me.

If I am understanding and we use the above second example we get:

Code:

OPT=c
OPTARG=d
OPTIND=3

Obviously in our example from GGirl we are saying that we would like any passed argument to be a number so we know
the above to be wrong (from the point of view not what we want)

Is there then a way to dispense with the '-c' and have our script now process '-d' which is also part of OPTIND=1?
Or is the solution here to just present the user with an error as '-c' must take an argument and we want it to be numeric?

I hope I too am making myself clear as i am trying not to over complicate.

crts · 10-25-2010, 04:42 AM

Quote:

Originally Posted by grail

Was this supposed to be:

Code:

script -cd filename   # this is valid

Because I think this is the one that loses me

No, I just wanted to point out that for options that don't take an argument you can use

Code:

script -a -d filename

as well as

Code:

script -ad filename

In both cases the non argument options will be interpreted correctly. But this behavior is not relevant for the handling of optional arguments. So let's just completely forget about it.

Quote:

If I am understanding and we use the above second example we get:

Code:

OPT=c
OPTARG=d
OPTIND=3

No, OPTIND will be 2. This is case III). However, I think I understand now where your problem is. It is the name OPTIND, right? You are reading way too much into it. Although the name might indicate that it points to the next option, it really does not. It simply points to the next positional parameter which holds the next option - more precise: the next set of options.

Quote:

Obviously in our example from GGirl we are saying that we would like any passed argument to be a number so we know
the above to be wrong (from the point of view not what we want)

Well, in this special case you could check if OPTARG is numeric. If not, you could handle it as in the c branch of the case/esac construct. But then what about

Code:

script -cda filename

Analyze them all manually? Or maybe set OPTIND back to our current PP, set a counter to indicate which option in the current PP has already been processed and strip it in the next loop? Very error prone, you might end up in an infinite loop. Also, as you can see, these are only some rough, not even half baked ideas. Not sure if they would work. So the general answer to the question

Quote:

Is there then a way to dispense with the '-c' and have our script now process '-d' which is also part of OPTIND=1?

is: (probably) not. At least I do not see an obvious one with my approach. And I a not sure if the benefits would justify the means of effort and complexity.
Which leaves us with

Quote:

Or is the solution here to just present the user with an error as '-c' must take an argument and we want it to be numeric?

Yes.