[SOLVED] How to convert the rows of data into the columns?

w1k0 · 09-25-2012, 06:17 AM

Quote:

Originally Posted by H_TeXMeX_H

If you have more plugins to write, just make another thread, and will be long just like this one

Not every idea is so appealing and so useless as a binary clock.

According to the performance tests results from post #56 my binary clock script is two times worse than the other binary clock scripts though it uses the rotate function designed by David the H. which is better than my original rotate function. So my own coding abilities are more than two times worse than the abilities of the competition. The wminfo package contains over 80 plugins. I wrote all of them except seven mentioned in post #56.

Theoretically I could post here these 80 plugins asking for help with the optimization of the code but it wouldn’t be neither honorable nor reasonable.

What I need to do is to improve my bash knowledge in order to be capable to produce the better code.

But when I invent some appealing and useless idea for the next time I’ll surely let you know.

w1k0 · 09-25-2012, 06:24 AM

Quote:

Originally Posted by grail

First line needs to be an interpreter just like bash, so try adding the following as the first line:

Code:

#!/usr/bin/awk -f

Of course it will be your path to awk if it is different from above

It’s exactly what I did in the second step mentioned above. As a result I saw:

./bin_clock.gawk

Code:

awk: ./bin_clock.gawk:6:   bin[0][1]=b;bin[0][2]=b;bin[0][3]=b;bin[0][4]=b
awk: ./bin_clock.gawk:6:         ^ syntax error
.
.
.
awk: ./bin_clock.gawk:15:   bin[9][1]=a;bin[9][2]=b;bin[9][3]=b;bin[9][4]=a
awk: ./bin_clock.gawk:15:                                             ^ syntax error
awk: ./bin_clock.gawk:25:         printf("%1s", bin[digit[j]][i])
awk: ./bin_clock.gawk:25:                                    ^ syntax error

Quote:

Although I think ntubski's version from post 15 is probably more concise

I attached to the wminfo package ntubski’s version as binclock-vertical-4.wmi. I intend to attach to the tarball discussed now PTrenholme’s version as well. But first it should work and then I should find the way to implement in it the variables switching the label and the separators according to the standards used in the other plugins from the binary clock family.

grail · 09-25-2012, 06:41 AM

Sorry ... missed that part (domestic blindness). Are you perhaps using an older version of gawk?

w1k0 · 09-25-2012, 07:36 AM

Quote:

Originally Posted by grail

Sorry ... missed that part (domestic blindness). Are you perhaps using an older version of gawk?

I use Slackware 13.37 and it provides gawk 3.1.8. What’s worse incoming Slackware 14.0 provides gawk 3.1.8 too. Free Software Foundation offers gawk 4.0.1. So the reply to your question is: “Probably yes”.

***

I installed gawk 4.0.1 and now PTrenholme’s script works. So my answer to your questions above the asterisks should be “Of course!” and below the asterisks: “No way!”.

w1k0 · 09-25-2012, 08:11 AM

@PTrenholme,

Your new script changed the situation in the samples directory:

Code:

    plugin                  | author     | language | CPU usage   | median
    ------------------------+------------+----------+-------------+-------
    binclock-vertical-8.wmi | PTrenholme | AWK      | 4.0% - 4.2% | 3.9%
    binclock-vertical-9.wmi | w1k0       | bash     | 3.8% - 4.4% | 3.9%

(The statistics for the competition are stored in the table from post #56.)

w1k0 · 09-25-2012, 08:27 AM

Quote:

Originally Posted by grail

Although I think ntubski's version from post 15 is probably more concise

Now – when wminfo can run AWK scripts thanks to ntubski’s patch – it’s possible to modify ntubski’s script to avoid wrapping AWK code with bash. It should improve the performance of that script.

***

By the way: do you know which gawk version introduced the changes that allow to run the discussed PTrenholme’s script?

ntubski · 09-25-2012, 10:32 AM

Quote:

Originally Posted by w1k0

The performance was mesured with the command:

top -b -n 40 | grep '^Cpu' | sort -n

I was thinking a better way of measuring performance would be to run wminfo for a fixed interval (say 5 minutes) and check the reported CPU usage from the time command. The man page doesn't say, but after experimenting it looks time commands generally include CPU usage of child processes (the plugins, in our case) in their reports.

PTrenholme · 09-25-2012, 11:21 AM

O.K., that one wasn't too efficient. This might work faster.

Code:

#!/bin/gawk -f
BEGIN {
  a="*"
  b=" "
# Try it with these values for a and b for a more colorfull output
#  a="\x1b[41m \x1b[0m"
#  b="\x1b[42m \x1b[0m"
  d=SUBSEP
  bin[0]=b d b d b d b
  bin[1]=b d b d b d a
  bin[2]=b d b d a d b
  bin[3]=b d b d a d a
  bin[4]=b d a d b d b
  bin[5]=b d a d b d a
  bin[6]=b d a d a d b
  bin[7]=b d a d a d a
  bin[8]=a d b d b d b
  bin[9]=a d b d b d a
# Uncomment the next two lines and the system, fflush and the bracket for continous run
#  while (1) { 
#    print "\x1b[2;J\x1b[1;1;G"
    date=strftime("%H%M%S")
    split(date,digit,"")
    print digit[1] digit[2] "|" digit[3] digit[4] "|" digit[5] digit[6]
    print "HH|MM|SS"
    split(bin[digit[1]],Hl,d)
    split(bin[digit[2]],Hr,d)
    split(bin[digit[3]],Ml,d)
    split(bin[digit[4]],Mr,d)
    split(bin[digit[5]],Sl,d)
    split(bin[digit[6]],Sr,d)
    for (i=1;i<=4;++i) {
      print Hl[i] Hr[i] "|" Ml[i] Mr[i] "|" Sl[i] Sr[i] 
    }
#    system("sleep 1s")
#    fflush()
#  }
}

P.S. Sorry for the confusion. I should have mentioned that the strftime was a gawk extension, and that multidimensional arrays where introduced in version four of gawk. Both of those features are not available in POSIX mode, nor, I believe, in mawk. (As you can see from the time, it was fairly late, and my wife was after me to "Turn of that [expletive deleted] computer and come to bed! Now!")

grail · 09-25-2012, 11:30 AM

Quote:

By the way: do you know which gawk version introduced the changes that allow to run the discussed PTrenholme’s script?

Unfortunately I was not able to track down exactly when array of arrays was introduced, but your testing would imply from version 4 onwards.

grail · 09-25-2012, 11:32 AM

Quote:

Turn of that [expletive deleted] computer and come to bed! Now!

I believe there are many here would have heard that from a family member / partner :-0

w1k0 · 09-25-2012, 03:28 PM

@ntubski,

Quote:

The man page doesn't say, but after experimenting it looks time commands generally include CPU usage of child processes (the plugins, in our case) in their reports.

I tested that and it works as you said. It’s incredible progress. Earlier I switched the system to the idle state and tested wminfo plugins one by one. Now I can work as always and test at the same time any number of wminfo plugins. Moreover the results of these new tests are much more precise. It’s fantastic!

Here’s my first attempt to write the benchmark using your TIMEFORMAT variable:

wminfo-benchmark:

Code:

#!/bin/bash

time=60

wrapper() {
for plugin in *.wmi
do
    TIMEFORMAT=$(printf '%-24.24s: %s' "$plugin" '%2R real, %2U user, %2S sys, %P%% CPU')
    command=$(grep '^# command: ' $plugin | sed 's/# command: //')
    time $command &
done
}

timer() {
    wrapper
    sleep $time
    killall wminfo
}

timer

That benchmark is designed to test wminfo plugins. The plugin has by default extension .wmi and the header such as:

Code:

#!/bin/bash

# wminfo plugin: binary clock (4-bits version by Noam Postavsky aka ntubski)

# command: wminfo -p binclock-vertical-4.wmi -u 1 -b#000000 -f#ffff00

On the basis on that the benchmark determines the command such as:

wminfo -p binclock-vertical-4.wmi -u 1 -b#000000 -f#ffff00

and runs one or more wminfo instances performing the tests for all *.wmi plugins from the current directory.

The benchmark performed on all binary clock plugins displays the following results:

./wminfo-benchmark

Code:

./benchmark.sh: line 6: 11388 Terminated              $command
binclock-vertical-2.wmi : 60.21 real, 0.45 user, 0.22 sys, 1.13% CPU
./benchmark.sh: line 6: 11380 Terminated              $command
binclock-vertical-1.wmi : 60.22 real, 0.43 user, 0.26 sys, 1.15% CPU
./benchmark.sh: line 6: 11395 Terminated              $command
binclock-vertical-3.wmi : 60.20 real, 1.35 user, 0.39 sys, 2.90% CPU
./benchmark.sh: line 6: 11400 Terminated              $command
binclock-vertical-4.wmi : 60.18 real, 1.40 user, 0.46 sys, 3.09% CPU
./benchmark.sh: line 6: 11411 Terminated              $command
binclock-vertical-5.wmi : 60.14 real, 1.64 user, 0.46 sys, 3.50% CPU
./benchmark.sh: line 6: 11422 Terminated              $command
binclock-vertical-6.wmi : 60.11 real, 1.39 user, 0.42 sys, 3.02% CPU
./benchmark.sh: line 6: 11430 Terminated              $command
binclock-vertical-7.wmi : 60.07 real, 1.63 user, 0.48 sys, 3.51% CPU
./benchmark.sh: line 6: 11441 Terminated              $command
binclock-vertical-8.wmi : 60.04 real, 0.74 user, 0.27 sys, 1.70% CPU
./benchmark.sh: line 6: 11447 Terminated              $command
binclock-vertical-9.wmi : 60.01 real, 3.62 user, 1.40 sys, 8.38% CPU

So binclock-vertical-8.wmi plugin by PTrenholme is much more efficient than according to my previous test (see: CPU usage).

The benchmark has two defects:

1. It not always displays the plugins in the correct order – it’s caused by multiple wminfo instances started one by one. Sometimes the next command runs wminfo faster then the previous one and that spoils the order of the plugins. I suppose it isn’t possible to avoid that.

2. It displays the unwanted messages. I can’t find the method to suppress them.

[EDIT]
I discovered the third defect:

3. The benchmark doesn’t cope with the command including quotation marks (parameters passed to the plugin). After the modification it can write these commands to the temporary file and then run it but I don’t like that solution too much.
[/EDIT]

I’ll be grateful if you help me to improve the above benchmark.

@PTrenholme,

According to the above comment your binclock-vertical-8.wmi plugin is much better than I thought. I’ll test all the plugins including your newest one once again when we’ll finish to tune wminfo-benchmark script.

***

[EDIT]
The pre-release of wminfo is here: http://linux-bsd-unix.strefa.pl/wminfo-pre3.1.0.tar.gz.
[/EDIT]

H_TeXMeX_H · 09-26-2012, 04:24 AM

Well, my recommendation is to improve timing within wminfo and implement a run time option, so you can get it to run for a certain amount of seconds accurately (cannot be done with currently implemented timing system). Then you can run it with the GNU time utility and time each plugin accurately.

H_TeXMeX_H · 09-26-2012, 05:56 AM

Well, here's my solution, I had to edit wminfo itself.

PHP Code:



#!/bin/sh 
 
ticks=10 
time="$(which time)" 
export TIME='%e real, %U user, %S sys, %P CPU' 
 
for plugin in *.wmi 
do 
    command="$(grep '^# command: ' "$plugin" | sed 's/# command: //')" 
    printf "$plugin : " 
    $time $command -t "$ticks" 2>&1 | awk '{ printf ("%.2f %s %.2f %s %.2f %s %.3f %% CPU\n",$1,$2,$3,$4,$5,$6,($3+$5)/$1)}' 
done 
 
exit 0

Code:

bash-4.1$ ./bench  
binclock-vertical-3.wmi : 9.38 real, 0.05 user, 0.03 sys, 0.009 % CPU
binclock-vertical-4.wmi : 10.00 real, 0.04 user, 0.04 sys, 0.008 % CPU
binclock-vertical-5.wmi : 10.00 real, 0.07 user, 0.04 sys, 0.011 % CPU
binclock-vertical-6.wmi : 9.99 real, 0.07 user, 0.02 sys, 0.009 % CPU
binclock-vertical-7.wmi : 10.00 real, 0.07 user, 0.04 sys, 0.011 % CPU

ntubski · 09-26-2012, 08:48 AM

Quote:

Originally Posted by H_TeXMeX_H

Well, my recommendation is to improve timing within wminfo and implement a run time option, so you can get it to run for a certain amount of seconds accurately (cannot be done with currently implemented timing system).

Based on your timings in the next post, it doesn't seem like this method is more accurate.

Quote:

Originally Posted by w1k0

Earlier I switched the system to the idle state and tested wminfo plugins one by one. Now I can work as always and test at the same time any number of wminfo plugins. Moreover the results of these new tests are much more precise. It’s fantastic!

Cool, I was just thinking about accuracy when I suggested using time, it didn't even occur to me to run things in parallel.

Quote:

1. It not always displays the plugins in the correct order – it’s caused by multiple wminfo instances started one by one. Sometimes the next command runs wminfo faster then the previous one and that spoils the order of the plugins. I suppose it isn’t possible to avoid that.

2. It displays the unwanted messages. I can’t find the method to suppress them.

[EDIT]
I discovered the third defect:

3. The benchmark doesn’t cope with the command including quotation marks (parameters passed to the plugin). After the modification it can write these commands to the temporary file and then run it but I don’t like that solution too much.
[/EDIT]

Should all be fixable:

Code:

#!/bin/bash

time=60

wrapper() {
for plugin in *.wmi
do
    TIMEFORMAT=$(printf '%-24.24s: %s' "$plugin" '%2R real, %2U user, %2S sys, %P%% CPU')
    command=$(sed -n 's/^# command: \(wminfo .*\)/\1/p' "$plugin")
    # eval will correctly handle quotes or any shell construct in the
    # command
    eval time "$command" &
done
}

timer() {
    # both the time reports and Terminated messages go on stderr
    wrapper 2>&1 | grep -v Terminated | sort -k1,1 &

    sleep $time
    killall wminfo

    # wait for all the reports to be printed before exiting
    wait 
}

timer

Oh, before I forget again, @grail: putting all the code on 1 line doesn't make it a one-liner! Justlikethissentenceisnot1word.

H_TeXMeX_H · 09-26-2012, 09:14 AM

Quote:

Originally Posted by ntubski

Based on your timings in the next post, it doesn't seem like this method is more accurate.

You are right, it isn't more accurate, because the real problem is that 'time' is not accurate. Oh well, your solution is better, I mean it doesn't require any core modifications.