[SOLVED] deleting columns over multiple files

atjurhs · 09-26-2012, 12:57 PM

here's the link

http://www.linuxquestions.org/questi...57#post4789957

I called the thread "deleting every other row, sometimes"

atjurhs · 09-28-2012, 09:50 AM

for some reason the output file from the first script on post 26 was not accepted by MatLab scripts. I'll go back and try the second script on post 26 and see if the MatLab script likes it any better. Otherwise I'll have to find one of the guys to help me debug the the problem, but I can't really work on this till Monday as my server is down

Tabby

konsolebox · 09-28-2012, 07:11 PM

@atjurhs: Try injecting a raw file first. When it works, see what differs it from the output of the script.

atjurhs · 10-01-2012, 03:34 PM

yipeeee, I found the problem

in the raw input file there is never a row that does not contain a value in at least one of the columns of data.

in the output file (from 1st of the script shown on post 26, (which runs much faster than the second)) there are several rows that will contain nothing but comas. so this output file is not the right input format for that MatLab code.

wrong format

Code:

header_3, header_5, header_8, header10_  , header11_0_0_    , header12, header13     , header14_, header15_
123 , 123456789 , 1234567 , 1234567 , 1 , 12345678 , 1 , 1 , 123456789
   ,          ,     , , ,        ,  ,  
1234567,        , 12345, , ,        ,  ,  
   ,          ,     , , ,        ,  ,  
   ,          ,     , , ,        ,  ,  
   ,          ,     , , ,        ,  ,  
   ,          ,     , , ,        , 1, 1 
   ,          ,     , , ,        ,  ,  
1234567,        ,     , , ,        ,  ,  
1234567,        , 12345, , ,        ,  ,  , 12345
123 , 123456789 , 1234567 , 1234567 , 1 , 12345678 , 1 , 1 , 123456789
   ,          ,     , , ,        ,  ,  
1234567,        , 12345, , ,        ,  ,

right format

Code:

header_3, header_5, header_8, header10_  , header11_0_0_    , header12, header13     , header14_, header15_
123 , 123456789 , 1234567 , 1234567 , 1 , 12345678 , 1 , 1 , 123456789 
1234567,        , 12345, , ,        ,  ,  
   ,          ,     , , ,        , 1, 1 
1234567,        ,     , , ,        ,  ,  
1234567,        , 12345, , ,        ,  ,  , 12345
123 , 123456789 , 1234567 , 1234567 , 1 , 12345678 , 1 , 1 , 123456789 
1234567,        , 12345, , ,        ,  ,

so all that needs to be added is to strip out those rows that only contain all comas and no values

Tabby

konsolebox · 10-01-2012, 08:54 PM

Then you could just add a test to that:

Code:

#!/bin/bash

# Change columns here. This is repetitive within the loop, but just for convenience.
REMOVE=(1 2 4 6 7 9)

# Extension of output file's name.
OUTPUTEXT='out'

IFS=','

for FILE; do
	while read LINE; do
		FIELDS=() # just to be safe
		read -a FIELDS <<< "${LINE//, /,}"
		for I in "${REMOVE[@]}"; do
			unset "FIELDS[$I - 1]"
		done
		LINE="${FIELDS[*]}"
		[[ $LINE == *[^\ ,]* ]] && echo "${LINE//,/, }"
	done < "$FILE" > "$FILE.$OUTPUTEXT"
done