Remove many lines based on content of other file

anderssk · 06-21-2011, 02:49 AM

Hi'

I a csv-file (A.csv) with a total of 4.600.000 lines.
Thats to many and only a few is nessesary

I have a txt-file with 150 lines (X.txt) (all lines is dataset from a mainframe and looks like abc.def.123.456

How do I remove lines from A.csv where none of the dataset from x.txt is present?

colucix · 06-21-2011, 03:11 AM

Code:

grep -vf X.txt A.csv > A_new.csv

MTK358 · 06-21-2011, 11:04 AM

Quote:

Originally Posted by anderssk

How do I remove lines from A.csv where none of the dataset from x.txt is present?

What does that mean?

anderssk · 06-22-2011, 05:16 AM

The content is one dataset pr. line
A mainframe dataset is build up by 8 char. a dot and 8 char.
ABCDEFGH.12345678.ABCDEFGH.12345678.ABCDEFGH

Max 44 char.

colucix · 06-22-2011, 05:34 AM

Just out of curiosity, does the code suggested in post #2 work?

MTK358 · 06-22-2011, 06:30 AM

Quote:

Originally Posted by anderssk

The content is one dataset pr. line
A mainframe dataset is build up by 8 char. a dot and 8 char.
ABCDEFGH.12345678.ABCDEFGH.12345678.ABCDEFGH

Max 44 char.

That doesn't help. What exactly is the format and what exactly do you want to do?

The way I understood it from your original post is that you want to remove lines from A.csv if they are not in file x.txt. Is that correct?

anderssk · 06-22-2011, 06:33 AM

Yes - that's correct. I have tried with the grep -vf but the A_new.txt file is emty :-(

MTK358 · 06-22-2011, 06:36 AM

MTK358 · 06-22-2011, 06:41 AM

Code:

while read line
do
    if grep -F "$line" x.txt > /dev/null
    then
        echo "$line"
    fi
done < <(cat A.csv)

anderssk · 06-22-2011, 07:11 AM

Last line give a syntax error : redirection unexpected

MTK358 · 06-22-2011, 07:12 AM

How about replacing the last line with:

Code:

done < A.csv

?

Reuti · 06-22-2011, 07:21 AM

Does the script start with #!/bin/sh instead of #!/bin/bash? <() redirection is present in bash only, not in its compatibility mode when it discovers it was started by /bin/sh.

MTK358 is right anyway: you don’t need it.

Nominal Animal · 06-22-2011, 12:49 PM

Quote:

Originally Posted by anderssk

I a csv-file (A.csv) with a total of 4.600.000 lines. [...]
I have a txt-file with 150 lines (X.txt) (all lines is dataset from a mainframe and looks like abc.def.123.456

It depends on how you interpret the fields in the CSV file, and what "dataset" means. You did not tell us either of those.

Do you wish to keep only the records in A.csv that contain a field with one of the abc.def.123.456 values? In other words, does each line of X.txt describe a field in the A.csv file?

David the H. · 06-23-2011, 10:34 AM

I think you should stop talking vaguely about the files and what you want to do with them, and post an actual example of the contents, demonstrating exactly what changes you need to make. Also show us the scripting attempts you've made so far.

And if you aren't using standard bash and standard linux (gnu) tools on standard *nix-based files, then give us the necessary background on that as well.

That should save us a lot of ping-pong Q&A and help you get the answers you need.