How to remove lines with while loop whch were previously added

blason · 11-16-2019, 05:33 AM

Hi Folks,

I am a little confused here with my script.

I am reading from file; running in while loop; creating the zone.
This is working fine.
However, since I keep adding new entries; due to while loop those are getting added again and again.

Here is what I mean -

Quote:

_EXPL_WHITELIST="/zone/expl_wl_bl/customWL.txt"

if [ -f ${_EXPL_WHITELIST} ]
then
if [ -s ${_EXPL_WHITELIST} ]
then
cat $_EXPL_WHITELIST | while read line;do echo -e $line'\t'CNAME'\t'rpz-passthru.;done >> $_FINAL_WHITELIST
fi
fi
rndc reload

in the first iteration, I get an output like

Quote:

productreviews.shopifycdn.com CNAME rpz-passthru.
large.edgecast.syn-cdn.com CNAME rpz-passthru.
www.anrdoezrs.net CNAME rpz-passthru.
scontent-frt3-1.xx.fbcdn.net CNAME rpz-passthru.

In the next iteration If I add "new.com" and "net.net" and re-run the script

Quote:

productreviews.shopifycdn.com CNAME rpz-passthru.
large.edgecast.syn-cdn.com CNAME rpz-passthru.
www.anrdoezrs.net CNAME rpz-passthru.
scontent-frt3-1.xx.fbcdn.net CNAME rpz-passthru.
new.net CNAME rpz-passthru.
new.com CNAME rpz-passthru.

In the next iteration again first lines get added again and again.
I need to find a solution -

so that only new entries will get appended.

Any idea folks?

blason · 11-16-2019, 06:12 AM

May be this is dirty way since I am not finding a better way but let me know if there are other fine ways to achieve this

I added # at the end of line

Quote:

while read line;do echo -e $line'\t'CNAME'\t'rpz-passthru. "#" ; done

And then at next iteration

Quote:

sed -i '/#$/d' added
cat file | while read line;do echo -e $line'\t'CNAME'\t'rpz-passthru. "#" ; done >> added

Firerat · 11-16-2019, 06:28 AM

Code:

#!/bin/bash
#

_EXPL_WHITELIST="/zone/expl_wl_bl/customWL.txt"

while read line
do  
  printf "${line}\tCNAME\trpz-passthru.\n"
done <${_EXPL_WHITELIST} > ${_FINAL_WHITELIST}


#

your problem was >> $_FINAL_WHITELIST
>> appends to file
> overwrites file

since you are not really doing much, you could just use sed

Code:

sed 's/.*/&\tCNAME\trpz-passthru./' "${_EXPL_WHITELIST}"

michaelk · 11-16-2019, 06:37 AM

You could just delete the final_whitelist file at the beginning of the script since it is recreated by the loop.

MadeInGermany · 11-16-2019, 07:31 AM

It is good to replace the non-portable "echo -e" with "printf".
But the printf format string (first argument) understands things like %s and %f in addition to \t and \n.
So if $line had a % character it would go wrong in the format string.
So better have variables in the following arguments:

Code:

printf "%s\tCNAME\trpz-passthru.\n" "$line"

or

Code:

printf "%s\t%s\t%s\n" "$line" "CNAME" "rpz-passthru."

pan64 · 11-16-2019, 07:52 AM

I think almost everything was already told, so here are just a few minor comments:
1. use shellcheck to check your shell script, that will give you some hints to write more robust code.
2. for example it will suggest to use: read -r line
3. a not-so-nice solution (comparing to the suggested ones) would be: you can sort -u the result, which will remove all the duplicates.

Firerat · 11-16-2019, 08:01 AM

yeap, very true

I was being very lazy

an awk

Code:

awk '{printf "%s\t%s\t%s\n",$0,"CNAME","rpz-passthru."}' $inputfile
# or
awk '{printf "%s\tCNAME\trpz-passthru.\n",$0}' $inputfile
#

and another bash loop

Code:

while read line
do
  echo "${line/%/$'\t'CNAME$'\t'rpz-passthru.}"
done <"$inputfile"

#

BW-userx · 11-16-2019, 08:55 AM

Quote:

Originally Posted by blason

Hi Folks,

I am a little confused here with my script.

I am reading from file; running in while loop; creating the zone.
This is working fine.
However, since I keep adding new entries; due to while loop those are getting added again and again.

Here is what I mean -

Code:

_EXPL_WHITELIST="/zone/expl_wl_bl/customWL.txt"

if [ -f ${_EXPL_WHITELIST} ]
then
if [ -s ${_EXPL_WHITELIST} ]
then
cat $_EXPL_WHITELIST | while read line;do echo -e $line'\t'CNAME'\t'rpz-passthru.;done >> $_FINAL_WHITELIST
fi
fi
rndc reload

in the first iteration, I get an output like

Code:

productreviews.shopifycdn.com CNAME rpz-passthru.
large.edgecast.syn-cdn.com CNAME rpz-passthru.
www.anrdoezrs.net CNAME rpz-passthru.
scontent-frt3-1.xx.fbcdn.net CNAME rpz-passthru.

In the next iteration If I add "new.com" and "net.net" and re-run the script

Code:

productreviews.shopifycdn.com CNAME rpz-passthru.
large.edgecast.syn-cdn.com CNAME rpz-passthru.
www.anrdoezrs.net CNAME rpz-passthru.
scontent-frt3-1.xx.fbcdn.net CNAME rpz-passthru.
new.net CNAME rpz-passthru.
new.com CNAME rpz-passthru.

In the next iteration again first lines get added again and again.
I need to find a solution -

so that only new entries will get appended.

Any idea folks?

you have two files
_EXPL_WHITELIST
_FINAL_WHITELIST

you cat $_EXPL_WHITELIST into the final list file. how are you getting text added into your first file? why not take how you're getting NEW text added into the first file then just append it to your last file instead, by passing that step that looks like it keeps a running tally of what you're doing, like your repeating a step.

appended new entries into file1
use file1 to append the entries into file2.
repeat

using 2 files with almost matching data

Code:

#!/usr/bin/env bash

file1=/home/userx/test/finalList
file2=/home/userx/test/customList

while read keep
do
  while read add
  do
    if [[ "$add" != "$keep" ]]
    then
	    echo "$add"
	    Add2Array+=( "$add" )
    fi
  
  done < $file2
done < $file1
#to get rid of dups due to method used to find
#not entered into final file
declare -A dupes

for i in "${Add2Array[@]}"; do
    if [[ -z ${dupes[$i]} ]]; then
        NEWARRAY+=("$i")
    fi
    dupes["$i"]=1
done
unset dupes # optional

printf "[%s]" "${Add2Array[@]}"
echo
printf "[%s]" "${NEWARRAY[@]}"
echo

or at end

Code:

printf "%s\n" "${NEWARRAY[@]}" >> $_FINAL_WHITELIST

you'd have to curtail it to your files, then run it to see what it does.

(last part of this solution here )

the longer you run this the bigger both of the files will get = the time to complete this step will increase.