Splitting the source file as per the code(1st substring) in 4th(AcctID) column

pan64 · 02-27-2018, 05:34 AM

there is no need to write script, grep will do it by itself.
as I told you already: grep --file=lookup-file input-file > output-file
just you need to use the lookup file I posted. see man grep about details (of --file)

keefaz · 02-27-2018, 05:47 AM

Depending on how large the files sizes are, perl language could be a good candidate too
But I agree with pan64, grep and creation of patterns files looks appealing with 4 lines of code to solve the whole problem

bishnumnnit2006 · 02-27-2018, 05:52 AM

Hey sorry but I am not sure how grep will work when we will getting different codes on every run, suppose we have 100 codes in a main file are we grepping all 100 codes in script, I am not sure if it works.

bishnumnnit2006 · 02-27-2018, 05:55 AM

This is what my actual code look like and its working fine.

#!/bin/bash

. /xmretl/Scripts/setTLMEtlEnv.ksh

. $XmrEtlScripts/TLMCmnFuncs.ksh

CurrDt=`date +%Y%m%d_%H%M%S`

LogFile=SplitTLMGrpRefFiles_Test_$CurrDt.log

echo "Start the execution of the script" >> $XmrEtlScriptLogs/$LogFile

echo "Client code in source file is "$CODE"" >> $XmrEtlScriptLogs/$LogFile

touch ${XmrEtlSrcFiles}/AMR1_CODE.TXT

touch ${XmrEtlSrcFiles}/AMR2_CODE.TXT

chmod 777 ${XmrEtlSrcFiles}/*CODE.TXT*

cat ${XmrEtlSrcFiles}/AMR1.TXT | awk 'BEGIN{RS=" "}{$1=$1}1' > ${XmrEtlSrcFiles}/AMR1_CODE.TXT

cat ${XmrEtlSrcFiles}/AMR2.TXT | awk 'BEGIN{RS=" "}{$1=$1}1' > ${XmrEtlSrcFiles}/AMR2_CODE.TXT

while read -r LINE

do

#CODE=`echo $LINE |cut -d'|' -f4|cut -d'-' -f1|grep -v AcctID`

CODE=`echo $LINE |awk -F '|' '{ split($4,a,"-"); print a[1]}'`

#echo "Client code in source file is "$CODE"" >> $XmrEtlScriptLogs/$LogFile

#touch ${XmrEtlSrcFiles}/AMR1_CODE.TXT

#touch ${XmrEtlSrcFiles}/AMR2_CODE.TXT

#chmod 777 ${XmrEtlSrcFiles}/*CODE.TXT*

#cat ${XmrEtlSrcFiles}/AMR1.TXT | awk 'BEGIN{RS=" "}{$1=$1}1' > ${XmrEtlSrcFiles}/AMR1_CODE.TXT

#cat ${XmrEtlSrcFiles}/AMR2.TXT | awk 'BEGIN{RS=" "}{$1=$1}1' > ${XmrEtlSrcFiles}/AMR2_CODE.TXT

while read -r LINE1

do

LKP_AMR1=`echo $LINE1`

if [[ $CODE = $LKP_AMR1 ]]

then

`touch /xmretl/TLMLandingZone/AMR1_ERU_6_GROUPREFERENCE_12012017_3.TXT` >> $XmrEtlScriptLogs/$LogFile

`chmod 775 /xmretl/TLMLandingZone/AMR1_ERU_6_GROUPREFERENCE_12012017_3.TXT`

`echo $LINE >> /xmretl/TLMLandingZone/AMR1_ERU_6_GROUPREFERENCE_12012017_3.TXT`

fi

done < ${XmrEtlSrcFiles}/AMR1_CODE.TXT

while read -r LINE2

do

LKP_AMR2=`echo $LINE2`

if [[ $CODE = $LKP_AMR2 ]]

then

`touch /xmretl/TLMLandingZone/AMR2_ERU_6_GROUPREFERENCE_12012017_3.TXT`

`chmod 775 /xmretl/TLMLandingZone/AMR2_ERU_6_GROUPREFERENCE_12012017_3.TXT`

`echo $LINE >> /xmretl/TLMLandingZone/ERU_6_GROUPREFERENCE_12012017_3.TXT`

fi

done < ${XmrEtlSrcFiles}/AMR2_CODE.TXT

done < ${XmrEtlSrcFiles}/ERU_6_GROUPREFERENCE_02262018_1.TXT

bishnumnnit2006 · 02-27-2018, 05:56 AM

The only issue is with the performance, so if you have any alternate solution please come out with the script, Many Thanks.

rtmistler · 02-27-2018, 06:01 AM

Quote:

Originally Posted by bishnumnnit2006

Hey sorry but I am not sure how grep will work when we will getting different codes on every run, suppose we have 100 codes in a main file are we grepping all 100 codes in script, I am not sure if it works.

You've been shown quite a few examples by various members. Please understand that you stand to raise the frustration level of them if you seemingly don't appear to be trying any of their suggestions. There are plenty of things you can do to test and debug, much of which I imagine you have done a bit of. I'd suggest that you update your fellow members with some progress with a few attempt updates. People are not here to do these tasks literally for you, we are here to help you to learn and do for yourself. You can use a smaller set of data to test a few tactics and then run it on your larger files. Bear in mind that I don't know everything about awk, it is one of my weakest "non-talents" in the Unix/Linux world. However when I choose to use it, or the tr command, I perform a web search saying exactly what I wish to do with the command. And if I've been too particular, I wean it down to be "how do I split a line with awk", "how do I print the 5th field with awk", and so forth. As you get to use LQ more and eventually offer advice to fellow members, you'll appreciate those who, while they have questions, they also work with some diligence at attaining their solution. Also, you'll feel much more satisfied if you partake an active role in solving your problem, you will own part of that solution, if not most of it.

Quote:

Originally Posted by bishnumnnit2006

The only issue is with the performance, so if you have any alternate solution please come out with the script, Many Thanks.

I wrote all of the above from your earlier reply. This that you just wrote is now bordering on rudeness towards forum members. Please consider what I've written above. If you feel that LQ members are here to hand you solutions, then this is not a forum where you can attain the help you are seeking.

You also persist with failing to put your code within [code] tags to properly separate it and retain formatting.

bishnumnnit2006 · 02-27-2018, 06:05 AM

That's fine, thanks everyone for your help, you can closed this discussion .

Have a nice time!

pan64 · 02-27-2018, 06:19 AM

I have already posted some improvements and already mentioned that....

BW-userx · 02-27-2018, 08:18 AM

Quote:

Originally Posted by bishnumnnit2006

Hey sorry but I am not sure how grep will work when we will getting different codes on every run, suppose we have 100 codes in a main file are we grepping all 100 codes in script, I am not sure if it works.

this is why one should and some do, write the code, in a command line or script, does not matter, then test it with some actual data pulled out of the chuck of it all and placed into a secure location so it does not effect the normal process.

even just that one grep line that is said that is all you need, then place it into a file by whatever name so you do not have to keep rewriting the entire line all of the time, just the filename to get it to do its work.

one shouldn't do anything like this on production runs without first testing it.

BW-userx · 02-27-2018, 08:52 AM

this is knowing what you are looking for that the information held within the two files have.

Code:

$ egrep 'GHU|TRP|ABC|AAR|TIA' source
6|ERU_MCSD_POS|ERU_MCSD_POS|AAR-IAS-AR8F92001T2-C
6|ERU6_ARCGEN|generic Accounts|ABC-IAS-AR8F92001T2-C
6|ERU6_ARCGEN|Archive Accounts|TRP-IAS-AR8F92001T2-C
6|ERU6_ARCGEN|Archive Generic|TIA-IAS-AR8F92001T2-C

astrogeek · 02-27-2018, 03:59 PM

@bishnumnnit2006:

As you have been аsked, please place your code snippets inside [CODE]...[/CODE] tags for better readability. You may type those yourself or click the "#" button in the edit controls.

You have received much good advice, including complete examples of how to approach your problem, but have not shown any effort of your own to use, or even consider the help provided. You persist in asking for others to provide complete solutions for you. This is not how LQ works.

Please review the Site FAQ for guidance in posting your questions and general forum usage.

We are happy to help, but if you are looking for others to do your work for you, this is not the place.