ProgrammingThis forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Got it , i just wrote it in a rush , the question i asked before was because the logfile related to a specific ip will change a bit because instead pointing to a single server file , i will point to a directory and look for ip on all files there .
Example for nginx access log files
Past = /var/log/nginx/access.log
Present = /var/log/nginx/*
This means that logfile will not be again :
ip - timestamp - date - string
will be now
filename that got string - ip - timestamp - date - string
witch means more stuff to be cleaned , and this is why that awk instruction in : ${loghttp} awk '/'$ip'/{print $7}'
I will grab all lines that contain a specific ip address from all logs in all services (mail,webservers,etc..) to 1 , and then i will run your code to search for patterns that match any of the lines in definitions.conf file .
The definitions.conf will not have only definitions that can be applied on webservers , it will also have patterns to identify intrusions on mail servers , pop3 or smtp services .
I need to do it this way (pick from everywhere to a single file) because it will happen a lot of occasions that something new was not yet added to definitions.conf and i will need to add it , and for that i will only need to look into 1 single file (ipres in last code) and choose a pattern to add it to definitions.conf .
Hope you all got +- the game here .
However , i am just commenting now .
Thanks for the tips above .
yup but there is a problem with it , i just dont need also to identify the pattern , i also need to bring back the variable 1 from definitions.conf , what really matters is the 1st variable in definitions that will tell me what intrusion was detected .
This last code you wrote will only identify the pattern and will not bring back the intrusion name . Also , this last code can not be applied to the script because i dont want to edit the script every time i need to add a new pattern .
Your previous code is perfect and runs great , i will use that one .
I just need to do some adjustments to the script for now , and with time build a new one with user config settings to add it to github just in case anyone needs it too .
But that is a long story .
ThinkPHP_RCE module/action/param1/${@die(md5(HelloThinkPHP))}
is the one you have to pay special attention to
how many entries do you expect in definition.conf ?
To that Intrusion ThinkPHP_RCE is just that one for now , but definitions.conf will have hundreds of lines with multiple patterns .
I update definition.conf file every time some smart guy tries anything new on server to see if he/she can exploit the server and take over controls .
Or i personally try to exploit my own server using multiples new exploits , then i watch how web servers log that exploit code in their logs , and then i choose a common pattern to add it to the definition.conf .
the 18 lines you see now in definition.conf will be 1000 in an year the way i see things everyday on the web .
ok , i get it now .
What you are speaking is that patterns like think php_rce that have special characters the code will not read it after that point , but if i change the code to understand it then i will have problems with all other patters that do not have those special characters .
Problems in : $,{,@
Quote:
module/action/param1/${@die(md5(HelloThinkPHP))}
dont worry , i will create 2 patterns for this exploit in definitions file .
you could follow @MadeInGermany's lead in #45 and have the patterns as simple strings
that is fine if you treat each one individually.
I was thinking ahead when I chose [[ ${LOG} =~ RE ]] , since crafting a regular expression to search multiple patterns is faster than searching each, one at a time.
maybe return to the RE once you are more comfortable with them ( I have to admit I do dislike working with RE ) and for now use one of the slow methods suggested by @MadeInGermany
we have a problem .
It was working but now is not .
It looks that when it is reading the patters from definition.conf file it is only reading the 1st variable and not the second as it should be .
what is happening is that in this line :
Quote:
PHP_Exploitation phppma/index.php
is reading PHP_Exploitation instead phppma/index.php
I am implementing the code in the final script now , and this problem poped up .
i run the script in bash -x and i found that .
Quote:
+ for P in "${Patterns[@]}"
+ [[ /sql/phpMyAdmin2/index.php?lang=en =~ PHP_Exploitation ]]
+ for P in "${Patterns[@]}"
+ [[ /sql/phpMyAdmin2/index.php?lang=en =~ PHP_Exploitation ]]
+ for P in "${Patterns[@]}"
+ [[ /sql/phpMyAdmin2/index.php?lang=en =~ PHP_Exploitation ]]
+ for P in "${Patterns[@]}"
+ [[ /sql/phpMyAdmin2/index.php?lang=en =~ PHP_Exploitation ]]
now , the code that matters was not changed :
Quote:
intdet () {
IP=$1
while read LOG
do
for P in "${Patterns[@]}"
do
[[ ${LOG} =~ "${P#* }" ]] && intr=${P% *}
done
# not greping for ip because the file to compare only contains all the requests from that ip
done < <(cat ipres )
return $?
}
# varsel is a variable containing an ip address from a file readed
varsel=$(sed -n ${opt}p iptemp.tmp)
grep "$varsel" $loghttp $loghttps | awk '{print$7}' > ipres
if [[ -s "ipres" ]]
then
Patterns=()
while read -a foo
do
Patterns+=("${foo}")
done < /def/definition.conf
while read IP
do
intdet $IP
done < <( echo $varsel )
fi
echo "Detectected $intr"
intdet = intrusion detection , i changed the function name .
the ipres file does not have any ip , only have the requests i grabbed out from all logs from that ip i selected
at some point I introduced you to having read set up an array
Code:
while read -a foo
do
Patterns+=("${foo}")
done < /def/definition.conf
this means that foo is an array with two elements
${foo[0]} and ${foo[1]}
post #51 is probably the one which confused you..
For now just drop the -a and it will work as you expect
grrr
Code:
#while read IP
#do
# intdet $IP
#done < <( echo $varsel )
# UUOE !!
intdet $varsel
# that assumes $varsel is a single ip
# done <<<${varsel}
# is an alternative
yup .
You`re good in this .
Also there was a problem here , i had to add one more line to php exploitation , i realized that in all 113 lines used by the intruser , not of then were similar to the ones i got in definitions .
I assume iptemp.tmp is just a list of ip address, and you are looping though each line number and using sed to grab that line
that was one of the first things we did
Code:
while read varsel
do
awk '/'$varsel'/ {print $7}' $loghttp $loghttps > ipres
#FIXME -----------------------------------------^ ^^^^^
# that will overwrite existing file, is that what you want?
done < iptemp.tmp
Code is not definitive , those lines are already changed now , because last code got all ips even those who did not make any bad request to server , and i had to change that code .
Why this code :
Code:
Patterns=()
while read foo
do
Patterns+=("${foo}")
done < /def/definition.conf
while read IP
do
intdet $varsel
done <<< ${varsel}
can only read 24 lines in definition.conf if i have 25 in there ?
I moved the line to the beginning of definition.conf and now it can be identified .
But 25 is now a new one and is not in patterns , it looks it can only read 24 ?
Code is not definitive , those lines are already changed now , because last code got all ips even those who did not make any bad request to server , and i had to change that code .
I don't know what that means
Quote:
Originally Posted by pedropt
Why this code :
Code:
Patterns=()
while read foo
do
Patterns+=("${foo}")
done < /def/definition.conf
while read IP
do
intdet $varsel
done <<< ${varsel}
this is getting very confusing
you are copy and pasting chunks of code and getting everything mixed up
Code:
Patterns=()
while read foo
do
Patterns+=("${foo}")
done < /def/definition.conf
# the above looks ok
# the below is not
while read IP
do
intdet $varsel
done <<< ${varsel}
varsel is what? a single IP ?
if so you don't need to have the loop
if it is a a number of IPs then, strange things may happen
read will probably put them all into IP, basicly IP is a copy of varsel
Code:
varsel="11.11.11.11 22.22.22.22 33.33.33.33"
while read IP
do
echo $IP
done <<< $varsel
for IP in $varsel
do
echo $IP
done
you need to understand all these bits of code you are mashing together
build up the script slowly, testing each bit does what you expect
avoid using all the "clever" sed cat echo grep | awk tricks you have been using
they are bad, at best just slow at worst will do odd things
Quote:
Originally Posted by pedropt
can only read 24 lines in definition.conf if i have 25 in there ?
I'm not sure I understand what you are asking
the definition.conf file is processed line by line
it doesn't finish until the whole of the file is read
be it 24, 25 or 9001 lines
Code is still changing , not all of it but some , depending on the problems i find ahead .
Yup , as you said it should read even if they were 9000 lines , but the fact is that is only reading 24 .
Stuff i did until now , and i still have to search the ip in firewall blocked ips and other stuff ahead , but since i found this problem then i stopped here .
Anyway here is the drop of that new function i am building .
Code:
# i tought it was important to drop these 2 new lines at the beginning to undestand the whole #logic of the code
loghttps="/var/log/apache2/roundcube-access.*"
loghttp="/var/log/nginx/access.*"
intdet () {
IP=$1
while read LOG
do
for P in "${Patterns[@]}"
do
[[ ${LOG} =~ "${P#* }" ]] && intr=${P% *}
done
done < ipres
return $?
}
srvlog() {
delf
echo -e "$green""Select one of the options"
echo ""
echo -e "$yellow""1$orange -$green Check Webservers logs"
echo -e "$yellow""2$orange -$green Check Mail logs"
echo -e "$yellow""3$orange -$green Check Streaming logs"
echo -e "$yellow""4$orange -$green Back To Menu"
echo ""
echo -ne "$green""Choose : "
read opt
case $opt in
1)
rm -rf /var/log/nginx/*.gz >/dev/null 2>&1
rm -rf /var/log/apache2/*.gz >/dev/null 2>&1
rm -rf ipres >/dev/null 2>&1
shd=$(netstat -anp | grep "ESTABLISHED" | grep "sshd" | awk '{print$5}' | cut -f1 -d":")
rm -rf iptemp.tmp >/dev/null 2>&1
echo -e "$green"""
echo -n "* - Collecting Webservers ip logs...."
grep "404" $loghttp | cut -d ':' -f2- | awk '{print$1}' > iptemp.tmp
grep "404" $loghttps | cut -d ':' -f2- | awk '{print$1}' >> iptemp.tmp
if [[ -z "$shd" ]]
then
cat iptemp.tmp | sort -u > iptemp2.tmp
else
cat iptemp.tmp | sort -u | sed -e "/$shd/d" > iptemp2.tmp
fi
rm -rf iptemp.tmp && mv iptemp2.tmp iptemp.tmp >/dev/null 2>&1
echo "Done"
cntfl=$(wc -l iptemp.tmp | awk '{print$1}')
echo ""
echo "Choose an ip to check its activity"
echo ""
for i in $(seq $cntfl)
do
var=$(sed -n ${i}p iptemp.tmp)
cnt=$(grep "$var" $loghttp $loghttps | wc -l | awk '{print$1}')
if [[ "$cnt" -le "1" ]]
then
cnt="1"
fi
if [[ $var =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
# temp file because i want to put the ips that hited more times in the # beginning , but i am still thinkinking in adding it , not sure
echo "$cnt $var" >> newip
# END
echo -e "$green""$i$white -$cyan $var $yellow ($orange$cnt$green times $yellow)"
fi
done
echo -ne "$green""Choose from 1 to $cntfl : "
read -r opt
# all these ifs are in case i am drunk or with my ahead in the moon
if [[ -z "$opt" ]]
then
echo "Invalid Input"
sleep 3
clear
srvlog
elif [[ "$opt" -lt "1" ]]
then
echo "Invalid Input"
sleep 3
clear
srvlog
elif [[ "$opt" -gt "$cntfl" ]]
then
echo "Invalid Input"
sleep 3
clear
srvlog
elif [[ "$opt" =~ [A-Za-z] ]]
then
echo "Invalid Input"
sleep 3
clear
srvlog
fi
# end of drunk part
varsel=$(sed -n ${opt}p iptemp.tmp)
grep "$varsel" $loghttp $loghttps | awk '{print$7}' > ipres
if [[ -s "ipres" ]]
then
Patterns=()
while read foo
do
Patterns+=("${foo}")
done < /def/definition.conf
while read IP
do
intdet $varsel
done <<< ${varsel}
fi
if [[ -z "$intr" ]]
then
echo "Database did not identified $varsel requests to server"
echo ""
cntfl=$(wc -l ipres | awk '{print$1}')
echo "....Displaying webserver logs for this IP...."
echo "---------------------------------------------"
for i in $(seq $cntfl)
do
rdinp=$(sed -n ${i}p ipres)
echo "$i - $rdinp"
done
echo "---------------------------------------------"
echo ""
echo "Do You want to add a new reference to Definition database?"
echo -ne "$green""Choose (y/n) : "
read -r opt
if [[ "$opt" == "y" ]] || [[ "$opt" == "Y" ]] || [[ "$opt" == "yes" ]] || [[ "$opt" == "Yes" ]] || [[ "$opt" == "YES" ]]
then
echo ""
echo -n "Choose a number between 1 & $cntfl to add :"
read -r opt1
echo -n "Write a name to identify this intrusion : "
read -r opt2
fi
if [[ "$opt" == "n" ]] || [[ "$opt" == "N" ]] || [[ "$opt" == "no" ]] || [[ "$opt" == "NO" ]] || [[ "$opt" == "No" ]]
then
echo "you choose no"
fi
else
echo "It was detected $intr by $varsel"
fi
;;
2)
echo ""
;;
3)
echo ""
;;
4)
clear
menu
;;
*)
clear
srvlog
;;
esac
}
Anyway , it should be simple for you to test the 25 line issue .
just write 24 lines with anything like "aaaa vhfjtrorooo" not equal to line 24 , and then in line 25 Write :
to the directory where you are going to place this access file .
you also must edit the loghttps variable , change it to same location of loghttp .
then
at the end of last code write after the end of the function srvlog
and of course write
Quote:
#!/bin/bash
at the beginning of file .
after execution permissions and deffile in /def/definition.conf , everything will work fine with a minor issue of the colour code error in the bash script , but to avoid that just remove the -e switches from echo and the variables with the colour codes .
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
, i just wrote it in a rush , the question i asked before was because the logfile related to a specific ip will change a bit because instead pointing to a single server file , i will point to a directory and look for ip on all files there .
