Within my template(callbacks), there is a line that ends with "IP:" I would like to append to. I tried this command:
cat callbacks | grep "IP:" | cut -d ":" -f 2 | echo $(ping -c2 host.com).
I thought i would be able to echo something at the end, but that didn't work. Could someone please shed some light on what i am doing wrong.
This is what i have so far:
for textfile in $(find . -iname "2013*-malware-callback*.txt")
do cat callbacks | cat - $textfile > tmpfile && mv tmpfile $textfile
done
The following takes the contents of $textfile, finds any occurrence of IP: and appends to it an IP address, and saves the result in tmpfile:
v="1.2.3.4"
cat "$textfile" | sed 's/IP:/IP: '"$v/" >tmpfile
The pipeline can be simplified:
sed 's/IP:/IP: '"$v/" <"$textfile" >tmpfile
Further, if the ultimate goal is to replace $textfile with the modified version, we can use sed's modify-in-place feature:
sed -i.bak 's/IP:/IP: '"$v/" "$textfile"
This modifies $textfile in place and, for safekeeping, leaves a backup copy of the original with extension .bak.
Related
I have several sequences to test to see if they are present in my file and I want to extract them in another file. The sequences start with a unique id that must be kept and end with ">" that I don't want to keep. I did a test but I have a problem with the regular expression
#!/bin/bash
cat data.fsa | grep "Qrob" | wc -l
for gene_id in 'gene1' 'gene2'
do
if cat "data.fsa" |grep $gene_id >/dev/null 2>&1
then
echo "data.fsa" | sed -n "s/.*${gene_id}\(.*\)>.*/\"\1\"/p"
else
continue
fi
done
How do I do this? Thanks for your help
I understand my error thanks to you ! Thank you.
sed -n "/^>$gene_id/,/^>/p" data.fsa >> test.fsa && sed -i '$d' test.fsa
I generate the file directly and I delete with sed -i '$d' test.fsa manually the last selection.
I have a file whose contents are like below.
$ cat test
static2 deploy
TDPlanValidator-Prod
I am trying upload contents from these directories to s3 bucket. The issue is s3 doesnt accept spaces and hence I am getting an error. For this to be done, I am trying to remove space between "static2 deploy". This file will have around 400 entries and some of them will have directories with space in it like "static2 deploy". The script which I have written is not able to do that. The script and the output is below.
for i in `cat test`;do var="$( echo "$i" | tr -d ' ' )"; echo $var;done
static2
deploy
TDPlanValidator-Prod
I have tried sed too but that also doesnt work. I want output as below so that I can push it in s3 bucket
static2deploy
Can someone please help me out here? I have been trying things since yesterday but have been unable to fix it.
You can achieve this by below sed command
echo "static2 deploy" | sed "/static2/s/ //g"
How will it work? sed first search for string static2 and once found it will search for all the spaces in that line and removes them.
So above command will output
static2deploy
But if you try with below:-
echo "static deploy" | sed "/static2/s/ //g"
Output would be
static deploy
So in your case you need to try with below:-
cat test | sed "/static2/s/ //g" > output.txt
Hope this will help.
for i in `cat test` loops over every string in the file and not every line.
This works:
cat test | while read line; do var="$( echo "$line" | tr -d ' ' )"; echo $var; done
or shorter if you only want to print the line:
cat test | while read line; do echo "$line" | tr -d ' '; done
Output:
static2deploy
TDPlanValidator-Prod
I'm trying to pass a xargs string replace into a sed replacement inside of a substitution, here's the non-working code.
CALCINT=$CALCINT$(seq $CALCLINES | xargs -Iz echo $CALCINT' -F "invoiceid'z'="'$(sed -n '/invoiceid'z'/s/.*name="invoiceid'z'"\s\+value="\([^"]\+\).*/\1/p' output.txt))
Everything works up until the sed inside the second substitution. the 'z' should be a number 1-20 based on the $CALCLINES variable. I know it has something to do with not escaping properly for sed but I'm having trouble wrapping my head around how sed wants things escaped in this situation.
Here's the surrounding lines of code:
curl -b mycookiefile -c mycookiefile http://localhost/calcint.php > output.txt
CALCLINES=`grep -o 'class="addinterest"' output.txt | wc -l`
CALCINT=$CALCINT$(seq $CALCLINES | xargs -Iz echo $CALCINT' -F "invoiceid'z'="'$(sed -n '/invoiceid17/s/.*name="invoiceid17"\s\+value="\([^"]\+\).*/\1/p' output.txt))
echo $CALCINT
Output: (What I get now)
-F "invoiceid1=" -F "invoiceid2=" -F "invoiceid3=" -F "invoiceid4=" -F "invoiceid5=" -F "invoiceid6=" -F "invoiceid7=" -F "invoiceid8=" -F "invoiceid9=" -F "invoiceid10=" -F "invoiceid11=" -F "invoiceid12=" -F "invoiceid13=" -F "invoiceid14=" -F "invoiceid15=" -F "invoiceid16=" -F "invoiceid17=" -F "invoiceid18=" -F "invoiceid19=" -F "invoiceid20="
What I'm hoping to see as output is something like this
-F "invoiceid1=2342" -F "invoiceid2=456456" -F "invoiceid3=78987" ...etc etc
-------------------------EDIT-----------------------
FWIW...here's the output.txt and other things I've tried.
for i in $(seq -f "%02g" ${CALCLINES});do
sed -n "/interest$i/s/.*name=\"interest$i\"\s\+value=\"\([^\"]\+\).*/\1/p" output.txt > output2.txt
done
output2.txt contains nothing
Thanks to #janos response for clearing things up but taking a step back makes it clear to me that the root of the issue here is that I'm struggling to get the invoice ids out. It's dynamically generated HTML "....name="invoiceid7" value="556"..." so there isn't anything consistent in those particular tags that I can grep on, which is why I was counting another tag that IS consistent then trying to use a variable sed to basically deduce the tag name then extract the value.
Annd..output.txt https://pastebin.com/ewUaddVi
------UPDATE-----
Working solution
Stuff sed into a loop. Note how I had to use ' to use variables in the sed string. That is well documented elsewhere on here. :)
for i in $(seq ${CALCLINES});do
e="interest"$i`
CALCINT=$CALCINT' -F "'$e'='
CALCINT=$CALCINT$(sed -n '/'$e'/s/.*name="'$e'"\s\+value="\([^"]\+\).*/\1/p' output.txt)'"'
done
Please read through the comments on the solution below, there is a cleaner way of doing this.
Your current approach cannot work, specifically this part:
... | xargs -Iz echo -F "invoiceid'z'="$(sed ...)"
The problem is that the $(sed ...) will not be evaluated for each line in the input during the execution of xargs.
The shell will evaluate this once, before it actually runs xargs.
And you need there dynamic values from your input.
You can make this work by taking a different approach:
Extract the invoice ids. For example, write a grep or sed pipeline that produces as output simply the list of invoice ids
Transform the invoice list to the -F "invoiceidNUM=..." form that you need
For the second step, Awk could be practical. The script could be something like this:
curl -b mycookiefile -c mycookiefile http://localhost/calcint.php > output.txt
args=$(sed ... output.txt | awk '{ print "-F \"invoice" NR "=" $0 "\"" }')
echo $args
For example if the sed step produces 2342, 456456, 78987, then the output will be:
-F "invoice1=2342" -F "invoice2=456456" -F "invoice3=78987"
I'm trying to get the output from a grep and sed pipe to go to the terminal and a text file.
Neither
grep -Filr "string1" * 2>&1 | tee ~/outputfile.txt | sed -i "s|string1|string2|g"
nor
grep -Filr "string1" * | sed -i "s|string1|string2|g" 2>&1 | tee ~/outputfile.txt
work. I get "sed: no input files" going to the terminal so sed is not getting the correct input. I just want to see and write out to a text file which files are modified from the search and replace. I know using find instead of grep would be more efficient since the search wouldn't be done twice, but I'm not sure how to output the file name using find and sed when there is a search hit.
EDIT:
Oops I forgot to include xargs in the code. It should have been:
grep -Filr "string1" * 2>&1 | tee ~/outputfile.txt | xargs sed -i "s|string1|string2|g"
and
grep -Filr "string1" * | xargs sed -i "s|string1|string2|g" 2>&1 | tee ~/outputfile.txt
To be clear, I'm looking for a solution that modifies the matched files with the search and replace, and then outputs the modified files' file names to the terminal and a log file.
The -i option to sed is only useful when sed operates on a file, not on standard input. Drop it, and your first option is correct.
I'd use a loop:
for i in `grep -lr string1 *`; do sed -i . 's/string1/string2/g' $i; echo $i >> ~/outputfile.txt; done
I'd advise against using the 'i' option for grep, because it would match files which the sed command won't actually modify.
You can do the same with find and exec, but that's a dangerous tool.
I almost forgot about this. I eventually went with a for loop in a bash script:
#!/bin/bash
for i in $( grep -Flr "string1" * ); do
sed -i "s|string1|string2|g" $i
echo $i
echo $i >> ~/outputfile.txt
done
I'm using the vertical pipe | as the separator, because I'm replacing URL paths with lots of forward slashes.
Thank you both for your help.
I am working on shell, I want to write one liner which will read the file contents of file A and execute grep command on file B.
for example, suppose there are two file
dataFile.log which have following value
abc
xyz
... and so on
now read abc and grep on searchFile.log like grep abc searchFile.log
I have shell script for the same but want one liner for it
for i in "cat dataFile.log" do grep $i searchFile.log done;
try this:
grep -f dataFile.log searchFile.log
Note that if you want to grep as fixed string, you need -F, if you want to match the text in dataFile.log as regex, use -E or -P
How about the following: it even ignores blank lines and # comments:
while read FILE; do if [[ "$FILE" != [/a-zA-Z0-9]* ]]; do continue; fi; grep -h pattern "$FILE"; done;
Beware: have not compiled this.
You can use grep -f option:
cat dataFile.log | grep -f searchFile.log
Edit
OK, now I understand the problem. You want to use every line from dataFile.log to grep in searchFile.log. I also see you have value1|value2|..., so instead of grep you need egrep.
Try with this:
for i in `cat dataFile.log`
do
egrep "$i" searchFile.log
done
Edit 2
Following chepner suggestion:
egrep -f dataFile.log searchFile.log