How to cut and overwrite a certain section from a file BASH
Content in dl.txt: "127. www.example.com"
I have tried:
#cat dl.txt|egrep -v "^[0-9]+.[ ]" > dl.txt
#cat dl.txt|egrep "www.example.com" > dl.txt
Could this maybe be done in awk ?
If you mean you want to alter the contents of the file dl.txt and delete "127. ", you can use sed:
sed -i.bak 's/127. //' dl.txt
Then you will see that dl.txt is changed and dl.txt.bak is a backup copy.
Likewise if you want to remove the "www.example.com"
sed -i.bak 's/www.example.com//' dl.txt
Or if you want to delete everything up to, and including the space on each line:
sed -i.bak 's/.* //' dl.txt
Or using awk:
awk '{print $1}' dl.txt
127.
awk '{print $2}' dl.txt
www.example.com
Or you can do it nearly in-place with awk like this, only overwriting the original if the awk is successful:
awk '{print $2}' dl.txt > $$.tmp && mv $$.tmp dl.txt
Below should get rid of 127. before the url, or any ip address for that matter:
sed -i 's/[0-9]\+\.\?\s*//g' dl.txt
Related
Input file
hello how are u
some what doing fine
so
thats all
huh
thats great
cool
gotcha im fine
I wanted to remove last 4 lines without re directing to another file or say in place edit.
I used head -n -3 input.txt but its removing only the last 2 lines.
Also wanted to understand is it possible to pipe head's output to sed
like head -n -3 input.txt | sed ...
Yes, I went thru sed's option to remove last n lines like below but couldn't understand the nuances of the command so went ahead with the alternative of head command
sed -e :a -e '$d;N;2,5ba' -e 'P;D' file
EDIT: Without creating a temp file solution:
awk -i inplace -v lines=$(wc -l < Input_file) 'FNR<=(lines-4)' Input_file
Could you please try following and let me know if this helps you.
tac Input_file | tail -n +5 | tac > temp_file && mv temp_file Input_file
Solution 2nd: Using awk.
awk -v lines=$(wc -l < Input_file) 'FNR<=(lines-4)' Input_file > temp_file && mv temp_file Input_file
Output of awk '{print $4}' is
b05808aa-c6ad-4d30-a334-198ff5726f7c
59996d37-9008-4b3b-ab22-340955cb6019
2b41f358-ff6d-418c-a0d3-ac7151c03b78
7ac4995c-ff2c-4717-a2ac-e6870a5670f0
I need to grep file st.log by these records. Something like awk '{print $4}' |xargs -i grep -w "pattern from awk" st.log I dont know how to pass pattern correctly?
What about
awk '{print $4}' | grep -F -f - st.log
Credits to Eric Renouf, who noticed that -f - can be used for standard input instead -f <(cat), Note: -f /dev/stdin also works and avoids launching a new process.
or closer to the question to have the output ordered
awk '{print $4}' | xargs -i grep -F {} st.log
maybe -w was not the option OP needed but -F
grep --help
-F, --fixed-strings PATTERN is a set of newline-separated strings
-w, --word-regexp force PATTERN to match only whole words
-w will match only line that contain exactly pattern
examples
grep -w . <<<a # matches
grep -w . <<<ab # doesn't match
grep -F . <<<a # doesn't match
grep -F . <<<a.b # matches
May be something along these lines be helpful
How to process each line received as a result of grep command
awk '{print $4} | while read -r line; do
grep $line st.log
done
here is a piece of log
21:36 b05808aa-c6ad-4d30-a334-198ff5726f7c new
22:21 59996d37-9008-4b3b-ab22-340955cb6019 new
21:12 2b41f358-ff6d-418c-a0d3-ac7151c03b78 new
12:36 7ac4995c-ff2c-4717-a2ac-e6870a5670f0 new
i print it by awk '{print $2}' st.log
so i got
b05808aa-c6ad-4d30-a334-198ff5726f7c
59996d37-9008-4b3b-ab22-340955cb6019
2b41f358-ff6d-418c-a0d3-ac7151c03b78
7ac4995c-ff2c-4717-a2ac-e6870a5670f0
now i need to pass it to grep, in this manner
awk '{print $2}' |xargs -i grep -w "pattern from awk" st.log
I need exactly how to pass each founded record from awk to grep. I do not need other solutions, because my task is more complicated, than this piece. Thank you.
With bash and grep:
grep -f <(awk '{print $2}' piece_of_log) st.log
No need for awk:
grep -Ff <(cut -d' ' -f2 log)
It seems you're looking for the replace string option:
-I replace-str
Replace occurrences of replace-str in the initial-arguments with
names read from standard input. Also, unquoted blanks do not
terminate input items; instead the separator is the newline
character. Implies -x and -L 1.
Like this:
awk '{print $2}' | xargs -I{} grep -w {} st.log
I want to split a file into two, but cannot find a way to do this.
Master.txt
Happy Birthday to you! [[#HAPPY]]
Stop it. [[#COMMAND]]
Make a U-turn. [[#COMMAND]]
I want to split into two files, with the 2nd file starting when it matches the regex pattern [[#
Output1.txt
Happy Birthday to you!
Stop it.
Make a U-turn.
Output2.txt
[[#HAPPY]]
[[#COMMAND]]
[[#COMMAND]]
I've tried using awk:
awk -v RS="[[#*" '{ print $0 > "temp" NR }'
but it doesn't give my desired output -- any help would be appreciated!
Here is one way with GNU awk:
awk -v RS='\\[\\[#|\n' 'NR%2{print $0>"Output1.txt";next}{print "[[#"$0>"Output2.txt"}' master
Test:
$ ls
master
$ cat master
Happy Birthday to you! [[#HAPPY]]
Stop it. [[#COMMAND]]
Make a U-turn. [[#COMMAND]]
$ awk -v RS='\\[\\[#|\n' 'NR%2{print $0>"Output1.txt";next}{print "[[#"$0>"Output2.txt"}' master
$ ls
master Output1.txt Output2.txt
$ head Out*
==> Output1.txt <==
Happy Birthday to you!
Stop it.
Make a U-turn.
==> Output2.txt <==
[[#HAPPY]]
[[#COMMAND]]
[[#COMMAND]]
A pure bash solution might be a little slower, but is very readable:
while read line; do
[[ $line =~ (.*)(\[\[#.*]]) ]]
printf "%s" "${BASH_REMATCH[1]}" >&3
printf "%s" "${BASH_REMATCH[2]}" >&4
done 3> output1.txt 4> output2.txt
you can write small script like this…
#!/bin/ksh
sed -i -e 's/ \[\[#/,\[\[#/' $1
cut -d, -f1 $1 > $1.part1
cut -d, -f2 $1 > $1.part2
---------------------------------------------
OR…use multi-command line
# sed -i -e 's/ \[\[#/,\[\[#/' Master.txt ; cut -d, -f1 Master.txt > output1.txt ; cut -d, -f1 Master.txt > output.txt
Simpler in sed, IMHO:
$ sed 's/^\([^[]*\).*/\1/' Master.txt > Output1.txt
$ sed 's/^[^[]*//' Master.txt > Output2.txt
sed -n 's/\[\[#/\
&/;P
/\n/ {s/.*\n//;H;}
$ {x;s/\n//;w Output2.txt
}' YourFile > Output1.txt
in 1 sed but awk is better suite for this task
This might work for you(GNU sed):
sed -n 's/\[\[#/\n&/;P;s/.*\n//w file3' file1 >file2
No need for gnu awk, this should work for any awk
awk -F'\\[\\[#' '{print $1>"Output1.txt";print "[[#"$2>"Output2.txt"}' Master.txt
cat Output1.txt
Happy Birthday to you!
Stop it.
Make a U-turn.
cat Output2.txt
[[#HAPPY]]
[[#COMMAND]]
[[#COMMAND]]
input.txt
1,Ram,Fail
2,John,Fail
3,Ron,Success
param.txt (New Input)
1,Sam,Success
2,John,Sucess
Now i want to replace the whole line in input.txt with those present in param.txt .
1st column will act like a primary key.
Output.txt
1,Sam,Success
2,John,Sucess
3,Ron,Success
I tried as
awk 'FNR==NR{a[$1]=$2 FS $3;next}{ print $0, a[$1]}' input.txt param.txt > Output.txt
But it is merging the file contents.
This might work for you (GNU sed):
sed 's|^\([^,]*,\).*|/^\1/c\\&|' param.txt | sed -f - input.txt
Explanation:
Convert param.txt into a sed script using the first field as an address to change the line in the input.txt. s|^\([^,]*,\).*|/^\1/c\\&|
Run the script against the input.txt. sed -f - input.txt
This can be done with one call to sort:
sort -t, -k1,1n -us param.txt input.txt
Use a stable numerical sort on the first comma-delimited field, and list param.txt before input.txt so that the correct, newer, lines are preferred when eliminating duplicates.
You could use join(1) to make this work:
$ join -t, -a1 -j1 Input.txt param.txt | sed -E 's/,.*?,.*?(,.*?,.*?)/\1/'
1,Sam,Success
2,John,Sucess
3,Ron,Success
sed as a pipe tail strips fields from Input.txt out of replaced lines.
This will work only if both input files are sorted by first field.
Pure awk isn't really the right tool for the job. If you must use only awk, https://stackoverflow.com/a/5467806/1301972 is a good starting point for your efforts.
However, Unix provides some tools that will help with feeding awk the right input for what you're trying to do.
$ join -a1 -t, <(sort -n input.txt) <(sort -n param.txt) |
awk -F, 'NF > 3 {print $1 "," $4 "," $5; next}; {print}'
Basically, you're feeding awk a single file with the lines joined on the keys from input.txt. Then awk can parse out the fields you want for proper display or for redirection to your output file.
This should work in awk
awk -F"," 'NR==FNR{a[$1]=$0;next} ($1 in a){ print a[$1]; next}1' param.txt input.txt
Test:
$ cat input.txt
1,Ram,Fail
2,John,Fail
3,Ron,Success
$ cat param.txt
1,Sam,Success
2,John,Sucess
$ awk -F"," 'NR==FNR{a[$1]=$0;next} ($1 in a){ print a[$1]; next}1' param.txt input.txt
1,Sam,Success
2,John,Sucess
3,Ron,Success