I am having trouble removing new lines from text in a shell script (using sed and grep) - shell

I am using zsh on macOS
I have a shell script that produces a text file with speedtest results in the following layout:
Download: 63.57 Mbps (data used: 69.3 MB)
Upload: 16.11 Mbps (data used: 23.0 MB)
I can manipulate the layout and produce this:
↓ 63.57 Mbps |
↑ 16.11 Mbps
Note the line break before the first line of text and the one after the pipe. In the Terminal only the final line is printed out: ↑ 16.11 Mbps
The script to transform the input is this:
DOWNLOAD=$(cat ~/Terminal_Projects/temp_speedtest_result.txt | grep Download | sed 's/ Download: /↓ /g' | sed 's/ (data used: //g' | sed -E 's/[0-9]{1,4}\.[0-9] MB)//g' | sed 's/\n\r\t//')
UPLOAD=$(cat ~/Terminal_Projects/temp_speedtest_result.txt | grep Upload | sed 's/ Upload: /↑ /g' | sed 's/ (data used: //g' | sed -E 's/[0-9]{1,4}\.[0-9] MB)//g' | tr '\n' ' ')
RESULT=$DOWNLOAD" | "$UPLOAD
echo $RESULT
I used multiple instances of sed because I couldn't get it to work in just one instance. You may know how to get it to work.
What I want to do is output the DOWNLOAD and UPLOAD variables on a single line. I have another very similar script that achieves that with exactly the same manipulation of variables.
What I have tried:
Using RESULT="$DOWNLOAD | $UPLOAD"
Using RESULT="${DOWNLOAD} | ${UPLOAD}"
Using tr '\n' ' ' instead of the sed command to remove \n
I tried removing the up and down arrows in case those symbols aren't supported - same behaviour.
I have tried using sed on the RESULT variable to try removing new lines. I also tried writing the contents of the RESULT variable to a new temp txt file and then retrieving the contents of the file and using grep to extract the results one by one in the hope the new lines would not be copied. Didn't work for me.
It looks like there are line breaks that I have been unable to remove but I could be wrong.
I am new to command line and shell scripts. Trying to apply my very limited knowledge to a new scenario. Any help would be appreciated.

tr seems to do the job and echo -n "$UPLOAD" shows on a single line, so I think you're on the right track and only need to fix the DOWNLOAD part.
I suggest you simplify the script a bit using something along these lines:
INPUT_FILE="~/Terminal_Projects/temp_speedtest_result.txt"
DOWNLOAD="$(grep Download $INPUT_FILE | awk '{print "↓" $2 " " $3}' | tr '\n' ' ')"
UPLOAD="$(grep Upload $INPUT_FILE | awk '{print "↓" $2 " " $3}' | tr '\n' ' ')"
echo "$DOWNLOAD | $UPLOAD"

How about
RESULT="$(grep -Ew '(Down|Up)load' <~/Terminal_Projects/temp_speedtest_result.txt | tr '\n' ' ')"
? This is more efficient (only one grep and tr process needed) and also fixes the bug you have in your solution by your use of a pipe in RESULT=$DOWNLOAD" | "$UPLOAD (which should have brought up an error message).

Related

grep return the string in between words

I am trying to use grep to filter out the RDS snapshot identifier from the rds describe-db-snapshots command output below:
"arn:aws:rds:ap-southeast-1:123456789:snapshot:rds:apple-pie-2018-05-06-17-12",
"rds:apple-pie-2018-05-06-17-12",
how to return the exact output as in
rds:apple-pie-2018-05-06-17-12
tried using
grep -Eo ",rds:"
but not able to
Following awk may also help you on same.
awk 'match($0,/^"rds[^"]*/){print substr($0,RSTART+1,RLENGTH-1)}' Input_file
Your grep -Eo ",rds:" is failing for different reasons:
You did not add a " in the string to match
Between the comma and rds you need to match the character.
You are trying to match the comma that can be on the previous line
Your sample input is 2 lines (with a newline in between), perhaps the real input is without the newline.
You want to match until the next double quote.
You can support both input-styles (with/without newline) with
grep -Eo '(,|^)"rds:[^"]*' rdsfile |cut -d'"' -f2
You can do this in one command with
sed -rn 's/.*(,|^)"(rds:[^"]*).*/\2/p' rdsfile
EDIT: Manipulting stdout and not the file is with similar commands:
yourcommand | grep -Eo '(,|^)"rds:[^"]*' |cut -d'"' -f2
# or
yourcommand | sed -rn 's/.*(,|^)"(rds:[^"]*).*/\2/p'
You can also test the original commands with yourcommand > rdsfile.
You might notice that rdsfile is missing data that you have seen on the screen, in that case add 2>&1
yourcommand 2>&1 | grep -Eo '(,|^)"rds:[^"]*' |cut -d'"' -f2
# or
yourcommand 2>&1 | sed -rn 's/.*(,|^)"(rds:[^"]*).*/\2/p'

Parse file by splitting string in file and get desired output using single command

I'm using bash to look into file and parse the results. Can someone tell me how to use cut/awk to split the string and get desired output by using single command? I can get through individual cut and get the below output (with 2 commands and concatenation) but i want to do using single command instead of two commands.
test.log:
1/98 | (PASSED) com.yahoo.qa.java.projects.stackoverview.questions.Password_01() | 21:20:20
Tried code:
str1=`cat test.log | tail -1 | cut -d '|' -f 1`
str2=`cat test.log | tail -1 | cut -d '|' -f 2 | sed -e 's/com.yahoo.qa.java.projects./''/g'`
str3="${str1} | ${str2}"
Expected:
1/98 | (PASSED) stackoverview.questions.Password_01
Since this is a simple substitution on an individual line it's better suited to sed than awk and not at all appropriate for cut:
$ sed 's/\(.*| [^ ]* \)com\.yahoo\.qa\.java\.projects\.\([^(]*\).*/\1\2/' file
1/98 | (PASSED) stackoverview.questions.Password_01
Following single awk may help you in same.
awk 'END{sub(/com\.yahoo\.qa\.java\.projects\./,"",$4);print $1,$2,$3,$4}' Input_file
OR for all kind of awks following may help you in same too.(As per SIR ED's suggestions):
awk '{value=$0} END{split(value, a," ");sub(/com.yahoo.qa.java.projects\./,"",a[4]);print a[1],a[2],a[3],a[4]}' Input_file
Using awk
$ awk -F "com[.]yahoo[.]qa[.]java[.]projects[.]" 'sub(/\(\).*/,"",$2)' file
1/98 | (PASSED) stackoverview.questions.Password_01

Printing a substring from log

I have a log line of the following format:
2016-08-04 19:12:02,537 INFO ...<Thread-4> - Got a message [......|clientTradeId=xxxxxxx|timeInForce=xxxx|.....TradeResponseMessage]
I would need to extract all line with the 'Got a message' key phrase;and then print out just the 'clientTradeId=xxxxxxx' part of the resulting shortlist.
How do I achieve this with scripting(grep and cut? - or is there a better option)
Considering data is in file data.log
grep -F "Got a message" data.log | grep -Po "clientTradeId=[^| ]+"
using cut
grep -F "Got a message" data.log | cut -f2 -d'|'
UPDATED COMMAND thanks #BenjaminW:
sed -rn 's/.*(clientTradeId=[0-9]*).*/p' file
Haven't used sed, but I have used regex.
Looking at the documentation for sed
cat file | sed '/.*(clientTradeId=[0-9]*).*/\1/'
What this does is pipe the file to sed, then, using regex, select the part that you wanted, then output it (I hope).

How to extract the exact api call from failed stack

I need to process the stack similar to below which has a string "oracle.apps.fnd.applcore.test.selenium.ApplcoreWebdriver" but I need the exact method call that failed. e.g. in this case the exact one is "oracle.apps.fnd.applcore.test.selenium.ApplcoreWebdriver.attachFile". There are multiple stacks that I have as part of html pages which I need to process and there will be multiple methods failing. How can I do it in shell script? This entire stack is in one line html entry which is causing grep to return the whole stack itself. I tried multiple things but nothing worked clearly. I guess awk or similar regex tool could be a way to go but not sure.
at oracle.adf.view.rich.automation.test.selenium.RichWebDriverTest.getElement(RichWebDriverTest.java:1414)
at oracle.apps.fnd.applcore.test.selenium.ApplcoreWebdriver.attachFile(ApplcoreWebdriver.java:1460)
at oracle.apps.fnd.applcore.attachments.ui.util.accessor.FndManageAttachmentsPopupAccessor.attachFile(FndManageAttachmentsPopupAccessor.java:1475)
at oracle.apps.fnd.applcore.attachments.ui.util.accessor.FndManageAttachmentsPopupAccessor.updateRowFileAttachment(FndManageAttachmentsPopupAccessor.java:550)
at oracle.apps.fnd.applcore.attachments.ui.AttachmentsBaseSelenium.testLMultiFileAdd_22108390(AttachmentsBaseSelenium.java:1782)
at oracle.javatools.test.WebDriverRunner.run(WebDriverRunner.java:122)
I cannot guarantee it being the most efficient of solutions, but am able so suit the need. Am using all grep, sed combined together.
Splitting the single big html file as multiple lines using cat & tr
cat file | tr ' ' '\n' | grep -w "$search_string" | sed 's/(.*)//'
For testing purposes am using the stack snippet which you shared in the OP.
$ cat file
at oracle.adf.view.rich.automation.test.selenium.RichWebDriverTest.getElement(RichWebDriverTest.java:1414)
at oracle.apps.fnd.applcore.test.selenium.ApplcoreWebdriver.attachFile(ApplcoreWebdriver.java:1460)
at oracle.apps.fnd.applcore.attachments.ui.util.accessor.FndManageAttachmentsPopupAccessor.attachFile(FndManageAttachmentsPopupAccessor.java:1475)
at oracle.apps.fnd.applcore.attachments.ui.util.accessor.FndManageAttachmentsPopupAccessor.updateRowFileAttachment(FndManageAttachmentsPopupAccessor.java:550)
at oracle.apps.fnd.applcore.attachments.ui.AttachmentsBaseSelenium.testLMultiFileAdd_22108390(AttachmentsBaseSelenium.java:1782)
at oracle.javatools.test.WebDriverRunner.run(WebDriverRunner.java:122)
Running the above command for the search string oracle.apps.fnd.applcore.test.selenium.ApplcoreWebdriver
$ cat file | tr ' ' '\n' | grep -w "oracle.apps.fnd.applcore.test.selenium.ApplcoreWebdriver" | sed 's/(.*)//'
oracle.apps.fnd.applcore.test.selenium.ApplcoreWebdriver.attachFile
Any suggestions simplifying it are welcome.
Well, I think I found a working solution:
for stack in $all_stacks
do
words=`echo $stack | tr ' ' '\n'`
for word in $words
do
search_string="oracle.apps.fnd.applcore.test.selenium.ApplcoreWebdriver"
if [[ $word == *${search_string}* ]];
then
echo $word
fi
done
done
The idea was to split the line (stack) into words separated by space and process each word by matching with the required string. Then the matching word finally prints the required value (including api call)
Update based on ans from #Inian :
Below is updated one command to achieve this by processing multiple html files which contains error stack to be processed.
find . -name '*-errors.html' | xargs cat | tr ' ' '\n' | grep -w "oracle.apps.fnd.applcore.test.selenium.ApplcoreWebdriver" | sed 's/(.*)//' | cut -d '<' -f1 | sort -u

vimdiff files given in a text file

I have a text file files.txt with following entries
"/home/dilawar/a.txt","/home/dilawar/b.txt"
"/home/dilawar/aa.txt","/home/dilawar/bb.txt"
Now I wish to see the diff of files on line 1. I tried the following
head -n 1 files.txt | cut -d, -f 2,3 | sed "s/,/\t/g" | xargs -I files vimdiff files
It is not working. I replaced vimdiff with diff, it did not work either. However this works
head -n 1 files.txt | cut -d, -f 1 | xargs -I file vim file
How to pass file as an argument to diff as two separate file paths rather than a single string?
PS : To make matter worse, I have space in some of file paths.
First take the first line, then recplace the symbols by a space, and feed it to vimdiff via a subshell.
vimdiff $(head -1 files.txt | tr '",' ' ')
The above elegant method will not work with names with a space. The below dirty one will.
awk -F, 'NR==1{print "vimdiff",$1,$2}' files.txt | bash
try this, see if it helps
sed '1{s/,/ /; s/^/diff /;q}' files.txt|sh
I also escaped the whitespace in filepath (first sed command)
head -n 1 files.txt | sed "s/ /\\\\ /g" | sed "s/[\",]/ /g" |xargs vimdiff

Resources