So I'm querying to get my gps location.
The shell script command is like this:
curl -s http://whatismycountry.com/ | sed -n 's/.*Coordinates \(.*\)<.*/\1/p'
Then to save the coordinates to a .csv file I write:
curl -s http://whatismycountry.com/ | sed -n 's/.*Coordinates \(.*\)<.*/\1/p' | sed -e 's/.*\[\([^ ]*\) \([^]]*\)\].*/\1,\2/' > cordinates.csv
which gives me a csv file with the co-ordinates.
the image of the .csv file pattern
Now the query is in a infinite loop and the intent is every time it queries it should save the new coordinates to the next block below.
Something like this
How do I write the regex part in the previous command for that in the script?
Thanks for much help. Totally a noob in regex.
This has nothing to do with regex. You should use >> for file append instead of > which rewrites file every time.
So your command will become
curl -s http://whatismycountry.com/ |\
sed -n 's/.*Coordinates \(.*\)<.*/\1/p' |\
sed -e 's/.*\[\([^ ]*\) \([^]]*\)\].*/\1,\2/' >> cordinates.csv
Related
I have these files
NotRequired.txt (having lines which need to be remove)
Need2CleanSED.txt (big file , need to clean)
Need2CleanGRP.txt (big file , need to clean)
content:
more NotRequired.txt
[abc-xyz_pqr-pe2_123]
[lon-abc-tkt_1202]
[wat-7600-1_414]
[indo-pak_isu-5_761]
I am reading above file and want to remove lines from Need2Clean???.txt, trying via SED and GREP but no success.
myFile="NotRequired.txt"
while IFS= read -r HKline
do
sed -i '/$HKline/d' Need2CleanSED.txt
done < "$myFile"
myFile="NotRequired.txt"
while IFS= read -r HKline
do
grep -vE \"$HKline\" Need2CleanGRP.txt > Need2CleanGRP.txt
done < "$myFile"
Looks as if the Variable and characters [] making some problem.
What you're doing is extremely inefficient and error prone. Just do this:
grep -vF -f NotRequired.txt Need2CleanGRP.txt > tmp &&
mv tmp Need2CleanGRP.txt
Thanks to grep -F the above treats each line of NotRequired.txt as a string rather than a regexp so you don't have to worry about escaping RE metachars like [ and you don't need to wrap it in a shell loop - that one command will remove all undesirable lines in one execution of grep.
Never do command file > file btw as the shell might decide to execute the > file first and so empty file before command gets a chance to read it! Always do command file > tmp && mv tmp file instead.
Your assumption is correct. The [...] construct looks for any characters in that set, so you have to preface ("escape") them with \. The easiest way is to do that in your original file:
sed -i -e 's:\[:\\[:' -e 's:\]:\\]:' "${myFile}"
If you don't like that, you can probably put the sed command in where you're directing the file in:
done < replace.txt|sed -e 's:\[:\\[:' -e 's:\]:\\]:'
Finally, you can use sed on each HKline variable:
HKline=$( echo $HKline | sed -e 's:\[:\\[:' -e 's:\]:\\]:' )
try gnu sed:
sed -Ez 's/\n/\|/g;s!\[!\\[!g;s!\]!\\]!g; s!(.*).!/\1/d!' NotRequired.txt| sed -Ef - Need2CleanSED.txt
Two sed process are chained into one by shell pipe
NotRequired.txt is 'slurped' by sed -z all at once and substituted its \n and [ meta-char with | and \[ respectively of which the 2nd process uses it as regex script for the input file, ie. Need2CleanSED.txt. 1st process output;
/\[abc-xyz_pqr-pe2_123\]|\[lon-abc-tkt_1202\]|\[wat-7600-1_414\]|\[indo-pak_isu-5_761\]/d
add -u ie. unbuffered, option to evade from batch process, sort of direct i/o
I'm trying to curl a list of URLs to aggregate the tabular data on them from a set of 7000+ URLs. The URLs are in a .txt file. My goal was to cURL each line and save them to a local folder after which I would grep and parse out the HTML tables.
Unfortunately, because of the format of the URLs in the file, duplicates exist (example.com/State/City.html. When I ran a short while loop, I got back fewer than 5500 files, so there are at least 1500 dupes in the list. As a result, I tried to grep the "/State/City.html" section of the URL and pipe it to sed to remove the / and substitute a hyphen to use with curl -O. cURL was trying to grab
Here's a sample of what I tried:
while read line
do
FILENAME=$(grep -o -E '\/[A-z]+\/[A-z]+\.htm' | sed 's/^\///' | sed 's/\//-/')
curl $line -o '$FILENAME'
done < source-url-file.txt
It feels like I'm missing something fairly straightforward. I've scanned the man page because I worried I had confused -o and -O which I used to do a lot.
When I run the loop in the terminal, the output is:
Warning: Failed to create the file State-City.htm
I think you dont need multitude seds and grep, just 1 sed should suffice
urls=$(echo -e 'example.com/s1/c1.html\nexample.com/s1/c2.html\nexample.com/s1/c1.html')
for u in $urls
do
FN=$(echo "$u" | sed -E 's/^(.*)\/([^\/]+)\/([^\/]+)$/\2-\3/')
if [[ ! -f "$FN" ]]
then
touch "$FN"
echo "$FN"
fi
done
This script should work and also take care of downloading same files multiple files.
just replace the touch command by your curl one
First: you didn't pass the url info to grep.
Second: try this line instead:
FILENAME=$(echo $line | egrep -o '\/[^\/]+\/[^\/]+\.html' | sed 's/^\///' | sed 's/\//-/')
i was trying to add the lines from the text file to the sed command
observered_list.txt
Uncaught SlingException
cannot render resource
IncludeTag Error
Recursive invocation
Reference component error
i need it to be coded like the following
sed '/Uncaught SlingException\|cannot render resource\|IncludeTag Error\|Recursive invocation\|Reference component error/ d'
help me to do this.
I would suggest you create a sed script and delete each pattern consecutively:
while read -r pattern; do
printf "/%s/ d;\n" "$pattern"
done < observered_list.txt >> remove_patterns.sed
# now invoke sed on the file you want to modify
sed -f remove_patterns.sed file_to_clean
Alternatively you could construct the sed command like this:
pattern=
while read -r line; do
pattern=$pattern'\|'$line
done < observered_list.txt
# strip of first and last \|
pattern=${pattern#\\\|}
pattern=${pattern%\\\|}
printf "sed '/%s/ d'\n" "$pattern"
# you still need to invoke the command, it's just printed
You can use grep for that:
grep -vFf /file/with/patterns.txt /file/to/process.txt
Explanation:
-v excludes lines of process.txt which match one of the patterns from output
-F treats patterns in patterns.txt as fixed strings instead of regexes (looks like this is desired here)
-f reads patterns from patterns.txt
Check man grep for further information.
I am trying to extract the value of a parameter from the text file.
Below is my text file with uri_param as parameter.
application.txt
---------------
uri_param=frontier://tenant=stripe;env=qa;svc=new-service#stripe-ftr-qa.stripe.nxz.com:80
command:
--------
egrep ^uri_param application.txt | sed -e 's/.*=//'
I am expecting the strign after first = as output i.e. frontier://tenant=stripe;env=qa;svc=new-service#stripe-ftr-qa.stripe.nxz.com:80, but the output I am getting is new-service#stripe-ftr-qa.stripe.nxz.com:80.
How can I fix this? What I found till now is .* in sed is greedy and it will match the longest string after =.
sed -n '/^uri_param=/ {s///p;q;}' YourFile
extract only the first occurance of the uri_param, remove this uri_param= (replace by nothing) and print it then quit.
OK for small and medium file (a grep is faster enough on a big file like 100 Mb)
sed -r 's/^[_0-9a-zA-Z]+=//g' File_Name
You can try this way
sed -r s'/[^=]+.(.*)/\1/g' File_Name
We can filter it in the grep part:
grep -o "^uri_param=.*:[0-9]\{1,\}" infile|sed -e "s/^uri_param=//"
Or use a more flexible tool like gawk:
gawk 'match($0, "^uri_param=(.*:[0-9]+)", r){print r[1]}' infile
NOTE: If your url doesn't finish allways with the port number the pattern should be adjusted.
Hello I am using bash to create CSV file by extracting data from an html file using grep. The problem is after getting the data then using sed to take out , in it and put a word like My_com it gose a crazy on me. here is my code.
time=$(grep -oP 'data-context-item-time=.*.data-context-item-views' index.html \
| cut -d'"' -f2)
title=$(grep -oP 'data-context-item-title=.*.data-context-item-id' index.html |\
cut -d'"' -f2)
sed "s/,/\My_commoms/g" $title
echo "$user,$views,$time,$title" >> test
I keep getting this error
sed: can't read Flipping: No such file or directory
sed: can't read the: No such file or directory
and so on
any advice on what wrong with my code
You can't use sed on text directly on the command line like that; sed expects a file, so it is reading your text as a file name. Try this for your second to last line:
echo $title | sed 's/,/My_com/g'
that way sed sees the text on a file (stdin in this case). Also note that I've used single quotes in the argument to sed; in this case I don't think it will make any difference, but in general it is good practice to make sure bash doesn't mess with the command at all.
If you don't want to use the echo | sed chain, you might also be able to rewrite it like this:
sed 's/,/My_com/g' <<< "$title"
I think that only works in bash, not dash etc. This is called a 'here-string', and bash passes the stuff on the right of the <<< to the command on its stdin, so you get the same effect.