I'm trying to read a file, each line of which is a CVE ID. For each CVE, I want to make a curl to get its severity and then store that result in a new CSV file with the format cve-id,cve-severity.
Below is the script I'm using, which reads the IDs correctly, but doesn't make the curl call correctly. When I run this, it just outputs empty values for each curl call.
I've tried using back ticks instead of the $(), but same result. What am I doing wrong here?
#!/bin/bash
filename="cves.csv"
while read line
do
echo "$line"
cve_result=$(curl -s "https://cve.circl.lu/api/cve/${line}")
echo "$cve_result"
done < $filename
Also tried these variations, all with same (empty) result:
cve_result=$(curl -s "https://cve.circl.lu/api/cve/${line}")
cve_result=`curl -s "https://cve.circl.lu/api/cve/${line}"`
cve_result=$(curl -s "https://cve.circl.lu/api/cve/$line")
cve_result=`curl -s "https://cve.circl.lu/api/cve/$line"`
cve_result=$(curl -s https://cve.circl.lu/api/cve/$line)
cve_result=`curl -s https://cve.circl.lu/api/cve/$line`
Here is a sample of the CSV file:
CVE-2014-0114
CVE-2014-9970
CVE-2015-1832
CVE-2015-2080
CVE-2015-7521
Your code works for me (ie, each curl call pulls down a bunch of data).
If I convert my (linux) file to contain windows/dos line endings (\r\n) then the curl calls don't generate anything.
At this point I'm guessing your input file has windows/dos line endings (you can verify by running head -2 cves.csv | od -c and you should see the sequence \r \n at the end of each line).
Assuming this is your issue then you need to remove the \r characters; a couple options:
dos2unix cves.csv - only have to run once as this will update the file
curl ... ${line//$'\r'/}" - use parameter substitution to strip out the \r
Related
I want to download files with wget in a bash script. The url's look like this:
https://xxx.blob.core.windows.net/somefolder/$somefile.webm?sv=xxx&sp=r..
Problem is the $ doller sign in the url
When I download the file with double quotes I get a 403 because the $ sign is probably interpreted.
wget "https://xxx.blob.core.windows.net/somefolder/$somefile.webm?sv=xxx&sp=r.."
When I single quote the url and download the file everything goes well:
wget 'https://xxx.blob.core.windows.net/somefolder/$somefile.webm?sv=xxx&sp=r..'
But the url should come from a line in a text file. So I read the file and pass the lines as url:
files=($(< file.txt))
# Read through the url.txt file and execute wget command for every filename
while IFS='=| ' read -r param uri; do
for file in "${files[#]}"; do
wget "${file}"
done
done < file.txt
I get the 403 here as well and don't know how to prevent the termimal from interpreting the dollar sign. How can I achieve this?
But the url should come from a line in a text file.
If you have file with 1 URL per line or are able to easily alter your file to hold 1 URL per line, then you might use -i file option, from wget man page
-i file
--input-file=file
Read URLs from a local or external file. If - is specified as file, URLs are read from the standard input. (Use ./- to
read from a file literally named -.)
If this function is used, no URLs need be present on the command line.
If there are URLs both on the command line and in an input file, those
on the command lines will be the first ones to be retrieved. If
--force-html is not specified, then file should consist of a series of URLs, one per line.(...)
So if you have single file, say urls.txt you might use it like so
wget -i urls.txt
and if you have few files you might concat them and shove through standard input like so
cat urls1.txt urls2.txt urls3.txt | wget -i -
If file(s) contain additional data then remember to process them so GNU wget will get only URLs.
I am trying to download PDF files from a list of URLs in a .txt file, with one URL per line. ('urls.txt')
When I use the following command, where the URL I used is an exact copy-paste of the first line of the .txt file:
$ curl http://www.isuresults.com/results/season1617/gpchn2016/gpchn2016_protocol.pdf -o 'test.pdf'
the pdf downloads perfectly. However when I use this command:
xargs -n 1 curl -O < urls.txt
Then I receive a 'curl: (3) URL using bad/illegal format or missing URL' error x times the amount of URLs listed in the .txt file. I have tested many of the URLS individually, and they all seem to download properly.
How can I fix this?
Edit - the first three lines of urls.txt reads as follows:
http://www.isuresults.com/results/season1718/gpf1718/gpf2017_protocol.pdf
http://www.isuresults.com/results/season1718/gpcan2017/gpcan2017_protocol.pdf
http://www.isuresults.com/results/season1718/gprus2017/gprus2017_protocol.pdf
SOLVED: As per the comment below, the issue was that the .txt file was in DOS/Windows format. I converted it using the line:
$ dos2unix urls.txt
and then the files downloaded perfectly using my original line of code. See this thread for more info: Are shell scripts sensitive to encoding and line endings?
Thank you to all who responded!
Try using
xargs -n 1 -t -a urls.txt curl -O
here the -a option reads list from a file rather than standard input
EDIT:
As #GordonDavisson mentioned, it looks like you may have a file with DOS line endings, you can potentially clean these up using sed before passing to xargs
sed 's/\r//' < urls.txt | xargs -n 1 -t curl -O
I'd like to run a unix command in a loop, replacing a variable for each iteration and then store the output into a file.
I'll be grabbing the HTTP headers of a series of URL's using curl -I and then I want each instance outputted to a new line of a file.
I know
I could store the output with | cat or redirect it into a file with >, but how would I run the loop?
I have a file with a list of URL's one per line (or I could comma separate them, alternatively).
You can write:
while IFS= read -r url ; do
curl -I "$url"
done < urls-to-query.txt > retrieved-headers.txt
(using the built-in read command, which reads a line from standard input — in this case redirected from urls-to-query.txt — and saves it to a variable — in this case $url).
Given a list of URLs in a file:
http://url1.com
http://url2.com
You could run
cat inputfile | xargs curl -I >> outputfile
That would read each line of the input file and append the results for each row into the outputfile
I've searched for hours looking for the answer to this question which seems frustratingly simple...
I have a bash script which I've simplified to find the line that's stopping it from working and am left with:
#!/bin/bash
#
sed -i -e "s/<link>/\n/g" /usb/lenny/rss/tmp/rss.tmp
If I run this script, nothing happens to the file rss.tmp - but if I call this exact same sed command from the terminal, it makes all the replacements as expected.
Anyone have any idea what I'm doing wrong here?
Based on the discussion the issue sounds like it is a cygwin shell problem.
The issue is that shell scripts may not have \r\n line terminations - they need \n terminations. Earlier versions of cygwin behaved differently.
The relevant section from a Cygwin FAQ at http://cs.nyu.edu/~yap/prog/cygwin/FAQs.html
Q: Mysterious errors in shell scripts, .bashrc, etc
A: You may get mysterious messages when bash reads
your .bashrc or .bash_profile, such as
"\r command not found"
(or similar). When you get rid of empty lines, the
complaints about "\r" disappears, but probably other
errors remain. What is going on?
The answer may lie in the fact that a text file (also
called ASCII file) can come in two formats:
in DOS format or in UNIX format.
Most editors can automatically detect the formats
and work properly in either format.
In the DOS format, a new line is represented by two characters:
CR (carriage return or ASCII code 13) and LF (line feed or ASCII code 15).
In the UNIX format, a new line is represented by only
one character, LF. When your .bashrc file is read,
bash thinks the extra character is the name of a command,
hence the error message.
In Cygwin or unix, you can convert a file INFILE in DOS format
to a file OUTFILE in Unix format by calling:
> tr -d '\15' OUTFILE
NOTE:
If you now compare the number of characters in INFILE and OUTFILE,
you will see that the latter has lost the correct
number of characters (i.e., the number of lines in INFILE):
> wc INFILE OUTFILE
Try using that instead:
sed -i -e "s/\<link\>/\n/g" /usb/lenny/rss/tmp/rss.tmp
You need to give an output file or the result will be only shown on the screen.
sed -e 's/<link>/\n/g' /usb/lenny/rss/tmp/rss.tmp > /usb/lenny/rss/tmp/output.tmp
to feed a file to the command you use "<", while to make a file u use ">" and sed is used as text formater not editor as far as i know
maybe something like this should work
cat < /usb/lenny/rss/tmp/rss.tmp | sed -i -e "s/<link>/\n/g" > /usb/lenny/rss/tmp/rssedit.tmp
cat gets the file and with sed editing it and ouput goes to rssedit.tmp
than check if rssedit.tmp has what u wanted
if it does and only if it does
next line of the your skript
should be
mv /usb/lenny/rss/tmp/rssedit.tmp /usb/lenny/rss/tmp/rss.tmp
which will replace made 1 with original, with renameing to original
I'm a Java developer and I'm using Ubuntu to develop. The project was created in Windows with Eclipse and it's using the Windows-1252 encoding.
To convert to UTF-8 I've used the recode program:
find Web -iname \*.java | xargs recode CP1252...UTF-8
This command gives this error:
recode: Web/src/br/cits/projeto/geral/presentation/GravacaoMessageHelper.java failed: Ambiguous output in step `CR-LF..data
I've searched about it and get the solution in Bash and Windows, Recode: Ambiguous output in step `data..CR-LF' and it says:
Convert line endings from CR/LF to a
single LF: Edit the file with Vim,
give the command :set ff=unix and save
the file. Recode now should run
without errors.
Nice, but I've many files to remove the CR/LF character from, and I can't open each to do it. Vi doesn't provide any option to command line for Bash operations.
Can sed be used to do this? How?
There should be a program called dos2unix that will fix line endings for you. If it's not already on your Linux box, it should be available via the package manager.
sed cannot match \n because the trailing newline is removed before the line is put into the pattern space, but it can match \r, so you can convert \r\n (DOS) to \n (Unix) by removing \r:
sed -i 's/\r//g' file
Warning: this will change the original file
However, you cannot change from Unix EOL to DOS or old Mac (\r) by this. More readings here:
How can I replace a newline (\n) using sed?
Actually, Vim does allow what you're looking for. Enter Vim, and type the following commands:
:args **/*.java
:argdo set ff=unix | update | next
The first of these commands sets the argument list to every file matching **/*.java, which is all Java files, recursively. The second of these commands does the following to each file in the argument list, in turn:
Sets the line-endings to Unix style (you already know this)
Writes the file out iff it's been changed
Proceeds to the next file
I'll take a little exception to jichao's answer. You can actually do everything he just talked about fairly easily. Instead of looking for a \n, just look for carriage return at the end of the line.
sed -i 's/\r$//' "${FILE_NAME}"
To change from Unix back to DOS, simply look for the last character on the line and add a form feed to it. (I'll add -r to make this easier with grep regular expressions.)
sed -ri 's/(.)$/\1\r/' "${FILE_NAME}"
Theoretically, the file could be changed to Mac style by adding code to the last example that also appends the next line of input to the first line until all lines have been processed. I won't try to make that example here, though.
Warning: -i changes the actual file. If you want a backup to be made, add a string of characters after -i. This will move the existing file to a file with the same name with your characters added to the end.
Update: The Unix to DOS conversion can be simplified and made more efficient by not bothering to look for the last character. This also allows us to not require using -r for it to work:
sed -i 's/$/\r/' "${FILE_NAME}"
The tr command can also do this:
tr -d '\15\32' < winfile.txt > unixfile.txt
and should be available to you.
You'll need to run tr from within a script, since it cannot work with file names. For example, create a file myscript.sh:
#!/bin/bash
for f in `find -iname \*.java`; do
echo "$f"
tr -d '\15\32' < "$f" > "$f.tr"
mv "$f.tr" "$f"
recode CP1252...UTF-8 "$f"
done
Running myscript.sh would process all the java files in the current directory and its subdirectories.
In order to overcome
Ambiguous output in step `CR-LF..data'
the simple solution might be to add the -f flag to force the conversion.
Try the Python script by Bryan Maupin found here (I've modified it a little bit to be more generic):
#!/usr/bin/env python
import sys
input_file_name = sys.argv[1]
output_file_name = sys.argv[2]
input_file = open(input_file_name)
output_file = open(output_file_name, 'w')
line_number = 0
for input_line in input_file:
line_number += 1
try: # first try to decode it using cp1252 (Windows, Western Europe)
output_line = input_line.decode('cp1252').encode('utf8')
except UnicodeDecodeError, error: # if there's an error
sys.stderr.write('ERROR (line %s):\t%s\n' % (line_number, error)) # write to stderr
try: # then if that fails, try to decode using latin1 (ISO 8859-1)
output_line = input_line.decode('latin1').encode('utf8')
except UnicodeDecodeError, error: # if there's an error
sys.stderr.write('ERROR (line %s):\t%s\n' % (line_number, error)) # write to stderr
sys.exit(1) # and just keep going
output_file.write(output_line)
input_file.close()
output_file.close()
You can use that script with
$ ./cp1252_utf8.py file_cp1252.sql file_utf8.sql
Go back to Windows, tell Eclipse to change the encoding to UTF-8, then back to Unix and run d2u on the files.