I am trying to pass a file into a program for data processing with bash and I am wondering if I have the correct syntax
/home/mumps/CS3150/Script/HW1/textfiles/CardioAndPulmonary.txt | /home/mumps/Medline2012/getDocs.mps > /home/mumps/CS3150/Scripts/HW1/textfiles/Titles.txt
The text files I am sending in are all valid and correctly formatted, but am just getting back a file error from the getDocs.mps (I should note that getDocs does work properly because it was something that my teacher passed out along with the debian vdi and other people aren't having a issue with it.)
getDocs does however call a text file that is located in Medline2012 as well which is where the error is coming from I believe.
Or just use bash redirection throughout without cat.
/home/mumps/Medline2012/getDocs.mps < /home/mumps/CS3150/Script/HW1/textfiles/CardioAndPulmonary.txt > /home/mumps/CS3150/Scripts/HW1/textfiles/Titles.txt
/home/mumps/Medline2012/getDocs.mps < /home/mumps/CS3150/Script/HW1/textfiles/CardioAndPulmonary.txt > /home/mumps/CS3150/Scripts/HW1/textfiles/Titles.txt
or
~/Medline2012/getDocs.mps < ~/CS3150/Script/HW1/textfiles/CardioAndPulmonary.txt > ~/CS3150/Scripts/HW1/textfiles/Titles.txt
or even
< ~/CS3150/Script/HW1/textfiles/CardioAndPulmonary.txt ~/Medline2012/getDocs.mps > ~/CS3150/Scripts/HW1/textfiles/Titles.txt
You either need to cat your .txt file, to pass the contents of it to the script via the pipe,
cat /home/mumps/CS3150/Script/HW1/textfiles/CardioAndPulmonary.txt | /home/mumps/Medline2012/getDocs.mps > output
or, depending on what's in the script, it might need to go as a command line parameter, i.e.
/home/mumps/Medline2012/getDocs.mps /home/mumps/CS3150/Script/HW1/textfiles/CardioAndPulmonary.txt > output
You are trying to execute your data file and feed the results to your script.
try
cat /home/mumps/CS3150/Script/HW1/textfiles/CardioAndPulmonary.txt | /home/mumps/Medline2012/getDocs.mps > /home/mumps/CS3150/Scripts/HW1/textfiles/Titles.txt
If you are still having trouble do a cd to the Medline2012 before you execute getDocs.mps. The reason is because when you access the getDoc.mps it calls to open the osu.medline database. This will cause a "file error" because the call in getDoc.mps does not include the path to osu.medline.
EDIT: A lot of people are telling you that you need to "cat" which is wrong. getDoc.mps has its own printing. If it didn't it wouldn't be printing "file error" for you. I also saw that you said that it is breaking after the loop. Did you test to make sure it isn't at the opening of the file. You can check by adding and indicating word in between the quotes in the first printing of "file error". You could change it to something like "file error 1." I realize you probably know that I just like to be thorough.
Related
I am trying to store the output of total count to a variable and then print the variable.
for i in `cat /home/retailer.sh`;do file = "$(grep $i /home/DD.inventory.loaded_master_list.txt|sort -u|wc -l)";echo $file;done
/home/retailer.sh
HAG
GAT
/home/DD.inventory.loaded_master_list.txt
/dailydata_hershey/prd/work/all/2109/delivery/POSDATA_FINAL.GAT.20200201.dat
/dailydata_hershey/prd/work/all/2109/delivery/POSDATA_FINAL.HAG.20200201.dat
Below error i get
=: cannot open = (No such file or directory)
file = "$(...)" - you have to use file="$(...)" (no spaces).
Otherwise it will be interpreted as a command. (e.g. file with an argument of =)
For future reference, shellcheck.net is a great resource for catching these kinds of things. Just copy-paste your script into the box and it will tell you all of the errors, how to fix them, and a handful of other warnings about potential bugs.
I have a program stored in programfile in which I want to pass command line arguments (with the contents of the file of varargs). I also want to take input on stdin from the contents of file p. I then want to store the final output into variable output.
This is what I have:
"$programfile" "${varargs}" < "${p}" > "$output"
I'm not sure if this is correct or not as I think my syntax is off somewhere?
Looks fine to me, as long as you meant that you are storing the final output into a file whose name is in the variable output. If you wanted to put the output into a variable you should use backticks or $().
As you have it, your output would go to a file named after the value of $output, not the variable itself. You could do something like:
output=$("$programfile" "${varargs}" < "${p}")
The redirector > is usually used to redirect the output to a file or device. For example,
ls > list.txt
But to store the result as a variable you will need to do:
result=`ls`
The usage of < is correct.
I do not know much bash scripting, but I know the task I would like to do would be greatly simplified by it. I would like to test a program against expected output using many test input files.
For example, I have files named "input1.txt, input2.txt, input3.text..." and expected output in files "output1.txt, output2.txt, output3.txt...". I would like to run my program with each of the input files and output a corresponding "test1.txt, test2.txt, test3.txt...". Then I would do a "cmp output1.txt test1.txt" for each file.
So I think it would start like this.. roughly..
for i in input*;
do
./myprog.py < "$i" > someoutputthing;
done
One question I have is: how would I match the numbers in the filename? Thanks for your help.
If the input file name pattern is inputX.txt, you need to remove input from the beginning. You do not have to remove the extension, as you want to use the same for output:
output=output${i#input}
See Parameter Expansion in man bash.
I would need to read certain data using curl. I'm basically reading keywords from file
while read line
do
curl 'https://gdata.youtube.com/feeds/api/users/'"${line}"'/subscriptions?v=2&alt=json' \
> '/home/user/archive/'"$line"
done < textfile.txt
Anyway I haven't found a way to form the url to curl so it would work. I've tried like every possible single and double quoted versions. I've tried basically:
'...'"$line"'...'
"..."${line}"..."
'...'$line'...'
and so on.. Just name it and I'm pretty sure that I've tried it.
When I'm printing out the URL in the best case it will be formed as:
/subscriptions?v=2&alt=jsoneeds/api/users/KEYWORD FROM FILE
or something similar. If you know what could be the cause of this I would appreciate the information. Thanks!
It's not a quoting issue. The problem is that your keyword file is in DOS format -- that is, each line ends with carriage return & linefeed (\r\n) rather than just linefeed (\n). The carriage return is getting read into the line variable, and included in the URL. The giveaway is that when you echo it, it appears to print:
/subscriptions?v=2&alt=jsoneeds/api/users/KEYWORD FROM FILE"
but it's really printing:
https://gdata.youtube.com/feeds/api/users/KEYWORD FROM FILE
/subscriptions?v=2&alt=json
...with just a carriage return between them, so the second overwrites the first.
So what can you do about it? Here's a fairly easy way to trim the cr at the end of the line:
cr=$'\r'
while read line
do
line="${line%$cr}"
curl "https://gdata.youtube.com/feeds/api/users/${line}/subscriptions?v=2&alt=json" \
> "/home/user/archive/$line"
done < textfile.txt
Your current version should work, I think. More elegant is to use a single pair of double quotes around the whole URL with the variable in ${}:
"https://gdata.youtube.com/feeds/api/users/${line}/subscriptions?v=2&alt=json"
Just use it like this, should be sufficient enough:
curl "https://gdata.youtube.com/feeds/api/users/${line}/subscriptions?v=2&alt=json" > "/home/user/archive/${line}"
If your shell gives you issues with & just put \&, but it works fine for me without it.
If the data from the file can contain spaces and you have no objection to spaces in the file name in the /home/user/archive directory, then what you've got should be OK.
Given the contents of the rest of the URL, you could even just write:
while read line
do
curl "https://gdata.youtube.com/feeds/api/users/${line}/subscriptions?v=2&alt=json" \
> "/home/user/archive/${line}"
done < textfile.txt
where strictly the ${line} could be just $line in both places. This works because the strings are fixed and don't contain shell metacharacters.
Since you're code is close to this, but you claim that you're seeing the keywords from the file in the wrong place, maybe a little rewriting for ease of debugging is in order:
while read line
do
url="https://gdata.youtube.com/feeds/api/users/${line}/subscriptions?v=2&alt=json"
file="/home/user/archive/${line}"
curl "$url" > "$file"
done < textfile.txt
Since the strings may end up containing spaces, it seems (do you need to expand spaces to + in the URL?), the quotes around the variables are strongly recommended. You can now run the script with sh -x (or add a line set -x to the script) and see what the shell thinks it is doing as it is doing it.
I have to ls command to get the details of certain types of files. The file name has a specific format. The first two words followed by the date on which the file was generated
e.g.:
Report_execution_032916.pdf
Report_execution_033016.pdf
Word summary can also come in place of report.
e.g.:
Summary_execution_032916.pdf
Hence in my shell script I put these line of codes
DATE=`date +%m%d%y`
Model=Report
file=`ls ${Model}_execution_*${DATE}_*.pdf`
But the value of Model always gets resolved to 'REPORT' and hence I get:
ls: cannot access REPORT_execution_*032916_*.pdf: No such file or directory
I am stuck at how the resolution of Model is happening here.
I can't reproduce the exact code here. Hence I have changed some variable names. Initially I had used the variable name type instead of Model. But Model is the on which I use in my actual code
You've changed your script to use Model=Report and ${Model} and you've said you have typeset -u Model in your script. The -u option to the typeset command (instead of declare — they're synonyms) means "convert the strings assigned to all upper-case".
-u When the variable is assigned a value, all lower-case characters are converted to upper-case. The lower-case attribute is disabled.
That explains the upper-case REPORT in the variable expansion. You can demonstrate by writing:
Model=Report
echo "Model=[${Model}]"
It would echo Model=[REPORT] because of the typeset -u Model.
Don't use the -u option if you don't want it.
You should probably fix your glob expression too:
file=$(ls ${Model}_execution_*${DATE}*.pdf)
Using $(…) instead of backticks is generally a good idea.
And, as a general point, learn how to Debug a Bash Script and always provide an MCVE (How to create a Minimal, Complete, and Verifiable Example?) so that we can see what your problem is more easily.
Some things to look at:
type is usually a reserved word, though it won't break your script, I suggest you to change that variable name to something else.
You are missing an $ before {DATE}, and you have an extra _ after it. If the date is the last part of the name, then there's no point in having an * at the end either. The file definition should be:
file=`ls ${type}_execution_*${DATE}.pdf`
Try debugging your code by parts: instead of doing an ls, do an echo of each variable, see what comes out, and trace the problem back to its origin.
As #DevSolar pointed out you may have problems parsing the output of ls.
As a workaround
ls | grep `date +%m%d%y` | grep "_execution_" | grep -E 'Report|Summary'
filters the ls output afterwards.
touch 'Summary_execution_032916.pdf'
DATE=`date +%m%d%y`
Model=Summary
file=`ls ${Model}_execution_*${DATE}*.pdf`
worked just fine on
GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu)
Part of question:
But the value of Model always gets resolved to 'REPORT' and hence I get:
This is due to the fact that in your script you have exported Model=Report
Part of question:
ls: cannot access REPORT_execution_*032916_*.pdf: No such file or directory
No such file our directory issue is due to the additional "_" and additional "*"s that you have put in your 3rd line.
Remove it and the error will be gone. Though, Model will still resolve to Report
Original 3rd line :
file=`ls ${Model}_execution_*${DATE}_*.pdf`
Change it to
file=`ls ${Model}_execution_${DATE}.pdf`
Above change will resolve the could not found issue.
Part of question
I am stuck at how the resolution of Model is happening here.
I am not sure what you are trying to achieve, but if you are trying to populate the file parameter with file name with anything_exection_someDate.pdf, then you can write your script as
DATE=`date +%m%d%y`
file=`ls *_execution_${DATE}.pdf`
If you echo the value of file you will get
Report_execution_032916.pdf Summary_execution_032916.pdf
as the answer
There were some other scripts which were invoked before the control reaches the line of codes which I mentioned in the question. In one such script there is a code
typeset -u Model
This sets the value of the variable model always to uppercase which was the reason this error was thrown
ls: cannot access REPORT_execution_032916_.pdf: No such file or directory
I am sorry that
i couldn't provide a minimal,complete and verifiable code