Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 3 years ago.
Improve this question
Here's my script:
if [[ $(jq '.haystack | index("needle")' /etc/xyz/daemon.json) = \0 ]] ; then
jq '.haystack += ["needle"]' /etc/xyz/daemon.json > daemon.json
mv -f daemon.json /etc/xyz/daemon.json
fi
I want to add needle to haystack array in a daemon.json file. However the problem is in the if construct of the shell. When I test the condition with echo True/False the terminal does show True or False based on the command in the condition. However, I cannot do any other command, such as a simple ls or mkdir. I execute the command from terminal, not saving into a bash file. Newline doesn't seem to need \.
I'm completely new to Linux terminal, is there something I'm missing here? Thanks!
Moving your input to a function to allow testing, and replacing \0 (which has undefined behavior) with 0 (which always expands precisely to itself when given as a literal in code):
get_current_json() {
printf '%s\n' '{"haystack": ["needle", "other1", "other2"]}'
}
if [[ $(jq '.haystack | index("needle")' < <(get_current_json) ) = 0 ]] ; then
echo "Found (at position 0)"
else
echo "Not found (at position 0)"
fi
...correctly emits Found. Thus, the issue cannot be reproduced given only the code provided in the question, without further context.
One potential piece of context: If you're running this in /etc/xyz, then > daemon.json is opening the file for output with the O_TRUNC flag, and thus emptying its contents, before jq begins execution (which can happen only after its output file descriptor is connected), and thus before jq is able to read any prior values which the open(..., O_TRUNC) would delete.
To avoid this, you should use a unique name for your temporary files, as created by mktemp. (Ideally, those names should be in the same directory as the destination file, to guarantee that they don't cross filesystem boundaries and that the final mv will be atomic). See How can I use a file in a command and redirect output to the same file without truncating it?
Related
We were assigned a project that works with various file. The code in question is this:
if [[ -f $first_arg ]]; then
open_file $first_arg
elif [[ -d $first_arg ]]; then
search_directory $first_arg
.....
It works fine with just regular files, but it comes into the second condition if I run the script like this (with the ~/.) :
./script01.sh ~/.config
So I'm wondering what goes on when bash checks -f and -d, what is considered a directory or file and what is not anymore.
~/.config is quite commonly a directory as Joe has suggested in the comments.
As to what goes on, bash apparently calls stat(2) on the file in question which returns a corresponding structure including st_mode field. Details for that are in inode(7):
The stat.st_mode field (for statx(2), the statx.stx_mode field) con‐
tains the file type and mode.
Namely:
The following mask values are defined for the file type:
...
S_IFREG 0100000 regular file
...
S_IFDIR 0040000 directory
All that is left is to check which bits are set.
This question already has answers here:
Bash syntax error: unexpected end of file
(21 answers)
Closed 6 years ago.
I have a VM with CentOS 6 and I am loading some scripts from bashrc.
Everything worked fine, but I wanted to copy-paste the same code and scripts in an older backup of same VM, but I got an error: "unexpected end of file". Also the same error had to deal another person when I wanted to share those scripts with him (he had the same VM).
So I started to debug a little and found that one row he didn't liked it was (it was parsing an array:
COUNTER=1
while [[ ! -z ${SCRIPT[$COUNTER]} ]]; do
Also he didn't liked this either (it's not exactly the same with "while" logic, but it does the job):
for i in ${Script[#]}; do
So, I replaced it with:
for ((i = 0; i < ${#SCRIPT[#]}; i++)); do
Now I tryed to get the error name with same piece of code and no more errors occurred.
Also I have this behavior which is the weirdest from all:
Code:
BASH_SCRIPTS_LOCATION='/mnt/hgfs/Shared-workspace/scripts/'
SCRIPT[0]='aliases.sh'
SCRIPT[1]='scripts_config.sh'
SCRIPT[2]='credentials.sh'
SCRIPT[3]='other_functions.sh'
SCRIPT[4]='ssh_functions.sh'
SCRIPT[5]='release_functions.sh'
SCRIPT[6]='test_functions.sh'
for ((i = 0; i < ${#SCRIPT[#]}; i++)); do
loadedScript=${BASH_SCRIPTS_LOCATION}${SCRIPT[$i]}
echo -e "$loadedScript"
done
Terminal output (seems the "concatenate" it is replacing the characters starting from the begging of first String/variable :
aliases.shShared-workspace/scripts/
scripts_config.shworkspace/scripts/
credentials.shed-workspace/scripts/
other_functions.shorkspace/scripts/
ssh_functions.sh-workspace/scripts/
release_functions.shkspace/scripts/
test_functions.shworkspace/scripts/
I think I am using something very inappropriate. But I am not sure what or what I should be looking for.
Any recommandation or advice is welcome.
Thanks!
It doesn't show here but your script has carriage return chars in the shell definition lines. Edit them out (using Notepad++ for instance or tr -d "\015" < yourscript.sh > newscript.sh)
You can redirect your script to a file you'll see all the text in the file.
Carriage return char (asc 13, \r) just resets the cursor without skipping to newline. Every text written after that overwrites the text in the current line. Windows uses that to complement the linefeed character. Windows text mode is like that
this is my first stackoverflow question, regarding bash scripting. I am a beginner in this language, so be kind with me.
I am trying to write a comparison script. I tried to store all the outputs into variables, but only the last one is stored.
Example code:
me:1234567
you:2345678
us:3456789
My code:
#!bin/bash
while read -r forName forNumber
do
aName="$forName"
echo "$aName"
aNumber="$forNumber"
echo "$aNumber"
done < "exampleCodeFile.txt"
echo "$aNumber"
For the first time, everything will be printed out fine. However, the second echo will only print out "3456789", but not all the numbers again. Same with $aName. This is a problem because i have another file, which i stored a bunch of numbers to compare $aNumber with, using the same method listed above, called $aMatcher, consisting:
aMatcher:
1234567
2345678
3456789
So if i tried to run a comparison:
if [ "$aNumber" == "$aMatcher" ]; then
echo "match found!"
fi
Expected output (with bash -x "scriptname"):
'['1234567 == 1234567']'
echo "match found!"
Actual output (with bash -x "scriptname"):
'['3456789 == 3456789']'
echo "match found!"
Of course my end product would wish to list out all the matches, but i wish to solve my current issue before attempting anything else. Thanks!
When you run your following code
aNumber="$forNumber"
You are over-writing the variable $aNumber for every line of the file exampleCodeFile.txt rather than appending.
If you really want the values to be appended, change the above line to
aNumber="$aNumber $forNumber"
And while matching with $aMatcher, you again have to use a for/while loop to iterate through every value in $aNumber and $aMatcher.
Here's the code snippet from a shell script. (It's from MPFR library's configure script and it starts with #!/bin/sh. The original script is over 17000 lines long.. It's used when building gcc.)
Because I have so many questions in a short piece of code, I have embedded my questions in the code. Please can somebody explain to me why the code is like this? Also, though I have a vague idea, I would appreciate if someone could explain what this code is doing (I understand it will be difficult because it's only a part of a big script).
if { { ac_try="$ac_link"
# <---- question 1 : why is the first curly bracket used for if condition? (probably just for grouping and using the last return code)
# <---- question 2 : Is this second bracket for locally used code(probably)?
case "(($ac_try" in # <---- question 3 : what is this "((" symbol?
*\"* | *\`* | *\\*) ac_try_echo=\$ac_try;;
*) ac_try_echo=$ac_try;;
esac
eval ac_try_echo="\"\$as_me:${as_lineno-$LINENO}: $ac_try_echo\""
$as_echo "$ac_try_echo"; } >&5 # <---- question 4 : what is this >&5 redirection? I know >&{1,2,3} but not 5.
(eval "$ac_link") 2>&5
# <----- question 5 : why use sub-shell here? not to use eval result?
ac_status=$?
$as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
test $ac_status = 0; }; then : # <---- question 6 : is this ':'(nop) here ?
....
some commands
....
else
....
some commands
....
fi
From Bash man page:
{ list; }
list is simply executed in the current shell environment. list
must be terminated with a newline or semicolon. This is known
as a group command. The return status is the exit status of
list. Note that unlike the metacharacters ( and ), { and } are
reserved words and must occur where a reserved word is permitted
to be recognized. Since they do not cause a word break, they
must be separated from list by whitespace or another shell
metacharacter.
{} is just to list a few commands to run, very much like cmd1; cmd2; cmd3. For example, if you write cmd1 ; cmd2 | cmd3, do you mean {cmd1; cmd2;} | cmd3 or cmd1; {cmd2 | cmd3;}.
{{ }} is just nested command list, easy: e.g. {cmd1; cmd2; {cmd3; cmd4;}; }
For question 3, (( is just in a source string to be matched with the following patterns. If you are asking why it is used, we need possible values of $ac_try to analyze why. Honestly, I don't see many shell scripts purposely adding (( in front of a source string to be matched for patterns.
For question 4,
>&5: if file descriptor 5 is not yet created (i.e. mentioned in any part of the script... => be careful, you need to care the scope, some codes runs in sub-shell, which is counted as a sub-shell context/scope), create an unnamed file (well, temp file, if you like), with descriptor 5. This file can be used in other part of the script as an input.
For example, see the part mentioning "exchanges STDIN and STDOUT" in my answer to another question here.
For question 5, the eval, I am not quite sure, just a quick guess (and it depends on what command it evals) by providing you an example why sub-shell makes some differences:
cmd="Foo=1; ls"
(eval $cmd) # this command runs in sub-shell and thus $Foo in current shell will not be changed.
eval $cmd # this command runs in current shell and thus $Foo is changed, and it will affect all subsequent commands.
For question 6, look carefully at the man page I mentioned at top of the answer, the {} list syntax, require a final ;. i.e. {cmd1; cmd2 ; } The last ; is required.
--- UPDATE ---
Question 6: Sorry for not seeing the colon... :-)
It's no op: see this link.
I'm writing shell scripts where quite regularly some stuff is written
to a file, after which an application is executed that reads that file. I find that through our company the network latency differs vastly, so a simple sleep 2 for example will not be robust enough.
I tried to write a (configurable) timeout loop like this:
waitLoop()
{
local timeout=$1
local test="$2"
if ! $test
then
local counter=0
while ! $test && [ $counter -lt $timeout ]
do
sleep 1
((counter++))
done
if ! $test
then
exit 1
fi
fi
}
This works for test="[ -e $somefilename ]". However, testing existence is not enough, I sometimes need to test whether a certain string was written to the file. I tried
test="grep -sq \"^sometext$\" $somefilename", but this did not work. Can someone tell me why?
Are there other, less verbose options to perform such a test?
You can set your test variable this way:
test=$(grep -sq "^sometext$" $somefilename)
The reason your grep isn't working is that quotes are really hard to pass in arguments. You'll need to use eval:
if ! eval $test
I'd say the way to check for a string in a text file is grep.
What's your exact problem with it?
Also you might adjust your NFS mount parameters, to get rid of the root problem. A sync might also help. See NFS docs.
If you're wanting to use waitLoop in an "if", you might want to change the "exit" to a "return", so the rest of the script can handle the error situation (there's not even a message to the user about what failed before the script dies otherwise).
The other issue is using "$test" to hold a command means you don't get shell expansion when actually executing, just evaluating. So if you say test="grep \"foo\" \"bar baz\"", rather than looking for the three letter string foo in the file with the seven character name bar baz, it'll look for the five char string "foo" in the nine char file "bar baz".
So you can either decide you don't need the shell magic, and set test='grep -sq ^sometext$ somefilename', or you can get the shell to handle the quoting explicitly with something like:
if /bin/sh -c "$test"
then
...
Try using the file modification time to detect when it is written without opening it. Something like
old_mtime=`stat --format="%Z" file`
# Write to file.
new_mtime=$old_mtime
while [[ "$old_mtime" -eq "$new_mtime" ]]; do
sleep 2;
new_mtime=`stat --format="%Z" file`
done
This won't work, however, if multiple processes try to access the file at the same time.
I just had the exact same problem. I used a similar approach to the timeout wait that you include in your OP; however, I also included a file-size check. I reset my timeout timer if the file had increased in size since last it was checked. The files I'm writing can be a few gig, so they take a while to write across NFS.
This may be overkill for your particular case, but I also had my writing process calculate a hash of the file after it was done writing. I used md5, but something like crc32 would work, too. This hash was broadcast from the writer to the (multiple) readers, and the reader waits until a) the file size stops increasing and b) the (freshly computed) hash of the file matches the hash sent by the writer.
We have a similar issue, but for different reasons. We are reading s file, which is sent to an SFTP server. The machine running the script is not the SFTP server.
What I have done is set it up in cron (although a loop with a sleep would work too) to do a cksum of the file. When the old cksum matches the current cksum (the file has not changed for the determined amount of time) we know that the writes are complete, and transfer the file.
Just to be extra safe, we never overwrite a local file before making a backup, and only transfer at all when the remote file has two cksums in a row that match, and that cksum does not match the local file.
If you need code examples, I am sure I can dig them up.
The shell was splitting your predicate into words. Grab it all with $# as in the code below:
#! /bin/bash
waitFor()
{
local tries=$1
shift
local predicate="$#"
while [ $tries -ge 1 ]; do
(( tries-- ))
if $predicate >/dev/null 2>&1; then
return
else
[ $tries -gt 0 ] && sleep 1
fi
done
exit 1
}
pred='[ -e /etc/passwd ]'
waitFor 5 $pred
echo "$pred satisfied"
rm -f /tmp/baz
(sleep 2; echo blahblah >>/tmp/baz) &
(sleep 4; echo hasfoo >>/tmp/baz) &
pred='grep ^hasfoo /tmp/baz'
waitFor 5 $pred
echo "$pred satisfied"
Output:
$ ./waitngo
[ -e /etc/passwd ] satisfied
grep ^hasfoo /tmp/baz satisfied
Too bad the typescript isn't as interesting as watching it in real time.
Ok...this is a bit whacky...
If you have control over the file: you might be able to create a 'named pipe' here.
So (depending on how the writing program works) you can monitor the file in an synchronized fashion.
At its simplest:
Create the named pipe:
mkfifo file.txt
Set up the sync'd receiver:
while :
do
process.sh < file.txt
end
Create a test sender:
echo "Hello There" > file.txt
The 'process.sh' is where your logic goes : this will block until the sender has written its output. In theory the writer program won't need modifiying....
WARNING: if the receiver is not running for some reason, you may end up blocking the sender!
Not sure it fits your requirement here, but might be worth looking into.
Or to avoid synchronized, try 'lsof' ?
http://en.wikipedia.org/wiki/Lsof
Assuming that you only want to read from the file when nothing else is writing to it (ie, the writing process has finished) - you could check whether nothing else has file handle to it ?