I'm trying to create a Bask script on Ubuntu server that should check if files where changed on a remote SVN repository. If so it should perform some tasks.
In my case run doxygen and regenerate the documentation.
I'm having trouble interpreting the svn status response.
I've been looking all week and tried several suggestion found but it is still not working. To be precise the tasks are always run no matter if their are changed files.
Here's my script so far:
#!/bin/bash
# Get the status:
svnstatus=$(svn status -u -q)
#Check for modified or new files:
#This part is not working correctly
added=$(printf "%s" "$svnstatus" | sed -n 's/[Added] *\(.*\)/\1/p')
modified=$(printf "%s" "$svnstatus" | sed -n 's/[Updated] *\(.*\)/\1/p')
if [ "x$added" != "x" -o "x$modified" != "x" ]
then
svn update
echo regenerated documentation using doxygen
fi
The main issue is with the sed part. I'm new to Ubuntu and that part is difficult for me. I copied that from a script I found on the Internet.
$svnstatus is
Status against revision: 75016
when nothing new and
* 75016 Resources/graphs/dot_tags.txt
Status against revision: 75017
when a file is changed.
Any advice is much appreciated
That sed command doesn't match the output svn status -u -q so it won't be doing anything useful.
Instead, you can use grep to check if there are any lines in the output with a * character in the 9th column, indicating a modified file.
This should work:
#!/bin/bash
# Get the status:
svnstatus=$(svn status -u -q)
# Check for modified or new files:
if echo "$svnstatus" | grep -q '^........\*'
then
svn update
echo regenerated documentation using doxygen
fi
Related
This question already has answers here:
How do I set a variable to the output of a command in Bash?
(15 answers)
Closed 5 months ago.
This is probably a simple one for a bash scripter, which I am not.
I'm running a cron job that downloads some data, and then depending on that data, may or may not modify a second file. After the job, I want to git commit one or both files. For the conditional commit, I tried this in a .sh script:
# attempt to capture whether MyNotes.txt was changed
# by counting lines in git status output
mywc=(git status -s MyNotes.txt | wc -l)
echo $mywc found!
if [ $mywc = 1 ]; then
echo Add file for commit
else
echo Nothing to add
fi
I'm pretty much getting nowhere; this thing seems to fail on the first line with syntax error near unexpected token '|'. If I run git status -s MyNotes.txt | wc -l on the command line, I get the numeric output I expect.
What am I doing wrong and how can I make this work?
If there's a more elegant way to determine whether a file changed, feel free to share.
Also, for my edification, how could I get this to work without the interim mywc variable? I.e., if I wanted to just do the command within the if, something like this:
if [[ $(git status -s MyNotes.txt | wc -l) = 1 ]]; then
...
Thanks!
What am I doing wrong and how can I make this work?
put a dollar before parenthesis.
foo=$(command)
The thing you are using looks like a bash array
declare -a letters=(a b c d)
If there's a more elegant way to determine whether a file changed, feel free to share.
Consider this:
$ git diff -s --exit-code README.md || echo has changed
has changed
$ git checkout README.md
Updated 1 path from the index
$ git diff -s --exit-code README.md || echo has changed
The OR (||) runs if the first command exits with a non-zero code.
Same thing essentially:
$ false || echo false exits with 1
false exits with 1
$ true || echo will not trigger
An aspect of bash that people overlook is that [[, ]], [ and ] are separate commands. They have return codes too. With this knowledge, you can leverage the return codes with if and any other command.
$ if true; then echo yes; else echo no; fi
yes
$ if false; then echo yes; else echo no; fi
no
So for detecting changes in a tracked file:
$ if git diff -s --exit-code README.md; then echo same as in index; else echo changed; fi
same as in index
$ echo 123 >> README.md
$ if git diff -s --exit-code README.md; then echo same as in index; else echo changed; fi
changed
With all of that said...
Just add the file. You don't need to check anything. If it hasn't changed, nothing will happen.
$ echo foo >> myfile
$ git add myfile
$ git commit -m 'maybe changed' myfile
[master b561cc1] maybe changed
1 file changed, 1 insertion(+), 1 deletion(-)
$ git add myfile
$ git commit -m 'maybe changed' myfile
no changes added to commit (use "git add" and/or "git commit -a")
if you need to avoid a non-zero exit code (such as with set -e), just put a || true after the command that you want to ignore the exit status of:
$ cat foo.sh
#!/bin/basho
set -e
echo foo >> myfile
git add myfile
git commit -m 'maybe changed' myfile
git add myfile
git commit -m 'maybe changed' myfile > /dev/null || true
echo no error here. it\'s fine..
false
echo fill never reach this.
Try running that script and see what happens
I search for a way for checking if file changed.
git diff --exit-code -s <path>
Now the bash scripter knows that every command returns a status code which can be checked with $?. In case everything went smoothly, 0 is returned. In that case we get 0 if file is not changed.
Every bash scripter knows too that you can use that with && and || operators (because of lazy evaluation) to write such construct:
git diff --exit-code -s <path> && echo "should add file"
About your edification, what you wrote is perfectly fine!
As CryptoFool pointed out in a comment, I failed to include a $ in my variable assignment. Simple fix in the first line of my script:
mywc=$(git status -s MyNotes.txt | wc -l)
As matt pointed out in a subsequent comment, doing a git add on a file that hasn't changed has no effect. It won't stage the file for commit. So instead of doing conditional logic to determine whether to git add myfile.txt, I'll just blindly execute git add myfile.txt, which will either stage the file if there are changes, or do nothing if there are no changes. Therefore, my entire script can be replaced with one line:
git add MyNotes.txt
I have a step in a Github Actions job:
- name: Check for changes
run: |
diff=$( git diff --name-only 'origin/main' )
changed_files=$( echo $diff | grep -c src/my_folder ) # this fails
# more script lines
# that are unrelated
This fails with Error: Process completed with exit code 1. only if grep finds nothing.
If there are matches in $diff, then this step works as intended. But of course it also needs to work without matches.
I can run this locally or inside a script without a problem, exit code is always 0 (on a Mac).
I fail to understand what the problem is. After some hours of trial and error and research I learned that apparently grep is tricky in Github actions, but I found no hint or proper documentation how I am supposed to solve this exact case.
If I change my failing line to
echo $( echo $diff | grep -c src/my_folder ) # this works and prints out the result
this gets executed without problems.
But how do I get my grep output into my variable even when there are no findings?
According to the doc, by default Github Actions enables set -e to run the step's commands. This is why an intermediate command failure may cause the whole step to fail. To take full control of your step's commands you can do like this:
- name: Check for changes
shell: bash {0}
run: |
diff=$( git diff --name-only 'origin/main' )
changed_files=$( echo $diff | grep -c src/my_folder )
# ...
# ...
Or you can just disable the default set -e at the beginning of the step's script:
- name: Check for changes
run: |
set +x
diff=$( git diff --name-only 'origin/main' )
changed_files=$( echo $diff | grep -c src/my_folder )
# ...
# ...
I would suggest to use the dorny/paths-filter action, like:
- uses: dorny/paths-filter#v2
id: changes
with:
filters: |
src:
- 'src/my_folder/**'
# run only if some file in 'src/my_folder' folder was changed
- if: steps.changes.outputs.src == 'true'
run: ...
Check the output section for details about changes
I don't know what problem grep has in Github actions but you can try something like :
...
changed_files=$( [ ! -e $diff ] && { echo $diff | grep -c src/my_folder } || echo 0 )
...
This way grep wouldn't run if $diff is empty. It would just store 0 in $changed_files
By using bash script, I'm trying to detect whether a file has been created on a directory or not while running commands. Let me illustrate the problem;
#!/bin/bash
# give base directory to watch file changes
WATCH_DIR=./tmp
# get list of files on that directory
FILES_BEFORE= ls $WATCH_DIR
# actually a command is running here but lets assume I've created a new file there.
echo >$WATCH_DIR/filename
# and I'm getting new list of files.
FILES_AFTER= ls $WATCH_DIR
# detect changes and if any changes has been occurred exit the program.
After that I've just tried to compare these FILES_BEFORE and FILES_AFTER however couldn't accomplish that. I've tried;
comm -23 <($FILES_AFTER |sort) <($FILES_BEFORE|sort)
diff $FILES_AFTER $FILES_BEFORE > /dev/null 2>&1
cat $FILES_AFTER $FILES_BEFORE | sort | uniq -u
None of them gave me a result to understand there is a change or not. What I need is detecting the change and exiting the program if any. I am not really good at this bash script, searched a lot on the internet however couldn't find what I need. Any help will be appreciated. Thanks.
Thanks to informative comments, I've just realized that I've missed the basics of bash script but finally made that work. I'll leave my solution here as an answer for those who struggle like me.:
WATCH_DIR=./tmp
FILES_BEFORE=$(ls $WATCH_DIR)
echo >$WATCH_DIR/filename
FILES_AFTER=$(ls $WATCH_DIR)
if diff <(echo "$FILES_AFTER") <(echo "$FILES_BEFORE")
then
echo "No changes"
else
echo "Changes"
fi
It outputs "Changes" on the first run and "No Changes" for the other unless you delete the newly added documents.
I'm trying to interpret your script (which contains some errors) into an understanding of your requirements.
I think the simplest way is simply to rediect the ls command outputto named files then diff those files:
#!/bin/bash
# give base directory to watch file changes
WATCH_DIR=./tmp
# get list of files on that directory
ls $WATCH_DIR > /tmp/watch_dir.before
# actually a command is running here but lets assume I've created a new file there.
echo >$WATCH_DIR/filename
# and I'm getting new list of files.
ls $WATCH_DIR > /tmp/watch_dir.after
# detect changes and if any changes has been occurred exit the program.
diff -c /tmp/watch_dir.after /tmp/watch_dir.before
If the any files are modified by the 'commands', i.e. the files exists in the 'before' list, but might change, the above will not show that as a difference.
In this case you might be better off using a 'marker' file created to mark the instance the monitoring started, then use the find command to list any newer/modified files since the market file. Something like this:
#!/bin/bash
# give base directory to watch file changes
WATCH_DIR=./tmp
# get list of files on that directory
ls $WATCH_DIR > /tmp/watch_dir.before
# actually a command is running here but lets assume I've created a new file there.
echo >$WATCH_DIR/filename
# and I'm getting new list of files.
find $WATCH_DIR -type f -newer /tmp/watch_dir.before -exec ls -l {} \;
What this won't do is show any files that were deleted, so perhaps a hybrid list could be used.
Here is how I got it to work. It's also setup up so that you can have multiple watched directories with the same script with cron.
for example, if you wanted one to run every minute.
* * * * * /usr/local/bin/watchdir.sh /makepdf
and one every hour.
0 * * * * /user/local/bin/watchdir.sh /incoming
#!/bin/bash
WATCHDIR="$1"
NEWFILESNAME=.newfiles$(basename "$WATCHDIR")
if [ ! -f "$WATCHDIR"/.oldfiles ]
then
ls -A "$WATCHDIR" > "$WATCHDIR"/.oldfiles
fi
ls -A "$WATCHDIR" > $NEWFILESNAME
DIRDIFF=$(diff "$WATCHDIR"/.oldfiles $NEWFILESNAME | cut -f 2 -d "")
for file in $DIRDIFF
do
if [ -e "$WATCHDIR"/$file ];then
#do what you want to the file(s) here
echo $file
fi
done
rm $NEWFILESNAME
I'm trying to grep for a version number from my subversion command, so that I can check we have the write subversion module loaded in a bash script.
Interactively, this is an example use:
> svn --version | head -n1 | grep -q '1.7'; echo $?
0
However, when I put this same line (and nothing else) in a script and run the script:
> ./setup_svn.sh
1
Also, the script runs noticeably faster than the interactive shell command. Does anyone have ideas of what I might be missing that explains this result?
Edit
It turns out that my interactive bash script was using the wrong svn command. Not sure exactly why, but I think that might be a question for the Unix StackExchange.
It's almost certainly to do with the module system on our workstations, running interactively I get:
> module list
Currently Loaded Modulefiles:
...
12) subversion/1.7.7
> which svn
/usr/bin/svn
> svn --version
svn, version 1.7.7 (r1393599)
...
Running in the script, I get:
> ./setup_svn.sh
Currently Loaded Modulefiles:
...
12) subversion/1.7.7
/usr/bin/svn
svn, version 1.6.17 (r1128011)
...
Further edit
It seems that if I start a new shell, I also get the same issues:
> bash
> module list
Currently Loaded Modulefiles:
...
12) subversion/1.7.7
> svn --version
svn, version 1.6.17 (r1128011)
I think I'll find out what our module system does to the environment and use that to work out what's going wrong.
I know it will not correct the issue, but I think the head command is useless. You can use the --quiet option.
$ svn --version --quiet
1.7.5
EDIT:
If it's a semicolon issue as mentioned by konsolebox, you can also use this syntax :
echo $(svn --version --quiet | grep -q '1.7')$?
It could be how the shell handles the semicolon. Try to place in two lines.
#!/bin/sh
svn --version | head -n1 | grep -q '1.7'
echo "$?"
If you're in bash also, make sure that pipefail is not enabled:
#!/bin/bash
shopt -u -o pipefail
svn --version | head -n1 | grep -q '1.7'
echo "$?"
I don't know if this exists yet, but I'd love to be able to do:
$ cat mygist.js | gh new gist
And have it return the URL (and possibly copy it to the clipboard / open it in the browser).
Seems like GitHub has a simple REST API, including methods for creating Gists. Just for fun:
$ curl -X POST \
--data-binary '{"files": {"file1.txt": {"content": "Hello, SO"}}}' \
https://api.github.com/gists
This successfully created this Gist. I guess it's enough to get you started.
Try this gem: https://github.com/defunkt/gist
Worked for me ^_^
Here's a simple bash script that takes a filename and makes it a gist.
function msg() {
echo -n '{"description":"","public":"false","files":{"file1.txt":{"content":"'
awk '{gsub(/"/,"\\\""); printf "%s\\n",$0}' "$1"
echo '"}}'
}
[ "$#" -ne 1 ] && echo "Syntax: gist.sh filename" && exit 1
[ ! -r "$1" ] && echo "Error: unable to read $1" && exit 2
msg "$1" | curl -v -d '#-' https://api.github.com/gists
FYI: gist replies with the post body, so if the file is big, perhaps grep just the relevant parts of the reply.
As Ronie said above, there is a gist gem which provides a gist command that you can use from your terminal to upload content to https://gist.github.com/
To upload the contents of a.rb just:
gist a.rb
More info http://defunkt.io/gist/
Have the same desire I found https://www.npmjs.com/package/gistup and fork the repository to https://github.com/CrandellWS/mkg because the developer did not want to support Windows which was the operating system being used at the time. So I reworked the npm package to work on windows as well as linux and apple...
Full source is available on GitHub:
https://github.com/CrandellWS/mkg
Installation is simple with npm
npm install -g mkg
Use is discribed on the npmjs package page:
https://www.npmjs.com/package/gistup
Once installed simply cd to which every directory you want to make a gist from...(remeber there are no subfolders with Gists)
and run the command:
mkg
and it will open your new gist in a broswer...additionally you will be able to control it like a normal git from there... just no subfolders...
https://stackoverflow.com/a/41233970/1815624
A super simple command I like to use for making gists out of diffs is:
git diff origin master -U15 | gist -t diff
Where 15 is the line spacing you can have before and after the change (So it's easier to see differences in larger files.)
-t is the type flag.