I have written a bash script that finds any executable files in our scripts directory, then performs a grep on the resulting files to display a description, if it was included in the file.
A "description" is identified in each file as a line beginning with "# DESC:"
For some reason, the script also includes the grep command that is being run (but only once). Does anyone know why this is?
Script and output shown below. Why does the second line in the output happen?
Script
#!/bin/bash
# Find any FILES that are EXECUTABLE in the SCRIPTS
# directory and display any description, if there is one
find /opt/scripts/. -perm -111 -type f -maxdepth 1 | while read line ;
do
file=$(basename "$line")
printf "\033[1m%10s\033[0m : " $file
grep "# DESC:" "$line" | cut -c 9-
done
Output
desc : Displays all the scripts and their descriptions
DESC:" "$line" | cut -c 9-
showhelp : Displays the script help file
test : Script to perform system testing
Reason
Presumably your grepping script is also in /opt/scripts?
So it finds itself, and finds the grep subject '#DESC' and prints that.
You could fix that by adding a # DESC line to the top of your grep script, and just outputting the first result found by each grep using 'grep -m1'
grep -m1 '# DESC' "$line" | cut -c 9-
<humour>Otherwise its just turtles all the way down... ;-) </humour>
Alternative Fix
You could also improve the grep by using a regular expression and anchoring to the beginning of the line:
egrep -m1 '^# DESC' "$line" | cut -c 9-
Related
I need to do as following, firstly I can read a server name from the text file line by line and do a grep command on it. The main task is to find the whole FQDN server name using only the hostname in the NB logs path and do other commands further. Here is the script I wrote but I am beginner in shell scripting and need your help. What do I missing here because I was thinking How to save a variable's value into another directory path and use it there? If I manually run these commands in command line it can give me what I do need it., thank you all..
#!/bin/sh
echo
echo "Looking for the whole backup client names?"
for server in `cat ./List_ServerNames.txt`;
do
cd /usr/openv/netbackup/logs/nbproxy/
return_fqdn=`grep -im1 '$server' "$(ls -Art /usr/openv/netbackup/logs/nbproxy/ | tail -n1 | cut -
d : -f8)" | cut -d " " -f 6 | cut -c -36 | cut -d "(" -f 1`
echo $return_fqdn
done
The output of script is down below:
Looking for the whole backup client names?
I try to search for files and seperate path and version as variable because each will be needed later for creating a directory and to unzip a .jar in desired path.
file=$(find /home/user/Documents/test/ -path *.jar)
version=$(echo "$file" | grep -P -o '[0-9].[0-9].[0-9].[0-9]')
path=$(echo "$file" | sed 's/\(.*\)[/].*/\1/')
newpath=$(echo "${path}/${version}")
echo "$newpath"
result
> /home/user/Documents/test/gb0500
> /home/user/Documents/test/gb0500 /home/user/Documents/test/gb0500
> /home/user/Documents/test /home/user/Documents/test/1.3.2.0
> 1.3.2.1
> 1.3.2.2
> 1.2.0.0
> 1.3.0.0
It's hilarious that it's only working at one line.
what else I tried:
file=$(find /home/v990549/Dokumente/test/ -path *.jar)
version=$(grep -P -o '[0-9].[0-9].[0-9].[0-9]')
path=$(sed 's/\(.*\)[/].*/\1/')
while read $file
do
echo "$path$version"
done
I have no experience in scripting. Thats what I figured out some days ago. I am just practicing and trying to make life easier.
find output:
/home/user/Documents/test/gb0500/gb0500-koetlin-log4j2-web-1.3.2.0-javadoc.jar
/home/user/Documents/test/gb0500/gb0500-koetlin-log4j2-web-1.3.2.1-javadoc.jar
/home/user/Documents/test/gb0500/gb0500-koetlin-log4j2-web-1.3.2.2-javadoc.jar
/home/user/Documents/test/gb0500-co-log4j2-web-1.2.0.0-javadoc.jar
/home/user/Documents/test/gb0500-commons-log4j2-web-1.3.0.0-javadoc.jar
As the both variables version and path are newline-separated, how about:
file=$(find /home/user/Documents/test/ -path *.jar)
version=$(echo "$file" | grep -P -o '[0-9].[0-9].[0-9].[0-9]')
path=$(echo "$file" | sed 's/\(.*\)[/].*/\1/')
paste -d "/" <(echo "$path") <(echo "$version")
Result:
/home/user/Documents/test/gb0500/1.3.2.0
/home/user/Documents/test/gb0500/1.3.2.1
/home/user/Documents/test/gb0500/1.3.2.2
/home/user/Documents/test/1.2.0.0
/home/user/Documents/test/1.3.0.0
BTW I do not recommend to store multiple filenames in a single variable
as a newline-separated variable due to several reasons:
Filenames may contain a newline character.
It is not easy to manipulate the values of each line.
For instance you could simply say
the third line as path=${file%/*} if file contains just one.
Hope this helps.
How do you use a command line argument as a file path and check for file existence in Bash?
I have the simple Bash script test.sh:
#!/bin/bash
set -e
echo "arg1=$1"
if [ ! -f "$1" ]
then
echo "File $1 does not exist."
exit 1
fi
echo "File exists!"
and in the same directory, I have a data folder containing stuff.txt.
If I run ./test.sh data/stuff.txt I see the expected output:
arg1=data/stuff.txt
"File exists!"
However, if I call this script from a second script test2.sh, in the same directory, like:
#!/bin/bash
fn="data/stuff.txt"
./test.sh $fn
I get the mangled output:
arg1=data/stuff.txt
does not exist
Why does the call work when I run it manually from a terminal, but not when I run it through another Bash script, even though both are receiving the same file path? What am I doing wrong?
Edit: The filename does not have spaces. Both scripts are executable. I'm running this on Ubuntu 18.04.
The filename was getting an extra whitespace character added to it as a result of how I was retrieving it in my second script. I didn't note this in my question, but I was retrieving the filename from folder list over SSH, like:
fn=$(ssh -t "cd /project/; ls -t data | head -n1" | head -n1)
Essentially, I wanted to get the filename of the most recent file in a directory on a remote server. Apparently, head includes the trailing newline character. I fixed it by changing it to:
fn=$(ssh -t "cd /project/; ls -t data | head -n1" | head -n1 | tr -d '\n' | tr -d '\r')
Thanks to #bigdataolddriver for hinting at the problem likely being an extra character.
I'm trying to write a bash script that downloads all the .txt files from a website 'http://www1.ncdc.noaa.gov/pub/data/ghcn/daily/'.
So far I have wget -A txt -r -l 1 -nd 'http://www1.ncdc.noaa.gov/pub/data/ghcn/daily/' but I'm struggling to find a way to print the name of each file to the screen (when downloading). That's the part I'm really stuck on. How would one print the names?
Thoughts?
EDIT this is what I have done so far, but I'm trying to remove a lot of stuff like ghcnd-inventory.txt</a></td><td align=...
wget -O- $LINK | tr '"' '\n' | grep -e .txt | while read line; do
echo Downloading $LINK$line ...
wget $LINK$line
done
LINK='http://www1.ncdc.noaa.gov/pub/data/ghcn/daily/'
wget -O- $LINK | tr '"' '\n' | grep -e .txt | grep -v align | while read line; do
echo Downloading $LINK$line ...
wget -nv $LINK$line
done
Slight optimization of Sundeep's answer:
LINK='http://www1.ncdc.noaa.gov/pub/data/ghcn/daily/'
wget -q -O- $LINK | sed -E '/.*href="[^"]*\.txt".*/!d;s/.*href="([^"]*\.txt)".*/\1/' | wget -nv -i- -B$LINK
The sed command eliminates all lines not matching href="xxx.txt" and extracts only the xxx.txt part of the others. It then passes the result to another wget that uses it as the list of files to retrieve. The -nv option tells wget to be as less verbose as possible. It will thus print the name of the file it currently downloads but almost nothing else. Warning: this works only for this particular web site and does not descend in the sub-directories.
I want to make UNIX script, which will automatically move my working directory files to newly created directories.
Example: In you dir you got files:
001-file.html,
001-file.rb,
002-file.html,
002-file.rb
And 2 files will be moved to ./NewDir/001-file and another 2 to ./NewDir/002-file
My problem is that after I get correct string from Unix commands I cannot assign it to variable.
Here is my code:
clear
echo "Starting script"
echo "Dir = "$(pwd)
read -p "Please enter count(max '999') of different file groups:" max_i
read -p "Enter new dir name:" outer_dir_name
for ((i=0; i<=$max_i;i++)) do
a1=$(($i/100))
a2=$((($i-$a1*100)/10))
a3=$(($i-($a2*10)-($a1*100)))
inner_dir_name=$((ls *[$a1][$a2][$a3]* 2>/dev/null | head -n 1 | cut -f1 -d"."))
echo $inner_dir_name
echo "--------------"
done
One pair of round parentheses is enough for command substitution.
inner_dir_name=$(ls *[$a1][$a2][$a3]* 2>/dev/null | head -n 1 | cut -f1 -d".")
It looks like you're going about the operation the hard way. I would probably do something like this, assuming that there are no spaces in the file names:
ls | sed 's/\..*$//' | sort -u |
while read prefix
do
mkdir -p $outer_dir_name/$prefix
mv $prefix.* $outer_dir_name/$prefix
done
The ls could be made more precise with:
ls [0-9][0-9][0-9]-file.*
If I was worried about blanks and other odd-ball characters in the file names, I'd have to use something more careful:
for file in [0-9][0-9][0-9]-file.*
do
prefix=${file%%.*}
[ -d "$outer_dir_name/$prefix" ] || mkdir -p "$outer_dir_name/$prefix"
mv "$file" "$outer_dir_name/$prefix"
done
This executes more mv commands, in general.