Please, help me with one simple script on bash. I wrote a script to backup one server with many vhosts. This script works fine, except one config file. When script parses this file, in results it adds some extra symbols
Cycle of parsing
for n in `find /etc/nginx/conf.d/ -type f -name *.conf`
do
dir_for_bckp=`cat $n | awk '{ print $1,$2 }' | egrep "^root" | awk '{ print $2 }' | sed 's/\;//g'`
name_bckp=`echo $dir_for_bckp | sed 's/\//\_/g'`
tar -zcvf $tmp_dir/backup/$name_bckp.tar.gz $dir_for_bckp/ 1>/dev/null
done
Output for problematic file in debug mode
+ for n in '`find /etc/nginx/conf.d/ -type f -name *.conf`'
++ cat /etc/nginx/conf.d/gitlab.conf
++ awk '{ print $1,$2 }'
++ awk '{ print $2 }'
++ sed 's/\;//g'
++ egrep '^root'
+ dir_for_bckp=$'/home/git/gitlab/public\r'
++ sed 's/\//\_/g'
+ name_bckp=$'_home_git_gitlab_public\r'
My question - why dir_for_bckp=$'/home/git/gitlab/public\r' ? Must be dir_for_bckp=/home/git/gitlab/public
Parsing other config files is ok!
Example
+ for n in '`find /etc/nginx/conf.d/ -type f -name *.conf`'
++ cat /etc/nginx/conf.d/site-com.conf
++ awk '{ print $1,$2 }'
++ egrep '^root'
++ sed 's/\;//g'
++ awk '{ print $2 }'
+ dir_for_bckp=/home/site.com/html
++ sed 's/\//\_/g'
+ name_bckp=_home_site.com_html
If I manually parse problematic file, output also ok.
cat /etc/nginx/conf.d/gitlab.conf | awk '{ print $1,$2 }' | egrep "^root" | awk '{ print $2 }' | sed 's/\;//g'
/home/git/gitlab/public
It seems the config file was created with MSWindows line ends, i.e. each line ends with \r\n, not only \n. Use fromdos or dos2unix to fix the file.
Related
I want to get the size of the directory's content. If I use the command line I can get like this:
ls -l | head -n 1 | awk '{ print $2 }'
So the output is the size of the directory's content:
16816
But I want to do this in a bash script:
x="ls -l DBw | head -n 1 | awk '{ print $2 }'"
while sleep 1; do
y=$(eval "$x");
echo $y
done
But the output of this script is the full line:
Total 16816
Why is this happening and how can I get just the second word?
x="ls -l DBw | head -n 1 | awk '{ print $2 }'"
It's happening because $2 is inside double quotes and so it's interpreted immediately. You could escape it with \$2, but better yet:
Don't store code in variables. Use a function.
x() {
ls -l DBw | head -n 1 | awk '{ print $2 }'
}
Then you can call x many times.
while sleep 1; do
x
done
That said, it's better not to parse the output of ls in the first place. Use du to compute the size of a directory:
$ du -sh /dir
1.3M /dir
$ du -sh /dir | cut -f1
1.3M
How to align this script output.
for instance in `find /bxp/*/*/*/prod/*/apache_*/httpd/htdocs/ -type f -name status.txt` ; do
echo "`hostname`: `ls -ltr | ${instance}` : `cat ${instance}`"
done
Output looks like:
r008abc, /bxp/xip/xip.pentaho-server_pentaho-server-assembly/pentaho.prod.jobengine/prod/xip.pentaho-server_web.partition_0.0.1/apache_5.3.3-2.2.
26/httpd/htdocs/status.txt, Web server is disabled
However i want the output be like:
r008abc| xip - xip.pentaho-server_web.partition_0.0.1 | Web server is disabled
xip - is nothing but the second column of the $instance - xip.pentaho-server_web.partition_0.0.1 is 6th column of the $instance. How can I achieve this. I tried awk command but it was not helpful. Your suggestion is appreciated.
Command I tried
for instance in `find /bxp/*/*/*/prod/*/apache_*/httpd/htdocs/ -type f -name status.txt` ; do
echo "`hostname`: `"ls -ltr | awk -F '/' '{print $3}"' ${instance}` : `cat ${instance}`"
done
something like this oneliner:
find /bxp/*/*/*/prod/*/apache_*/httpd/htdocs/ -type f -name status.txt | awk -F/ -v host=$(hostname) '{cmd="cat \047" $0 "\047"; if ((cmd | getline out) > 0){printf "%s| %s - %s | %s\n", host,$3, $7, out} close(cmd);}'
Explanation of awk-script:
awk -F/ # use / as field separator
-v host=$(hostname) # set var host
'{
cmd="cat \047" $0 "\047" # define cmd
if ((cmd | getline out) > 0) # read output of cmd
printf "%s| %s - %s | %s\n",host,$3,$7,out; # print formatted result
close(cmd);
}'
I have list of names in a file that I need to create directories from. The list looks like
Ada Lovelace
Jean Bartik
Leah Culver
I need the folders to be the exact same, preserving the whitespace(s). But with
awk '{print $0}' myfile | xargs mkdir
I create separate folders for each word
Ada
Lovelace
Jean
Bartik
Leah
Culver
Same happens with
awk '{print $1 " " $2}' myfile | xargs mkdir
Where is the error?
Using gnu xargs you can use -d option to set delimiter as \n only. This way you can avoid awk also.
xargs -d '\n' mkdir -p < file
If you don't have gnu xargs then you can use tr to convert all \n to \0 first:
tr '\n' '\0' < file | xargs -0 mkdir
#birgit:try: Completely based on your sample Input_file provided.
awk -vs1="\"" 'BEGIN{printf "mkdir ";}{printf("%s%s%s ",s1,$0,s1);} END{print ""}' Input_file | sh
awk '{ system ( sprintf( "mkdir \"%s\"", $0)) }' YourFile
# OR
awk '{ print"mkdir "\"" $0 "\"" | "/bin/sh" }' YourFile
# OR for 1 subshell
awk '{ Cmd = sprintf( "%s%smkdir \"%s\"", Cmd, (NR==1?"":"\n"), $0) } END { system ( Cmd ) }' YourFile
Last version is better due to creation of only 1 subshell.
If there are a huge amount of folder (shell parameter limitation), you could loop and create smaller command several times
I am looking for a command which helps me use if in awk and equates it to the current date.
A/B folder has files with different dates. I need to filter out files of the present day whenever script runs
A) Gives an output with all the dates,
date=`date +"%Y-%m-%d"`
s3cmd ls --recursive s3://A/B/ | grep A-B | grep .tar | awk '{ if ($1 -eq "$date" ) print $1" "$2 " " $3 " " $4 }' | sort -r
B)Replaces $1 which contains dates with "$date" to all of them
date=`date +"%Y-%m-%d"`
s3cmd ls --recursive s3://A/B/ | grep A-B | grep .tar | awk '{ if ($1 = "$date" ) print $1" "$2 " " $3 " " $4 }' | sort -r
C)Does not give any output. leaves blank
date=`date +"%Y-%m-%d"`
s3cmd ls --recursive s3://A/B/ | grep A-B | grep .tar | awk '{ if ($1 == "$date" ) print $1" "$2 " " $3 " " $4 }' | sort -r
if I remove "" it does not give me any output in all the cases.
The shell doesn't substitute variables inside double quotes. You should assign an awk variable from the shell variable. Also, the equality comparison is ==, not -eq or =.
awk -v date="$date" '$1 == date { print $1" "$2 " " $3 " " $4 }'
You don't really need awk for that. Just use find and say
find /path/to/search/ -type f ! -newermt $(date +"%Y-%m-%d")
$(..) is command substitution and what it will do is expand to current date in the format YYYY-MM-DD.
! -newermt is find option to look for files older than specified date
-type f will only look for files
awk is not shell. It is a completely separate tool with it's own syntax and capabilities. Therefore you should not expect to be able to use shell variables or shell syntax in an awk script. Try this:
s3cmd ls --recursive s3://A/B/ |
awk -v date="$date" -v OFS=" " '/A-B/ && /.tar/ && ($1 == date) { print $1, $2, $3, $4 }' |
sort -r
You probably actually meant \.tar instead of .tar though and as #jaypal said, this is a job for find, not ls piped to awk.
I'm using a few commands to cat a few files, like this:
cat somefile | grep example | awk -F '"' '{ print $2 }' | xargs cat
It nearly works, but my issue is that I'd like to add a newline after each file.
Can this be done in a one liner?
(surely I can create a new script or a function that does cat and then echo -n but I was wondering if this could be solved in another way)
cat somefile | grep example | awk -F '"' '{ print $2 }' | while read file; do cat $file; echo ""; done
Using GNU Parallel http://www.gnu.org/software/parallel/ it may be even faster (depending on your system):
cat somefile | grep example | awk -F '"' '{ print $2 }' | parallel "cat {}; echo"
awk -F '"' '/example/{ system("cat " $2 };printf "\n"}' somefile