I wrote what I thought was a quick script I could run on a bunch of machines. Instead it print what looks like might be directory contents in a recursive search:
version=$(mysql Varnish -B --skip-column-names -e "SELECT value FROM sys_param WHERE param='PatchLevel'" | sed -n 's/^.*\([0-9]\.[0-9]*\).*$/\1/p')
if [[ $(echo "if($version == 6.10) { print 1; } else { print 0; }" | bc) -eq 1 ]]; then
status=$(dpkg-query -l | awk '{print $2}' | grep 'sg-status-polling');
cons=$(dpkg-query -l | awk '{print $2}' | grep 'sg-consolidated-poller');
if [[ "$status" != "" && "$cons" != "" ]]; then
echo "about to change /var/www/Varnish/lib/Extra/SG/ObjectPoller2.pm"; echo;
cp /var/www/Varnish/lib/Extra/SG/ObjectPoller2.pm /var/www/Varnish/lib/Extra/SG/ObjectPoller2.pm.bkup;
sed -ir '184s!\x91\x93!\x91\x27--timeout=35\x27\x93!' /var/www/Varnish/lib/Extra/SG/ObjectPoller2.pm;
sed -n 183,185p /var/www/Varnish/lib/Extra/SG/ObjectPoller2.pm; echo;
else
echo "packages not found. Assumed to be not applicable";
fi
else
echo "This is 4.$version, skipping";
fi
The script is supposed to make sure Varnish is version 4.6.10 and has 2 custom .deb packages installed (not through apt-get). then makes a backup and edits a single line in a perl module from [] to ['--timeout=35']
it looks like its tripping up on the sed replace one liner.
There are two major problems (minor ones addressed in comments). The first is that you use the decimal code for [] instead of the hexa, so you should use \x5b\x5d instead of \x91\x93. The second problem is that if you do use the proper codes, sed will still interpret those syntactically as []. So you can't escape escaping. Here's what you should call:
sed -ri'.bkup' '184s!\[\]![\x27--timeout=35\x27]!' /var/www/Varnish/lib/Extra/SG/ObjectPoller2.pm
And this will create the backup for you (but you should double check).
Related
I'm still new to the shell and need some help.
I have a file stapel_old.
Also I have in the same directory files like english_old_sync, math_old_sync and vocabulary_old_sync.
The content of stapel_old is:
english
math
vocabulary
The content of e.g. english is:
basic_grammar.md
spelling.md
orthography.md
I want to manipulate all files which are given in stapel_old like in this example:
take the first line of stapel_old 'english', (after that math, and so on)
convert in this case english to english_old_sync, (or after that what is given in second line, e.g. math to math_old_sync)
search in english_old_sync line by line for the pattern '.md'
And append to each line after .md :::#a1
The result should be e.g. of english_old_sync:
basic_grammar.md:::#a1
spelling.md:::#a1
orthography.md:::#a1
of math_old_sync:
geometry.md:::#a1
fractions.md:::#a1
and so on. stapel_old should stay unchanged.
How can I realize that?
I tried with sed -n, while loop (while read -r line), and I'm feeling it's somehow the right way - but I still get errors and not the expected result after 4 hours inspecting and reading.
Thank you!
EDIT
Here is the working code (The files are stored in folder 'olddata'):
clear
echo -e "$(tput setaf 1)$(tput setab 7)Learning directories:$(tput sgr 0)\n"
# put here directories which should not become flashcards, command: | grep -v 'name_of_directory_which_not_to_learn1' | grep -v 'directory2'
ls ../ | grep -v 00_gliederungsverweise | grep -v 0_weiter | grep -v bibliothek | grep -v notizen | grep -v Obsidian | grep -v z_nicht_uni | tee olddata/stapel_old
# count folders
echo -ne "\nHow much different folders: " && wc -l olddata/stapel_old | cut -d' ' -f1 | tee -a olddata/stapel_old
echo -e "Are this learning directories correct? [j ODER y]--> yes; [Other]-->no\n"
read lernvz_korrekt
if [ "$lernvz_korrekt" = j ] || [ "$lernvz_korrekt" = y ];
then
read -n 1 -s -r -p "Learning directories correct. Press any key to continue..."
else
read -n 1 -s -r -p "Learning directories not correct, please change in line 4. Press any key to continue..."
exit
fi
echo -e "\n_____________________________\n$(tput setaf 6)$(tput setab 5)Found cards:$(tput sgr 0)$(tput setaf 6)\n"
#GET && WRITE FOLDER NAMES into olddata/stapel_old
anzahl_zeilen=$(cat olddata/stapel_old |& tail -1)
#GET NAMES of .md files of every stapel and write All to 'stapelname'_old_sync
i=0
name="var_$i"
for (( num=1; num <= $anzahl_zeilen; num++ ))
do
i="$((i + 1))"
name="var_$i"
name=$(cat olddata/stapel_old | sed -n "$num"p)
find ../$name/ -name '*.md' | grep -v trash | grep -v Obsidian | rev | cut -d'/' -f1 | rev | tee olddata/$name"_old_sync"
done
(tput sgr 0)
I tried to add:
input="olddata/stapel_old"
while IFS= read -r line
do
sed -n "$line"p olddata/stapel_old
done < "$input"
The code to change only the english_old_sync is:
lines=$(wc -l olddata/english_old_sync | cut -d' ' -f1)
for ((num=1; num <= $lines; num++))
do
content=$(sed -n "$num"p olddata/english_old_sync)
sed -i "s/"$content"/""$content":::#a1/g"" olddata/english_old_sync
done
So now, this need to be a inner for-loop, of a outer for-loop which holds the variable for english, right?
stapel_old should stay unchanged.
You could try a while + read loop and embed sed inside the loop.
#!/usr/bin/env bash
while IFS= read -r files; do
echo cp -v "$files" "${files}_old_sync" &&
echo sed '/^.*\.md$/s/$/:::#a1/' "${files}_old_sync"
done < olddata/staple_old
convert in this case english to english_old_sync, (or after that what is given in second line, e.g. math to math_old_sync)
cp copies the file with a new name, if the goal is renaming the original file name from the content of the file staple_old then change cp to mv
The -n and -i flag from sed was ommited , include it, if needed.
The script also assumes that there are no empty/blank lines in the content of staple_old file. If in case there are/is add an addition test after the line where the do is.
[[ -n $files ]] || continue
It also assumes that the content of staple_old are existing files. Just in case add an additional test.
[[ -e $files ]] || { printf >&2 '%s no such file or directory.\n' "$files"; continue; }
Or an if statement.
if [[ ! -e $files ]]; then
printf >&2 '%s no such file or directory\n' "$files"
continue
fi
See also help test
See also help continue
Combining them all together should be something like:
#!/usr/bin/env bash
while IFS= read -r files; do
[[ -n $files ]] || continue
[[ -e $files ]] || {
printf >&2 '%s no such file or directory.\n' "$files"
continue
}
echo cp -v "$files" "${files}_old_sync" &&
echo sed '/^.*\.md$/s/$/:::#a1/' "${files}_old_sync"
done < olddata/staple_old
Remove the echo's If you're satisfied with the output so the script could copy/rename and edit the files.
My current code:
while read -r rbv_line || [[ -n "$rbv_line" ]]; do
if [[ "${rbv_line}" =~ ${rbv_reg} ]]; then
rbv_downcase="${BASH_REMATCH[0],,}" &&
ruby_version="${rbv_downcase//[^0-9a-z\.\-]/}" &&
((reg_matches="${reg_matches}"+1))
printf "\n"
printf "Setting Ruby version: %s\n" "${ruby_version}"
break
fi
done < "${1}"
It does what I want. But I would love to know if I can simplify this code even more, hoping someone can help me understand the syntax.
If you see these two lines:
rbv_downcase="${BASH_REMATCH[0],,}" &&
ruby_version="${rbv_downcase//[^0-9a-z\.\-]/}" &&
Initially I tried to combine those into one using something like this:
ruby_version="${BASH_REMATCH[0],,//[^0-9a-z\.\-]/}"
That does not work.
Is there a way to combine those two parameter expansions (,, and the //[^0-9a-z\.\-]/) or is passing it through an intermediary variable the right approach?
You can view the latest version of the code here:
https://github.com/octopusnz/scripts
You cannot combine multiple parameter expansions, but...
... you can simplify this code!
The biggest gain is by using already available tools.
Instead of looping, let's use grep. It's supposed to do something when RegEx pattern is occurred, so:
grep -E "$rbv_reg" "$1" # -E is for extended RegEx
I guess your pattern isn't case sensitive, so let's disable it with -i flag.
The loop breaks after match, so let's pass -m 1 to stop processing file after first match.
You want to convert uppercase to lowercase, so let's pipe it through tr:
grep -m 1 -E -i "$rbv_reg" "$1" | tr '[:upper:]' '[:lower:]'
You then replace some characters with //[^0-9a-z\.\-]/, piping it to sed will do the trick:
grep -m 1 -E -i "$rbv_reg" "$1" | tr '[:upper:]' '[:lower:]' | sed 's/[^0-9a-z\.\-]//g'
And at the very end, let's grab the output to variable:
ruby_version="$( grep -m 1 -E -i '$rbv_reg' '$1' | tr '[:upper:]' '[:lower:]' | sed 's/[^0-9a-z\.\-]//g' )"
Since you are printing new line anyway, let's use simple echo instead of printf
All what's left is if [ -n "$ruby_version" ] to increment reg_matches
At the end, we got:
ruby_version="$(
grep -m 1 -E -i '$rbv_reg' '$1' |
tr '[:upper:]' '[:lower:]' |
sed 's/[^0-9a-z\.\-]//g'
)"
if [ -n "$ruby_version" ]; then
reg_matches="$((reg_matches+1))"
echo
echo "Setting Ruby version: $ruby_version"
fi
The advantage of above code is the fact it isn't really Bash dependent and should work in any POSIX Bourne compatible shell.
Is it possible in bash to filter out a part of a string with another given string ?
I have a fixed list of motifs defined in a string. The order IS important and I want to keep only the parts that are passed as a parameter ?
myDefaultList="s,t,a,c,k" #order is important
toRetains="k,t,c,u" #provided by the user, order is not enforced
retained=filter $myDefaultList $toRetains # code to filter
echo $retained # will print t,c,k"
I can write an ugly method that will use IFS, arrays and loops, but I wonder if there's a 'clever' way to do that, using built-in commands ?
here is another approach
tolines() { echo $1 | tr ',' '\n'; }
grep -f <(tolines "$toRetains") <(tolines "$myDefaultList") | paste -sd,
will print
t,c,k
assign to a variable as usual.
Since you mention in your comments that you are open to sed/awk , check also this with GNU awk:
$ echo "$a"
s,t,a,c,k
$ echo "$b"
k,t,c,u
$ awk -v RS=",|\n" 'NR==FNR{a[$1];next}$1 in a{printf("%s%s",$1,RT)}' <(echo "$b") <(echo "$a")
t,c,k
#!/bin/bash
myDefaultList="s,t,a,c,k"
toRetains="s,t,c,u"
IFS=","
for i in $myDefaultList
do
echo $toRetains | grep $i > /dev/null
if [ "$?" -eq "0" ]
then
retained=$retained" "$i
fi
done
echo $retained | sed -e 's/ /,/g' -e 's/,//1'
I have checked it running for me. Kindly check.
I have this file:
CLUSTERS=SP1,SP2,SP3
FNAME_SP1="REWARDS_BTS_SP1_<GTS>.dat"
FNAME_SP2="DUMP_LOG_SP2_<GTS>.dat"
FNAME_SP3="TEST_CASE_TABLE_SP3_<GTS>.dat"
What I want to get from these are:
REWARDS_BTS_SP1_
DUMP_LOG_SP2_
TEST_CASE_TABLE_SP3_
I loop through the CLUSTERS field, get the values, and use it to find the appropriate FNAME_<CLUSTERNAME> value. Basically, the CLUSTERS value are ALWAYS before the _<GTS> part of the string. Any string pattern will do, provided that the CLUSTERS value come before the _<GTS> at the end of the string.
Any suggestions? Here's a part of the script.
function loadClusters() {
for i in `echo ${!CLUSTER*}`
do
CLUSTER=`echo ${i} | grep $1`
if [[ -n ${CLUSTER} ]]; then
CLUSTER=${!i}
break;
fi
done
echo -e ${CLUSTER}
}
function loadClustersCampaign() {
for i in `echo ${!BPOINTS*}`
do
BPOINTS=`echo ${i} | grep $1`
if [[ -n ${BPOINTS} ]]; then
BPOINTS=${!i}
break;
fi
done
for i in `echo ${!FNAME*}`
do
FNAME=`echo ${i} | grep $1`
if [[ -n ${FNAME} ]]; then
FNAME=${!i}
break;
fi
done
echo -e ${BPOINTS}"|"${FNAME}
}
#get clusters
clusters=$(loadClusters $1)
for i in `echo $clusters | sed 's/,/ /g'`
do
file=$(loadClustersCampaign ${i/-/_} | awk -F"|" '{print $2}') ;
echo $file;
#then get the part of the $file variable
done
Fun with Shell Parameter Expansions
You can use matching-prefix notation and indirect expansion to get at the variables you want, and use the "remove suffix" expansion on each result to collect just the portions of the filename that you want. For example:
FNAME_SP1='REWARDS_BTS_SP1_<GTS>.dat'
FNAME_SP2='DUMP_LOG_SP2_<GTS>.dat'
FNAME_SP3='TEST_CASE_TABLE_SP3_<GTS>.dat'
for cluster in "${!FNAME_SP#}"; do
echo ${!cluster%%<GTS>*}
done
This will print out the following:
REWARDS_BTS_SP1_
DUMP_LOG_SP2_
TEST_CASE_TABLE_SP3_
but you could issue any valid shell command inside the loop instead of using echo.
See Also
http://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html
If you like an awk solution for this ,may be below will be useful.
> echo 'FNAME_SP1="REWARDS_BTS_SP1_<GTS>.dat"' | awk -F"<GTS>" '{split($1,a,"=\"");print substr(a[2],2)}'
REWARDS_BTS_SP1_
Furthur more detail below:
> cat temp
LUSTERS=SP1,SP2,SP3
FNAME_SP1="REWARDS_BTS_SP1_<GTS>.dat"
FNAME_SP2="DUMP_LOG_SP2_<GTS>.dat"
FNAME_SP3="TEST_CASE_TABLE_SP3_<GTS>.dat"
> awk -F"<GTS>" '/FNAME_SP/{split($1,a,"=");print substr(a[2],2)}' temp
REWARDS_BTS_SP1_
DUMP_LOG_SP2_
TEST_CASE_TABLE_SP3_
>
I have the following data in a Tab delimited file:
_ DATA _
Col1 Col2 Col3 Col4 Col5
blah1 blah2 blah3 4 someotherText
blahA blahZ blahJ 2 someotherText1
blahB blahT blahT 7 someotherText2
blahC blahQ blahL 10 someotherText3
I want to make sure that the data in 4th column of this file is always an integer. I know how to do this in perl
Read each line, Store value of 4th column in a variable
check if that variable is an integer
if above is true, continue the loop
else break out of the loop with message saying file data not correct
But how would I do this in a shell script using standard linux/unix filter? My guess would be to use grep, but I am not sure how?
cut -f4 data | LANG=C grep -q '[^0-9]' && echo invalid
LANG=C for speed
-q to quit at first error in possible long file
If you need to strip the first line then use tail -n+2 or you could get hacky and use:
cut -f4 data | LANG=C sed -n '1b;/[^0-9]/{s/.*/invalid/p;q}'
awk is the tool most naturally suited for parsing by columns:
awk '{if ($4 !~ /^[0-9]+$/) { print "Error! Column 4 is not an integer:"; print $0; exit 1}}' data.txt
As you get more complex with your error detection, you'll probably want to put the awk script in a file and invoke it with awk -f verify.awk data.txt.
Edit: in the form you'd put into verify.awk:
{
if ($4 !~/^[0-9]+$/) {
print "Error! Column 4 is not an integer:"
print $0
exit 1
}
}
Note that I've made awk exit with a non-zero code, so that you can easily check it in your calling script with something like this in bash:
if awk -f verify.awk data.txt; then
# action for success
else
# action for failure
fi
You could use grep, but it doesn't inherently recognize columns. You'd be stuck writing patterns to match the columns.
awk is what you need.
I can't upvote yet, but I would upvote Jefromi's answer if I could.
Sometimes you need it BASH only, because tr, cut & awk behave differently on Linux/Solaris/Aix/BSD/etc:
while read a b c d e ; do [[ "$d" =~ ^[0-9] ]] || echo "$a: $d not a numer" ; done < data
Edited....
#!/bin/bash
isdigit ()
{
[ $# -eq 1 ] || return 0
case $1 in
*[!0-9]*|"") return 0;;
*) return 1;;
esac
}
while read line
do
col=($line)
digit=${col[3]}
if isdigit "$digit"
then
echo "err, no digit $digit"
else
echo "hey, we got a digit $digit"
fi
done
Use this in a script foo.sh and run it like ./foo.sh < data.txt
See tldp.org for more info
Pure Bash:
linenum=1; while read line; do field=($line); if ((linenum>1)); then [[ ! ${field[3]} =~ ^[[:digit:]]+$ ]] && echo "FAIL: line number: ${linenum}, value: '${field[3]}' is not an integer"; fi; ((linenum++)); done < data.txt
To stop at the first error, add a break:
linenum=1; while read line; do field=($line); if ((linenum>1)); then [[ ! ${field[3]} =~ ^[[:digit:]]+$ ]] && echo "FAIL: line number: ${linenum}, value: '${field[3]}' is not an integer" && break; fi; ((linenum++)); done < data.txt
cut -f 4 filename
will return the fourth field of each line to stdout.
Hopefully that's a good start, because it's been a long time since I had to do any major shell scripting.
Mind, this may well not be the most efficient compared to iterating through the file with something like perl.
tail +2 x.x | sort -n -k 4 | head -1 | cut -f 4 | egrep "^[0-9]+$"
if [ "$?" == "0" ]
then
echo "file is ok";
fi
tail +2 gives you all but the first line (since your sample has a header)
sort -n -k 4 sorts the file numerically on the 4th column, letters will rise to the top.
head -1 gives you the first line of the file
cut -f 4 gives you the 4th column, of the first line
egrep "^[0-9]+$" checks if the value is a number (integers in this case).
If egrep finds nothing, $? is 1, otherwise it's 0.
There's also:
if [ `tail +2 x.x | wc -l` == `tail +2 x.x | cut -f 4 | egrep "^[0-9]+$" | wc -l` ] then
echo "file is ok";
fi
This will be faster, requiring two simple scans through the file, but it's not a single pipeline.
#OP, use awk
awk '$4+0<=0{print "not ok";exit}' file