I am trying to build a pipeline for modifying kafka topics. Part of the steps includes getting the current number of partitions of the topic to determine if it can be updated or not. For some reason, jenkins does not like my use of variables in the script. I am getting an error on --topic ${TOPIC}. the variable is correctly enclosed by {} so i don't understand what is going on.
error being given:
WorkflowScript: 32: illegal string body character after dollar sign;
solution: either escape a literal dollar sign "\$5" or bracket the value expression "${5}" # line 32, column 36.
--topic ${TOPIC} | head -1)
snippet of pipeline:
script {
sh """
if [ -n "\$(printf '%s\n' ${NUM_PARTITIONS} | sed 's/[0-9]//g')" ]; then
echo 'Num partitions is not numeric'
exit 1
fi
"""
sh """
desc=\$(kafka-topics.sh \
--bootstrap-server ${KAFKA_SERVERS} \
--command-config sasl.properties \
--describe \
--topic ${TOPIC} | head -1)
if [ \$? -ne 0 ]; then
echo 'describe failed'
exit 1
fi
np=\$(echo desc | gawk 'match($0, /PartitionCount:\s*([[:digit:]]*)\s*/, a) {print a[1]}')
if [ ${NUM_PARTITIONS} -le np ]; then
echo 'num partitions <= configured partitions. alter skipped'
else
kafka-topics.sh \
--bootstrap-server ${KAFKA_SERVERS} \
--command-config sasl.properties \
--alter \
--topic ${TOPIC} \
--partitions ${NUM_PARTITIONS}
fi
"""
}
got it working. Weird escaping rules...
script {
sh """
if [ -n "\$(printf '%s\n' ${NUM_PARTITIONS} | sed 's/[0-9]//g')" ]; then
echo 'Num partitions is not numeric'
exit 1
fi
if [ -n "\$(printf '%s\n' ${RETENTION} | sed 's/[0-9]//g')" ]; then
echo 'Retention is not numeric'
exit 1
fi
"""
sh """
desc=\$(${KAFKA_SCRIPTS_LOC}/kafka-topics.sh \
--bootstrap-server ${KAFKA_SERVERS} \
--command-config ${SASL_LOC} \
--describe \
--topic $TOPIC)
if [ \$? -ne 0 ]; then
echo 'describe failed'
exit 1
fi
np=\$(echo \${desc} | head -1 | gawk 'match(\$0, /PartitionCount:\\s*([[:digit:]]*)\\s*/, a) {print a[1]}')
if [ ${NUM_PARTITIONS} -le \${np} ]; then
echo 'num partitions <= configured partitions. alter skipped'
else
${KAFKA_SCRIPTS_LOC}/kafka-topics.sh \
--bootstrap-server ${KAFKA_SERVERS} \
--command-config ${SASL_LOC} \
--alter \
--topic ${TOPIC} \
--partitions ${NUM_PARTITIONS}
fi
"""
}
Related
I have a make target, which i usually need to run twice to get accurate outcome. I.e the 1st run if accurate thenn on the 2nd run, if the variable is changed, it still displays the previous output, which is wrong, is there a way to get rid of cache or clear it in between.
.PHONY:check-tf-lint
check-tf-lint: configure ## TF Linting
$(eval list_of_dir := $(shell cd ${deployment} && ls -ld */ | awk '{print $$NF}'| grep -v 'test_cases'| sed 's|/||g'))
$(shell touch ${quality-metrics}/formatting.txt)
#for i in aws_bot; do \
make set-tf-version -e infra_module_path=$$i; \
terraform fmt -check -list=false ${deployment}/$$i ; \
if [ "$$?" != "0" ]; then \
echo "Need Formatting in $$i" >> ${quality-metrics}/formatting.txt; \
terraform fmt -check ${deployment}/$$i >> ${quality-metrics}/formatting.txt; \
echo "" >> ${quality-metrics}/formatting.txt; \
fi \
done
$(eval TMP := $(shell (cat ${quality-metrics}/formatting.txt | wc -l)))
echo "${TMP}"
#if [ "$(TMP)" = "0" ]; then \
echo "All Good! No Formatting Needed."; \
else \
echo "Kindly Format Below Mentioned code and check in Again"; \
cat ${quality-metrics}/formatting.txt; \
fi
$(shell rm -rf ${quality-metrics}/formatting.txt)
#if [ "$(TMP)" != "0" ]; then \
exit 1; \
fi
Rule of thumb: you should never use eval or shell functions in a make recipe. If you are doing that it's a pretty sure sign that something has gone wrong somewhere.
In your case the reason you see this behavior is that make will expand ALL variables and functions for all lines in a recipe before the first line in the recipe is invoked. So as far as make is concerned your recipe is handled like this:
.PHONY:check-tf-lint
check-tf-lint: configure ## TF Linting
$(eval list_of_dir := $(shell cd ${deployment} && ls -ld */ | awk '{print $$NF}'| grep -v 'test_cases'| sed 's|/||g'))
$(shell touch ${quality-metrics}/formatting.txt)
$(eval TMP := $(shell (cat ${quality-metrics}/formatting.txt | wc -l)))
$(shell rm -rf ${quality-metrics}/formatting.txt)
#for i in aws_bot; do \
make set-tf-version -e infra_module_path=$$i; \
terraform fmt -check -list=false ${deployment}/$$i ; \
if [ "$$?" != "0" ]; then \
echo "Need Formatting in $$i" >> ${quality-metrics}/formatting.txt; \
terraform fmt -check ${deployment}/$$i >> ${quality-metrics}/formatting.txt; \
echo "" >> ${quality-metrics}/formatting.txt; \
fi \
done
echo "${TMP}"
#if [ "$(TMP)" = "0" ]; then \
echo "All Good! No Formatting Needed."; \
else \
echo "Kindly Format Below Mentioned code and check in Again"; \
cat ${quality-metrics}/formatting.txt; \
fi
#if [ "$(TMP)" != "0" ]; then \
exit 1; \
fi
You should always write your recipes using shell facilities and not make facilities. Set shell variables, don't use eval to set make variables, and run shell commands directly (you're in a recipe after all!) rather than using make's shell function.
You may need to put all the lines in a single script (with semicolon / backslash) to allow this to work. Or consider .ONESHELL but that's a much bigger set of changes.
This worked for me !
.PHONY:check-tf-lint
check-tf-lint: ## TF Linting
$(eval list_of_dir := $(shell cd ${deployment} && ls -ld */ | awk '{print $$NF}'| grep -v 'test_cases'| sed 's|/||g'))
#for i in $(list_of_dir); do \
make set-tf-version -e infra_module_path=$$i; \
terraform fmt -check -list=false ${deployment}/$$i ; \
if [ "$$?" != "0" ]; then \
echo "Need Formatting in $$i" >> ${quality-metrics}/formatting.txt; \
terraform fmt -check ${deployment}/$$i >> ${quality-metrics}/formatting.txt; \
echo "" >> ${quality-metrics}/formatting.txt; \
fi \
done
#if [ -e "${quality-metrics}/formatting.txt" ]; then \
export MNC=`cat ${quality-metrics}/formatting.txt | wc -l`; \
if [ "$${MNC}" = "0" ]; then \
echo "All Good! No Formatting Needed."; \
else \
echo ""; \
echo "Kindly Format Below Mentioned code and check in Again"; \
cat ${quality-metrics}/formatting.txt; \
fi; \
rm -rf ${quality-metrics}/formatting.txt; \
if [ "$${MNC}" != "0" ]; then \
exit 1; \
fi \
fi
In the code below I am attempting to assign variables to the two yad values Radius and Amount.
This can be done with awk by printing the yad values to file but I want to avoid this if I can.
The string (that is, both yad values) is assigned a variable and trimmed of characters, as required, using sed. However, the script stops at this line;
radius=$(sed 's|[amount*,]||g')
Two questions
is there a better way of tackling this; and
why is the script not completing? I have not been able to figure out the syntax.
EDIT: don't need the loop and working on the sed syntax
#!/bin/bash
#ifs.sh
values=`yad --form --center --width=300 --title="Test" --separator=' ' \
--button=Skip:1 \
--button=Apply:0 \
--field="Radius":NUM \
'0!0..30!1!0' \
--field="Amount":NUM \
'0!0..5!0.01!2'`
radius=$(echo "$values" | sed 's|[amount*,]||g')
amount=$(echo "$values" | sed 's/.a://')
if [ $? = 1 ]; then
echo " " >/dev/null 2>&1; else
echo "Radius = $radius"
echo "Amount = $amount"
fi
exit
Alternatives
# with separator
# radius="${values%????????}"
# amount="${values#????????}"
# without separator
# radius=$(echo "$values" | sed s'/........$//')
# amount=$(echo "$values" | sed 's/^........//')
It's easier than you think:
$ values=( $(echo '7.000000 0.100000 ') )
$ echo "${values[0]}"
7.000000
$ echo "${values[1]}"
0.100000
Replace $(echo '7.000000 0.100000 ') with yad ... so the script would be:
values=( $(yad --form --center --width=300 --title="Test" --separator=' ' \
--button=Skip:1 \
--button=Apply:0 \
--field="Radius":NUM \
'0!0..30!1!0' \
--field="Amount":NUM \
'0!0..5!0.01!2') )
if [ $? -eq 0 ]; then
echo "Radius = ${values[0]}"
echo "Amount = ${values[1]}"
fi
EDIT: Changed answer based on #Ed Morton
#!/bin/bash
#ifs.sh
values=($(yad --form --center --width=300 --title="Test" --separator=' ' \
--button=Skip:1 \
--button=Apply:0 \
--field="Radius":NUM \
'0!0..30!1!0' \
--field="Amount":NUM \
'0!0..5!0.01!2'))
if [ $? -eq 0 ]; then
radius="${values[0]}"
amount="${values[1]}"
fi
exit
bash -x Output
+ '[' 0 -eq 0 ']'
+ radius=7.000000
+ amount=1.000000
+ exit
I am trying to write a target in a makefile which will read a variable(having IPs) from one of the .mk file and if a space separated list found split it and take some action.
Issue i am facing that the string do not split and in for loop do not get the value either.
Have tried following
GWTS_FE_IPS=2600:40f0:3e::2 2600:40f0:3e::3 2600:40f0:3e::4 2600:40f0:3e::5
test:
$(eval IPS=$(shell echo "$(GW_IPS)" |awk -F " " '{print NF}'))
if [ ${IPS} -gt 1 ]; then \
echo "Multiple Ips [$(GW_IPS)]"; \
for ip in $(shell echo "${GW_IPS}" | sed -e 's/ /\n/g'); \
do \
echo ".... $(ip) ...."; \
done \
else \
echo "Single IP [$(GW_IPS)]"; \
fi
Result i get is
2600:40f0:3e::2n2600:40f0:3e::3n2600:40f0:3e::4n2600:40f0:3e::5
if [ 4 -gt 1 ]; then \
echo "Multiple Ips [2600:40f0:3e::2 2600:40f0:3e::3 2600:40f0:3e::4 2600:40f0:3e::5]"; \
for ip in 2600:40f0:3e::2n2600:40f0:3e::3n2600:40f0:3e::4n2600:40f0:3e::5; \
do \
echo ".... ...."; \
done \
else \
echo "Single IP [2600:40f0:3e::2 2600:40f0:3e::3 2600:40f0:3e::4 2600:40f0:3e::5]"; \
fi
Multiple Ips [2600:40f0:3e::2 2600:40f0:3e::3 2600:40f0:3e::4 2600:40f0:3e::5]
.... ....
Can any one give some pointers.
You are trying to do too many things at once without testing any of them. When you try new tools, try them one at a time.
GWTS_FE_IPS=2600:40f0:3e::2 2600:40f0:3e::3 2600:40f0:3e::4 2600:40f0:3e::5
IPS := $(words $(GWTS_FE_IPS))
test:
#if [ ${IPS} -gt 1 ]; then \
echo "Multiple Ips [$(GW_IPS)]"; \
for ip in $(GWTS_FE_IPS) ; \
do \
echo ".... $$ip ...."; \
done \
else \
echo "Single IP [$(GW_IPS)]"; \
fi
I have several experiments. Each experiment has several replicate files. I want to place all these replicate files into one text file in the following way.
Lets say there are 3 experiments and each experiment has 2 replicate files.(Experiment and replicate number can be more than this)
/home/data/study1/EXP1_30/EXP1_replicate_1_30.txt
/home/data/study1/EXP1_30/EXP1_replicate_2_30.txt
/home/data/study1/EXP1_60/EXP1_replicate_1_60.txt
/home/data/study1/EXP1_60/EXP1_replicate_2_60.txt
/home/data/study1/EXP2_30/EXP2_replicate_1_30.txt
/home/data/study1/EXP2_30/EXP2_replicate_2_30.txt
/home/data/study1/EXP2_60/EXP2_replicate_1_60.txt
/home/data/study1/EXP2_60/EXP2_replicate_2_60.txt
/home/data/study1/EXP3_30/EXP3_replicate_1_30.txt
/home/data/study1/EXP3_30/EXP3_replicate_2_30.txt
/home/data/study1/EXP3_60/EXP3_replicate_1_60.txt
/home/data/study1/EXP3_60/EXP3_replicate_2_60.txt
output file1.txt will look like
/home/data/study1/EXP1/EXP1_replicate_1_30.txt,/home/data/study1/EXP1/EXP1_replicate_2_30.txt \
/home/data/study1/EXP2/EXP2_replicate_1_30.txt,/home/data/study1/EXP2/EXP2_replicate_2_30.txt \
/home/data/study1/EXP3/EXP3_replicate_1_30.txt,/home/data/study1/EXP3/EXP3_replicate_2_30.txt
output file2.txt will look like
/home/data/study1/EXP1/EXP1_replicate_1_60.txt,/home/data/study/EXP1/EXP1_replicate_2_60.txt \
/home/data/study1/EXP2/EXP2_replicate_1_60.txt,/home/data/study1/EXP2/EXP2_replicate_2_60.txt \
/home/data/study1/EXP3/EXP3_replicate_1_60.txt,/home/data/study1/EXP3/EXP3_replicate_2_60.txt
....
My code with for loops:
ID=(30 60)
exp=("EXP1" "EXP2" "EXP3")
d=""
for txtfile in /home/data/study1/${exp[0]}/${exp[0]}*_${ID[0]}.txt
do
printf "%s%s" "$d" "$txtfile"
d=","
done
printf " \\"
printf "\n"
d=""
for txtfile in /home/data/study1/${exp[1]}/${exp[1]}*_${ID[0]}.txt
do
printf "%s%s" "$d" "$txtfile"
d=","
done
printf " \\"
printf "\n"
d=""
for txtfile in /home/data/study1/${exp[2]}/${exp[2]}*_${ID[0]}.txt
do
printf "%s%s" "$d" "$txtfile"
d=","
done
I am using for loops with index numbers for each experiment and replicates which is very time consuming. Is there any easy way?
I think that this does what you want:
#!/bin/bash
ids=( 30 60 )
dir=/home/data/study1
# join glob on comma, add slash at end
# modified from http://stackoverflow.com/a/3436177/2088135
join() { local IFS=,; echo "$* "'\'; } #' <- to fix syntax highlighting
i=0
for id in "${ids[#]}"; do
s=$(for exp in "$dir"/EXP*"$id"; do join "$exp/"*"$id".txt; done)
# trim off final slash and output to file
echo "${s%?}" > file$((++i)).txt
done
Output (note that when testing, I set dir=.):
$ cat file1.txt
./EXP1_30/EXP1_replicate_1_30.txt,./EXP1_30/EXP1_replicate_2_30.txt \
./EXP2_30/EXP2_replicate_1_30.txt,./EXP2_30/EXP2_replicate_2_30.txt \
./EXP3_30/EXP3_replicate_1_30.txt,./EXP3_30/EXP3_replicate_2_30.txt
$ cat file2.txt
./EXP1_60/EXP1_replicate_1_60.txt,./EXP1_60/EXP1_replicate_2_60.txt \
./EXP2_60/EXP2_replicate_1_60.txt,./EXP2_60/EXP2_replicate_2_60.txt \
./EXP3_60/EXP3_replicate_1_60.txt,./EXP3_60/EXP3_replicate_2_60.txt
You can use the following bash script:
#!/bin/bash
i=0; n=0; files=""
sort -t_ -k5 files.txt | while read line ; do
files="$files $line"
i=$((i+1))
if [ $((i%6)) -eq 0 ] ; then
n=$((n+1))
cat $files > "$n.txt"
files=""
fi
done
You can also make use of a subshell and do it from the command line (your data in dat/experiment.txt) with:
$ ( first=0; cnt=0; grep 30 dat/experiment.txt | sort | while read line; do \
[ "$first" = 0 ] && first=1 || { [ "$cnt" = 0 ] && echo ' \'; }; echo -n $line; \
((cnt++)); [ "$cnt" = 1 ] && echo -n ","; [ "$cnt" = 2 ] && cnt=0; done; \
echo "" ) >outfile1.txt
$ ( first=0; cnt=0; grep 60 dat/experiment.txt | sort | while read line; do \
[ "$first" = 0 ] && first=1 || { [ "$cnt" = 0 ] && echo ' \'; }; echo -n $line; \
((cnt++)); [ "$cnt" = 1 ] && echo -n ","; [ "$cnt" = 2 ] && cnt=0; done; \
echo "" ) >outfile2.txt
Admittedly, the one liner ended up being longer than originally anticipated to match your line continuations -- exactly. If you omit the line continuations in the outfiles, the line reduces to (e.g.):
$ (cnt=0; grep 30 dat/experiment.txt | sort | while read line; do echo -n $line; \
((cnt++)); [ "$cnt" = 1 ] && echo -n ","; [ "$cnt" = 2 ] && echo "" && cnt=0; \
done ) >outfile1.txt
output:
$ cat outfile1.txt
/home/data/study1/EXP1_30/EXP1_replicate_1_30.txt,/home/data/study1/EXP1_30/EXP1_replicate_2_30.txt \
/home/data/study1/EXP2_30/EXP2_replicate_1_30.txt,/home/data/study1/EXP2_30/EXP2_replicate_2_30.txt \
/home/data/study1/EXP3_30/EXP3_replicate_1_30.txt,/home/data/study1/EXP3_30/EXP3_replicate_2_30.txt \
$ cat outfile2.txt
/home/data/study1/EXP1_60/EXP1_replicate_1_60.txt,/home/data/study1/EXP1_60/EXP1_replicate_2_60.txt \
/home/data/study1/EXP2_60/EXP2_replicate_1_60.txt,/home/data/study1/EXP2_60/EXP2_replicate_2_60.txt \
/home/data/study1/EXP3_60/EXP3_replicate_1_60.txt,/home/data/study1/EXP3_60/EXP3_replicate_2_60.txt \
I got an error message:
-d was unexpected at this time
while running the following (see line 3: if [-d ".git"])
autoversion:
#( \
if [ -d ".git" ] && which git > /dev/null ; then \
DETECTED_VERSION=$$(git describe --always --tags --dirty) ; \
else \
DETECTED_VERSION=$$(grep -v "^#" "$(RELEASED_VERSION_FILE)") ; \
if basename $$(pwd) | grep -q "^[[:alnum:]]*-bedtools-[[:alnum:]]*$$" ; then \
DETECTED_VERSION=$${DETECTED_VERSION}-zip-$$(basename "$$(pwd)" | sed 's/^[[:alnum:]]*-bedtools-//') ; \
fi ; \
fi ; \
\
CURRENT_VERSION="" ; \
[ -e "$(VERSION_FILE)" ] && CURRENT_VERSION=$$(grep "define VERSION_GIT " "$(VERSION_FILE)" | cut -f3 -d" " | sed 's/"//g') ; \
\
echo "DETECTED_VERSION = $$DETECTED_VERSION" ; \
echo "CURRENT_VERSION = $$CURRENT_VERSION" ; \
if [ "$${DETECTED_VERSION}" != "$${CURRENT_VERSION}" ] ; then \
echo "Updating version file." ; \
echo "#ifndef VERSION_GIT_H" > $(VERSION_FILE) ; \
echo "#define VERSION_GIT_H" >> $(VERSION_FILE) ; \
echo "#define VERSION_GIT \"$${DETECTED_VERSION}\"" >> $(VERSION_FILE) ; \
echo "#endif /* VERSION_GIT_H */" >> $(VERSION_FILE) ; \
fi )
What is the problem?