Having a string of space separated paths, relative or absolute, example:
/aaaa/bbbb/ccc /ddas/sdsa/dasd ./dasd/dsd dasd/dsda/dsd dsd/dsad/erer/rerer ../dasd/dsad ../../sdasd/sdsd
How can I process this in bash in order to prepend every one of these paths with -I? Example output should be:
-I/aaaa/bbbb/ccc -I/ddas/sdsa/dasd -I./dasd/dsd dasd/dsda/dsd dsd/dsad/erer/rerer -I../dasd/dsad -I../../sdasd/sdsd
Thanks
Edit for context:
As some of you may already have guessed, the purpose of this is to prepend folder paths with the -I flag for gcc commands.
I'm using this in a makefile. The following (slightly modified from anubhava's suggestion) works perfectly:
#to include subdirectories in source
TEMP := $(shell find $(SOURCE_PATH)* -type d)
TEMP := $(shell echo $(TEMP) | awk 1 ORS=' ')
TEMP := $(shell printf -- "-I%s " ${TEMP} )
ifdef TEMP
INC_PATHS += $(TEMP)
endif
You can use printf:
s='/aaaa/bbbb/ccc /ddas/sdsa/dasd ./dasd/dsd dasd/dsda/dsd dsd/dsad/erer/rerer ../dasd/dsad ../../sdasd/sdsd'
printf -v output -- "-I%s " $s
echo "$output"
-I/aaaa/bbbb/ccc -I/ddas/sdsa/dasd -I./dasd/dsd -Idasd/dsda/dsd -Idsd/dsad/erer/rerer -I../dasd/dsad -I../../sdasd/sdsd
Or if using an array:
arr=(/aaaa/bbbb/ccc /ddas/sdsa/dasd ./dasd/dsd dasd/dsda/dsd dsd/dsad/erer/rerer ../dasd/dsad ../../sdasd/sdsd)
printf -v output -- "-I%s " "${arr[#]}"
If you have the paths in an array:
paths=(/aaaa/bbbb/ccc /ddas/sdsa/dasd ./dasd/dsd dasd/dsda/dsd dsd/dsad/erer/rerer ../dasd/dsad ../../sdasd/sdsd)
then you can use bash's find-and-replace syntax:
includes=("${paths[#]/#/-I}")
You can provide an array as a series of arguments to a command (or function):
compile $the_file "${includes[#]}"
You can do a similar transform on $# (in quotes) in a bash function
with_includes() {
# If you need to do something with the first few arguments,
# collect them here and then call shift:
# the_file=$1; shift
# But you need to check $# to make sure the arguments exist :)
local includes=("{#/#/-I}")
compile $the_file "${includes[#]}"
}
#!/usr/local/bin/bash
echo "/aaaa/bbbb/ccc /ddas/sdsa/dasd ./dasd/dsd dasd/dsda/dsd dsd/dsad/erer/rerer ../dasd/dsad ../../sdasd/sdsd" | \
sed s'/ / -I/'g
I agree with the others that this is bad form, however. Your paths should ideally be on a seperate line.
#!/usr/local/bin/bash
echo "/aaaa/bbbb/ccc /ddas/sdsa/dasd ./dasd/dsd dasd/dsda/dsd \
dsd/dsad/erer/rerer ../dasd/dsad ../../sdasd/sdsd" | \
tr ' ' '\n' > tempfile
let "lines=$(wc -l tempfile | cut -d' ' -f1)"
let "lines=lines+1"
let "counter=1"
function loop() {
[ "${counter}" -lt "${lines}" ] && \
echo "Current loop iteration: ${counter}." && \
sed -n "${counter}p" tempfile | sed s'#^# -I#'g >> tempfile2 && \
let "counter=counter+1" && \
loop
}
loop
[ -f ./tempfile ] && \
echo "File exists: ./tempfile." && \
rm ./tempfile
[ ! -f ./tempfile ] && \
echo "File deleted: ./tempfile."
Then you can just edit the beginning of each line in tempfile2 to do whatever, and run it as a shell script.
Inb4 "Oh God no, he's using test! That could accidentally create an infinite loop! WE'RE ALL GOING TO DIE!"
Infinite loops aren't the end of the world. The worst it will cause is a segfault. Occasionally they can even be useful. OP, you will need to make sure you include "&& \" (without quotes) at the end of every command within a test block though; except for the last one of course.
I learned to do it this way because I've spent time in Sparta OpenBSD, where rm and most other utilities don't have -v flags, and we also aren't afraid of infinite loops. ;)
Related
I have a fairly large list of websites in "file.txt" and wanted to check if the words "Hello World!" in the site in the list using looping and curl.
i.e in "file.txt" :
blabla.com
blabla2.com
blabla3.com
then my code :
#!/bin/bash
put() {
printf "list : "
read list
run=$(cat $list)
}
put
scan_list() {
for run in $(cat $list);do
if [[ $(curl -skL ${run}) =~ "Hello World!" ]];then
printf "${run} Hello World! \n"
else
printf "${run} No Hello:( \n"
fi
done
}
scan_list
this takes a lot of time, is there a way to make the checking process faster?
Use xargs:
% tr '\12' '\0' < file.txt | \
xargs -0 -r -n 1 -t -P 3 sh -c '
if curl -skL "$1" | grep -q "Hello World!"; then
echo "$1 Hello World!"
exit
fi
echo "$1 No Hello:("
' _
Use tr to convert returns in the file.txt to nulls (\0).
Pass through xargs with -0 option to parse by nulls.
The -r option prevents the command from being ran if the input is empty. This is only available on Linux, so for macOS or *BSD you will need to check that file.txt is not empty before running.
The -n 1 permits only one file per execution.
The -t option is debugging, it prints the command before it is ran.
We allow 3 simultaneous commands in parallel with the -P 3 option.
Using sh -c with a single quoted multi-line command, we substitute $1 for the entries from the file.
The _ fills in the $0 argument, so our entries are $1.
I went through some posts, but still didn't get how it work.
My request is:
for i in *.json
do
file = `echo $i |cut -d _ -f2`
echo ${file}
# do the rest tasks
done
How to convert above script to target of Makefile?
Here is what I tried
foo:
for i in *.json; do \
$(eval FILE = $(shell echo $$i |cut -d _ -f2)); \
echo $(FILE) ;\
done
But it doesn't work
Using $(eval) or $(shell) is ... not even wrong.
foo:
for i in *.json; do \
file=$$(echo "$$i" |cut -d _ -f2); \
echo "$$file"; \
done
Notice the quoting of the filename variables, and the absence of spaces around the = assignment operator, and the doubling of any dollar sign in order to pass it through from make to the shell.
However, the shell provides a much better mechanism for this;
foo:
for i in *.json; do \
j=$${i#*_}; \
echo "$${j%%_*}"; \
done
or perhaps
foo:
printf '%s\n' *.json \
| sed 's/[^_]*_\([^_]*\)_.*/\1/'
If you only expect a single underscore, both of these can be further simplified.
Or maybe you are just looking for
makefile_variable := $(foreach x,$(wildcard *.json),$(word 2,$(subst _, ,$x)))
Below is the file
TMPQM>CSQN205I COUNT= 213, RETURN=00000000, REASON=00000000
CSQM401I ?TMPQM
QUEUE(Q1) TYPE(QLOCAL)
QSGDISP(QMGR) CURDEPTH(0)
CSQM401I ?TMPQM
QUEUE(Q2) TYPE(QLOCAL)
QSGDISP(QMGR) CURDEPTH(23)
CSQM401I ?TMPQM
QUEUE(Q3) TYPE(QLOCAL)
QSGDISP(QMGR) CURDEPTH(150)
CSQM401I ?TMPQM
My intention is to get the values,
Q=Q1
V=0
Q=Q2
V=23
Q=Q3
V=150
Can you use bash? Not the most efficient, but here's a bash script:
#!/bin/bash
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
count=1
for queue in `grep "QUEUE" input.txt`
do
# Strip the beginning up to (
q1="${queue#*(}"
# Strip the end from ) on
q2="${q1%%)*}"
q[$count]=$q2
count=$((count+1))
done
count=1
for value in `grep "CURDEPTH" input.txt`
do
# Strip the beginning up to (
v1="${value##*(}"
# Strip the end from ) on
v2="${v1%)*}"
v[$count]=$v2
count=$((count+1))
done
for index in 1 2 3
do
echo "Q=${q[$index]}"
echo "V=${v[$index]}"
done
IFS=$SAVEIFS
The fancy for loops are just to deal with looping through lines with spaces in bash.
A combination of pcregrep and sed will do the job. pcregrep supports multiline matches with the -M option
-bash-3.2$ pcregrep -M -o '^QUEUE[(](.+?)[)].*\n.*CURDEPTH[(](\d+?)[)]$' trial.txt | sed -e 's/^QUEUE(\([^)]\+\)).*$/Q=\1/g' -e 's/.*CURDEPTH(\([^)]\+\))/V=\1/g'
Q=Q1
V=0
Q=Q2
V=23
Q=Q3
V=150
My solution:
grep -oP '(?<=QUEUE|CURDEPTH)\S+' input.txt | tr -d '()' | (while read NAME && read VAL; do echo $NAME=$VAL; done)
It's a little more flexible than #cravoori's solution because you can format the output easily. That line's output is...
Q1=0
Q2=23
Q3=150
...which you can source and then read directly using $Q1, $Q2, etc.
I'm writing a bash script to modify a config file which contains a bunch of key/value pairs. How can I read the key and find the value and possibly modify it?
A wild stab in the dark for modifying a single value:
sed -c -i "s/\($TARGET_KEY *= *\).*/\1$REPLACEMENT_VALUE/" $CONFIG_FILE
assuming that the target key and replacement value don't contain any special regex characters, and that your key-value separator is "=". Note, the -c option is system dependent and you may need to omit it for sed to execute.
For other tips on how to do similar replacements (e.g., when the REPLACEMENT_VALUE has '/' characters in it), there are some great examples here.
Hope this helps someone. I created a self contained script, which required config processing of sorts.
#!/bin/bash
CONFIG="/tmp/test.cfg"
# Use this to set the new config value, needs 2 parameters.
# You could check that $1 and $2 is set, but I am lazy
function set_config(){
sudo sed -i "s/^\($1\s*=\s*\).*\$/\1$2/" $CONFIG
}
# INITIALIZE CONFIG IF IT'S MISSING
if [ ! -e "${CONFIG}" ] ; then
# Set default variable value
sudo touch $CONFIG
echo "myname=\"Test\"" | sudo tee --append $CONFIG
fi
# LOAD THE CONFIG FILE
source $CONFIG
echo "${myname}" # SHOULD OUTPUT DEFAULT (test) ON FIRST RUN
myname="Erl"
echo "${myname}" # SHOULD OUTPUT Erl
set_config myname $myname # SETS THE NEW VALUE
Assuming that you have a file of key=value pairs, potentially with spaces around the =, you can delete, modify in-place or append key-value pairs at will using awk even if the keys or values contain special regex sequences:
# Using awk to delete, modify or append keys
# In case of an error the original configuration file is left intact
# Also leaves a timestamped backup copy (omit the cp -p if none is required)
CONFIG_FILE=file.conf
cp -p "$CONFIG_FILE" "$CONFIG_FILE.orig.`date \"+%Y%m%d_%H%M%S\"`" &&
awk -F '[ \t]*=[ \t]*' '$1=="keytodelete" { next } $1=="keytomodify" { print "keytomodify=newvalue" ; next } { print } END { print "keytoappend=value" }' "$CONFIG_FILE" >"$CONFIG_FILE~" &&
mv "$CONFIG_FILE~" "$CONFIG_FILE" ||
echo "an error has occurred (permissions? disk space?)"
sed "/^$old/s/\(.[^=]*\)\([ \t]*=[ \t]*\)\(.[^=]*\)/\1\2$replace/" configfile
So I can not take any credit for this as it is a combination of stackoverflow answers and help from irc.freenode.net #bash channel but here are bash functions now to both set and read config file values:
# https://stackoverflow.com/a/2464883
# Usage: config_set filename key value
function config_set() {
local file=$1
local key=$2
local val=${#:3}
ensureConfigFileExists "${file}"
# create key if not exists
if ! grep -q "^${key}=" ${file}; then
# insert a newline just in case the file does not end with one
printf "\n${key}=" >> ${file}
fi
chc "$file" "$key" "$val"
}
function ensureConfigFileExists() {
if [ ! -e "$1" ] ; then
if [ -e "$1.example" ]; then
cp "$1.example" "$1";
else
touch "$1"
fi
fi
}
# thanks to ixz in #bash on irc.freenode.net
function chc() { gawk -v OFS== -v FS== -e 'BEGIN { ARGC = 1 } $1 == ARGV[2] { print ARGV[4] ? ARGV[4] : $1, ARGV[3]; next } 1' "$#" <"$1" >"$1.1"; mv "$1"{.1,}; }
# https://unix.stackexchange.com/a/331965/312709
# Usage: local myvar="$(config_get myvar)"
function config_get() {
val="$(config_read_file ${CONFIG_FILE} "${1}")";
if [ "${val}" = "__UNDEFINED__" ]; then
val="$(config_read_file ${CONFIG_FILE}.example "${1}")";
fi
printf -- "%s" "${val}";
}
function config_read_file() {
(grep -E "^${2}=" -m 1 "${1}" 2>/dev/null || echo "VAR=__UNDEFINED__") | head -n 1 | cut -d '=' -f 2-;
}
at first I was using the accepted answer's sed solution: https://stackoverflow.com/a/2464883/2683059
however if the value has a / char it breaks
in general it's easy to extract the info with grep and cut:
cat "$FILE" | grep "^${KEY}${DELIMITER}" | cut -f2- -d"$DELIMITER"
to update you could do something like this:
mv "$FILE" "$FILE.bak"
cat "$FILE.bak" | grep -v "^${KEY}${DELIMITER}" > "$FILE"
echo "${KEY}${DELIMITER}${NEWVALUE}" >> "$FILE"
this would not maintain the order of the key-value pairs obviously. add error checking to make sure you don't lose your data.
I have done this:
new_port=$1
sed "s/^port=.*/port=$new_port/" "$CONFIG_FILE" > /yourPath/temp.x
mv /yourPath/temp.x "$CONFIG_FILE"
This will change port= to port=8888 in your config file if you choose 8888 as $1 for example.
Suppose your config file is in below format:
CONFIG_NUM=4
CONFIG_NUM2=5
CONFIG_DEBUG=n
In your bash script, you can use:
CONFIG_FILE=your_config_file
. $CONFIG_FILE
if [ $CONFIG_DEBUG == "y" ]; then
......
else
......
fi
$CONFIG_NUM, $CONFIG_NUM2, $CONFIG_DEBUG is what you need.
After your read the values, write it back will be easy:
echo "CONFIG_DEBUG=y" >> $CONFIG_FILE
Most scripts that parse /proc/cmdline break it up into words and then filter out arguments with a case statement, example:
CMDLINE="quiet union=aufs wlan=FOO"
for x in $CMDLINE
do
»···case $x in
»···»···wlan=*)
»···»···echo "${x//wlan=}"
»···»···;;
»···esac
done
The problem is when the WLAN ESSID has spaces. Users expect to set wlan='FOO
BAR' (like a shell variable) and then get the unexpected result of 'FOO with the above code, since the for loop splits on spaces.
Is there a better way of parsing the /proc/cmdline from a shell script falling short of almost evaling it?
Or is there some quoting tricks? I was thinking I could perhaps ask users to entity quote spaces and decode like so: /bin/busybox httpd -d "FOO%20BAR". Or is that a bad solution?
There are some ways:
cat /proc/PID/cmdline | tr '\000' ' '
cat /proc/PID/cmdline | xargs -0 echo
These will work with most cases, but will fail when arguments have spaces in them. However I do think that there would be better approaches than using /proc/PID/cmdline.
set -- $(cat /proc/cmdline)
for x in "$#"; do
case "$x" in
wlan=*)
echo "${x#wlan=}"
;;
esac
done
Most commonly, \0ctal escape sequences are used when spaces are unacceptable.
In Bash, printf can be used to unescape them, e.g.
CMDLINE='quiet union=aufs wlan=FOO\040BAR'
for x in $CMDLINE; do
[[ $x = wlan=* ]] || continue
printf '%b\n' "${x#wlan=}"
done
Since you want the shell to parse the /proc/cmdline contents, it's hard to avoid eval'ing it.
#!/bin/bash
eval "kernel_args=( $(cat /proc/cmdline) )"
for arg in "${kernel_args[#]}" ; do
case "${arg}" in
wlan=*)
echo "${arg#wlan=}"
;;
esac
done
This is obviously dangerous though as it would blindly run anything that was specified on the kernel command-line like quiet union=aufs wlan=FOO ) ; touch EVIL ; q=( q.
Escaping spaces (\x20) sounds like the most straightforward and safe way.
A heavy alternative is to use some parser, which understand shell-like syntax.
In this case, you may not even need the shell anymore.
For example, with python:
$ cat /proc/cmdline
quiet union=aufs wlan='FOO BAR' key="val with space" ) ; touch EVIL ; q=( q
$ python -c 'import shlex; print shlex.split(None)' < /proc/cmdline
['quiet', 'union=aufs', 'wlan=FOO BAR', 'key=val with space', ')', ';', 'touch', 'EVIL', ';', 'q=(', 'q']
Use xargs -n1:
[centos#centos7 ~]$ CMDLINE="quiet union=aufs wlan='FOO BAR'"
[centos#centos7 ~]$ echo $CMDLINE
quiet union=aufs wlan='FOO BAR'
[centos#centos7 ~]$ echo $CMDLINE | xargs -n1
quiet
union=aufs
wlan=FOO BAR
[centos#centos7 ~]$ xargs -n1 -a /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-3.10.0-862.14.4.el7.x86_64
root=UUID=3260cdba-e07e-408f-93b3-c4e9ff55ab10
ro
consoleblank=0
crashkernel=auto
rhgb
quiet
LANG=en_US.UTF-8
You could do something like the following using bash, which would turn those arguments in to variables like $cmdline_union and $cmdline_wlan:
bash -c "for i in $(cat /proc/cmdline); do printf \"cmdline_%q\n\" \"\$i\"; done" | grep = > /tmp/cmdline.sh
. /tmp/cmdline.sh
Then you would quote and/or escape things just like you would in a normal shell.
In posh:
$ f() { echo $1 - $3 - $2 - $4
> }
$ a="quiet union=aufs wlan=FOO"
$ f $a
quiet - wlan=FOO - union=aufs -
You can define a function and give your $CMDLINE unquoted as an argument to the function. Then you'll invoke shell's parsing mechanisms. Note, that you should test this on the shell it will be working in -- zsh does some funny things with quoting ;-).
Then you can just tell the user to do quoting like in shell:
#!/bin/posh
CMDLINE="quiet union=aufs wlan=FOO"
f() {
while test x"$1" != x
do
case $1 in
union=*) echo ${1##union=}; shift;;
*) shift;;
esac
done
}
f $CMDLINE
(posh - Policy-compliant Ordinary SHell, a shell stripped of any features beyond standard POSIX)
Found here a nice way to do it with awk, unfortunately it will work only with doublequotes:
# Replace spaces outside double quotes with newlines
args=`cat /proc/cmdline | tr -d '\n' | awk 'BEGIN {RS="\"";ORS="\"" }{if (NR%2==1){gsub(/ /,"\n",$0);print $0} else {print $0}}'`
IFS='
'
for line in $args; do
key=${line%%=*}
value=${line#*=}
value=`echo $value | sed -e 's/^"//' -e 's/"$//'`
printf "%20s = %s\n" "$key" "$value"
done