I've written a template engine script that uses cut to extract certain elements from a file, but I want to use grep in place of the cut. Here is the code I have written:
#!/bin/bash
IFS=# #makes # a delimiter.
while read line
do
dataspace=`echo ${line} | cut -d'=' -f1`
value=`echo ${line} | cut -d"=" -f2`
printf -v $dataspace "$value" #make the value stored in value into the name of a dataspace.
done < 'template.vars' #read template.vars for standard input.
skipflag=false #initialize the skipflag to false
while read line #while it is reading standard input one line at a time
Just came to the conclusion that the code blocks system here does not support bash.
Anyway, since stackoverflow isn't letting me put Bash into codeblocks, I am not putting the entire script since it would look nasty. Based on what is currently high-lighted, how would I go about changing the part using the cut command into a line using the grep command?
As has been noted, you should give more information for a better answer. Going with what you have, I would say that awk is a better option than grep
dataspace=$(awk '$0=$1' FS== <<< "$line")
value=$(awk '$0=$2' FS== <<< "$line")
Related
I have very little experience with bash, but am attempting to put together a pipeline that reads in and executes commands line by line from an input file. The input file, called "seeds.txt", is set up like so
program_to_execute -seed_value 4496341759106 -a_bunch_of_arguments c(1,2,3) ; #Short_description_of_run
program_to_execute -seed_value 7502828106749 -a_bunch_of_arguments c(4,5,6) ; #Short_description_of_run
I separated the #Short_descriptions from the commands by a semi-colon (;) since the arguments contain commas (,). When I run the following script I get a "No such file or directory" error
#!/bin/bash
in="${1:-seeds.txt}"
in2=$(cut -d';' -f1 "${in}")
while IFS= read -r SAMP
do
$SAMP
done < $in2
I know that seeds.txt is getting read in fine, so I'm not sure why I'm getting a missing file/directory message. Could anyone point me in the right direction?
Using GNU Parallel it looks like this:
cut -d';' -f1 seeds.txt | parallel
you can try as below with eval...not very safe though, see this for more info
while read line; do eval "$line" ; done < <(cut -d';' -f1 seeds.txt)
Just in case you want to avoid eval
while read -ra line; do command "${line[#]}"; done < <(cut -d';' -f1 seeds.txt)
Note this solution does not work if the program/utility is not an executable within your PATH, e.g. you wan to use a function or an alias. Not sure if the eval solution can do that too. Kudos to the cut solution!
I have a log file with a lot of lines with the following format:
IP - - [Timestamp Zone] 'Command Weblink Format' - size
I want to write a script.sh that gives me the number of times each website has been clicked.
The command:
awk '{print $7}' server.log | sort -u
should give me a list which puts each unique weblink in a separate line. The command
grep 'Weblink1' server.log | wc -l
should give me the number of times the Weblink1 has been clicked. I want a command that converts each line created by the Awk command above to a variable and then create a loop that runs the grep command on the extracted weblink. I could use
while IFS='' read -r line || [[ -n "$line" ]]; do
echo "Text read from file: $line"
done
(source: Read a file line by line assigning the value to a variable) but I don't want to save the output of the Awk script in a .txt file.
My guess would be:
while IFS='' read -r line || [[ -n "$line" ]]; do
grep '$line' server.log | wc -l | ='$variabel' |
echo " $line was clicked $variable times "
done
But I'm not really familiar with connecting commands in a loop, as this is my first time. Would this loop work and how do I connect my loop and the Awk script?
Shell commands in a loop connect the same way they do without a loop, and you aren't very close. But yes, this can be done in a loop if you want the horribly inefficient way for some reason such as a learning experience:
awk '{print $7}' server.log |
sort -u |
while IFS= read -r line; do
n=$(grep -c "$line" server.log)
echo "$line" clicked $n times
done
# you only need the read || [ -n ] idiom if the input can end with an
# unterminated partial line (is illformed); awk print output can't.
# you don't really need the IFS= and -r because the data here is URLs
# which cannot contain whitespace and shouldn't contain backslash,
# but I left them in as good-habit-forming.
# in general variable expansions should be doublequoted
# to prevent wordsplitting and/or globbing, although in this case
# $line is a URL which cannot contain whitespace and practically
# cannot be a glob. $n is a number and definitely safe.
# grep -c does the count so you don't need wc -l
or more simply
awk '{print $7}' server.log |
sort -u |
while IFS= read -r line; do
echo "$line" clicked $(grep -c "$line" server.log) times
done
However if you just want the correct results, it is much more efficient and somewhat simpler to do it in one pass in awk:
awk '{n[$7]++}
END{for(i in n){
print i,"clicked",n[i],"times"}}' |
sort
# or GNU awk 4+ can do the sort itself, see the doc:
awk '{n[$7]++}
END{PROCINFO["sorted_in"]="#ind_str_asc";
for(i in n){
print i,"clicked",n[i],"times"}}'
The associative array n collects the values from the seventh field as keys, and on each line, the value for the extracted key is incremented. Thus, at the end, the keys in n are all the URLs in the file, and the value for each is the number of times it occurred.
I am trying to write a bash script to ease a process with IP information gathering.
Right now I have made a script which runs throught the one column of IP address in multiple files, looks for geo and host information and stores it to a new file.
What would be nice is also to have a script that generates a result from files with a 3 columns - date, time, ip address. Separator is space.
I tried this an that but no. I am a total newbie :)
This is my original script:
#!/usr/bin/env bash
find *.txt -print0 | while read -d $'\0' file;
do
for i in $( cat "$file")
do echo -e "$i,"$( geoiplookup -f "/usr/share/GeoIP/GeoLiteCity.dat" $i | cut -d' ' -f6,8-9)" "$(nslookup $i | grep name | awk '{print $4}')"" >> "res/res-"$file".txt";
done
done
Input file example
2014-03-06 12:13:27 213.102.145.172
2014-03-06 12:18:24 83.177.253.118
2014-03-25 15:42:01 213.102.155.173
2014-03-25 15:55:47 213.101.185.223
2014-03-26 15:21:43 90.130.182.2
Can you please help me on this?
It's not entirely clear what the current code is attempting to do, but here is a hopefully useful refactoring which could be at least a starting point.
#!/usr/bin/env bash
find *.txt -print0 | while read -d $'\0' file;
do
while read date time ip; do
geo=$(geoiplookup -f "/usr/share/GeoIP/GeoLiteCity.dat" "$ip" |
cut -d' ' -f6,8-9)
addr=$(nslookup "$ip" | awk '/name/ {print $4}')
#addr=$(dig +short -x "$ip")
echo "$ip $geo $addr"
done <"$file" >"res/res-$file.txt"
done
My copy of nslookup does not output four fields but I assume that part of your script is correct. The output from dig +short is better suitable for machine processing, so maybe switch to that instead. Perhaps geoiplookup also offers an option to output machine-readable results, or maybe there is an alternative interface which does.
I assume it was a mistake that your script would output partially comma-separated, partially whitespace-separated results, so I changed that, too. Maybe you should use CSV or JSON instead if you intend for other tools to be able to read this output.
Trying to generate a file named res/res-$file.txt will only work if file is not in any subdirectory, so I'm guessing you will want to fix that with basename; or perhaps the find loop should be replaced with a simple for file in *.txt instead.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question appears to be off-topic because it lacks sufficient information to diagnose the problem. Describe your problem in more detail or include a minimal example in the question itself.
Closed 9 years ago.
Improve this question
I am trying to cat a file to create a copy of itself, but at the same time replace some values
my command is:
cat ${FILE} | sed "s|${key}|${value}|g" > ${TEMP_FILE}
However, when I open the temp file, none of the keys have been replaced- just a straight swap. I have echoed the values of key and value and they are correct - they come from an array element.
Yet if I use a plain string not a variable, it works fine for one type of key - i.e:
cat ${FILE} | sed "s|example_key|${value}|g" > ${TEMP_FILE}
The example_key instances within the file are replaced which is what I want.
However, when I try to use my array $key parameter, it does nothing. No idea why :-(
Command usage:
declare -a props
...
....
for x in "${props[#]}"
do
key=`echo "${x}" | cut -d '=' -f 1`
value=`echo "${x}" | cut -d '=' -f 2`
# global replace on the $FILE
cat ${FILE} | sed "s|${key}|${value}|g" > ${TEMP_FILE}
#cat ${FILE} | sed "s|example_key|${value}|g" > ${TEMP_FILE}
done
array elements are stored in the following format: $key=$value
key='echo "${x}" | cut -d '=' -f 1
value='echo "${x}" | cut -d '=' -f 2
Use back-ticks, not single-quotes, if you want to do command substitution.
key=`echo "${x}" | cut -d '=' -f 1`
value=`echo "${x}" | cut -d '=' -f 2`
Also note that as you loop over the series of key=value pairs, you're overwriting your temp file each time, using only one substitution applied to the original file.. So after the loop is finished, the best you can hope for is that only the last substitution will be applied.
I'd also suggest not doing this in multiple passes -- do it by passing multiple expressions to sed:
for x in "${props[#]}" ; do
subst="$subst -e 's=$x=g'"
done
sed $subst "${FILE}" > "${TEMP_FILE}"
I'm using a trick, by using = as the delimiter for the sed substitution expression, we don't have to separate the key from the value. The command simply becomes:
sed -e 's=foo=1=g' -e 's=bar=2=g' "${FILE}" > "${TEMP_FILE}"
Thanks to #BillKarwin for spotting the crux of the problem: each iteration of the loop wipes out the previous iterations' replacements, because the result of a single key-value pair replacement replaces the entire output file every time.
Try the following:
declare -a props
# ...
cp "$FILE" "$TEMP_FILE"
for x in "${props[#]}"; do
IFS='=' read -r key value <<<"$x"
sed -i '' "s|${key}|${value}|g" "${TEMP_FILE}"
done
Copies the input file to the output file first, then replaces the output file in-place (using sed's -i option) in every iteration of the loop.
I also streamlined the code to parse each line into key and value, using read.
Also note that I consistently double-quoted all variable references.
#anubhava makes a good general point: depending on the variable values, a different regex delimiter may be needed (in your case: if the keys or values contained '|', you couldn't use '|' to delimit the regexes).
Update: #BillKarwin makes a good point: performing the replacements one by one, in a loop, is inefficient.
Here's a one-liner that avoids loops altogether:
sed -f <(awk -F'=' '{ if ($0) print "s/" $1 "/" substr($0, 1+length($1)) "/g" }' \
"$FILE") "$FILE" > "$TEMP_FILE"
Uses awk to build up the entire set of substitution commands for sed (one per line).
Then feeds the result via process substitution as a command file to sed with -f.
Handles the case where values have embedded = chars. correctly.
I have a configuration file that contains lines like "hallo;welt;" and i want to do a grep on this file.
Whenever i try something like grep "$1;$2" my.config or echo "$1;$2 of even line="$1;$2" my script fails with something like:
: command not found95: line 155: =hallo...
How can i tell bash to ignore ; while evaluating "..." blocks?
EDIT: an example of my code.
# find entry
$line=$(grep "$1;$2;" $PERMISSIONSFILE)
# splitt line
reads=$(echo $line | cut -d';' -f3)
writes=$(echo $line | cut -d';' -f4)
admins=$(echo $line | cut -d';' -f5)
# do some stuff on the permissions
# replace old line with new line
nline="$1;$2;$reads;$writes;$admins"
sed -i "s/$line/$nline/g" $TEMPPERM
my script should be called like this: sh script "table" "a.b.*.>"
EDIT: another, simpler example
$test=$(grep "$1;$2;" temp.authorization.config)
the temp file:
table;pattern;read;write;stuff
the call sh test.sh table pattern results in: : command not foundtable;pattern;read;write;stuff
Don't use $ on the left side of an assignment in bash -- if you do it'll substitute the current value of the variable rather than assigning to it. That is, use:
test=$(grep "$1;$2;" temp.authorization.config)
instead of:
$test=$(grep "$1;$2;" temp.authorization.config)
Edit: also, variable expansions should be in double-quotes unless there's a good reason otherwise. For example, use:
reads=$(echo "$line" | cut -d';' -f3)
instead of:
reads=$(echo $line | cut -d';' -f3)
This doesn't matter for semicolons, but does matter for spaces, wildcards, and a few other things.
A ; inside quotes has no meaning at all for bash. However, if $1 contains a doublequote itself, then you'll end up with
grep "something";$2"
which'll be parsed by bash as two separate commands:
grep "something" ; other"
^---command 1----^ ^----command 2---^
Show please show exactly what your script is doing around the spot the error is occurring, and what data you're feeding into it.
Counter-example:
$ cat file.txt
hello;welt;
hello;world;
hell;welt;
$ cat xx.sh
grep "$1;$2" file.txt
$ bash -x xx.sh hello welt
+ grep 'hello;welt' file.txt
hello;welt;
$
You have not yet classified your problem accurately.
If you try to assign the result of grep to a variable (like I do) your example breaks.
Please show what you mean. Using the same data file as before and doing an assignment, this is the output I get:
$ cat xx.sh
grep "$1;$2" file.txt
output=$(grep "$1;$2" file.txt)
echo "$output"
$ bash -x xx.sh hello welt
+ grep 'hello;welt' file.txt
hello;welt;
++ grep 'hello;welt' file.txt
+ output='hello;welt;'
+ echo 'hello;welt;'
hello;welt;
$
Seems to work for me. It also demonstrates why the question needs an explicit, complete, executable, minimal example so that we can see what the questioner is doing that is different from what people answering the question think is happening.
I see you've provided some sample code:
# find entry
$line=$(grep "$1;$2;" $PERMISSIONSFILE)
# splitt line
reads=$(echo $line | cut -d';' -f3)
writes=$(echo $line | cut -d';' -f4)
admins=$(echo $line | cut -d';' -f5)
The line $line=$(grep ...) is wrong. You should omit the $ before line. Although it is syntactically correct, it means 'assign to the variable whose name is stored in $line the result of the grep command'. That is unlikely to be what you had in mind. It is, occasionally, useful. However, those occasions are few and far between, and only for people who know what they're doing and who can document accurately what they're doing.
For safety if nothing else, I would also enclose the $line values in double quotes in the echo lines. It may not strictly be necessary, but it is simple protective programming.
The changes lead to:
# find entry
line=$(grep "$1;$2;" $PERMISSIONSFILE)
# split line
reads=$( echo "$line" | cut -d';' -f3)
writes=$(echo "$line" | cut -d';' -f4)
admins=$(echo "$line" | cut -d';' -f5)
The rest of your script was fine.
It seems like you are trying to read a semicolon-delimited file, identify a line starting with 'table;pattern;' where table is a string you specify and pettern is a regular expression grep will understand. Once the line is identified you wish to replaced the 3rd, 4th and 5th fields with different data and write the updated line back to the file.
Does this sound correct?
If so, try this code
#!/bin/bash
in_table="$1"
in_pattern="$2"
file="$3"
while IFS=';' read -r -d$'\n' tuple pattern reads writes admins ; do
line=$(cut -d: -f1<<<"$tuple")
table=$(cut -d: -f2<<<"$tuple")
# do some stuff with the variables
# e.g., update the values
reads=1
writes=2
admins=12345
# replace the old line with the new line
sed -i'' -n $line'{i\
'"$table;$pattern;$reads;$writes;$admins"'
;d;}' "$file"
done < <(grep -n '^'"${in_table}"';'"${in_pattern}"';' "${file}")
I chose to update by line number here to avoid problems of unknown characters in the left hand of the substitution.