This question already has answers here:
When to wrap quotes around a shell variable?
(5 answers)
Closed 4 years ago.
I have a file which was created by another scripts with file names. Only thing is that it contains * for similar file names. now i want to iterate a for loop on this file to execute a few operation but i want to do it on different servers for that i simply need exact same name from that file. For this, i used grep and passed it to 'for' loop for one by one execution. But instead before going for the loop it actually executes it on current server itself and gives wrong output. I don't know what i'm missing.
# node_name=auxiliary;grep -F $node_name /opt/log_automation/ME_log_sheet.csv|awk -F "," '{print $2}'
/opt/tpa/logs/install_SPS.*.log
/opt/tpa/logs/localization.log
/opt/tpa/logs/SPI.*.log
/opt/tpa/logs/TPA_system.log
#
This seems to be working as expected as i want * to be present in the name. Now if i put it in a variable or pass it to the loop it expands and gives actual present file which i don't want. I want it to return simply the name with * on it.
# node_name=auxiliary;myvar=$(grep -F $node_name /opt/log_automation/ME_log_sheet.csv|awk -F "," '{print $2}');echo $myvar
/opt/tpa/logs/install_SPS.2019.02.09-12:12.log /opt/tpa/logs/localization.log /opt/tpa/logs/SPI.2019.02.09-12:10.log /opt/tpa/logs/TPA_system.log
# node_name=auxiliary;for i in $(grep -F $node_name /opt/log_automation/ME_log_sheet.csv|awk -F "," '{print $2}');do echo $i;done
/opt/tpa/logs/install_SPS.2019.02.09-12:12.log
/opt/tpa/logs/localization.log
/opt/tpa/logs/SPI.2019.02.09-12:10.log
/opt/tpa/logs/TPA_system.log
Please help me resolve this. What am i doing wrong?
P.S.: '-F' with grep i added after a few changes. It wasn't working either way.
Quote all variables:
node_name=auxiliary
for i in "$(grep -F $node_name /opt/log_automation/ME_log_sheet.csv|awk -F "," '{print $2}')"
do
echo "$i"
done
I think you should be using set -f to disable filename expansion : https://unix.stackexchange.com/questions/333867/what-does-set-f-in-korn-shell-do
However as triplee suggest, using such characters (eg asterisk) in filenames is a very bad practice..
Related
I'm trying to temporarily disable dhcp on all connections in a computer using bash, so I need the process to be reversible. My approach is to comment out lines that contain BOOTPROTO=dhcp, and then insert a line below it with BOOTPROTO=none. I'm not sure of the correct syntax to make sed understand the line number stored in the $insertLine variable.
fileList=$(ls /etc/sysconfig/network-scripts | grep ^ifcfg)
path="/etc/sysconfig/network-scripts/"
for file in $fileList
do
echo "looking for dhcp entry in $file"
if [ $(cat $path$file | grep ^BOOTPROTO=dhcp) ]; then
echo "disabling dhcp in $file"
editLine=$(grep -n ^BOOTPROTO=dhcp /$path$file | cut -d : -f 1 )
#comment out the original dhcp value
sed -i "s/BOOTPROTO=dhcp/#BOOTPROTO=dhcp/g" $path$file
#insert a line below it with value of none.
((insertLine=$editLine+1))
sed "$($insertLine)iBOOTPROTO=none" $path$file
fi
done
Any help using sed or other stream editor greatly appreciated. I'm using RHEL 6.
The sed editor should be able to do the job, without having to to be combine bash, grep, cat, etc. Easier to test, and more reliable.
The whole scripts can be simplified to the below. It performs all operations (substitution and the insert) with a single pass using multiple sed scriptlets.
#! /bin/sh
for file in $(grep -l "^BOOTPROTO=dhcp" /etc/sysconfig/network-scripts/ifcfg*) ; do
sed -i -e "s/BOOTPROTO=dhcp/#BOOTPROTO=dhcp/g" -e "/BOOTPROTO=dhcp/i BOOTPROTO=none" $file
done
As side note consider NOT using path as variable to avoid possible confusion with the 'PATH` environment variable.
Writing it up, your attempt with the following fails:
sed "$($insertLine)iBOOTPROTO=none" $path$file
because:
$($insertLine) encloses $insertLIne in a command substitution which when $insertLIne is evaluated it returns a number which is not a command generating an error.
your call to sed does not include the -i option to edit the file $path$file in place.
You can correct the issues with:
sed -i "${insertLine}i BOOTPROTO=none" $path$file
Which is just sed - i (edit in place) and Ni where N is the number of the line to insert followed by the content to insert and finally what file to insert it in. You add ${..} to insertLine to protect the variable name from the i that follows and then the expression is double-quoted to allow variable expansion.
Let me know if you have any further questions.
(and see dash-o's answer for refactoring the whole thing to simply use sed to make the change without spawning 10 other subshells)
This question already has answers here:
How do I split a string on a delimiter in Bash?
(37 answers)
Closed 6 years ago.
I have a .txt file that has a list containing a hash and a password so it looks likes this:
00608cbd5037e18593f96995a080a15c:9e:hoboken
00b108d5d2b5baceeb9853b1ad9aa9e5:1c:wVPZ
Out of this txt file I need to extract only the passwords and add them in a new text file so that I have a list that would look like this:
hoboken
wVPZ
etc
etc
etc
etc
How to do this in bash, a scripting language or simply with a text editor?
Given your examples, the following use of cut would achieve what you want:
cut -f3 -d':' /folder/file >> /folder/result
The code above would delete anything before (and including) the second colon : on each line, which would work on your case, given your examples. The result is stored on /folder/result.
Edit: I edited this answer to make it simpler.
I suggest to use awk to get always last column from your file:
awk -F ':' '{print $NF}' file
Output:
hoboken
wVPZ
With sed, to remove the string up to ::
sed 's/.*://' file
You could also use grep:
$ grep -o [^:]*$ file
hoboken
wVPZ
-o print only matching part
[^:] anything but :
* all matching characters
$ end of record
I have a bash script that I'm trying to put together that finds all of the images in a folder and then puts the names of those files into a pre-formatted CSV. I actually have the more complicated parts of it figured out and working well... I'm stuck on a really basic part. I have a variable that I need to increment for each file found, simple enough right? I've tried a bunch of different things and cannot for the life of me get it to increment. Here's the script I'm working with:
EDITED to show less context
i=0
find "$(pwd)" -maxdepth 1 -type f -exec file {} \; | awk -F: '{if ($2 ~/image/) print $1}' | grep -o -E '[^/]*$' | sed -e "s/^/$((++i))/" > "$(pwd)/inventory-$(date +%Y%m%d)-$(date +%I%M).csv"
I've tried incrementing it with i++, i=+1, i=i+1 as well as putting the dollar sign before the different iterations of the i variable... nothing seems to actually increment the variable. My best guess is that this isn't a true loop so it doesn't save the changes to the variable? Any guidance would be greatly appreciated!
The $((++i)) is performed by your shell. But the shell executes this line only once. This line cannot do what you need.
I would increment in inside awk, print it alongside the file name, and then combine output in the further commands.
Having a bit of bother trying to get the following to work.
I have a file containing hostname:timestamp as below:
hostname1:1445072150
hostname2:1445076364
I am trying to create a bash script that will query this file (using a cron job) to check if the timestamp is over 1 hour old and if so, remove the line.
Below is what I have so far but it doesn't appear to be removing the line in the file.
#!/bin/bash
hosts=/tmp/hosts
current_timestamp=$(date +%s)
while read line; do
hostname=`echo $line | sed -e 's/:.*//g'`
timestamp=`echo $line | cut -d ":" -f 2`
diff=$(($current_timestamp-$timestamp))
if [ $diff -ge 3600 ]; then
echo "$hostname - Timestamp over an hour old. Deleting line."
sed -i '/$hostname/d' $hosts
fi
done <$hosts
I have managed to get the timestamp part working correctly in identifying hosts that are over an hour old but having trouble removing the time from the file.
I suspect it may be due to the while loop keeping the file open but not 100% sure how to work around it. Also tried making a copy of the file and editing that but still nothing.
ALTERNATIVELY: If there is a better way to get this to work and produce the same result, I am open to suggestions :)
Any help would be much appreciated.
Cheers
The problem in your script was just this line:
sed -i '/$hostname/d' $hosts
Variables inside single-quotes are not expanded to their values,
so the command is trying to replace literally "$hostname", instead of its value. If you replace the single-quotes with double-quotes,
the variable will get expanded to its value, which is what you need here:
sed -i "/$hostname/d" $hosts
There are improvements possible:
#!/bin/bash
hosts=/tmp/hosts
current_timestamp=$(date +%s)
while read line; do
set -- ${line/:/ }
hostname=$1
timestamp=$2
((diff = current_timestamp - timestamp))
if ((diff >= 3600)); then
echo "$hostname - Timestamp over an hour old. Deleting line."
sed -i "/^$hostname:/d" $hosts
fi
done <$hosts
The improvements:
More strict pattern in the sed command, to make it more robust and to avoid some potential errors
Simpler way to extract hostname part and timestamp part without any sub-shells
Simpler arithmetic operations by enclosing within ((...))
You ask for alternatives — use awk:
awk -F: -v ts=$(date +%s) '$2 <= ts-3600 { next }' $hosts > $hosts.$$
mv $hosts.$$ $hosts
The ts=$(date +%s) sets the awk variable ts to the value from date. The script skips any lines where the value in the second column (after the first colon) is smaller than the threshold. You could do the subtraction once in a BEGIN block if you wanted to. Decide whether <= or < is correct for your purposes.
If you need to know which lines are deleted, you can add
printf "Deleting %s - timestamp %d older than %d\n", $1, $2, (ts-3600) >/dev/stderr;
before the next to print the information on standard error. If you must write that to standard output, then you need to arrange for retained lines to be written to a file with print > file as an alternative action after the filter condition (passing -v file="$hosts.$$" as another pair of arguments to awk). The tweaks that can be made are endless.
If the file is of any significant size, it will be quicker to copy the relevant subsection of the file once to a temporary file and then to the final file than to edit the file in place multiple times as in the original code. If the file is small enough, there isn't a problem.
This question already has answers here:
Parsing variables from config file in Bash
(7 answers)
Closed 8 years ago.
I'm new to bash scripting, my task is to read a line one of my configuration file,
this is the way....I want...
let's assume "sample.conf" is the file, in file there is a line..
like this,
webURL: "http://www.sampledomain:8080"
What I want is , I need to get the value of webURL, that means "http://www.sampledomain:8080" then if the URL is not equal to "http://www.sampledomain:8080" this I need to changed it with the correct value. Hope u will help me, any help would be greatly appreciated.
Thank You.
Use awk like this:
URL=$(awk -F\" '/^webURL/{print $2}' sample.conf)
echo $URL
http://www.sampledomain:8080
The $() means "put the result of running the enclosed command into the variable URL". The awk then sets the separator for fields to the double quote sign. It then looks in your file for a line that starts with "webURL" and when it finds it, it outputs everything between the second pair of double quotes, i.e. field 2 on the line.
EDITED to answer further question
If you want to parse out a subscribe_key value from a java file called thing.java, that looks like this:
subscribe_key = 'sub-f-xx-xx-xx-xx-xx';
you can use this:
key=$(awk -F\' '/^subscribe_key/{print $2}' thing.java)
echo $key
sub-f-xx-xx-xx-xx-xx
Note that I changed the double quote after -F to single quote to tell awk that fields are now separated by single quotes.
Note that if you have values that are marked by single quotes and double quotes IN THE SAME FILE you can tell awk to use either single quotes or double quotes as field separators like this:
value=$(awk -F"\'|\"" '/^subscribe_key/{print $2}' yourFile)
echo $value
sub-f-xx-xx-xx-xx-xx
value=$(awk -F"\'|\"" '/^webURL/{print $2}' yourFile)
echo $value
http://www.sampledomain:8080
url=`grep webURL sample.conf | sed -e 's/webURL: //'`
using sed
sed 's#\(webURL: \).*#\1 "http://www.sampledomain:8080"#' file