I am trying to extract the number of lines from a file, and then use it in a variable. However, it keeps passing the file name, not the number of lines. I read through this question, but the examples are not working.
for i in $BASE/$TEMPLATE_DIR-$i$PRUNING_METHOD/*.txt
do
NUM_LINES=$(wc -l < $i)
echo $NUM_LINES
UPLOAD_CMD="sh sshpass_setup_verification.sh $EXP_ID-$i$PRUNING_METHOD__$NUM_LINES__features";
echo "$UPLOAD_CMD"
break 1;
done
Prints:
15 #the correct number of lines
sh sshpass_setup_verification.sh di-60sec-max/TemplateUser1.txt #where TemplateUser1.txt is the name of the file
Should print:
15
sh sshpass_setup_verification.sh di-60sec-max__15__features
A summary of what people are telling you in the comments:
for i in "${BASE}/${TEMPLATE_DIR}-${i}${PRUNING_METHOD}"/*.txt
do
num_lines=$(wc -l < "$i")
echo "$num_lines"
upload_cmd="sh sshpass_setup_verification.sh ${EXP_ID}-${i}${PRUNING_METHOD}__${num_lines}__features"
echo "$upload_cmd"
break
done
The key thing here is using double quotes around your parameter expansions and curly braces to disambiguate in situations where characters such as _ could be interpreted as part of a variable name but shouldn't. I've added them in several places where they aren't strictly needed but they do no harm.
I've also changed your variable names to lowercase. This will help you one day when you decide to have a variable called PATH and suddenly all your commands stop working.
Related
I've written a script to go through all the files in the directory the script is located in, identify if a file name contains a certain string and then modify the filename. When I run this script, the files that are supposed to be modified are disappearing. It appears my usage of the mv command is incorrect and the files are likely going to an unknown directory.
#!/bin/bash
string_contains="dummy_axial_y_position"
string_dontwant="dummy_axial_y_position_time"
file_extension=".csv"
for FILE in *
do
if [[ "$FILE" == *"$string_contains"* ]];then
if [[ "$FILE" != *"$string_dontwant"* ]];then
filename= echo $FILE | head -c 15
combined_name="$filename$file_extension"
echo $combined_name
mv $FILE $combined_name
echo $FILE
fi
fi
done
I've done my best to go through the possible errors I've made in the MV command but I haven't had any success so far.
There are a couple of problems and several places where your script can be improved.
filename= echo $FILE | head -c 15
This pipeline runs echo $FILE adding the variable filename having the null string as value in its environment. This value of the variable is visible only to the echo command, the variable is not set in the current shell. echo does not care about it anyway.
You probably want to capture the output of echo $FILE | head -c 15 into the variable filename but this is not the way to do it.
You need to use command substitution for this purpose:
filename=$(echo $FILE | head -c 15)
head -c outputs only the first 15 characters of the input file (they can be on multiple lines but this does not happen here). head is not the most appropriate way for this. Use cut -c-15 instead.
But for what you need (extract the first 15 characters of the value stored in the variable $FILE), there is a much simpler way; use a form of parameter expansion called "substring expansion":
filename=${FILE:0:15}
mv $FILE $combined_name
Before running mv, the variables $FILE and $combined_name are expanded (it is called "parameter expansion"). This means that the variable are replaced by their values.
For example, if the value of FILE is abc def and the value of combined_name is mnp opq, the line above becomes:
mv abc def mnp opq
The mv command receives 4 arguments and it attempts to move the files denoted by the first three arguments into the directory denoted by the fourth argument (and it probably fails).
In order to keep the values of the variables as single words (if they contain spaces), always enclose them in double quotes. The correct command is:
mv "$FILE" "$combined_name"
This way, in the example above, the command becomes:
mv "abc def" "mnp opq"
... and mv is invoked with two arguments: abc def and mnp opq.
combined_name="$filename$file_extension"
There isn't any problem in this line. The quotes are simply not needed.
The variables filename and file_extension are expanded (replaced by their values) but on assignments word splitting is not applied. The value resulted after the replacement is the value assigned to variable combined_name, even if it contains spaces or other word separator characters (spaces, tabs, newlines).
The quotes are also not needed here because the values do not contain spaces or other characters that are special in the command line. They must be quoted if they contain such characters.
string_contains="dummy_axial_y_position"
string_dontwant="dummy_axial_y_position_time"
file_extension=".csv"
It is not not incorrect to quote the values though.
for FILE in *
do
if [[ "$FILE" == *"$string_contains"* ]];then
if [[ "$FILE" != *"$string_dontwant"* ]]; then
This is also not wrong but it is inefficient.
You can use the expression from the if condition directly in the for statement (and get rid of the if statement):
for FILE in *"$string_contains"*; do
if [[ "$FILE" != *"$string_dontwant"* ]]; then
...
If you have read and understood the above (and some of the linked documentation) you will be able to figure out yourself where were your files moved :-)
What is wrong with these lines?
FASTAQ1 = "/home/mdb1c20/my_onw_NGS_pipeline/files/fastq/W2115220_S5_L001_R1_001.fastq"
FASTAQ2 = "/home/mdb1c20/my_onw_NGS_pipeline/files/fastq/W2115220_S5_L001_R2_001.fastq"
DIRECTORY = "/home/mdb1c20/my_onw_NGS_pipeline/scripts_my_first_NGS"
They are in a .conf file with other similar variables. The only difference is that these three are created with printf
printf 'FASTAQ1 = "%s"\n' "$FASTA1" >> "$DIRECTORY/$filename1/scripts/shortcut.config"
printf 'FASTAQ2 = "%s"\n' "$FASTA2" >> "$DIRECTORY/$filename1/scripts/shortcut.config"
printf 'DIRECTORY = "%s"\n' "$DIRECTORY" >> "$DIRECTORY/$filename1/scripts/shortcut.config"
When a script I am using open the .confi file its says that FASTAQ1: command not found
Apart from these three, the rest of variables were created manually in a archive .conf file but the script add these three on the go. The only thing I haven't tried because I don't know how to do that is to remove the white spaces before and after the equal simbol?
In bash, this:
var = value
is not the same as this:
var=value
The first example runs a command named "var" and passes it two arguments "=" and "value".
The second example sets a variable called "var" to "value".
It was hard to find this detail in the Bash manual, but the difference is between simple commands and assigning to variables, or shell parameters.
If you intended to source your configuration file, you should have used printf this way:
printf 'FASTAQ1=%q\n' "$FASTA1" >> "$DIRECTORY/$filename1/scripts/shortcut.config"
This allows you to store the value safely regardless if it has spaces or quotes.
The error was caused by the assignment command being interpretted as a simple command instead because of the spaces around the equal sign.
Alternatively for Bash 4.4+, you can use #Q expansion:
echo "FASTAQ1=${FASTA1#Q}" >> "$DIRECTORY/$filename1/scripts/shortcut.config"
I need to take specific variables from one script and use them in a different script.
Example:
Original script:
VARA=4 # Some description of VARA
VARB=6 # Some description of VARB
SOMEOTHERVAR="Foo"
/call/to/some/program
I want to write a second script that needs VARA and VARB, but not SOMEOTHERVAR or the call to the program.
I can already do:
eval $(grep 'VARA=' origscript.sh)
eval $(grep 'VARB=' origscript.sh)
This seems to work, but when I want to do both, like this, it only sets the first:
eval $(grep 'VAR[AB]=' origscript.sh)
because it seems to concatenate the two lines that grep returns. (Which probably means that the comments save the first assignments.)
Put quotes around it, so that the newlines in the output of grep will not be turned into spaces.
eval "$(grep 'VAR[AB]=' origscript.sh)"
I am trying to modify the variables of one shell script, using another script. This is what I have so far:
script1.sh
#!/bin/bash
var=123.45.67.890
script2.sh
#!/bin/bash
currVar=000.00.00.000
. /./script1.sh
var=$currVar
I understand that I am not modifying Script 1 here, but simply temporarily modifying var. How can I modify this var in script 1, via script 2?
Solution
. /./script1.sh
echo $var | sed "s/$var/$currVar/g" /./script1.sh > "temp.txt" && mv temp.txt /./script1.sh
Just use sed in 2nd script (script2.sh) as
currVar="000.00.00.000"
sed -r -i.bak "s/var=([[:graph:]]+)/var=$currVar/" script1.sh
var=000.00.00.000
where [[:graph:]] is a character class for [[:alnum:]] & [[:punct:]] to match values for var with printable characters/meta-characters.
Since you mentioned it is a proper IP address, use a proper regEx as
sed -r "s/(\b[0-9]{1,3}\.){3}[0-9]{1,3}\b/$currVar/" script1.sh
var=000.00.00.000
(\b[0-9]{1,3}\.){3}[0-9]{1,3} implies match 3 groups consisting of digits from 0-9, which each group could have from 1-3 digits each, preceded by a dot . and the 4th group also the same as the last. Remember the each group I am mentioning represents an IP octet
Normally, variables in scripts have local scope. If you export the variable, you can extend this scope to include any child processes. However, it looks like you might want to use the modified value when script1.sh runs. If that is the case, you can use the new var value as an input to script1.sh when you run it.
if [[ -z "$1" ]];
then
var=$1
else
var=123.45.67.890
This will check if you gave any parameters when you ran script1.sh, and if you did, then it should set your var equal to this value instead of the default IP.
I'm constructing a bash script file a bit at a time. I'm learning as I
go. But I can't find anything online to help me at this point: I need to
extract a substring from a large string, and the two methods I found using ${} (curly brackets) just won't work.
The first, ${x#y}, doesn't do what it should.
The second, ${x:p} or ${x:p:n}, keeps reporting bad substitution.
It only seems to work with constants.
The ${#x} returns a string length as text, not as a number, meaning it does not work with either ${x:p} or ${x:p:n}.
Fact is, it's seems really hard to get bash to do much math at all. Except for the for statements. But that is just counting. And this isn't a task for a for loop.
I've consolidated my script file here as a means of helping you all understand what it is that I am doing. It's for working with PureBasic source files, but you only have to change the grep's "--include=" argument, and it can search other types of text files instead.
#!/bin/bash
home=$(echo ~) # Copy the user's path to a variable named home
len=${#home} # Showing how to find the length. Problem is, this is treated
# as a string, not a number. Can't find a way to make over into
# into a number.
echo $home "has length of" $len "characters."
read -p "Find what: " what # Intended to search PureBasic (*.pb?) source files for text matches
grep -rHn $what $home --include="*.pb*" --exclude-dir=".cache" --exclude-dir=".gvfs" > 1.tmp
while read line # this checks for and reads the next line
do # the closing 'done' has the file to be read appended with "<"
a0=$line # this is each line as read
a1=$(echo "$a0" | awk -F: '{print $1}') # this gets the full path before the first ':'
echo $a0 # Shows full line
echo $a1 # Shows just full path
q1=${line#a1}
echo $q1 # FAILED! No reported problem, but failed to extract $a1 from $line.
q1=${a0#a1}
echo $q1 # FAILED! No reported problem, but failed to extract $a1 from $a0.
break # Can't do a 'read -n 1', as it just reads 1 char from the next line.
# Can't do a pause, because it doesn't exist. So just run from the
# terminal so that after break we can see what's on the screen .
len=${#a1} # Can get the length of $a1, but only as a string
# q1=${line:len} # Right command, wrong variable
# q1=${line:$len} # Right command, right variable, but wrong variable type
# q1=${line:14} # Constants work, but all $home's aren't 14 characters long
done < 1.tmp
The following works:
x="/home/user/rest/of/path"
y="~${x#/home/user}"
echo $y
Will output
~/rest/of/path
If you want to use "/home/user" inside a variable, say prefix, you need to use $ after the #, i.e., ${x#$prefix}, which I think is your issue.
The hejp I got was most appreciated. I got it done, and here it is:
#!/bin/bash
len=${#HOME} # Showing how to find the length. Problem is, this is treated
# as a string, not a number. Can't find a way to make over into
# into a number.
echo $HOME "has length of" $len "characters."
while :
do
echo
read -p "Find what: " what # Intended to search PureBasic (*.pb?) source files for text matches
a0=""; > 0.tmp; > 1.tmp
grep -rHn $what $home --include="*.pb*" --exclude-dir=".cache" --exclude-dir=".gvfs" >> 0.tmp
while read line # this checks for and reads the next line
do # the closing 'done' has the file to be read appended with "<"
a1=$(echo $line | awk -F: '{print $1}') # this gets the full path before the first ':'
a2=${line#$a1":"} # renove path and first colon from rest of line
if [[ $a0 != $a1 ]]
then
echo >> 1.tmp
echo $a1":" >> 1.tmp
a0=$a1
fi
echo " "$a2 >> 1.tmp
done < 0.tmp
cat 1.tmp | less
done
What I don't have yet is an answer as to whether variables can be used in place of constants in the dollar-sign, curly brackets where you use colons to mark that you want a substring of that string returned, if it requires constants, then the only choice might be to generate a child scriot using the variables, which would appear to be constants in the child, execute there, then return the results in an environmental variable or temporary file. I did stuff like that with MSDOS a lot. Limitation here is that you have to then make the produced file executable as well using "chmod +x filename". Or call it using "/bin/bash filename".
Another bash limitation found it that you cannot use "sudo" in the script without discontinuing execution of the present script. I guess a way around that is use sudo to call /bin/bash to call a child script that you produced. I assume then that if the child completes, you return to the parent script where you stopped at. Unless you did "sudo -i", "sudo -su", or some other variation where you become super user. Then you likely need to do an "exit" to drop the super user overlay.
If you exit the child script still as super user, would typing "exit" but you back to completing the parent script? I suspect so, which makes for some interesting senarios.
Another question: If doing a "while read line", what can you do in bash to check for a keyboard key press? The "read" option is already taken while in this loop.