I have a couple of files with certain keywors (one is MAT1).
For this key word i like to read the ID corresponding to it, put this together with the filename into an array.
I tried the following (I am not very familiar with bash programming):
#!/bin/bash
Num=0
arr=( $(find . -name '*.mat' | sort) )
for i in "${arr[#]}"
do
file=$(basename "${i}")
while read -r line
do
name="$line"
IFS=' ' read -r -a array <<< "$line"
for index in "${!array[#]}"
do
if [ ${array[index]} == "MAT1" ]
then
out[$num] = "${array[index+1]} $file "
let num++
#printf "%-32s %8i\n" "$file" "${array[index+1]}"
fi
done
done < "$i"
done
With this get the message
make_mat_list.bsh: line 21: out[0]: command not found
make_mat_list.bsh: line 21: out[1]: command not found
What is wrong here?
bash is white-space sensitive, your line below cannot have spaces.
out[$num] = "${array[index+1]} $file "
As for the reason for the error, the shell treats that particular line as the first word being a command out[$num] i.e. out[1]..etc and rest of it as arguments to it = and "${array[index+1]} $file ", which does not make any sense. Remove the spaces and do jsut
out[$num]="${array[index+1]} $file"
Related
I'am trying to get the first character of each string using regex and BASH_REMATCH in shell script.
My input text file contain :
config_text = STACK OVER FLOW
The strings STACK OVER FLOW must be uppercase like that.
My output should be something like this :
SOF
My code for now is :
var = config_text
values=$(grep $var test_file.txt | tr -s ' ' '\n' | cut -c 1)
if [[ $values =~ [=(.*)]]; then
echo $values
fi
As you can see I'am using tr and cut but I'am looking to replace them with only BASH_REMATCH because these two commands have been reported in many links as not functional on MacOs.
I tried something like this :
var = config_text
values=$(grep $var test_file.txt)
if [[ $values =~ [=(.*)(\b[a-zA-Z])]]; then
echo $values
fi
VALUES as I explained should be :
S O F
But it seems \b does not work on shell script.
Anyone have an idea how to get my desired output with BASH_REMATCH ONLY.
Thanks in advance for any help.
A generic BASH_REMATCH solution handling any number of words and any separator.
local input="STACK OVER FLOW" pattern='([[:upper:]]+)([^[:upper:]]*)' result=""
while [[ $input =~ $pattern ]]; do
result+="${BASH_REMATCH[1]::1}${BASH_REMATCH[2]}"
input="${input:${#BASH_REMATCH[0]}}"
done
echo "$result"
# Output: "S O F"
Bash's regexes are kind of cumbersome if you don't know how many words there are in the input string. How's this instead?
config_text="STACK OVER FLOW"
sed 's/\([^[:space:]]\)[^[:space:]]*/\1/g' <<<"$config_text"
First Put a valid shebang and paste your script at https://shellcheck.net for validation/recommendation.
With the assumption that the line starts with config and ends with FLOW e.g.
config_text = STACK OVER FLOW
Now the script.
#!/usr/bin/env bash
values="config_text = STACK OVER FLOW"
regexp="config_text = ([[:upper:]]{1})[^ ]+ ([[:upper:]]{1})[^ ]+ ([[:upper:]]{1}).+$"
while IFS= read -r line; do
[[ "$line" = "$values" && "$values" =~ $regexp ]] &&
printf '%s %s %s\n' "${BASH_REMATCH[1]}" "${BASH_REMATCH[2]}" "${BASH_REMATCH[3]}"
done < test_file.txt
If there is Only one line or the target string/pattern is at the first line of the test_file.txt, the while loop is not needed.
#!/usr/bin/env bash
values="config_text = STACK OVER FLOW"
regexp="config_text = ([[:upper:]]{1})[^ ]+ ([[:upper:]]{1})[^ ]+ ([[:upper:]]{1}).+$"
IFS= read -r line < test_file.txt
[[ "$line" = "$values" && "$values" =~ $regexp ]] &&
printf '%s %s %s\n' "${BASH_REMATCH[1]}" "${BASH_REMATCH[2]}" "${BASH_REMATCH[3]}"
Make sure you have and running/using Bashv4+ since MacOS, defaults to Bashv3
See How can I read a file (data stream, variable) line-by-line (and/or field-by-field)?
Another option rather than bash regex would be to utilize bash parameter expansion substring ${parameter:offset:length} to extract the desired characters:
$ read -ra arr <text.file ; printf "%s%s%s\n" "${arr[2]:0:1}" "${arr[3]:0:1}" "${arr[4]:0:1}"
SOF
I have problem with url formatting in bash script. In below code url request:
text="$(lynx --dump https://address/"${array[${i}]}")"
returns HTTP Error 400. The request URL is invalid. I assume that on
"${array[${i}]}"
is something wrong in url part. But I can't figure out what is right format.
#!/bin/bash
saveIFS="$IFS"
IFS=$'\n'
array=($(<words))
IFS="$saveIFS"
elements=${#array[#]}
for (( i=0;i<$elements;i++))
do
text="$(lynx --dump https://address/"${array[${i}]}")"
echo "$text" >> "outputfilename"
fi
done
I also tried:
text="$(lynx --dump https://address/${array[${i}]})"
Try
#!/bin/bash
IFS=$'\n' read -rd '' -a array <words
elements=${#array[#]}
for (( i=0;i<$elements;i++))
do
text="$(lynx --dump https://address/"${array[${i}]}")"
echo "$text" >> "outputfilename"
done
The array variable wasn't being set with array=($(<words))
You can use read or readarray, but this example is with read
Incidentally, putting IFS=$'\n' before read without a command separator ; sets $IFS only for the read command, removing the need to save and re-set $IFS
You don't need an array at all; the following will work in any POSIX-compatible shell, assuming you have one URL component per line:
while IFS= read -r line; do
text=$(lynx --dump https://address/"$line")
echo "$text"
done < words >> output filename
My two cents...
I prefer use printf -v for this, and this could be build like a filter:
catWeb() {
while IFS= read -r word;do
printf -v url "https://address/%s" "$word"
lynx --dump "$url"
done
}
catWeb <words >outputfilename
I was reading windows file. Lines ended with CR LF. So address contains
\r
character. I can remove it:
array[${i}]=${array[${i}]%$'\r'}
Or I can reformat input file so lines end only with LF.
Main structure of working script reading from CR LF file is
#!/bin/bash
IFS=$'\n' read -rd '' -a array <words
elements=${#array[#]}
for (( i=0;i<$elements;i++))
do
array[${i}]=${array[${i}]%$'\r'}
text="$(lynx --dump https://adrress/"${array[${i}]}")"
if [ ${#text} -gt 1 ]
then
echo "$text" >> "filename"
else
echo "${array[${i}]}" >> "filename2"
fi
done
I'm trying to read a list of files from stdin, with each file delimited by a newline, however I'm noticing that only the first element is getting appended to the list. I noticed this by simply entering two strings and then q. Can anyone explain why?
files=()
read input
while [ "$input" != "q" ] ;
do
files+=( "$input" )
read input
done
for f in $files ;
do
echo "the list of files is:"
echo "$f"
echo "The length of files is ${#files} " #prints 1, even if 2+ are entered
done
Actually your files+=( "$input" ) expression is adding elements to your array but you are not iterating it correctly.
Your last loop should be:
for f in "${files[#]}"; do
echo "element is: $f"
done
Test (thanks to #fedorqui)
$ a+=(1)
$ a+=("hello")
$ a+=(3)
$ for i in "${a[#]}"; do echo "$i"; done
1
hello
3
I'm trying to write a small script that will count entries in a log file, and I'm incrementing a variable (USCOUNTER) which I'm trying to use after the loop is done.
But at that moment USCOUNTER looks to be 0 instead of the actual value. Any idea what I'm doing wrong? Thanks!
FILE=$1
tail -n10 mylog > $FILE
USCOUNTER=0
cat $FILE | while read line; do
country=$(echo "$line" | cut -d' ' -f1)
if [ "US" = "$country" ]; then
USCOUNTER=`expr $USCOUNTER + 1`
echo "US counter $USCOUNTER"
fi
done
echo "final $USCOUNTER"
It outputs:
US counter 1
US counter 2
US counter 3
..
final 0
You are using USCOUNTER in a subshell, that's why the variable is not showing in the main shell.
Instead of cat FILE | while ..., do just a while ... done < $FILE. This way, you avoid the common problem of I set variables in a loop that's in a pipeline. Why do they disappear after the loop terminates? Or, why can't I pipe data to read?:
while read country _; do
if [ "US" = "$country" ]; then
USCOUNTER=$(expr $USCOUNTER + 1)
echo "US counter $USCOUNTER"
fi
done < "$FILE"
Note I also replaced the `` expression with a $().
I also replaced while read line; do country=$(echo "$line" | cut -d' ' -f1) with while read country _. This allows you to say while read var1 var2 ... varN where var1 contains the first word in the line, $var2 and so on, until $varN containing the remaining content.
Always use -r with read.
There is no need to use cut, you can stick with pure bash solutions.
In this case passing read a 2nd var (_) to catch the additional "fields"
Prefer [[ ]] over [ ].
Use arithmetic expressions.
Do not forget to quote variables! Link includes other pitfalls as well
while read -r country _; do
if [[ $country = 'US' ]]; then
((USCOUNTER++))
echo "US counter $USCOUNTER"
fi
done < "$FILE"
minimalist
counter=0
((counter++))
echo $counter
You're getting final 0 because your while loop is being executed in a sub (shell) process and any changes made there are not reflected in the current (parent) shell.
Correct script:
while read -r country _; do
if [ "US" = "$country" ]; then
((USCOUNTER++))
echo "US counter $USCOUNTER"
fi
done < "$FILE"
I had the same $count variable in a while loop getting lost issue.
#fedorqui's answer (and a few others) are accurate answers to the actual question: the sub-shell is indeed the problem.
But it lead me to another issue: I wasn't piping a file content... but the output of a series of pipes & greps...
my erroring sample code:
count=0
cat /etc/hosts | head | while read line; do
((count++))
echo $count $line
done
echo $count
and my fix thanks to the help of this thread and the process substitution:
count=0
while IFS= read -r line; do
((count++))
echo "$count $line"
done < <(cat /etc/hosts | head)
echo "$count"
USCOUNTER=$(grep -c "^US " "$FILE")
Incrementing a variable can be done like that:
_my_counter=$[$_my_counter + 1]
Counting the number of occurrence of a pattern in a column can be done with grep
grep -cE "^([^ ]* ){2}US"
-c count
([^ ]* ) To detect a colonne
{2} the colonne number
US your pattern
Using the following 1 line command for changing many files name in linux using phrase specificity:
find -type f -name '*.jpg' | rename 's/holiday/honeymoon/'
For all files with the extension ".jpg", if they contain the string "holiday", replace it with "honeymoon". For instance, this command would rename the file "ourholiday001.jpg" to "ourhoneymoon001.jpg".
This example also illustrates how to use the find command to send a list of files (-type f) with the extension .jpg (-name '*.jpg') to rename via a pipe (|). rename then reads its file list from standard input.
The following program reads a file and it intends to store the all values (each line) into a variable but doesn't store the last line. Why?
file.txt :
1
2
.
.
.
n
Code :
FileName=file.txt
if test -f $FileName # Check if the file exists
then
while read -r line
do
fileNamesListStr="$fileNamesListStr $line"
done < $FileName
fi
echo "$fileNamesListStr" // 1 2 3 ..... n-1 (but it should print up to n.)
Instead of reading line-by-line, why not read the whole file at once?
[ -f $FileName ] && fileNameListStr=$( tr '\n' ' ' < $FileName )
One probable cause is that there misses a newline after the last line n.
Use the following command to check it:
tail -1 file.txt
And the following fixes:
echo >> file.txt
If you really need to keep the last line without newline, I reorganized the while loop here.
#!/bin/bash
FileName=0
if test -f $FileName ; then
while [ 1 ] ; do
read -r line
if [ -z $line ] ; then
break
fi
fileNamesListStr="$fileNamesListStr $line"
done < $FileName
fi
echo "$fileNamesListStr"
The issue is that when the file does not end in a newline, read returns non-zero and the loop does not proceed. The read command will still read the data, but it will not process the loop. This means that you need to do further processing outside of the loop. You also probably want an array instead of a space separated string.
FileName=file.txt
if test -f $FileName # Check if the file exists
then
while read -r line
do
fileNamesListArr+=("$line")
done < $FileName
[[ -n $line ]] && fileNamesListArr+=("$line")
fi
echo "${fileNameListArr[#]}"
See the "My text files are broken! They lack their final newlines!" section of this article:
http://mywiki.wooledge.org/BashFAQ/001
As a workaround, before reading from the text file a newline can be appended to the file.
echo "\n" >> $file_path
This will ensure that all the lines that was previously in the file will be read. Now the file can be read line by line.