alternative to readarray, because it does not work on mac os x - macos

I have a varsValues.txt file
cat varsValues.txt
aa=13.7
something=20.6
countries=205
world=1
languages=2014
people=7.2
oceans=3.4
And I would like to create 2 arrays, vars and values. It should contain
echo ${vars[#]}
aa something countries world languages people oceans
echo ${values[#]}
13.7 20.6 205 1 2014 7.2 3.4
I use
Npars=7
readarray -t vars < <(cut -d '=' -f1 varsValues.txt)
readarray -t values < <(cut -d '=' -f2 varsValues.txt)
for (( yy=0; yy<$Npars; yy++ )); do
eval ${vars[$yy]}=${values[$yy]}
done
echo $people
7.2
But I would like it without readarray which does not work on Mac (os x) and IFS (interfield separater).
Any other solution? awk? perl? which I can use in my bash script.
Thanks.

You could use a read loop.
while IFS=\= read var value; do
vars+=($var)
values+=($value)
done < VarsValues.txt

Here's the awk version. Note that NPars is not hardcoded.
vars=($(awk -F= '{print $1}' varsValues.txt))
values=($(awk -F= '{print $2}' varsValues.txt))
Npars=${#vars[#]}
for ((i=0; i<$Npars; i++)); do
eval ${vars[$i]}=${values[$i]}
done
echo $people

You can use declare builtin:
declare -a vars=( $(cut -d '=' -f1 varsValues.txt) )
declare -a values=( $(cut -d '=' -f2 varsValues.txt) )
Although, as commenters have pointed out declare -a is superfluous.
vars=( $(cut -d '=' -f1 varsValues.txt) )
values=( $(cut -d '=' -f2 varsValues.txt) )
Works just as well.

Try:
IFS=$'\n' vars=($(cut -d '=' -f1 varsValues.txt))
IFS=$'\n' values=($(cut -d '=' -f2 varsValues.txt))

perl -0777 -nE '#F= split /[=\r\n]/; say "#F[grep !($_%2), 0..$#F]"; say "#F[grep $_%2, 0..$#F]"' varsValues.txt
or by reading same file twice,
perl -F'=' -lane 'print $F[0]' varsValues.txt
perl -F'=' -lane 'print $F[1]' varsValues.txt

Let's start with this:
$ awk -F'=' '{values[$1]=$2} END{print values["people"]}' file
7.2
$ awk -F'=' '{values[$1]=$2} END{for (name in values) print name, values[name]}' file
languages 2014
oceans 3.4
world 1
something 20.6
countries 205
people 7.2
aa 13.7
Now - what else do you need to do?

Figured I'd toss this in here: https://raw.githubusercontent.com/AdrianTP/new-environment-setup/master/utils/readarray.sh
#!/bin/bash
# from: https://peniwize.wordpress.com/2011/04/09/how-to-read-all-lines-of-a-file-into-a-bash-array/
readarray() {
local __resultvar=$1
declare -a __local_array
let i=0
while IFS=$'\n' read -r line_data; do
__local_array[i]=${line_data}
((++i))
done < $2
if [[ "$__resultvar" ]]; then
eval $__resultvar="'${__local_array[#]}'"
else
echo "${__local_array[#]}"
fi
}
I keep this in a "utils" folder in my "new-environment-setup" Github repo, and I just clone it down and import it whenever I need to read a file into an array of lines an array get a new computer or wipe my drive. It should thus act as a backfill for readarray's shortcomings on Mac.
Import looks like:
# From: https://stackoverflow.com/a/12694189/771948
DIR="${BASH_SOURCE%/*}"
if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi
. "$DIR/utils/readarray.sh"
Usage looks like readarray "<output_var_name>" "<input_file_name>".
Yes it's a little rough. Sorry about that. It may not even work correctly anymore, but it did at one point, so I thought I would share it here to plant the idea of simply...writing your own backfill.

Mac uses an outdated version of bash (due to licencing reasons) by default which is lacking the readarray command.
This solution worked best for me (Mac user):
Check version of bash (probably version 3 from 2007)
bash --version
Download latest version of bash
brew install bash
Open a new terminal (which will load the new environment), then check the new version of bash (should be version 5 or higher)
bash --version
Check location(s) of bash
which -a bash
Output:
/usr/local/bin/bash
/bin/bash
You can see that you now have two versions of bash. Usually, both of these paths are in your PATH variable.
Check PATH variable
echo $PATH
The /usr/local/bin/bash should be standing before the /bin/bash in this variable. The shell is searching for executables in the order of occurrence in the PATH variable and takes the first one it finds.
Make sure to use a bash shell (rather than zsh) when using this command.
Try out the readarray command by e.g. redirecting the output of the ls
command with command substitution to the readarray command to generate an array containing a list with the filenames of the current folder:
readarray var < <(ls); echo ${var[#]}
Also, if you want to write a bash script make sure to use the correct Shebang:
#!/usr/local/bin/bash

Related

Script does not work with ` but works in single command

In my bash, the whole script won't work... When I use `
My script is
#!/bin/bash
yesterday=$(date --date "$c days ago" +%F)
while IFS= read -r line
do
dir=$(echo $line | awk -F, '{print $1 }')
country=$(echo $line | awk -F, '{print $2 }')
cd path/$dir
cat `ls -v | grep email.csv` > e.csv
done < "s.csv"
Above output is blank.
If i use ""
output is No such file or directory
but if I use only 1 line on the terminal it works
cat `ls -v | grep email.csv` > e.csv
I also try with / , but didnt work either...
You should generally avoid ls in scripts.
Also, you should generally prefer the modern POSIX $(command substitution) syntax like you already do in several other places in your script; the obsolescent backtick `command substitution` syntax is clunky and somewhat more error-prone.
If this works in the current directory but fails in others, it means that you have a file matching the regex in the current directory, but not in the other directory.
Anyway, the idiomatic way to do what you appear to be attempting is simply
cat *email?csv* >e.csv
If you meant to match a literal dot, that's \. in a regular expression. The ? is a literal interpretation of what your grep actually did; but in the following, I will assume you actually meant to match *email.csv* (or in fact probably even *email.csv without a trailing wildcard).
If you want to check if there are any files, and avoid creating e.csv if not, that's slightly tricky; maybe try
for file in *email.csv*; do
test -e "$file" || break
cat *email.csv* >e.csv
break
done
Alternatively, look into the nullglob feature of Bash. See also Check if a file exists with wildcard in shell script.
On the other hand, if you just want to check whether email.csv exists, without a wildcard, that's easy:
if [ -e email.csv ]; then
cat email.csv >e.csv
fi
In fact, that can even be abbreviated down to
test -e email.csv && cat email.csv >e.csv
As an aside, read can perfectly well split a line into tokens.
#!/bin/bash
yesterday=$(date --date "$c days ago" +%F)
while IFS=, read -r dir country _
do
cd "path/$dir" # notice proper quoting, too
cat *email.csv* > e.csv
# probably don't forget to cd back
cd ../..
done < "s.csv"
If this is in fact all your script does, probably do away with the silly and slightly error-prone cd;
while IFS=, read -r dir country _
do
cat "path/$dir/"*email.csv* > "path/$dir/e.csv"
done < "s.csv"
See also When to wrap quotes around a shell variable.

Extract the higher number of a folder name in bash

I have a folder with the following structure:
$ ls
DRS-1.1.0 DRS-1.3.0 DRS-1.3.1 DRS-1.3.2
with time, the folder version will increase (e.g. DRS-1.3.5, DRS-1.4.1, etc...)
How can I store the name of the highest version into a variable? (in the current case DRS-1.3.5)
the output should be:
$ echo $VERSION
DRS-1.3.2
Thanks
With GNU sort's version sort:
printf '%s\n' DRS-* | sort -V | tail -n 1
Output:
DRS-1.3.2
You can use awk to get the max number:
ver=$(printf '%s\n' DRS-* | awk -F- '$2 > max{max=$2; f=$0} END{print f}')
echo "$ver"
DRS-1.3.2
Bash sorts its glob expansions alphabetically, so we can just index the last element of such an expansion:
files=(*)
VERSION=${files[$((${#files[*]}-1))]}
printf '%s\n' "$VERSION"
This has an advantage over methods that rely on filtering line-based output in that it will work all filenames, even if they include newline characters.
The downside is that it's limited to collation ordering, and doesn't understand version numbers in the way that (say) dpkg --compare-versions does; that will require iterating through the array, like this:
#!/bin/bash
set -eu
shopt -s nullglob
files=(DRS-*)
max=
for f in "${files[#]}"
do
dpkg --compare-versions "${f#DRS-}" gt "${max#DRS-}" || continue
max=$f
done
printf '%s\n' "$max"
(which still works with filenames containing newlines, of course)
Demo:
$ ls -1 DRS-*
DRS-1.1.0
DRS-1.12.0
DRS-1.3.0
DRS-1.3.1
DRS-1.3.2
$ ./find_max
DRS-1.12.0

Extracting a pattern (grep output) in Linux from shell?

Grep output is usually like this:
after/ftplugin/python.vim:49: setlocal number
Is it possible for me extract the file name and line number from this result using standard linux utilities ? Looking for a generic solution that works pretty well .
I can think of using awk to get the first string like :
Input
echo 'after/ftplugin/python.vim:49: setlocal number' | awk 'print $1'
'after/ftplugin/python.vim:49:'
$
Expected
after/ftplugin/python.vim and 49
Goal : Open in Vim
I am writing a small function that transforms the grep output to something vim can understand - mostly for academic purpose . I know there are thinks like Ack.vim out there which does something similar . What are the standard light weight utils out there ?
Edit: grep -n "text to find" file.ext |cut -f1 -d: seems to do it if you dont mind double parsing the string . Sed though needs to be used !
If you're using Bash you can do it this way:
IFS=: read FILE NUM __ < <(exec grep -Hn "string to find" file)
vim "+$NUM" "$FILE"
Or POSIX:
IFS=: read FILE NUM __ <<EOD
$(grep -Hn "string to find" file)
EOD
vim "+$NUM" "$FILE"
Style © konsolebox :)
This will do:
echo 'after/ftplugin/python.vim:49: setlocal number' | awk -F: '{print $1,"and",$2}'
after/ftplugin/python.vim and 49
But give us data before grep. It may be that we can cut it more down. No need for both grep and awk
If by "reverse parse" you mean you want to start from the end (and can safely assume that the file content contains no colons), parameter expansion makes that easy:
line='after/ftplugin/python.vim:49: setlocal number'
name_and_lineno=${line%:*}
name=${name_and_lineno%:*}
lineno=${name_and_lineno##*:}
Being all in-process (using shell built-in functionality), this is much faster than using external tools such as sed, awk, etc.
To connect it all together, consider a loop such as the following:
while read -r line; do
...
done < <(grep ...)
Now, to handle all possible filenames (including ones with colons) and all possible content (including strings with colons), you need a grep with GNU extensions:
while IFS='' read -u 4 -r -d '' file \
&& read -u 4 -r -d ':' lineno \
&& read -u 4 -r line; do
vim "+$lineno" "$file"
done 4< <(grep -HnZ -e "string to find" /dev/null file)
This works as follows:
Use grep -Z (a GNU extension) to terminate each filename with a NUL rather than a :
Use IFS='' read -r -d '' to read until the first NUL when reading filenames
Use read -r -d ':' lineno to read until a colon when reading line numbers
Read until the next newline when reading lines
Redirect contents on FD #4 to avoid overriding stdin, stdout or stderr (so vim will still work properly)
Use the -u 4 argument on all calls to read to handle contents from FD #4
How about this?:
echo 'after/ftplugin/python.vim:49: setlocal number' | cut -d: -f1-2 | sed -e 's/:/ and /'
Result:
after/ftplugin/python.vim and 49

BASH: Split MAC Address -> 000E0C7F6676 to 00:0E:0C:7F:66:76

Hy,
Can someone help me with splitting mac addresses from a log file? :-)
This:
000E0C7F6676
should be:
00:0E:0C:7F:66:76
Atm i split this with OpenOffice but with over 200 MAC Address' this is very boring and slow...
It would be nice if the solution is in bash. :-)
Thanks in advance.
A simple sed script ought to do it.
sed -e 's/[0-9A-F]\{2\}/&:/g' -e 's/:$//' myFile
That'll take a list of mac addresses in myFile, one per line, and insert a ':' after every two hex-digits, and finally remove the last one.
$ mac=000E0C7F6676
$ s=${mac:0:2}
$ for((i=1;i<${#mac};i+=2)); do s=$s:${mac:$i:2}; done
$ echo $s
00:00:E0:C7:F6:67:6
Pure Bash. This snippet
mac='000E0C7F6676'
array=()
for (( CNTR=0; CNTR<${#mac}; CNTR+=2 )); do
array+=( ${mac:CNTR:2} )
done
IFS=':'
string="${array[*]}"
echo -e "$string"
prints
00:0E:0C:7F:66:76
$ perl -lne 'print join ":", $1 =~ /(..)/g while /\b([\da-f]{12})\b/ig' file.log
00:0E:0C:7F:66:76
If you prefer to save it as a program, use
#! /usr/bin/perl -ln
print join ":" => $1 =~ /(..)/g
while /\b([\da-f]{12})\b/ig;
Sample run:
$ ./macs file.log
00:0E:0C:7F:66:76
imo, regular expressions are the wrong tool for a fixed width string.
perl -alne 'print join(":",unpack("A2A2A2A2A2A2",$_))' filename
Alternatively,
gawk -v FIELDWIDTHS='2 2 2 2 2 2' -v OFS=':' '{$1=$1;print }'
That's a little funky with the assignment to change the behavior of print. Might be more clear to just print $1,$2,$3,$4,$5,$6
Requires Bash version >= 3.2
#!/bin/bash
for i in {1..6}
do
pattern+='([[:xdigit:]]{2})'
done
saveIFS=$IFS
IFS=':'
while read -r line
do
[[ $line =~ $pattern ]]
mac="${BASH_REMATCH[*]:1}"
echo "$mac"
done < macfile.txt > newfile.txt
IFS=$saveIFS
If your file contains other information besides MAC addresses that you want to preserve, you'll need to modify the regex and possibly move the IFS manipulation inside the loop.
Unfortunately, there's not an equivalent in Bash to sed 's/../&:/' using something like ${mac//??/??:/}.
a='0123456789AB'
m=${a:0:2}:${a:2:2}:${a:4:2}:${a:6:2}:${a:8:2}:${a:10:2}
result:
01:23:45:67:89:AB

How to extract and chop version string from file in bash

I have a simple text file ./version containing a version number. Unfortunately, sometimes the version number in the file is followed by whitespaces and newlines like
1.1.3[space][space][newline]
[newline]
[newline]
What is the best, easiest and shortest way to extract the version number into a bash variable without the trailing spaces and newlines? I tried
var=`cat ./version | tr -d ' '`
which works for the whitespaces but when appending a tr -d '\n' it does not work.
Thanks,
Chris
$ echo -e "1.1.1 \n\n" > ./version
$ read var < ./version
$ echo -n "$var" | od -a
0000000 1 . 1 . 1
0000005
Pure Bash, no other process:
echo -e "1.2.3 \n\n" > .version
version=$(<.version)
version=${version// /}
echo "'$version'"
result: '1.2.3'
I still do not know why, but after deleting and recreating the version file this worked:
var=`cat ./version | tr -d ' ' | tr -d '\n'`
I'm confused... what can you do different when creating a text file. However, it works now.
I like the pure bash version from fgm's answer.
I provide this one-line perl command to remove also other characters if any:
perl -pe '($_)=/([0-9]+([.][0-9]+)+)/'
The extracted version number is trimmed/stripped (no newline or carriage return symbols):
$> V=$( bash --version | perl -pe '($_)=/([0-9]+([.][0-9]+)+)/' )
$> echo "The bash version is '$V'"
The bash version is '4.2.45'
I provide more explanation and give other more sophisticated (but still short) one-line perl commands in my other answer.

Resources