Hy,
Can someone help me with splitting mac addresses from a log file? :-)
This:
000E0C7F6676
should be:
00:0E:0C:7F:66:76
Atm i split this with OpenOffice but with over 200 MAC Address' this is very boring and slow...
It would be nice if the solution is in bash. :-)
Thanks in advance.
A simple sed script ought to do it.
sed -e 's/[0-9A-F]\{2\}/&:/g' -e 's/:$//' myFile
That'll take a list of mac addresses in myFile, one per line, and insert a ':' after every two hex-digits, and finally remove the last one.
$ mac=000E0C7F6676
$ s=${mac:0:2}
$ for((i=1;i<${#mac};i+=2)); do s=$s:${mac:$i:2}; done
$ echo $s
00:00:E0:C7:F6:67:6
Pure Bash. This snippet
mac='000E0C7F6676'
array=()
for (( CNTR=0; CNTR<${#mac}; CNTR+=2 )); do
array+=( ${mac:CNTR:2} )
done
IFS=':'
string="${array[*]}"
echo -e "$string"
prints
00:0E:0C:7F:66:76
$ perl -lne 'print join ":", $1 =~ /(..)/g while /\b([\da-f]{12})\b/ig' file.log
00:0E:0C:7F:66:76
If you prefer to save it as a program, use
#! /usr/bin/perl -ln
print join ":" => $1 =~ /(..)/g
while /\b([\da-f]{12})\b/ig;
Sample run:
$ ./macs file.log
00:0E:0C:7F:66:76
imo, regular expressions are the wrong tool for a fixed width string.
perl -alne 'print join(":",unpack("A2A2A2A2A2A2",$_))' filename
Alternatively,
gawk -v FIELDWIDTHS='2 2 2 2 2 2' -v OFS=':' '{$1=$1;print }'
That's a little funky with the assignment to change the behavior of print. Might be more clear to just print $1,$2,$3,$4,$5,$6
Requires Bash version >= 3.2
#!/bin/bash
for i in {1..6}
do
pattern+='([[:xdigit:]]{2})'
done
saveIFS=$IFS
IFS=':'
while read -r line
do
[[ $line =~ $pattern ]]
mac="${BASH_REMATCH[*]:1}"
echo "$mac"
done < macfile.txt > newfile.txt
IFS=$saveIFS
If your file contains other information besides MAC addresses that you want to preserve, you'll need to modify the regex and possibly move the IFS manipulation inside the loop.
Unfortunately, there's not an equivalent in Bash to sed 's/../&:/' using something like ${mac//??/??:/}.
a='0123456789AB'
m=${a:0:2}:${a:2:2}:${a:4:2}:${a:6:2}:${a:8:2}:${a:10:2}
result:
01:23:45:67:89:AB
Related
for example our bash script's name is masodik and there is a text.txt with these lines:
qwer
qw2qw
12345
qwert432
Then I write ./masodik text.txt and i got
qw2qw
12345
I tried it many ways and I dont know why this is not working
#!/bin/bash
for i in read u ; do
echo $i $u | grep '^[a-zA-Z0-9][a-zA-Z0-9][0-9]'
done
$ grep -E '^.{2}[0-9]' text.txt
qw2qw
12345
, and in script it could be something like:
#!/bin/sh
grep -E '^.{2}[0-9]' "$1"
To print lines whose third character is a digit:
grep ^..[0-9] text.txt
^ matches the start of the line. The dot . matches any character. [0-9] matches any digit.
You can do it with awk quite easily as well:
awk '/^..[0-9]/' file
Result
With your input in file:
$ awk '/^..[0-9]/' file
qw2qw
12345
(sed works as well, sed -n '/^..[0-9]/p' file)
The problem with the code here:
#!/bin/bash
for i in read u ; do
echo $i $u | grep '^[a-zA-Z0-9][a-zA-Z0-9][0-9]'
done
...is that the for syntax is wrong:
read u is treated as a word list. So the $u variable is never set, so $u stays empty.
The for loop will run twice -- the 1st time $i will be set to the string "read", the 2nd time $i will be set to the string "u". Since neither string contains a number, the grep returns nothing.
The code never reads text.txt.
See Sasha Khapyorsky's answer for actual working code.
If for some odd reason all external utils, (grep, awk, etc.), are forbidden, this pure POSIX code would work:
#!/bin/sh
while read u ; do
case "$u" in
[a-zA-Z0-9][a-zA-Z0-9][0-9]*) echo "$u" ;;
esac
done
If perl is installed into the system then shell script will look like
#!/bin/bash
perl -e 'print if /^.{2}\d/' text.txt
I have file names like these:
func/sub-01_task-biommtloc_run-01_bold_space-T1w_preproc.nii.gz
func/sub-01_task-pfobloc_run-01_bold_space-T1w_preproc.nii.gz
func/sub-01_task-rest_run-01_bold_space-T1w_preproc.nii.gz
and from each file name I want to extract the part until and including the word bold so that in the end I have:
func/sub-01_task-biommtloc_run-01_bold
func/sub-01_task-pfobloc_run-01_bold
func/sub-01_task-rest_run-01_bold
Any ideas how to do that?
The easiest thing to do is to just remove bold and everything after, then replace bold. Obviously, this only works if the terminating string is fixed, as in this case.
$ f=func/sub-01_task-biommtloc_run-01_bold_space-T1w_preproc.nii.gz
$ echo "${f%%bold*}"
func/sub-01_task-biommtloc_run-01_
$ echo "${f%%bold*}bold"
func/sub-01_task-biommtloc_run-01_bold
Is something like this what you want?
echo func/sub-01_task-biommtloc_run-01_bold_space-T1w_preproc.nii.gz | sed -e 's#bold_.*$#bold#'
Hope this helps
This is (needlessly) clever: remove the prefix ending with "bold"
and then so some substring index arithmetic based on the length of the suffix that's left over:
$ file=func/sub-01_task-biommtloc_run-01_bold_space-T1w_preproc.nii.gz
$ tmp=${file#*bold}
$ keep=${file:0:${#file}-${#tmp}}
$ echo "$keep"
func/sub-01_task-biommtloc_run-01_bold
If $file does not contain "bold", then $keep will be empty: we can give it the value of $file if it is empty:
$ file=foobar
$ tmp=${file#*bold}
$ keep=${file:0:${#file}-${#tmp}}
$ : ${keep:=$file}
$ echo "$keep"
foobar
But seriously, do what chepner suggests.
using Perl
> echo "func/sub-01_task-biommtloc_run-01_bold_space-T1w_preproc.nii.gz" | perl -e 'while (<>) { $_=~s/(.*bold)(.*)/\1/g; print } '
func/sub-01_task-biommtloc_run-01_bold
>
This is similar to glenn's solution, but a bit "less clever" in that it doesn't use substrings, just nested substitutions:
$ while IFS= read -r fname; do echo "${fname%"${fname#*bold}"}"; done < infile
func/sub-01_task-biommtloc_run-01_bold
func/sub-01_task-pfobloc_run-01_bold
func/sub-01_task-rest_run-01_bold
The substitution "${fname%"${fname#*bold}"}" says:
Remove "${fname#*bold}" from the end of each filename, where
"${fname#*bold}" is everything up to and including bold removed from the front of the filename
Example for the first filename with explicit intermediate steps:
$ fname=func/sub-01_task-biommtloc_run-01_bold_space-T1w_preproc.nii.gz
$ echo "${fname#*bold}"
_space-T1w_preproc.nii.gz
$ echo "${fname%"${fname#*bold}"}"
func/sub-01_task-biommtloc_run-01_bold
f=func/sub-01_task-biommtloc_run-01_bold_space-T1w_preproc.nii.g
echo "${f//bold*/bold}"
I would recommend using sed for this task. First take all of your input filenames and stick them in a file, call it namelist.txt in the current directory. The following will work, as long as your sed supports extended regular expressions (which most will, particularly GNU sed). Note that the flag for extended regular expressions may differ a bit between platforms, check your sed manual page. On my Linux, it is -r.
bash -c "sed -r 's/(sub-01_task-.{1,10}_run-01_bold).+/\\1/' namelist.txt"
I have a varsValues.txt file
cat varsValues.txt
aa=13.7
something=20.6
countries=205
world=1
languages=2014
people=7.2
oceans=3.4
And I would like to create 2 arrays, vars and values. It should contain
echo ${vars[#]}
aa something countries world languages people oceans
echo ${values[#]}
13.7 20.6 205 1 2014 7.2 3.4
I use
Npars=7
readarray -t vars < <(cut -d '=' -f1 varsValues.txt)
readarray -t values < <(cut -d '=' -f2 varsValues.txt)
for (( yy=0; yy<$Npars; yy++ )); do
eval ${vars[$yy]}=${values[$yy]}
done
echo $people
7.2
But I would like it without readarray which does not work on Mac (os x) and IFS (interfield separater).
Any other solution? awk? perl? which I can use in my bash script.
Thanks.
You could use a read loop.
while IFS=\= read var value; do
vars+=($var)
values+=($value)
done < VarsValues.txt
Here's the awk version. Note that NPars is not hardcoded.
vars=($(awk -F= '{print $1}' varsValues.txt))
values=($(awk -F= '{print $2}' varsValues.txt))
Npars=${#vars[#]}
for ((i=0; i<$Npars; i++)); do
eval ${vars[$i]}=${values[$i]}
done
echo $people
You can use declare builtin:
declare -a vars=( $(cut -d '=' -f1 varsValues.txt) )
declare -a values=( $(cut -d '=' -f2 varsValues.txt) )
Although, as commenters have pointed out declare -a is superfluous.
vars=( $(cut -d '=' -f1 varsValues.txt) )
values=( $(cut -d '=' -f2 varsValues.txt) )
Works just as well.
Try:
IFS=$'\n' vars=($(cut -d '=' -f1 varsValues.txt))
IFS=$'\n' values=($(cut -d '=' -f2 varsValues.txt))
perl -0777 -nE '#F= split /[=\r\n]/; say "#F[grep !($_%2), 0..$#F]"; say "#F[grep $_%2, 0..$#F]"' varsValues.txt
or by reading same file twice,
perl -F'=' -lane 'print $F[0]' varsValues.txt
perl -F'=' -lane 'print $F[1]' varsValues.txt
Let's start with this:
$ awk -F'=' '{values[$1]=$2} END{print values["people"]}' file
7.2
$ awk -F'=' '{values[$1]=$2} END{for (name in values) print name, values[name]}' file
languages 2014
oceans 3.4
world 1
something 20.6
countries 205
people 7.2
aa 13.7
Now - what else do you need to do?
Figured I'd toss this in here: https://raw.githubusercontent.com/AdrianTP/new-environment-setup/master/utils/readarray.sh
#!/bin/bash
# from: https://peniwize.wordpress.com/2011/04/09/how-to-read-all-lines-of-a-file-into-a-bash-array/
readarray() {
local __resultvar=$1
declare -a __local_array
let i=0
while IFS=$'\n' read -r line_data; do
__local_array[i]=${line_data}
((++i))
done < $2
if [[ "$__resultvar" ]]; then
eval $__resultvar="'${__local_array[#]}'"
else
echo "${__local_array[#]}"
fi
}
I keep this in a "utils" folder in my "new-environment-setup" Github repo, and I just clone it down and import it whenever I need to read a file into an array of lines an array get a new computer or wipe my drive. It should thus act as a backfill for readarray's shortcomings on Mac.
Import looks like:
# From: https://stackoverflow.com/a/12694189/771948
DIR="${BASH_SOURCE%/*}"
if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi
. "$DIR/utils/readarray.sh"
Usage looks like readarray "<output_var_name>" "<input_file_name>".
Yes it's a little rough. Sorry about that. It may not even work correctly anymore, but it did at one point, so I thought I would share it here to plant the idea of simply...writing your own backfill.
Mac uses an outdated version of bash (due to licencing reasons) by default which is lacking the readarray command.
This solution worked best for me (Mac user):
Check version of bash (probably version 3 from 2007)
bash --version
Download latest version of bash
brew install bash
Open a new terminal (which will load the new environment), then check the new version of bash (should be version 5 or higher)
bash --version
Check location(s) of bash
which -a bash
Output:
/usr/local/bin/bash
/bin/bash
You can see that you now have two versions of bash. Usually, both of these paths are in your PATH variable.
Check PATH variable
echo $PATH
The /usr/local/bin/bash should be standing before the /bin/bash in this variable. The shell is searching for executables in the order of occurrence in the PATH variable and takes the first one it finds.
Make sure to use a bash shell (rather than zsh) when using this command.
Try out the readarray command by e.g. redirecting the output of the ls
command with command substitution to the readarray command to generate an array containing a list with the filenames of the current folder:
readarray var < <(ls); echo ${var[#]}
Also, if you want to write a bash script make sure to use the correct Shebang:
#!/usr/local/bin/bash
How can I replace/delete characters in a file while leaving comment lines unchanged? I'm looking for a something to the effect of the following lines (where 'X' is replaced for 'Y' in file.txt), just substantially faster:
while read line
do
if [[ ${line:0:1} = "#" ]]
then
echo "$line"
else
echo "$line" | tr "X" "Y"
fi
done < file.txt
Thank you!
Equivalent, more accurate (and faster) will be this sed command as compared to your script:
sed '/^ *#/!{s/X/Y/g;}' file.txt
This means match any line that doesn't have 0 or more spaces followed by # at the start of line and replace X with Y globally.
i am willing to bet perl will be faster than all above :
perl -i -pe 's/X/Y/g unless /^#/' file.txt
for fast replacement, use sed, and only replace in lines not starting with "#":
cat foo.txt | sed -e '/^#/! s/X/Y/g'
sed -i '/^#/! s/{what_to_replace}/{to_what_to_replace}/g' file.txt
awk version:
awk '!/^ *#/{gsub(/X/,"Y")}1' file.txt
Do look for word boundaries to prevent sub strings of your substitution from getting replaced. For example, with gawk you can use \< and \>
I wanted to print the name from the entire address by shell scripting. So user1#12.12.23.234 should give output "user1" and similarly 11234#12.123.12.23 should give output 11234
Reading from the terminal:
$ IFS=# read user host && echo "$user"
<user1#12.12.23.234>
user1
Reading from a variable:
$ address='user1#12.12.23.234'
$ cut -d# -f1 <<< "$address"
user1
$ sed 's/#.*//' <<< "$address"
user1
$ awk -F# '{print $1}' <<< "$address"
user1
Using bash in place editing:
EMAIL='user#server.com'
echo "${EMAIL%#*}
This is a Bash built-in, so it might not be very portable (it won't run with sh if it's not linked to /bin/bash for example), but it is probably faster since it doesn't fork a process to handle the editing.
Using sed:
echo "$EMAIL" | sed -e 's/#.*//'
This tells sed to replace the # character and as many characters that it can find after it up to the end of line with nothing, ie. removing everything after the #.
This option is probably better if you have multiple emails stored in a file, then you can do something like
sed -e 's/#.*//' emails.txt > users.txt
Hope this helps =)
I tend to use expr for this kind of thing:
address='user1#12.12.23.234'
expr "$address" : '\([^#]*\)'
This is a use of expr for its pattern matching and extraction abilities. Translated, the above says: Please print out the longest prefix of $address that doesn't contain an #.
The expr tool is covered by Posix, so this should be pretty portable.
As a note, some historical versions of expr will interpret an argument with a leading - as an option. If you care about guarding against that, you can add an extra letter to the beginning of the string, and just avoid matching it, like so:
expr "x$address" : 'x\([^#]*\)'