shell $ character overwrites the previous variable - shell

I'm trying to write a script on shell but I'm stucked on a point.
I have a program creating data daily and puting it to a directory like this: home/meee/data/2013/07/22/mydata
My problem is I'm trying to change directory using date. Here is my script:
#!/bin/sh
x=$(date -u -v-2H "+%Y-%m-%d")
echo $x
year=$(echo $x | cut -d"-" -f1)
month=$(echo $x | cut -d"-" -f2)
day=$(echo $x | cut -d"-" -f3)
echo $year
echo $month
echo $day
d1='/home/sensor/data/'${year}/${month}
echo $d1
There is no problem related to year, day, month, they are working. But the output of echo d1 is /07me/sensor/data/2013. Similarly, when I write echo $year$day it gives 2312 (characters of day is overwritten on the first two characater of the year)
I tried many other syntax like instead of ' character put " or leave it empty. Removing { and so on. But nothing changed.
Shortly, when I write two variable ($var1 $var2) in same line the second $ behaves like go to the beginning of the line and start overwriting the first variable.
I've been looking for that but there is nothing related to that or I couldn't find anything related and there are a lot of solution in Stackoverflow that solves the problem using $var1$var2
What am I doing wrong, or how can I solve that.
I'm working on FreeBSD 9.0-RELEASE amd64 and using sh
Any help will be appreciated.
Thanks

Somehow, your commands are introducing carriage returns to your variables, which affect the output when the variable is not the last thing echoed. You can confirm this by passing the value through hexdump or od:
printf "%s" "$x" | hexdump -C # Look for 0d in the output.
printf "%s" "$year" | hexdump -C # Look for 0d in the output.
printf "%s" "$month" | hexdump -C # Look for 0d in the output.
printf "%s" "$day" | hexdump -C # Look for 0d in the output.
I don't think this will fix the problem, but you can get the year, month, and day without forking so many external programs:
IFS=- read year month day <<EOF
$(date -u -v-2H "+%Y-%m-%d")
EOF
or more simply
read year month day <<EOF
$(date -u -v-2H "+%Y %m %d")
EOF

You’re likely using MinGW or some other not-quite-Unix environment for Windows®, which is introducing Carriage Return (CR, \r) characters at end-of-line (from the Unix PoV).
So change this to either:
x=$(date -u -v-2H "+%Y-%m-%d" | sed $'s/\r$//')
echo $x
year=$(echo $x | cut -d"-" -f1 | sed $'s/\r$//')
month=$(echo $x | cut -d"-" -f2 | sed $'s/\r$//')
day=$(echo $x | cut -d"-" -f3 | sed $'s/\r$//')
Or, even better:
x=$(date -u -v-2H "+%Y %m %d ")
echo $x
set -- $x
year=$1
month=$2
day=$3
Note the extra space after %d which ensures that the CR will become $4 instead of attached to the day.

Related

What is causing my md5 hash to come out incorrectly?

#!/bin/bash
# Program's Purpose: Compute time elapsed between epoch time and current time
# Produce an MD5 hash from that time
# Get code from that hash
# compute time elapsed in seconds between epoch time and current time
#EPOCH=$(date -u -d '2001-02-03 04:05:06' '+%F %H:%M:%S')
#CURRENT=$(date -u -d '2010-06-13 12:55:34' '+%F %H:%M:%S')
# code: dd15
EPOCH=$(date -u -d '1999-12-31 23:59:59' '+%F %H:%M:%S')
CURRENT=$(date -u -d '2013-05-06 07:43:25' '+%F %H:%M:%S')
# interval is time elapsed minus time elapsed % 60
echo $EPOCH
echo $CURRENT
read YEAR1 M1 DAY1 HOUR1 MIN1 SEC1 <<< "${EPOCH//[:-]/ }"
read YEAR2 M2 DAY2 HOUR2 MIN2 SEC2 <<< "${CURRENT//[:-]/ }"
echo $YEAR1 $M1 $DAY1 $HOUR1 $MIN1 $SEC1
# date in seconds from
sec1=$(date -d "$EPOCH" -u +%s)
sec2=$(date -d "$CURRENT" -u +%s)
echo $sec1
echo $sec2
# get the difference from the two times
difference=$((sec2 - sec1))
difference=$((difference - ((difference % 60))))
echo $difference
# get the hash from the time
hash=$(echo -n $difference | md5sum | tr -d '\n')
hash=$(echo -n $hash | md5sum | tr -d '\n')
echo $hash
# creating two strings, one with all of the letters
# the other with all of the numbers
letter=$(echo $hash | sed 's/[0-9]*//g' | cut -c1-2)
echo $letter
num=$(echo $hash | sed 's/[^0-9]*//g')
echo $num
#num=$(echo $num | cut -c1-2)
#echo $num
# getting the last two numbers in reverse order
num1=$(echo ${num: -1})
num=$(echo "${num::-1}")
echo $num
num2=$(echo ${num: -1})
code="$letter$num1$num2"
echo $code
I'm trying to write a program that takes an epoch time and
current time, computes the difference in seconds, and then creates a
code by doing a double md5 hash on the time in seconds. By what times
I have entered in currently, the difference in seconds should be 421,
141, 406, and the 'code' is supposed to be based on 60-second
intervals, so the code I'm trying to generate should come from 421,
141, 380.
The resulting hash should be
042876ca07cb2d993601fb40a1525736, but I am getting
d49904f9e7e62d0ff16e523d89be08eb. Can anyone tell me what I'm doing
wrong exactly?
I read that bash leaves a newline at the end of
strings, so I ran echo with -n option to remove that newline, but I am
still not getting the preferred results.
The output of md5sum on many platforms is not just the MD5 sum. For example, on a GNU/Linux system, you get
debian$ echo -n 401 | md5sum
816b112c6105b3ebd537828a39af4818 -
Notice that the output has two fields: The actual MD5 sum, followed by two spaces, followed by the input file name (where - stands for standard input).
(By contrast, on OSX, and I would expect most *BSDs, you get
yosemite$ echo -n 401 | md5
816b112c6105b3ebd537828a39af4818
Here, you'll notice that the name of the MD5 utility is different.)
The fix should be simple. I have refactored your code to (a) prefer the portable printf over the less portable echo -n; (b) remove the completely superfluous final tr -d '\n' (newlines are trimmed off the end of the captured variable by the shell already); and (c) fold everything into a single pipeline.
hash=$(printf '%s' "$difference" | md5sum | cut -d ' ' -f 1 | tr -d '\n' |
md5sum | cut -d ' ' -f 1)
echo "$hash"
For completeness, this code also has proper quoting; it's not strictly necessary here (but it would have been if you really needed to preserve the spacing in the value you originally obtained from md5sum, for example) but omitting quotes is a common newbie problem which should be avoided.
(Capturing a variable just so you can echo it is also a common newbie antipattern; but your code will want to use the variable hash subsequently.)
Repeated code is always a bad smell; maybe provide a function which performs the same job as the *BSD md5 utility;
md5 () { md5sum "$#" | cut -d ' ' -f 1; }
hash=$(printf '%s' "$difference" | md5 | tr -d '\n' | md5)

Parsing date and time format - Bash

I have date and time format like this(yearmonthday):
20141105 11:30:00
I need assignment year, month, day, hour and minute values to variable.
I can do it year, day and hour like this:
year=$(awk '{print $1}' log.log | sed 's/^\(....\).*/\1/')
day=$(awk '{print $1}' log.log | sed 's/^.*\(..\).*/\1/')
hour=$(awk '{print $2}' log.log | sed 's/^\(..\).*/\1/')
How can I do this for month and minute?
--
And I need that every line of my log file:
20141105 11:30:00 /bla/text.1
20141105 11:35:00 /bla/text.2
20141105 11:40:00 /bla/text.3
....
I'm trying read line by line this log file and do this:
mkdir -p "/bla/backup/$year/$month/$day/$hour/$minute"
mv $file "/bla/backup/$year/$month/$day/$hour/$minute"
Here is my not working code:
#!/bin/bash
LOG=/var/log/LOG
while read line
do
year=${line:0:4}
month=${line:4:2}
day=${line:6:2}
hour=${line:9:2}
minute=${line:12:2}
file=$(awk '{print $3}')
if [ -f "$file" ]; then
printf -v path "%s/%s/%s/%s/%s" $year $month $day $hour $minute
mkdir -p "/bla/backup/$path"
mv $file "/bla/backup/$path"
fi
done < $LOG
You don't need to call out to awk to date at all, use bash's substring operations
d="20141105 11:30:00"
yr=${d:0:4}
mo=${d:4:2}
dy=${d:6:2}
hr=${d:9:2}
mi=${d:12:2}
printf -v dir "/bla/%s/%s/%s/%s/%s/\n" $yr $mo $dy $hr $mi
echo "$dir"
/bla/2014/11/05/11/30/
Or directly, without all the variables.
printf -v dir "/bla/%s/%s/%s/%s/%s/\n" ${d:0:4} ${d:4:2} ${d:6:2} ${d:9:2} ${d:12:2}
Given your log file:
while read -r date time file; do
d="$date $time"
printf -v dir "/bla/%s/%s/%s/%s/%s/\n" ${d:0:4} ${d:4:2} ${d:6:2} ${d:9:2} ${d:12:2}
mkdir -p "$dir"
mv "$file" "$dir"
done < filename
or, making a big assumption that there are no whitespace or globbing characters in your filenames:
sed -r 's#(....)(..)(..) (..):(..):.. (.*)#mv \6 /blah/\1/\2/\3/\4/\5#' | sh
date command also do this work
#!/bin/bash
year=$(date +'%Y' -d'20141105 11:30:00')
day=$(date +'%d' -d'20141105 11:30:00')
month=$(date +'%m' -d'20141105 11:30:00')
minutes=$(date +'%M' -d'20141105 11:30:00')
echo "$year---$day---$month---$minutes"
You can use only one awk
month=$(awk '{print substr($1,5,2)}' log.log)
year=$(awk '{print substr($1,0,4)}' log.log)
minute=$(awk '{print substr($2,4,2)}' log.log)
etc
I guess you are processing the log file, which each line starts with the date string. You may have already written a loop to handle each line, in your loop, you could do:
d="$(awk '{print $1,$2}' <<<"$line")"
year=$(date -d"$d" +%Y)
month=$(date -d"$d" +%m)
day=$(date -d"$d" +%d)
min=$(date -d"$d" +%M)
Don't repeat yourself.
d='20141105 11:30:00'
IFS=' ' read -r year month day min < <(date -d"$d" '+%Y %d %m %M')
echo "year: $year"
echo "month: $month"
echo "day: $day"
echo "min: $min"
The trick is to ask date to output the fields you want, separated by a character (here a space), to put this character in IFS and ask read to do the splitting for you. Like so, you're only executing date once and only spawn one subshell.
If the date comes from the first line of the file log.log, here's how you can assign it to the variable d:
IFS= read -r d < log.log
eval "$(
echo '20141105 11:30:00' \
| sed 'G;s/\(....\)\(..\)\(..\) \(..\):\(..\):\(..\) *\(.\)/Year=\1\7Month=\2\7Day=\3\7Hour=\4\7Min=\5\7Sec=\6/'
)"
pass via a assignation string to evaluate. You could easily adapt to also check the content by replacing dot per more specific pattern like [0-5][0-9] for min and sec, ...
posix version so --posix on GNU sed
I wrote a function that I usually cut and paste into my script files
function getdate()
{
local a
a=(`date "+%Y %m %d %H %M %S" | sed -e 's/ / /'`)
year=${a[0]}
month=${a[1]}
day=${a[2]}
hour=${a[3]}
minute=${a[4]}
sec=${a[5]}
}
in the script file, on a line of it's own
getdate
echo "year=$year,month=$month,day=$day,hour=$hour,minute=$minute,second=$sec"
Of course, you can modify what I provided or use answer [6] above.
The function takes no arguments.

Error message "date: Argument list too long" bash

So, everything works fine in the code, except for one tiny little thing.
This part:
if [ "$LIMITHOURS" -gt "0" -a "$LIMITHOURS" -lt "24" ]; then
x=$(($LIMITHOURS*60*60))
fi
SDATE=$( echo "01/jan/2003:11:00:06 +0100"| sed 's/[/]/ /g' |sed 's/:/ /')
EDATE=$(date --date "$SDATE - $x seconds" +"%d%m%Y%H%M%S")
#echo "$SDATE"
#echo "$EDATE"
while read LINE; do
CDATE=$( awk '{print $4}'| sed 's/[[]//' | sed 's/[/]//g' |sed 's/://g' )
DATE=$(date --date "$CDATE" +"%d%m%Y%H%M%S")
#echo "$CDATE"
done < "$FILENAME"
When I try to run the script, I get the error message "date: Argument list too long
" and I know that the problem is in the while loop, with:
DATE=$(date --date "$CDATE" +"%d%m%Y%H%M%S")
Anyone who know any solution for this? I want the date format in ddmmYYYYHHMMSS, eg. 23102002120022
You can find rest of the script here: http://pastebin.com/PMk2QDre
This code:
while read LINE; do
CDATE=$( awk '{print $4}'| sed 's/[[]//' | sed 's/[/]//g' |sed 's/://g' )
DATE=$(date --date "$CDATE" +"%d%m%Y%H%M%S")
#echo "$CDATE"
done < "$FILENAME"
will read one line from $FILENAME into the variable LINE, but then the first call to awk is reading the rest of the lines. The resulting CDATE value is probably too large to fit in a single command line, never mind it containing too many dates. You probably wanted
echo "$LINE" | awk '{print $4}' | ...
A simpler way to strip the undesirable characters from LINE, however, is
CDATE=${LINE//[\/[:]}

Shell script: how to read only a portion of text from a variable

I'm developing a little script using ash shell (not bash).
Now i have a variable with the following composition:
VARIABLE = "number string status"
where number could be any number (actually between 1 and 18 but in the future that number could be higher) the string is a name and status is or on or off
The name usually is only lowercase letter.
Now my problem is to read only the string content in the variable, removing the number and the status.
How i can obtain that?
Two ways; one is to leverage $IFS and use a while loop - this will work for a single line quite happily - as:
echo "Part1 Part2 Part3" | while read a b c
do
echo $a
done
alternatively, use cut as follows:
a=`echo $var | cut -d' ' -f2`
echo $a
How about using cut?
name=$(echo "$variable" | cut -d " " -f 2)
UPDATE
Apparently, Ash doesn't understand $(...). Hopefully you can do this instead:
name=`echo "$variable" | cut -d " " -f 2`
How about :
name=$(echo "$variable" | awk '{print $2}')
#!/bin/sh
myvar="word1 word2 word3 wordX"
set -- $myvar
echo ${15} # outputs word 15

hash each line in text file

I'm trying to write a little script which will open a text file and give me an md5 hash for each line of text. For example I have a file with:
123
213
312
I want output to be:
ba1f2511fc30423bdbb183fe33f3dd0f
6f36dfd82a1b64f668d9957ad81199ff
390d29f732f024a4ebd58645781dfa5a
I'm trying to do this part in bash which will read each line:
#!/bin/bash
#read.file.line.by.line.sh
while read line
do
echo $line
done
later on I do:
$ more 123.txt | ./read.line.by.line.sh | md5sum | cut -d ' ' -f 1
but I'm missing something here, does not work :(
Maybe there is an easier way...
Almost there, try this:
while read -r line; do printf %s "$line" | md5sum | cut -f1 -d' '; done < 123.txt
Unless you also want to hash the newline character in every line you should use printf or echo -n instead of echo option.
In a script:
#! /bin/bash
cat "$#" | while read -r line; do
printf %s "$line" | md5sum | cut -f1 -d' '
done
The script can be called with multiple files as parameters.
You can just call md5sum directly in the script:
#!/bin/bash
#read.file.line.by.line.sh
while read line
do
echo $line | md5sum | awk '{print $1}'
done
That way the script spits out directly what you want: the md5 hash of each line.
this worked for me..
cat $file | while read line; do printf %s "$line" | tr -d '\r\n' | md5 >> hashes.csv; done

Resources