SO I am making a program that tests the average REad/Write speed of the hard drive using the dd command and my code is as follows(bash):
a=1
b=1
numval=3
for i in `seq 1 3`;
do
TEST$i=$(dd if=/dev/zero of=speedtest bs=1M count=100 conv=fdatasync)
#I think that this is the problem line
done
RESULT=$(($TEST1 + $TEST2))
RESULT=$(($RESULT + $TEST3))
RESULT=$(($RESULT / $numval))
echo $RESULT > Result
The code above returns the following errors (in between the dd outputs):
TEST1=: command not found
TEST2=: command not found
TEST3=: command not found
Please help (believe it or not) this is for a school project
edit: I understand that my variable does not have a valid name. but Im wondering if there is a way to do this without this shit: "^$({$-%})$" REGEX? is there way to do it without that?
You have (at least) two problems.
TEST$i=... is not valid bash syntax for a variable assignment. And if the first "word" in a command line is not a valid assignment, then it's treated as a command name. So bash goes ahead and substitutes the value of $i for $i and the output of the dd command for $(dd ...) (see below), ending up with the successive "commands" TEST1=, TEST2= and TEST3=. Those aren't known commands, so it complains.
In an assignment, the only characters you can put before the = are letters, numbers and _ (unless it is an array assignment), which means that you cannot use parameter substitution to create a variable name. (But you could use an array.)
You seem to be assuming that the dd command will output the amount of time it took, or something like that. It doesn't. In fact, it doesn't output anything on stdout. It will output several lines on stderr, but stderr isn't captured with $(...)
First problem: you can't use a variable name that's defined in terms of other variables (TEST$i=...) without jumping through some special hops. There are several ways around this. You could use the declare command (declare TEST$i=...), or use an array (TEST[i]=... and then e.g. RESULT=$((TEST[1] + TEST[2]))), or what I'd recommend: accumulate the times as you go without bothering with the numbered TEST1 etc variables:
numval=3
result=0
for i in `seq 1 $numval`; do
test=$(dd if=/dev/zero of=speedtest bs=1M count=100 conv=fdatasync)
result=$((result + test))
done
result=$((result / numval))
(Note that I prefer to use lowercase variable names in shell scripts, to avoid accidentally using one of the shell's predefined variables and making a mess. Also, inside $(( )), variables are automatically replaced, so you don't need $ there.)
However, this still won't work because...
Second problem: dd doesn't output a number. In fact, it doesn't output anything to standard output (which $( ) captures). What it does is output a bunch of numbers and other such things to standard error. Your version of dd is a bit different from mine, but its stderr output is probably something like this:
$ dd if=/dev/zero of=/dev/null bs=1m count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 0.011789 secs (8894645697 bytes/sec)
... and you presumably want to pick out the bytes/sec figure. Depending on your dd's exact output, something like this might work:
$ dd if=/dev/zero of=/dev/null bs=1m count=100 2>&1 | sed -n 's/^.*(\([0-9]*\) bytes.*$/\1/p'
8746239457
What this does is redirect dd's error output to standard output (2>&1), then pipe (|) that to a somewhat messy sed command that looks for something matching "(", then a bunch of digits, then " bytes", and outputs just the digits part.
Here's the full script I wind up with:
#!/bin/bash
numval=3
result=0
for i in `seq 1 $numval`; do
test=$(dd if=/dev/zero of=speedtest bs=1M count=100 conv=fdatasync 2>&1 | sed -n 's/^.*(\([0-9]*\) bytes.*$/\1/p')
result=$((result + test))
done
result=$((result / numval))
echo "$result" >Result
Related
I have been using socat to pull ASCII streams over UDP and write them to files. The following is one such line.
socat UDP-RECV:$UDP_PORT,reuseaddr - | cat >> $INSTRUMENT_$JDAY_RAW &
Each stream being received already has its data timestamped by the sender using ts (part of moreutils) with the year, Julian day, hour, min, second, and msec. If the Julian day changes, the JDAY variable on the receiving end doesn't get reinitialized and cat merrily keeps piping data into the same file with yesterday's timestamp.
Here is an example of the udp stream being received by socat. It is being recorded at 20hz.
2015 317 06 34 43 303 winch680 000117.9 00000000 00000000.0
2015 317 06 34 43 353 winch680 000117.5 00000000 00000000.0
Is there some way in bash I can take each line received by socat, examine the jday timestamp field, and change the output file according to that timestamp?
You may parse the input stream using the read built-in program in bash. You may obtain further information with $ help read. It normally separates tokens using whitespace. If you provided a two-line preview of what your output looks like, it might be easier to help.
The variables $INSTRUMENT, and $JDAY have to be defined before that cat command is launched, because cat will open the file before it starts writing to it.
If $JDAY and $INSTRUMENT are somehow to be extracted from each line, you can use the following bash snippet (assuming lines read by socat look like <INSTRUMENT> <JDAY> <TS> yaddi yadda ...):
function triage_per_day () {
while read INSTRUMENT JDAY TS REST; do
echo "$TS $REST" >> "${INSTRUMENT}_${JDAY}_RAW";
done
}
triage_per_day < <(socat UDP-RECV:"${UDP_PORT}",reuseaddr -)
If you want to get fancier, you can use file handles to help bash run a bit faster. You can use file descriptor redirections to keep outputting to the same file as long as the day is the same. This will minimize the number of file opens and closes bash has to do.
function triage_per_day () {
local LAST_JDAY=init
exec 5>&1 # save stdout
exec 1>&2 # echos are sent to stderr until JDAY is redefined
while read INSTRUMENT JDAY TS REST; do
if [[ "$JDAY" != "$LAST_JDAY" ]]; then
# we need to change output file
# send stdout to file in append mode
exec 1>>"${INSTRUMENT}_${JDAY}_RAW"
LAST_JDAY="${JDAY}"
fi
echo "$TS $REST"
done
exec 1>&5 # restore stdout
exec 5>&- # close stdout copy
}
triage_per_day < <(socat UDP-RECV:"${UDP_PORT}",reuseaddr -)
If you wish to tokenize your lines over different characters than whitespace, say ',' commas, you can locally modify the special variable IFS:
function extract_ts () {
local IFS=,; # special bash variable: internal-field-separator
# $REST will contain everything after the third token. it is a good
# practice to specify one more name than your last token of interest.
while read TOK1 TS REST; do
echo "timestamp is $TS";
done
}
If you need fancier processing of each line to extract timestamps and other fields, you may instead execute external programs (python/perl/cut/awk/grep, etc.), but this will be much slower than simply sticking with the bash builtin functions like read or echo. If you have to do this, and speed is an issue, you may consider changing your script to a different language that gives you the expressiveness you need. You may wish to also look into bash Pattern substitution in the manual if you need fancy regular expressions.
function extract_ts () {
# store each line in the variable $LINE
while read LINE; do
TS="$(echo "$LINE" | ...)";
echo "Timestamp is $TS";
done
}
Recommended practices
Also, I should mention that it is good practice to surround your bash variables in double quotes (like in the answer) if you intend to use them as filename parameters. This is especially true if the names contain spaces or special characters -- like could be expected from a filename derived from dates or times. In cases where your variables expand to nothing (due to human or programming error), positional parameters will be missing, with sometimes bad repercussions.
Consider:
# copy two files to the directory (bad)
$ cp file1 file2 $MYDIR
If $MYDIR is undefined, then this command amounts to overwriting file2 with the contents of file1. Contrast this with cp file1 file2 "$MYDIR" which will fail early because the target "" does not exist.
Another source for problems that I see in your question is the variable names followed by underscores _, like $INSTRUMENT. Those should be surrounded in curly braces { }.
INSTRUMENT=6
BAR=49
echo $INSTRUMENT_$BAR # prints '49', but you may have expected 6_49
Because _ are valid characters in variable names, bash will attempt to greedily 'glue' the '_' after INSTRUMENT to match the longest valid variable name possible, which would be $INSTRUMENT_. This variable is undefined however, and expands to the empty string, so you're left with the rest, $BAR. This example can be correctly rewritten as:
INSTRUMENT=6
BAR=49
echo ${INSTRUMENT}_${BAR} # prints 6_49
or even better (avoiding future surprises if values ever change)
echo "${INSTRUMENT}_${BAR}" # prints 6_49
Not with cat. You'll need a [not bash] script (e.g. perl/python or C program).
Replace:
socat UDP-RECV:$UDP_PORT,reuseaddr - | cat >> $INSTRUMENT_$JDAY_RAW &
With:
socat UDP-RECV:$UDP_PORT,reuseaddr - | myscript &
Where myscript looks like:
while (1) {
get_data_from_socat_on_stdin();
if (jdaynew != jdayold) {
close_output_file();
jdayold = jdaynew;
}
if (output_file_not_open)
open_output_file(jdaynew);
write_data_to_output_file();
}
This is the code that worked for me.
The input udp stream looks like this:
2015 317 06 34 43 303 winch680 000117.9 00000000 00000000.0
#!/bin bash
# This code creates a function which reads the fields in the
# udp stream into a table
# and uses the fields in the table to determine output.
UDP_PORT=5639
function DATAOUT () {
while read YR JDY MIN SEC MSEC INST TENS SPEED LINE; do
echo "$YR $JDY $HR $MIN $SEC $MSEC $INST $TENS $SPEED $LINE" >> "${INST}_${JDY}_RAW";
done
}
DATAOUT < <(socat udp-recv:${UDP_PORT},reuseaddr -)
I have a script which uses parameters. In the shell I run the script like this: ./script 1 2 3 4. I prefer to use a file which contain 1 2 3 4 in a single line and run: ./script `cat file`.
After I call this script in a for loop like this: for i in `./script `cat file` ` but it doesn't work. What is the good syntax?
You cannot nest tilde (command substitution) like this. You can do this bash:
for i in $(./script $(<file))
$(<file) is another way of getting the output of $(cat file)
Use $() instead of \`` for Command Substitution. This, among other things, is one of the many reasons it is better.
That being said using cat file to get a list of words is, at best, a poor idea and, at worst, a broken (and potentially dangerous) one. It will not work with any words that require spaces or use shell globbing characters.
I suggest not doing it in the first place.
man seq
generate a sequence of numbers.
#!/bin/bash
for i in $(seq 1 10)
do
echo $i
done
I am making a script to copy a lot of files from a path to another one.
Any files on the path has, on the first line, a lot of "garbage" untill the word "Return-Path....."
File content example:
§°ç§°*é*é*§°ç§°çççççççReturn-PathOTHERTHINGS
REST
OF
THE
FILE
EOF
Probably sed or awk could help on this.
THE PROBLEM:
I want the whole content of the file, except for anything previous then "Return-Path" and it should be stripped ONLY on the first line, in this way:
Return-PathOTHERTHINGS
REST
OF
THE
FILE
EOF
Important thing: anything before Return-Path is "binary", infact files are seen as binary...
How to solve?
Ok, it's a new day, and now I do feel like coding this for you :-)
The algorithm is described in my other answer to your same question.
#!/bin/bash
################################################################################
# behead.sh
# Mark Setchell
#
# Utility to remove stuff preceding specified string near start of binary file
#
# Usage: behead.sh <infile> <outfile>
################################################################################
IN=$1
OUT=$2
SEARCH="Return-Path"
for i in {0..80}; do
str=$(dd if="$1" bs=1 count=${#SEARCH} iseek=$i 2> /dev/null)
if [ "$str" == $SEARCH ]; then
# The following line will go faster if you exchange "bs" and "iseek"
# parameters, because it will work in bigger blocks, it just looks
# wrong, so I haven't done it.
dd if="$1" of="$OUT" bs=1 iseek=$i 2> /dev/null
exit $?
fi
done
echo String not found, sorry.
exit 1
You can test it works like this:
#
# Create binary with 15 bytes of bash, then "Return-Path", then entire bash in file "bashed"
(dd if=/bin/bash bs=1 count=15 2>/dev/null; echo -n 'Return-Path'; cat /bin/bash) > bashed
#
# Chop off junk at start of "bashed" and save in "restored"
./behead.sh bashed restored
#
# Check the restored "bash" is exactly 11 bytes longer than original,
# as it has "Return-Path" at the beginning
ls -l bashed restored
If you save my script as "behead.sh" you will need to make it executable like this:
chmod +x behead.sh
Then you can run it like this:
./behead.sh inputfile outputfile
By the way, there is no concept of "a line" in a binary file, so I have assumed the first 80 characters - you are free to change it, of course!
Try:
sed '1s/.*Return-Path/Return-Path/'
This command substitutes anything before "Return-Path" with "Return-Path" only on the first line.
I don't feel like coding this at this minute, but can give you a hint maybe. "Return-Path" is 11 characters. You can get 11 characters from a file at offset "n" with
dd if=file bs=1 count=11 iseek=n
So if you do a loop with "n" starting at zero and increasing till the result matches "Return-Path" you can calculate how many bytes you need to remove off the front. Then you can do that with another "dd".
Alternatively, have a look at running the file through "xxd", editing that with "sed" and then running it back through "xxd" the other way with "xxd -r".
What might be the most concise way in bash to convert a number into a bitfield character string like 1101?
In effect I am trying to do the opposite of
echo $[2#1101]
Why: I need to send a parameter to a program that takes bitfields in the form of a full string like "0011010110" but often only need to enable one or few bits as in:
SUPPRESSbits=$[1<<16] runscript.sh # OR
SUPPRESSbits=$[1<<3 + 1<<9] runscript.sh # much more readable when I know what bits 3 and 9 toggle in the program
Then runscript.sh then sees in its env a SUPPRESSbits=65536 rather than SUPPRESSbits="1000000000000000" and ends in parse error.
The easy way:
$ dc <<<2o123p
1111011
$ bc <<<'obase=2; 123'
1111011
I doubt about bash but you always can use perl:
a=123; b=$(perl -e 'printf "%b", "'$a'"'); echo $b
1111011
Suppose I've got a list of files
file1
"file 1"
file2
a for...in loop breaks it up between whitespace, not newlines:
for x in $( ls ); do
echo $x
done
results:
file
1
file1
file2
I want to execute a command on each file. "file" and "1" above are not actual files. How can I do that if the filenames contains things like spaces or commas?
It's a little trickier than I think find -print0 | xargs -0 could handle, because I actually want the command to be something like "convert input/file1.jpg .... output/file1.jpg" so I need to permutate the filename in the process.
Actually, Mark's suggestion works fine without even doing anything to the internal field separator. The problem is running ls in a subshell, whether by backticks or $( ) causes the for loop to be unable to distinguish between spaces in names. Simply using
for f in *
instead of the ls solves the problem.
#!/bin/bash
for f in *
do
echo "$f"
done
UPDATE BY OP: this answer sucks and shouldn't be on top ... #Jordan's post below should be the accepted answer.
one possible way:
ls -1 | while read x; do
echo $x
done
I know this one is LONG past "answered", and with all due respect to eduffy, I came up with a better way and I thought I'd share it.
What's "wrong" with eduffy's answer isn't that it's wrong, but that it imposes what for me is a painful limitation: there's an implied creation of a subshell when the output of the ls is piped and this means that variables set inside the loop are lost after the loop exits. Thus, if you want to write some more sophisticated code, you have a pain in the buttocks to deal with.
My solution was to take the "readline" function and write a program out of it in which you can specify any specific line number that you may want that results from any given function call. ... As a simple example, starting with eduffy's:
ls_output=$(ls -1)
# The cut at the end of the following line removes any trailing new line character
declare -i line_count=$(echo "$ls_output" | wc -l | cut -d ' ' -f 1)
declare -i cur_line=1
while [ $cur_line -le $line_count ] ;
do
# NONE of the values in the variables inside this do loop are trapped here.
filename=$(echo "$ls_output" | readline -n $cur_line)
# Now line contains a filename from the preceeding ls command
cur_line=cur_line+1
done
Now you have wrapped up all the subshell activity into neat little contained packages and can go about your shell coding without having to worry about the scope of your variable values getting trapped in subshells.
I wrote my version of readline in gnuc if anyone wants a copy, it's a little big to post here, but maybe we can find a way...
Hope this helps,
RT