I have a script that aims to find out which key is pressed. The problem is that I don't manage it quickly because I need the timeout in the fifth digit that makes it not react quickly, or not react at all.
#!/bin/bash
sudo echo Start
while true
do
file_content=$(sudo timeout 0.5s cat /dev/input/event12 | hexdump)
content_split=$(echo $file_content | tr " " "\n")
word_counter=0
for option in $content_split
do
word_counter=$((word_counter+1))
if [ $word_counter -eq 25 ]
then
case $option in
"0039")echo "<space>";;
"001c")echo "<return>";;
"001e")echo "a";;
"0030")echo "b";;
"002e")echo "c";;
"0020")echo "d";;
"0012")echo "e";;
"0021")echo "f";;
"0022")echo "g";;
"0023")echo "h";;
"0017")echo "i";;
"0024")echo "j";;
"0025")echo "k";;
"0026")echo "l";;
"0032")echo "m";;
"0031")echo "n";;
"0018")echo "o";;
"0019")echo "p";;
"0010")echo "q";;
"0013")echo "r";;
"001f")echo "s";;
"0014")echo "t";;
"0016")echo "u";;
"002f")echo "v";;
"0011")echo "w";;
"002d")echo "x";;
"002c")echo "y";;
"0015")echo "z";;
esac
fi
done
done
Do not run cat in a timeout in a loop - that's just an invalid way to look at the problem. No matter how "fast" your program runs it will always miss some events that way. Overall polling approach is just invalid here.
The parsing and linux philosophy is build around streams that transfers bytes. The always available streams are stdin, stdout and stderr. They allow to pass data from one context to another. The shell most common | operator allows to bind together output from one program to input of another program - and this is the way™ you should work in shell. Shell primary use is to "connect" programs together.
So you could do:
# read from the input
sudo cat /dev/input/mouse1 |
# transform input to hex data one byte at at time so
# we could parse it in **shell**
xxd -c1 -p |
# read one byte and parse it in **shell**
while IFS= read -r line; do
: parse line
done
but shell is very slow and while read is very slow. If you want speed (and events from input are going to be fast) do not use shell and use a good programming language - python, perl, ruby, at least awk - these are common scripting languages. The case $option in construct looks like a mapping from hex values to output strings. I could see:
# save one `cat` process by just calling xxd
sudo xxd -c1 -p dev/input/mouse1 |
awk 'BEGIN{
map["1c"]="<return>";
map["1e"]="a";
# etc. add all other cases to mapping
}
{ if ($0 in a) print map[$0] }
'
Related
I'm having difficulty grasping how pipes work. Initially I thought of them as per the title but I couldn't get a simple example to work e.g.
mkdir temp
cd temp
echo "rubbish" > txtfile
ls | cat
I'm wondering why it returns the output from 'ls' rather than the output of 'cat txtfile' (i.e. "rubbish"). I've read many pipe tutorials but none of them seem to go beyond "STDOUT of LHS becomes STDIN for RHS" and I'm left wondering what is STDIN of RHS. Does it become the first argument? Where does it slot in when RHS of pipe has options or more than one argument. Is there any kind of macro substitution taking place or is my thinking wide of the mark.
Edit: I'm still none the wiser 5 comments later. I'll certainly take a look at Roadowl's pv utility but for now if I type
ls | cut -c 2-4
I get
xtf
which I'd expect. So, does cut take its input from stdin but cat doesn't?
Edit2: I stuck the question up on askubuntu (I originally put it up here by mistake). The answer there https://askubuntu.com/questions/1316848/does-output-from-lhs-of-pipe-become-an-arg-for-rhs-of-pipe throws a bit more light on it.
Edit3: While reading the answers here and ask ubuntu and the links therein it struck me (again) how woeful bash (& cohorts) are. It's almost like they're designed to trip you up. I only started using bash a couple of months back and every time I write a script I have to read endless web pages to get it to work or discover where I'm going wrong. Take a simple [[ $1=="..." ]] condition. You forget the spaces round the operator and the else condition might wipe some files you want without so much as a warning. Yes, you can do great things with it without a lot of typing but at times it's like using a tightrope to get from skyscraper A to skyscraper B to avoid using 2 lifts. What's up with gold c code like cat(ls())? That said, thanks to everyone who contributed.
I guess, you meant while performing
ls | cat
ls should return txtfile and which should go as a file input to cat command.
But, the things happening in the background are different :
First your shell creates a pipe using pipe(int pipefd[2]) system-call. This pipe has 2 ends, one is read and another is write.
When ls command is executing, it writes its output to the write end of the pipe and cat simultaneously reads from the read end of the pipe.
So, here STDOUT of ls is the write end whereas STDIN for cat is read end of the pipe.
While reading from the pipe cat will consider it as a stream of bytes and not as a name of the file.
So basically, cat is printing whatever is coming as a stream of bytes.
Read about pipe() over here : pipe(2) — Linux manual page
ls | cut -c 2-4
Here, cut reads its standard input, gets the line txtfile, takes characters 2 to 4 from it, producing xtf, and prints that on standard output. That's what the command line option tells it to do.
ls | cat
Here, cat reads its standard input, gets the line txtfile, and prints that on standard output, unchanged. That's what cat does. If there were further lines, it would do the same for those.
Both read standard input unless one or more file names are given as arguments. That standard input is connected to the terminal (the same one where you enter the command line), unless you use pipes or redirections to change that.
So, run the command cut -c 2-4, and enter the line abcdefghijkl, and it will print out bcd. Because without any arguments, it reads its standard input, which is the terminal, by default. Similarly for running just cat, you'll get back the same line you entered.
Running ls | cut -c 2-4 changes where the standard input comes from, but it doesn't create any new command line arguments (other than the -c and 2-4 you gave). Command line arguments are not the same as the standard input.
So, echo txtfile | cat is not the same as running cat txtfile, any more than running echo txtfile | cut -c 2-4 is the same as running cut -c 2-4 txtfile. For some reason, you seem to expect the pipe should work differently for cat than it does for cut.
This question already has answers here:
How do I iterate over a range of numbers defined by variables in Bash?
(20 answers)
Closed 3 years ago.
I am still learning how to shell script and I have been given a challenge to make it easier for me to echo "Name1" "Name2"..."Name15" and I'm not too sure where to start, I've had ideas but I don't want to look silly if I mess it up. Any help?
I haven't actually tried anything just yet it's all just been mostly thought.
#This is what I wrote to start
#!/bin/bash
echo "Name1"
echo "Name2"
echo "Name3"
echo "Name4"
echo "Name5"
echo "Name6"
echo "Name7"
echo "Name8"
echo "Name9"
echo "Name10"
echo "Name11"
echo "Name12"
echo "Name13"
echo "Name14"
echo "Name15"
My expected results are obviously just for it to output "Name1" "Name2" etc. But I'm looking for a more creative way to do it. If possible throw in a few ways to do it so I can learn. Thank you.
The easiest (possibly not the most creative) way to do this is to use printf:
printf "%s\n" name{1..15}
This relies on bash brace expansion {1..15} to have the 15 strings.
Use a for loop
for i in {1..15};do echo "Name$i";done
A few esoteric solutions, from the least to the most unreasonable :
base64 encoded string :
base64 -d <<<TmFtZTEKTmFtZTIKTmFtZTMKTmFtZTQKTmFtZTUKTmFtZTYKTmFtZTcKTmFtZTgKTmFtZTkKTmFtZTEwCk5hbWUxMQpOYW1lMTIKTmFtZTEzCk5hbWUxNApOYW1lMTUK
The weird chain is your expected result encoded in base64, an encoding generally used to represent binary data as text. base64 -d <<< weirdChain is passing the weird chain as input to the base64 tool and asking it to decode it, which displays your expected result
generate an infinite stream of "Name", truncate it, use line numbers :
yes Name | awk 'NR == 16 { exit } { printf("%s%s\n", $0, NR) }'
yes outputs an infinite stream of what it's passed as argument (or y by default, used to automatize interactive scripts asking for [y/n] confirmation). The awk command exits once it reaches the 16th line, and otherwise prints its input (provided by yes) followed by the line number. The truncature could as easily be done with head -15, and I've tried using the nl "number line" utility or grep -n to number lines, but they always added the line numbers as prefix which required an extra re-formatting step.
read random binary data and hope to stumble on all the lines you want to output :
timeout 1d strings /dev/urandom | grep -Eo "Name(1[0-5]|[1-9])" | sort -uV
strings /dev/urandom will extract ascii sequences from the binary random source /dev/urandom, grep will filter those which respect the format of a line of your expected output and sort will reorder those lines in the correct order. Since sort needs to have a received its whole input before it reorders it and /dev/urandom won't stop producing data, we use timeout 1d to stop reading from /dev/urandom after a whole day in hope it has sifted through enough random data to find your 15 lines (I'm not sure that's even remotely likely).
use an HTTP client to retrieve this page, extract the bash script you posted and execute it.
my_old_script=$(curl "https://stackoverflow.com/questions/57818680/" | grep "#This is what I wrote to start" -A 18 | tail -n+4)
eval "$my_old_script"
curl is a command line tool that can be used as an HTTP client, grep with its -A 18 parameter will select the "This is what I wrote to start" text and the 18 lines that follow, tail will remove the first 3 lines, and eval will execute your script.
While it will be much more efficient than the previous solution, it's an even less reasonable solution because high-rep users can edit your question to make this solution execute arbitrary code on your computer. Ideally you'd be using an HTML-aware parser rather than basic string manipulation to extract the code, but we're not talking about best practices here...
I have a stream that is null delimited, with an unknown number of sections. For each delimited section I want to pipe it into another pipeline until the last section has been read, and then terminate.
In practice, each section is very large (~1GB), so I would like to do this without reading each section into memory.
For example, imagine I have the stream created by:
for I in {3..5}; do seq $I; echo -ne '\0';
done
I'll get a steam that looks like:
1
2
3
^#1
2
3
4
^#1
2
3
4
5
^#
When piped through cat -v.
I would like to pipe each section through paste -sd+ | bc, so I get a stream that looks like:
6
10
15
This is simply an example. In actuality the stream is much larger and the pipeline is more complicated, so solutions that don't rely on streams are not feasible.
I've tried something like:
set -eo pipefail
while head -zn1 | head -c-1 | ifne -n false | paste -sd+ | bc; do :; done
but I only get
6
10
If I leave off bc I get
1+2+3
1+2+3+4
1+2+3+4+5
which is basically correct. This leads me to believe that the issue is potentially related to buffering and the way each process is actually interacting with the pipes between them.
Is there some way to fix the way that these commands exchange streams so that I can get the desired output? Or, alternatively, is there a way to accomplish this with other means?
In principle this is related to this question, and I could certainly write a program that reads stdin into a buffer, looks for the null character, and pipes the output to a spawned subprocess, as the accepted answer does for that question. Given the general support of streams and null delimiters in bash, I'm hoping to do something that's a little more "native". In particular, if I want to go this route, I'll have to escape the pipeline (paste -sd+ | bc) in a string instead of just letting the same shell interpret it. There's nothing too inherently bad about this, but it's a little ugly and will require a bunch of somewhat error prone escaping.
Edit
As was pointed out in an answer, head makes no guarantees about how much it buffers. Unless it only buffers single byte at a time, which would be impractical, this will never work. Thus, it seems like the only solution would be to read it into memory, or write a specific program.
The issue with your original code is that head doesn't guarantee that it won't read more than it outputs. Thus, it can consume more than one (NUL-delimited) chunk of input, even if it's emitting only one chunk of output.
read, by contrast, guarantees that it won't consume more than you ask it for.
set -o pipefail
while IFS= read -r -d '' line; do
bc <<<"${line//$'\n'/+}"
done < <(build_a_stream)
If you want native logic, there's nothing more native than just writing the whole thing in shell.
Calling external tools -- including bc, cut, paste, or others -- involves a fork() penalty. If you're only processing small amounts of data per invocation, the efficiency of the tools is overwhelmed by the cost of starting them.
while read -r -d '' -a numbers; do # read up to the next NUL into an array
sum=0 # initialize an accumulator
for number in "${numbers[#]}"; do # iterate over that array
(( sum += number )) # ...using an arithmetic context for our math
done
printf '%s\n' "$sum"
done < <(build_a_stream)
For all of the above, I tested with the following build_a_stream implementation:
build_a_stream() {
local i j IFS=$'\n'
local -a numbers
for ((i=3; i<=5; i++)); do
numbers=( )
for ((j=0; j<=i; j++)); do
numbers+=( "$j" )
done
printf '%s\0' "${numbers[*]}"
done
}
As discussed, the only real solution seemed to be writing a program to do this specifically. I wrote one in rust called xstream-util. After installing it with cargo install xstream-util, you can pipe the input into
xstream -0 -- bash -c 'paste -sd+ | bc'
to get the desired output
6
10
15
It doesn't avoid having to run the program in bash, so it still needs escaping if the pipeline is complicated. Also, it currently only supports single byte delimiters.
I have been using socat to pull ASCII streams over UDP and write them to files. The following is one such line.
socat UDP-RECV:$UDP_PORT,reuseaddr - | cat >> $INSTRUMENT_$JDAY_RAW &
Each stream being received already has its data timestamped by the sender using ts (part of moreutils) with the year, Julian day, hour, min, second, and msec. If the Julian day changes, the JDAY variable on the receiving end doesn't get reinitialized and cat merrily keeps piping data into the same file with yesterday's timestamp.
Here is an example of the udp stream being received by socat. It is being recorded at 20hz.
2015 317 06 34 43 303 winch680 000117.9 00000000 00000000.0
2015 317 06 34 43 353 winch680 000117.5 00000000 00000000.0
Is there some way in bash I can take each line received by socat, examine the jday timestamp field, and change the output file according to that timestamp?
You may parse the input stream using the read built-in program in bash. You may obtain further information with $ help read. It normally separates tokens using whitespace. If you provided a two-line preview of what your output looks like, it might be easier to help.
The variables $INSTRUMENT, and $JDAY have to be defined before that cat command is launched, because cat will open the file before it starts writing to it.
If $JDAY and $INSTRUMENT are somehow to be extracted from each line, you can use the following bash snippet (assuming lines read by socat look like <INSTRUMENT> <JDAY> <TS> yaddi yadda ...):
function triage_per_day () {
while read INSTRUMENT JDAY TS REST; do
echo "$TS $REST" >> "${INSTRUMENT}_${JDAY}_RAW";
done
}
triage_per_day < <(socat UDP-RECV:"${UDP_PORT}",reuseaddr -)
If you want to get fancier, you can use file handles to help bash run a bit faster. You can use file descriptor redirections to keep outputting to the same file as long as the day is the same. This will minimize the number of file opens and closes bash has to do.
function triage_per_day () {
local LAST_JDAY=init
exec 5>&1 # save stdout
exec 1>&2 # echos are sent to stderr until JDAY is redefined
while read INSTRUMENT JDAY TS REST; do
if [[ "$JDAY" != "$LAST_JDAY" ]]; then
# we need to change output file
# send stdout to file in append mode
exec 1>>"${INSTRUMENT}_${JDAY}_RAW"
LAST_JDAY="${JDAY}"
fi
echo "$TS $REST"
done
exec 1>&5 # restore stdout
exec 5>&- # close stdout copy
}
triage_per_day < <(socat UDP-RECV:"${UDP_PORT}",reuseaddr -)
If you wish to tokenize your lines over different characters than whitespace, say ',' commas, you can locally modify the special variable IFS:
function extract_ts () {
local IFS=,; # special bash variable: internal-field-separator
# $REST will contain everything after the third token. it is a good
# practice to specify one more name than your last token of interest.
while read TOK1 TS REST; do
echo "timestamp is $TS";
done
}
If you need fancier processing of each line to extract timestamps and other fields, you may instead execute external programs (python/perl/cut/awk/grep, etc.), but this will be much slower than simply sticking with the bash builtin functions like read or echo. If you have to do this, and speed is an issue, you may consider changing your script to a different language that gives you the expressiveness you need. You may wish to also look into bash Pattern substitution in the manual if you need fancy regular expressions.
function extract_ts () {
# store each line in the variable $LINE
while read LINE; do
TS="$(echo "$LINE" | ...)";
echo "Timestamp is $TS";
done
}
Recommended practices
Also, I should mention that it is good practice to surround your bash variables in double quotes (like in the answer) if you intend to use them as filename parameters. This is especially true if the names contain spaces or special characters -- like could be expected from a filename derived from dates or times. In cases where your variables expand to nothing (due to human or programming error), positional parameters will be missing, with sometimes bad repercussions.
Consider:
# copy two files to the directory (bad)
$ cp file1 file2 $MYDIR
If $MYDIR is undefined, then this command amounts to overwriting file2 with the contents of file1. Contrast this with cp file1 file2 "$MYDIR" which will fail early because the target "" does not exist.
Another source for problems that I see in your question is the variable names followed by underscores _, like $INSTRUMENT. Those should be surrounded in curly braces { }.
INSTRUMENT=6
BAR=49
echo $INSTRUMENT_$BAR # prints '49', but you may have expected 6_49
Because _ are valid characters in variable names, bash will attempt to greedily 'glue' the '_' after INSTRUMENT to match the longest valid variable name possible, which would be $INSTRUMENT_. This variable is undefined however, and expands to the empty string, so you're left with the rest, $BAR. This example can be correctly rewritten as:
INSTRUMENT=6
BAR=49
echo ${INSTRUMENT}_${BAR} # prints 6_49
or even better (avoiding future surprises if values ever change)
echo "${INSTRUMENT}_${BAR}" # prints 6_49
Not with cat. You'll need a [not bash] script (e.g. perl/python or C program).
Replace:
socat UDP-RECV:$UDP_PORT,reuseaddr - | cat >> $INSTRUMENT_$JDAY_RAW &
With:
socat UDP-RECV:$UDP_PORT,reuseaddr - | myscript &
Where myscript looks like:
while (1) {
get_data_from_socat_on_stdin();
if (jdaynew != jdayold) {
close_output_file();
jdayold = jdaynew;
}
if (output_file_not_open)
open_output_file(jdaynew);
write_data_to_output_file();
}
This is the code that worked for me.
The input udp stream looks like this:
2015 317 06 34 43 303 winch680 000117.9 00000000 00000000.0
#!/bin bash
# This code creates a function which reads the fields in the
# udp stream into a table
# and uses the fields in the table to determine output.
UDP_PORT=5639
function DATAOUT () {
while read YR JDY MIN SEC MSEC INST TENS SPEED LINE; do
echo "$YR $JDY $HR $MIN $SEC $MSEC $INST $TENS $SPEED $LINE" >> "${INST}_${JDY}_RAW";
done
}
DATAOUT < <(socat udp-recv:${UDP_PORT},reuseaddr -)
In Shell scripting (Linux/Ubuntu , Bash) , why do we use echo and bc commands together ? I am new to Shell scripting and have a basic understanding of pipes .
I know that bc is kind of a seperate language . How does the following statement actually work (Just an example) ?
echo 5+6 | bc
The first command (echo) writes the expression "5+6" to its standard output. This is then piped to bc's standard input, read from there, and evaluated.
This is used since bc doesn't take the expression as a direct input, it always reads from files and/or standard input.
You can use that program combination for another set of powerful operations, for example you can convert from hexadecimal to binary like this
echo "ibase=16; obase=2; A15" | bc
It will print: 101000010101
As for the process of echoing and using the | operator, it just make the output of the echocommand an input for the bc program, you can achieve the same using for example: bc <<< "5 + 2"
bc does not read operations from command line arguments, instead it reads it from an input file or in an interactive session
Another example of this useful combination is the calculation of really big quantities, like:
echo "2^1024" | bc
A note about the <<<: it passes a string on a single line as an input file to the command, if the program reads its input from a file, with <<< you can convert a string to a "file" and then pass this "file" to the program.
echo is not required here and can be replaced by an here document:
bc <<%
5+6
%
or with modern shells:
bc <<< 5+6
Before bc command we can see very simple PIPE:
|
The name pipe is very accurate! Like a normal pipe this one is transferring water from source to target. In computer science water is called data or information.
Like every good pipe, both endings of it are special. Through those endings we can connect pipe to other pipes, taps, joints, etc.
One ending of this pipe is connected to bc which has matching ending. Bc is a big piece of software, so it has many different connection points for different pipes. Also for this simple | pipe.
On the other hand 5+6 is not any piece of software. It is pure data/water. You can imagine what will happen if you pure water to one end of the pipe without fixed connection! Lack of water pressure...
We need some software which has good connection to that pipe. Echo is very simple application, doing practically nothing, like decent echo should do... But it has basic and functional ending matching that simple pipe.
A little bit of text formatting
echo -n "pi=" ; bc -l <<< "scale=10; 4*a(1)"
pi=3.1415926532