In Shell scripting (Linux/Ubuntu , Bash) , why do we use echo and bc commands together ? I am new to Shell scripting and have a basic understanding of pipes .
I know that bc is kind of a seperate language . How does the following statement actually work (Just an example) ?
echo 5+6 | bc
The first command (echo) writes the expression "5+6" to its standard output. This is then piped to bc's standard input, read from there, and evaluated.
This is used since bc doesn't take the expression as a direct input, it always reads from files and/or standard input.
You can use that program combination for another set of powerful operations, for example you can convert from hexadecimal to binary like this
echo "ibase=16; obase=2; A15" | bc
It will print: 101000010101
As for the process of echoing and using the | operator, it just make the output of the echocommand an input for the bc program, you can achieve the same using for example: bc <<< "5 + 2"
bc does not read operations from command line arguments, instead it reads it from an input file or in an interactive session
Another example of this useful combination is the calculation of really big quantities, like:
echo "2^1024" | bc
A note about the <<<: it passes a string on a single line as an input file to the command, if the program reads its input from a file, with <<< you can convert a string to a "file" and then pass this "file" to the program.
echo is not required here and can be replaced by an here document:
bc <<%
5+6
%
or with modern shells:
bc <<< 5+6
Before bc command we can see very simple PIPE:
|
The name pipe is very accurate! Like a normal pipe this one is transferring water from source to target. In computer science water is called data or information.
Like every good pipe, both endings of it are special. Through those endings we can connect pipe to other pipes, taps, joints, etc.
One ending of this pipe is connected to bc which has matching ending. Bc is a big piece of software, so it has many different connection points for different pipes. Also for this simple | pipe.
On the other hand 5+6 is not any piece of software. It is pure data/water. You can imagine what will happen if you pure water to one end of the pipe without fixed connection! Lack of water pressure...
We need some software which has good connection to that pipe. Echo is very simple application, doing practically nothing, like decent echo should do... But it has basic and functional ending matching that simple pipe.
A little bit of text formatting
echo -n "pi=" ; bc -l <<< "scale=10; 4*a(1)"
pi=3.1415926532
Related
I have a script that aims to find out which key is pressed. The problem is that I don't manage it quickly because I need the timeout in the fifth digit that makes it not react quickly, or not react at all.
#!/bin/bash
sudo echo Start
while true
do
file_content=$(sudo timeout 0.5s cat /dev/input/event12 | hexdump)
content_split=$(echo $file_content | tr " " "\n")
word_counter=0
for option in $content_split
do
word_counter=$((word_counter+1))
if [ $word_counter -eq 25 ]
then
case $option in
"0039")echo "<space>";;
"001c")echo "<return>";;
"001e")echo "a";;
"0030")echo "b";;
"002e")echo "c";;
"0020")echo "d";;
"0012")echo "e";;
"0021")echo "f";;
"0022")echo "g";;
"0023")echo "h";;
"0017")echo "i";;
"0024")echo "j";;
"0025")echo "k";;
"0026")echo "l";;
"0032")echo "m";;
"0031")echo "n";;
"0018")echo "o";;
"0019")echo "p";;
"0010")echo "q";;
"0013")echo "r";;
"001f")echo "s";;
"0014")echo "t";;
"0016")echo "u";;
"002f")echo "v";;
"0011")echo "w";;
"002d")echo "x";;
"002c")echo "y";;
"0015")echo "z";;
esac
fi
done
done
Do not run cat in a timeout in a loop - that's just an invalid way to look at the problem. No matter how "fast" your program runs it will always miss some events that way. Overall polling approach is just invalid here.
The parsing and linux philosophy is build around streams that transfers bytes. The always available streams are stdin, stdout and stderr. They allow to pass data from one context to another. The shell most common | operator allows to bind together output from one program to input of another program - and this is the way™ you should work in shell. Shell primary use is to "connect" programs together.
So you could do:
# read from the input
sudo cat /dev/input/mouse1 |
# transform input to hex data one byte at at time so
# we could parse it in **shell**
xxd -c1 -p |
# read one byte and parse it in **shell**
while IFS= read -r line; do
: parse line
done
but shell is very slow and while read is very slow. If you want speed (and events from input are going to be fast) do not use shell and use a good programming language - python, perl, ruby, at least awk - these are common scripting languages. The case $option in construct looks like a mapping from hex values to output strings. I could see:
# save one `cat` process by just calling xxd
sudo xxd -c1 -p dev/input/mouse1 |
awk 'BEGIN{
map["1c"]="<return>";
map["1e"]="a";
# etc. add all other cases to mapping
}
{ if ($0 in a) print map[$0] }
'
I'm having difficulty grasping how pipes work. Initially I thought of them as per the title but I couldn't get a simple example to work e.g.
mkdir temp
cd temp
echo "rubbish" > txtfile
ls | cat
I'm wondering why it returns the output from 'ls' rather than the output of 'cat txtfile' (i.e. "rubbish"). I've read many pipe tutorials but none of them seem to go beyond "STDOUT of LHS becomes STDIN for RHS" and I'm left wondering what is STDIN of RHS. Does it become the first argument? Where does it slot in when RHS of pipe has options or more than one argument. Is there any kind of macro substitution taking place or is my thinking wide of the mark.
Edit: I'm still none the wiser 5 comments later. I'll certainly take a look at Roadowl's pv utility but for now if I type
ls | cut -c 2-4
I get
xtf
which I'd expect. So, does cut take its input from stdin but cat doesn't?
Edit2: I stuck the question up on askubuntu (I originally put it up here by mistake). The answer there https://askubuntu.com/questions/1316848/does-output-from-lhs-of-pipe-become-an-arg-for-rhs-of-pipe throws a bit more light on it.
Edit3: While reading the answers here and ask ubuntu and the links therein it struck me (again) how woeful bash (& cohorts) are. It's almost like they're designed to trip you up. I only started using bash a couple of months back and every time I write a script I have to read endless web pages to get it to work or discover where I'm going wrong. Take a simple [[ $1=="..." ]] condition. You forget the spaces round the operator and the else condition might wipe some files you want without so much as a warning. Yes, you can do great things with it without a lot of typing but at times it's like using a tightrope to get from skyscraper A to skyscraper B to avoid using 2 lifts. What's up with gold c code like cat(ls())? That said, thanks to everyone who contributed.
I guess, you meant while performing
ls | cat
ls should return txtfile and which should go as a file input to cat command.
But, the things happening in the background are different :
First your shell creates a pipe using pipe(int pipefd[2]) system-call. This pipe has 2 ends, one is read and another is write.
When ls command is executing, it writes its output to the write end of the pipe and cat simultaneously reads from the read end of the pipe.
So, here STDOUT of ls is the write end whereas STDIN for cat is read end of the pipe.
While reading from the pipe cat will consider it as a stream of bytes and not as a name of the file.
So basically, cat is printing whatever is coming as a stream of bytes.
Read about pipe() over here : pipe(2) — Linux manual page
ls | cut -c 2-4
Here, cut reads its standard input, gets the line txtfile, takes characters 2 to 4 from it, producing xtf, and prints that on standard output. That's what the command line option tells it to do.
ls | cat
Here, cat reads its standard input, gets the line txtfile, and prints that on standard output, unchanged. That's what cat does. If there were further lines, it would do the same for those.
Both read standard input unless one or more file names are given as arguments. That standard input is connected to the terminal (the same one where you enter the command line), unless you use pipes or redirections to change that.
So, run the command cut -c 2-4, and enter the line abcdefghijkl, and it will print out bcd. Because without any arguments, it reads its standard input, which is the terminal, by default. Similarly for running just cat, you'll get back the same line you entered.
Running ls | cut -c 2-4 changes where the standard input comes from, but it doesn't create any new command line arguments (other than the -c and 2-4 you gave). Command line arguments are not the same as the standard input.
So, echo txtfile | cat is not the same as running cat txtfile, any more than running echo txtfile | cut -c 2-4 is the same as running cut -c 2-4 txtfile. For some reason, you seem to expect the pipe should work differently for cat than it does for cut.
This question already has answers here:
How do I iterate over a range of numbers defined by variables in Bash?
(20 answers)
Closed 3 years ago.
I am still learning how to shell script and I have been given a challenge to make it easier for me to echo "Name1" "Name2"..."Name15" and I'm not too sure where to start, I've had ideas but I don't want to look silly if I mess it up. Any help?
I haven't actually tried anything just yet it's all just been mostly thought.
#This is what I wrote to start
#!/bin/bash
echo "Name1"
echo "Name2"
echo "Name3"
echo "Name4"
echo "Name5"
echo "Name6"
echo "Name7"
echo "Name8"
echo "Name9"
echo "Name10"
echo "Name11"
echo "Name12"
echo "Name13"
echo "Name14"
echo "Name15"
My expected results are obviously just for it to output "Name1" "Name2" etc. But I'm looking for a more creative way to do it. If possible throw in a few ways to do it so I can learn. Thank you.
The easiest (possibly not the most creative) way to do this is to use printf:
printf "%s\n" name{1..15}
This relies on bash brace expansion {1..15} to have the 15 strings.
Use a for loop
for i in {1..15};do echo "Name$i";done
A few esoteric solutions, from the least to the most unreasonable :
base64 encoded string :
base64 -d <<<TmFtZTEKTmFtZTIKTmFtZTMKTmFtZTQKTmFtZTUKTmFtZTYKTmFtZTcKTmFtZTgKTmFtZTkKTmFtZTEwCk5hbWUxMQpOYW1lMTIKTmFtZTEzCk5hbWUxNApOYW1lMTUK
The weird chain is your expected result encoded in base64, an encoding generally used to represent binary data as text. base64 -d <<< weirdChain is passing the weird chain as input to the base64 tool and asking it to decode it, which displays your expected result
generate an infinite stream of "Name", truncate it, use line numbers :
yes Name | awk 'NR == 16 { exit } { printf("%s%s\n", $0, NR) }'
yes outputs an infinite stream of what it's passed as argument (or y by default, used to automatize interactive scripts asking for [y/n] confirmation). The awk command exits once it reaches the 16th line, and otherwise prints its input (provided by yes) followed by the line number. The truncature could as easily be done with head -15, and I've tried using the nl "number line" utility or grep -n to number lines, but they always added the line numbers as prefix which required an extra re-formatting step.
read random binary data and hope to stumble on all the lines you want to output :
timeout 1d strings /dev/urandom | grep -Eo "Name(1[0-5]|[1-9])" | sort -uV
strings /dev/urandom will extract ascii sequences from the binary random source /dev/urandom, grep will filter those which respect the format of a line of your expected output and sort will reorder those lines in the correct order. Since sort needs to have a received its whole input before it reorders it and /dev/urandom won't stop producing data, we use timeout 1d to stop reading from /dev/urandom after a whole day in hope it has sifted through enough random data to find your 15 lines (I'm not sure that's even remotely likely).
use an HTTP client to retrieve this page, extract the bash script you posted and execute it.
my_old_script=$(curl "https://stackoverflow.com/questions/57818680/" | grep "#This is what I wrote to start" -A 18 | tail -n+4)
eval "$my_old_script"
curl is a command line tool that can be used as an HTTP client, grep with its -A 18 parameter will select the "This is what I wrote to start" text and the 18 lines that follow, tail will remove the first 3 lines, and eval will execute your script.
While it will be much more efficient than the previous solution, it's an even less reasonable solution because high-rep users can edit your question to make this solution execute arbitrary code on your computer. Ideally you'd be using an HTML-aware parser rather than basic string manipulation to extract the code, but we're not talking about best practices here...
I have a stream that is null delimited, with an unknown number of sections. For each delimited section I want to pipe it into another pipeline until the last section has been read, and then terminate.
In practice, each section is very large (~1GB), so I would like to do this without reading each section into memory.
For example, imagine I have the stream created by:
for I in {3..5}; do seq $I; echo -ne '\0';
done
I'll get a steam that looks like:
1
2
3
^#1
2
3
4
^#1
2
3
4
5
^#
When piped through cat -v.
I would like to pipe each section through paste -sd+ | bc, so I get a stream that looks like:
6
10
15
This is simply an example. In actuality the stream is much larger and the pipeline is more complicated, so solutions that don't rely on streams are not feasible.
I've tried something like:
set -eo pipefail
while head -zn1 | head -c-1 | ifne -n false | paste -sd+ | bc; do :; done
but I only get
6
10
If I leave off bc I get
1+2+3
1+2+3+4
1+2+3+4+5
which is basically correct. This leads me to believe that the issue is potentially related to buffering and the way each process is actually interacting with the pipes between them.
Is there some way to fix the way that these commands exchange streams so that I can get the desired output? Or, alternatively, is there a way to accomplish this with other means?
In principle this is related to this question, and I could certainly write a program that reads stdin into a buffer, looks for the null character, and pipes the output to a spawned subprocess, as the accepted answer does for that question. Given the general support of streams and null delimiters in bash, I'm hoping to do something that's a little more "native". In particular, if I want to go this route, I'll have to escape the pipeline (paste -sd+ | bc) in a string instead of just letting the same shell interpret it. There's nothing too inherently bad about this, but it's a little ugly and will require a bunch of somewhat error prone escaping.
Edit
As was pointed out in an answer, head makes no guarantees about how much it buffers. Unless it only buffers single byte at a time, which would be impractical, this will never work. Thus, it seems like the only solution would be to read it into memory, or write a specific program.
The issue with your original code is that head doesn't guarantee that it won't read more than it outputs. Thus, it can consume more than one (NUL-delimited) chunk of input, even if it's emitting only one chunk of output.
read, by contrast, guarantees that it won't consume more than you ask it for.
set -o pipefail
while IFS= read -r -d '' line; do
bc <<<"${line//$'\n'/+}"
done < <(build_a_stream)
If you want native logic, there's nothing more native than just writing the whole thing in shell.
Calling external tools -- including bc, cut, paste, or others -- involves a fork() penalty. If you're only processing small amounts of data per invocation, the efficiency of the tools is overwhelmed by the cost of starting them.
while read -r -d '' -a numbers; do # read up to the next NUL into an array
sum=0 # initialize an accumulator
for number in "${numbers[#]}"; do # iterate over that array
(( sum += number )) # ...using an arithmetic context for our math
done
printf '%s\n' "$sum"
done < <(build_a_stream)
For all of the above, I tested with the following build_a_stream implementation:
build_a_stream() {
local i j IFS=$'\n'
local -a numbers
for ((i=3; i<=5; i++)); do
numbers=( )
for ((j=0; j<=i; j++)); do
numbers+=( "$j" )
done
printf '%s\0' "${numbers[*]}"
done
}
As discussed, the only real solution seemed to be writing a program to do this specifically. I wrote one in rust called xstream-util. After installing it with cargo install xstream-util, you can pipe the input into
xstream -0 -- bash -c 'paste -sd+ | bc'
to get the desired output
6
10
15
It doesn't avoid having to run the program in bash, so it still needs escaping if the pipeline is complicated. Also, it currently only supports single byte delimiters.
I am new to shellscript
I have this program
for(( s=0x001; s <=0x00f; s++))
do
echo $s
done
I want to print s values as 1,2,3,4.....a,b,c,d,e,f
But when I run above program,I have seen the output as 1,2,3,4,5,6......13,14,15.
I want to print the hex values only.
How to pass hex values in pipe.Lets say I have to pass this hex in pipe along with some other arguments.How to do that?
Lets says I have to pass some arguments to access device driver in pipe.
echo "8 $s q" | /usr/sbin/tejas/test /dev/slot$1/tupp_fpga$devNo | grep "at offset"
Here s should contain hex values.How to do it.
I think printf "%x" $s should do. It is part of the POSIX standard, and implemented as a built-in command in bash:
$ type printf
printf is a shell builtin
…so the documentation can be found in man bash, at least for the Bash shell.
Use the printf command. It's standard and works more or less like the C function.
printf "%x\n" $s
I'm not quite sure to understand the updated question but...
... if you require the given format, something like that will do the job:
print "8 %x q" $s | /usr/sbin/tejas/test /dev/slot$1/tupp_fpga$devNo | grep "at offset"
If your not familiar with the printf-style functions please refer to one of the many web pages describing the various format available.
On this particular case, the %x in the format string will be replaced by the hexadecimal representation of the next argument. Given that the s variable hold the value 1010 (or 0xA16 or 0138 -- which are indeed the same), the printf internal command will write the string 8 a s to the standard input of your /usr/sbin/tejas/test tool.