I have a shell command that produces line based output. Let me call it magic for the sake of the argument. For whatever output it produces, how can I just append another value to it? I would like to do it in a pipe. I tried for a long time to google this without any luck. It seems like I must be missing some obvious way to do it. Ideally, there would be another unix command called append which given as standard input the output of any other command the same output along with its arguments.
What I am imagining:
magic
This returns:
apple
cherry
banana
magic | append taro
This returns:
apple
cherry
banana
taro
Does this append command already exist with a different name? If so, what is it called?
Converting my comment to answer so that solution is easy to find for future visitors.
You may use grouping of command using { ...; } to group multiple commands:
{ magic; echo 'taro'; }
And if you want to redirect output to a file then use:
{ magic; echo 'taro'; } > outfile
The best way is not to use a pipe, but anubhava's { magic; echo 'taro'; }.
However, since you asked a pipe, you've opened up Pandora's box of possibilities.
magic|sed '$ataro'
is the first.
magic| awk '{print} END{print "taro"}'
as second.
Or a bash function:
hop(){
while read line; do
echo $line
done
echo $1
}
magic | hop taro
And so on.
Related
This question already has answers here:
How do I iterate over a range of numbers defined by variables in Bash?
(20 answers)
Closed 3 years ago.
I am still learning how to shell script and I have been given a challenge to make it easier for me to echo "Name1" "Name2"..."Name15" and I'm not too sure where to start, I've had ideas but I don't want to look silly if I mess it up. Any help?
I haven't actually tried anything just yet it's all just been mostly thought.
#This is what I wrote to start
#!/bin/bash
echo "Name1"
echo "Name2"
echo "Name3"
echo "Name4"
echo "Name5"
echo "Name6"
echo "Name7"
echo "Name8"
echo "Name9"
echo "Name10"
echo "Name11"
echo "Name12"
echo "Name13"
echo "Name14"
echo "Name15"
My expected results are obviously just for it to output "Name1" "Name2" etc. But I'm looking for a more creative way to do it. If possible throw in a few ways to do it so I can learn. Thank you.
The easiest (possibly not the most creative) way to do this is to use printf:
printf "%s\n" name{1..15}
This relies on bash brace expansion {1..15} to have the 15 strings.
Use a for loop
for i in {1..15};do echo "Name$i";done
A few esoteric solutions, from the least to the most unreasonable :
base64 encoded string :
base64 -d <<<TmFtZTEKTmFtZTIKTmFtZTMKTmFtZTQKTmFtZTUKTmFtZTYKTmFtZTcKTmFtZTgKTmFtZTkKTmFtZTEwCk5hbWUxMQpOYW1lMTIKTmFtZTEzCk5hbWUxNApOYW1lMTUK
The weird chain is your expected result encoded in base64, an encoding generally used to represent binary data as text. base64 -d <<< weirdChain is passing the weird chain as input to the base64 tool and asking it to decode it, which displays your expected result
generate an infinite stream of "Name", truncate it, use line numbers :
yes Name | awk 'NR == 16 { exit } { printf("%s%s\n", $0, NR) }'
yes outputs an infinite stream of what it's passed as argument (or y by default, used to automatize interactive scripts asking for [y/n] confirmation). The awk command exits once it reaches the 16th line, and otherwise prints its input (provided by yes) followed by the line number. The truncature could as easily be done with head -15, and I've tried using the nl "number line" utility or grep -n to number lines, but they always added the line numbers as prefix which required an extra re-formatting step.
read random binary data and hope to stumble on all the lines you want to output :
timeout 1d strings /dev/urandom | grep -Eo "Name(1[0-5]|[1-9])" | sort -uV
strings /dev/urandom will extract ascii sequences from the binary random source /dev/urandom, grep will filter those which respect the format of a line of your expected output and sort will reorder those lines in the correct order. Since sort needs to have a received its whole input before it reorders it and /dev/urandom won't stop producing data, we use timeout 1d to stop reading from /dev/urandom after a whole day in hope it has sifted through enough random data to find your 15 lines (I'm not sure that's even remotely likely).
use an HTTP client to retrieve this page, extract the bash script you posted and execute it.
my_old_script=$(curl "https://stackoverflow.com/questions/57818680/" | grep "#This is what I wrote to start" -A 18 | tail -n+4)
eval "$my_old_script"
curl is a command line tool that can be used as an HTTP client, grep with its -A 18 parameter will select the "This is what I wrote to start" text and the 18 lines that follow, tail will remove the first 3 lines, and eval will execute your script.
While it will be much more efficient than the previous solution, it's an even less reasonable solution because high-rep users can edit your question to make this solution execute arbitrary code on your computer. Ideally you'd be using an HTML-aware parser rather than basic string manipulation to extract the code, but we're not talking about best practices here...
I try to create a one-liner for opening less on the last screen of an multi-screen output coming from standard input. The reason for this is that I am working on a program that produces a long AST and I need to be able to traverse up and down through it but I would prefer to start at the bottom. I came up with this:
$ python a.py 2>&1 | tee >(lines=+$(( $(wc -l) - $LINES))) | less +$lines
First, I need to compute number of lines in output and subtract $LINES from it so I know what's the uppermost line of the last screen. I will need to reuse a.py output later so I use tee with process substitution for that purpose. As the last step I point less to open an original stdout on a particular line. Of course, it doesn't work in Bash because $lines is not set in last step as every subcommand is run in a subshell. In ZSH, even though pipe commands are not run in a subshell, process substitution still is and therefore it doesn't work neither. It's not a homework or a work task, I just wonder whether it's possible to do what I want without creating a temporary file in Bash or ZSH. Any ideas?
less supports this innately. The + syntax you're using accepts any less command you could enter while it's running, including G for go-to-end.
... | less +G
does exactly what you want.
This is actually mentioned explicitly as an example in the man page (search for "+G").
The real answer to your question should be the option +G to less, but you indicated that the problem definition is not representative for the abstract problem you want to solve. Therefore, please consideer this alternative problem:
python a.py 2>&1 | \
awk '
{a[NR]=$0}
END{
print NR
for (i=1;i<=NR;i++)print a[i]
}
' | {
read -r l
less -j-1 +$l
}
The awk command is printing the number of lines, and then all the lines in sequence. We define the first line to contain some meta information. This is piped to a group of commands delimited by { and }. The first line is consumed by read, which stores it in variable $l. The rest of the lines are taken by less, where this variable can be used. -j-1 is used, so the matched line is at the bottom of the screen.
this is my first post so hopefully I will make my question clear.
I am new to shell scripts and my task with this one is to add a new value to every line of a csv file. The value that needs added is based on the first 3 digits of the filename.
I bit of background. The csv files I am receiving are eventually being loaded into partitioned oracle tables. The start of the file name (e.g. BATTESTFILE.txt) contains the partitioned site so I need to write a script that takes the first 3 characters of the filename (in this example BAT) and add this to the end of each line of the file.
The closest I have got so far is when I stripped the code to the bare basics of what I need to do:
build_files()
{
OLDFILE=${filename[#]}.txt
NEWFILE=${filename[#]}.NEW.txt
ABSOLUTE='path/scripts/'
FULLOLD=$ABSOLUTE$OLDFILE
FULLNEW=$ABSOLUTE$NEWFILE
sed -e s/$/",${j}"/ "${FULLOLD}" > "${FULLNEW}"
}
set -A site 'BAT'
set -A filename 'BATTESTFILE'
for j in ${site[#]}; do
for i in ${filename[#]}; do
build_files ${j}
done
done
Here I have set up an array site as there will be 6 'sites' and this will make it easy to add additionals sits to the code as the files come through to me. The same is to be siad for the filename array.
This codes works, but it isn't as automated as I need. One of my most recent attempts has been below:
build_files()
{
OLDFILE=${filename[#]}.txt
NEWFILE=${filename[#]}.NEW.txt
ABSOLUTE='/app/dss/dsssis/sis/scripts/'
FULLOLD=$ABSOLUTE$OLDFILE
FULLNEW=$ABSOLUTE$NEWFILE
sed -e s/$/",${j}"/ "${FULLOLD}" > "${FULLNEW}"
}
set -A site 'BAT'
set -A filename 'BATTESTFILE'
for j in ${site[#]}; do
for i in ${filename[#]}; do
trust=echo "$filename" | cut -c1-3
echo "$trust"
if ["$trust" = 'BAT']; then
${j} = 'BAT'
fi
build_files ${j}
done
done
I found the code trust=echo "$filename" | cut -c1-3 through another question on StackOverflow as I was researching, but it doesn't seem to work for me. I added in the echo to test what trust was holding, but it was empty.
I am getting 2 errors back:
Line 17 - BATTESTFILE: not found
Line 19 - test: ] missing
Sorry for the long winded questions. Hopefully It contains helpful info and shows the steps I have taken. Any questions, comment away. Any help or guidance is very much appreciated. Thanks.
When you are new with shells, try avoiding arrays.
In an if statement use spaces before and after the [ and ] characters.
Get used to surrounding your shell variables with {} like ${trust}
I do not know how you fill your array, when the array is hardcoded, try te replace with
SITE=file1
SITE="${SITE} file2"
And you must tell unix you want to have the rightside eveluated with $(..) (better than backtics):
trust=$(echo "${filename}" | cut -c1-3)
Some guidelines and syntax help can be found at Google
Just use shell parameter expansion:
$ var=abcdefg
$ echo "${var:0:3}"
abc
Assuming you're using a reasonably capable shell like bash or ksh, for example
Just in case it is useful for anyone else now or in the future, I got my code to work as desired by using the below. Thanks Walter A below for his answer to my main problem of getting the first 3 characters from the filename and using them as a variable.
This gave me the desired output of taking the first 3 characters of the filename, and adding them to the end of each line in my csv file.
## Get the current Directory and file name, create a new file name
build_files()
{
OLDFILE=${i}.txt
NEWFILE=${i}.NEW.txt
ABSOLUTE='/app/dss/dsssis/sis/scripts/'
FULLOLD=$ABSOLUTE$OLDFILE
FULLNEW=$ABSOLUTE$NEWFILE
## Take the 3 characters from the filename and
## add them onto the end of each line in the csv file.
sed -e s/$/";${j}"/ "${FULLOLD}" > "${FULLNEW}"
}
## Loop to take the first 3 characters from the file names held in
## an array to be added into the new file above
set -A filename 'BATTESTFILE'
for i in ${filename[#]}; do
trust=$(echo "${i}" | cut -c1-3)
echo "${trust}"
j="${trust}"
echo "${i} ${j}"
build_files ${i} ${j}
done
Hope this is useful for someone else.
Hi I have the following list of stocks that is generated and it is placed in file called awk_1
dfs
fsd
dsf
sdf
I then run the following one liner which generates the correct ULR links
while read i ; do
echo $(http://uk.finance.yahoo.com/echartss=$i#symbol=$i\;range=my\;compare=\;indicator=volume\;charttype\=area\;crosshair\=on\;ohlcvalues\=0\;logscale\=off\;source\=undefined\;) tee stock_urls;
done < awk_1
However is does not put the out put in the file called stock_urls ?
Also it generate and strange output on the screen, below is a small section of the output that I get to standard output. It puts "./large_cap_stocks.sh: 51: ./large_cap_stocks.sh:" at the front and "not found" at the end , why might that be happening.
I have searching high and low for why this is not working any help would be really appreciated.
Thanks
You probably meant to write like this:
while read i; do
echo "http://uk.finance.yahoo.com/echarts?s=$i#symbol=$i\;range=my\;compare=\;indicator=volume\;charttype\=area\;crosshair\=on\;ohlcvalues\=0\;logscale\=off\;source\=undefined\;"
done < awk_1 | tee stock_urls
That is:
In the echo command, use "..." to quote your text instead of $(...) which is something else
Use the pipe operator | to pass the output to tee, and you can pipe the entire loop this way, no need to do for individual echo lines.
I have this line in my script.sh
printf "%d" "$endMS_line"
$endMS_line is a number. I get that number with
endMS_line=`cat file | awk '{if($1=='"$variable"') print NR}'`
And to print it I use
printf "%d" "$endMS_line"
or
echo $endMS_line
So everything works perfectly in the standard output. The problem is when I want to save that number into a file (because I want to use the result in another script, may be there is a clever way to do it than write a file and then read the number from the file, etc..)
But for now I am trying to do that. How? Well I write this in the standard output.
myscript.sh inputs > file.txt
But when I try to see the file (when I open the file) I see the result plus weird characteres
[H[2J867
The correct number in this example is 867. Anyone know how can I fix this?
Thank you!
At the begginning of the script I had the command:
clear
removing that and using:
echo "$endMS_line"
Then in the standard output:
myscript.sh input > file.txt
works perfectly.