When I set any multiline text as a variable in fish, it removes the new line characters and replaces them with space, how can I stop it from doing that? Minimal complete example:
~ ) set lines (cat .lorem); set start 2; set end 4;
~ ) cat .lorem
once upon a midnight dreary while i pondered weak and weary
over many a quaint and curious volume of forgotten lore
while i nodded nearly napping suddenly there came a tapping
as of some one gently rapping rapping at my chamber door
tis some visiter i muttered tapping at my chamber door
~ ) cat .lorem | sed -ne $start\,{$end}p\;{$end}q # Should print lines 2..4
over many a quaint and curious volume of forgotten lore
while i nodded nearly napping suddenly there came a tapping
as of some one gently rapping rapping at my chamber door
~ ) echo $lines
once upon a midnight dreary while i pondered weak and weary over many a quaint and curious volume of forgotten lore while i nodded nearly napping suddenly there came a tapping as of some one gently rapping rapping at my chamber door tis some visiter i muttered tapping at my chamber door
fish splits command substitutions on newlines. This means that $lines is a list. You can read more about lists here.
When you pass a list to a command, each entry in the list becomes a separate argument. echo space-separates its arguments. That explains the behavior you're seeing.
Note that other shells do the same thing here. For example, in bash:
lines=$(cat .lorem)
echo $lines
If you want to prevent the splitting, you can temporarily set IFS to empty:
begin
set -l IFS
set lines (cat .lorem)
end
echo $lines
now $lines will contain newlines.
As faho says, read can also be used and is a little shorter:
read -z lines < ~/.lorem
echo $lines
but consider whether splitting on newlines might actually be what you want. As faho hinted, your sed script can be replaced with array slices:
set lines (cat .lorem)
echo $lines[2..4] # prints lines 2 through 4
From fish 3.4, we can use "$(innercommand)" syntax.
set lines "$(echo -e 'hi\nthere')"
https://fishshell.com/docs/current/relnotes.html
https://fishshell.com/docs/current/language.html#command-substitution
Pipe it to string split0.
set lines (echo -e 'hi\nthere')
set -S lines
# $lines: set in global scope, unexported, with 2 elements
# $lines[1]: length=2 value=|hi|
# $lines[2]: length=5 value=|there|
set lines (echo -e 'hi\nthere' | string split0)
set -S lines
# $lines: set in global scope, unexported, with 1 elements
# $lines[1]: length=9 value=|hi\nthere\n|
This is noted in the document:
If the output is piped to string split or string split0 as the last step, those splits are used as they appear instead of splitting lines.
It is not just removing the newlines, it is splitting on them.
Your variable $lines is now a list, with each line being an element in that list.
See
set lines (cat .lorem)
for line in $lines
echo $line
end
echo $lines[2]
printf "%s\n" $lines[2..4]
Related
I am taking a rather large file which is basically a list of products sold in various quantities and I want to add a fixed number to every existing dollar amount mentioned in the file (so everything with a dollar sign in front of it, to make things even more confusing .) The file contents are very predictable and are arranged as such:
16-point printed 2 side three and a half by two matte finish no round corners turn around 2-4 business days one set
250 $9.40 500 $11.05 750 $13.58 1000 $14.40 2500 $33.25 5000 $43.00 10000 $73.00 15000 $108.00 20000 $140.00 25000 $172.50
and that goes on until forever and a day. All I want to do is add lets say 5 bucks to each dollar amount, and spit out a new file. I am pretty sure that I want to evaluate it one word at a time, but entire lines can be totally skipped, due to quantities/values only appearing every second line. NOt all items have the same number of quantities so a fixed loop can't work.
I have been able to read and regurgitate the file, and I have played with reading single characters at a time. but seeing as this script will be useful now and down the road, I want to do it right and I'm new to BASH.
I'm sure it will be a combination of
#!/bin/sh
while read -r line; do
for word in $line; do
echo -n "'$word'"
done
done < "textfile.txt.bak"
and
n=1
while IFS= read -r "variable$n"; do
n=$((n + 1))
done < textfile.txt
for ((i=1; i<n; i++)); do
var="variable$i"
printf '%s\n' "${!var}"
done
I know I should be searching for the '/$' and reading in the numbers that follow, convert string to a real number, add 5, and then convert back to string, print a $ and the string, rinse wash repeat until next year. I'm pulling my hair out trying to find the best way to approach it in Bash.
PLEASE HELP!
RJM
While possible in bash, I'd recommend to use something else. For example, in Perl it boils down to:
perl -pe 's/\$\K([0-9.]+)/sprintf "%.2f", $1+5/ge if /\$[0-9]/' -- file
-p processes the input line by line, printing eachline after processing.
... if /\$[0-9]/ runs the ... part if the current line contains a dollar sign followed by a digit.
/g does a global replacement, i.e. all the possible occurrences.
/e interprets the replacement as code.
\K forgets what matched so far, in other words it only replaces the number, not the dollar sign.
It's pretty gnarly in bash due to the integer-only arithmetic
while read -ra words; do
for i in "${!words[#]}"; do
if [[ ${words[i]} =~ ^\$([0-9]+)(\.[0-9]+)? ]]; then
printf -v words[i] '$%d%s' $((BASH_REMATCH[1] + 5)) "${BASH_REMATCH[2]}"
fi
done
echo "${words[*]}"
done < file
16-point printed 2 side three and a half by two matte finish no round corners turn around 2-4 business days one set
250 $14.40 500 $16.05 750 $18.58 1000 $19.40 2500 $38.25 5000 $48.00 10000 $78.00 15000 $113.00 20000 $145.00 25000 $177.50
To send to a new file, change the last line to
done < file > new.file
You can also use awk for tnis task.
awk 'NR%2==0{for(i=2;i<=NF;i+=2){split($i,a,"$");$i=sprintf("%s%.2f","$",a[2]+5)}}1' file
So, what I'm trying to do is read in a file, loop through it comparing it line by line, but only in the third column. Sorry if this doesn't make sense, but maybe this will help. I have a file of names:
JOHN SMITH SMITH
JIM JOHNSON JOHNSON
JIM SMITH SMITH
I want to see if (first, col3)SMITH is equal to JOHNSON, if not, move onto the next name. If (first, col3) SMITH is equal to (second, col3) SMITH, then I'll do something with that.
Again, I'm sorry if this doesn't make much sense, but I tried to explain it as best as I could.
I was attempting to see if they were equal, but obviously that didn't work. Here is what I have so far, but I got stuck:
while read -a line
do
if [ ${line[2]} == ${line[2]} ]
then
echo -e "${line[2]}" >> names5.txt
else
echo "Not equal."
fi
done < names4.txt
Store your immediately prior line in a separate variable, so you can compare against it:
#!/usr/bin/env bash
old_line=( )
while read -r -a line
do
if [ "${line[2]}" = "${line[2]}" ]; then
printf '%s\n' "${line[2]}"
else
echo "Not equal." >&2
fi
old_line=( "${line[#]}" )
done <names4.txt >>names5.txt
Some other changes of note:
Instead of re-opening names5.txt every time you want to write a single line to it, we're opening it just once, for the whole loop. (You could make this >names5.txt if you want to clear it at the top of the loop and append from there, which is likely to be desirable behavior).
We're avoiding echo -e. See the APPLICATION USE and RATIONALE sections of the POSIX standard for echo for background on why echo use is not recommended for new development when contents are not tightly constrained (known not to contain any backslashes, for example).
We're quoting both sides of the test operation. This is mandatory with [ ] to ensure correct operation of words can be expanded as globs (ie. if you have a word *, you don't want it replaced with a list of files in your current directory in the final command), or if they can contain spaces (not so much a concern here, since you're using the same IFS value for the read -a as the unquoted expansion). Even if using [[ ]], you want to quote the right-hand side so it's treated as a literal string and not a pattern.
We're passing -r to read, which ensures that backslashes are not silently removed (changing \t in the input to just t, for example).
When you want to compare each third field with all previous third fields, you need to store the old third fields in an array. You can use awk for this.
When you only want to see the repeated third fields, you can use other tools:
cut -d" " -f3 names4.txt | sort | uniq -d
EDIT:
When you onlu want to print doubles from 2 consecutive lines, it is even easier:
cut -d" " -f3 names4.txt | uniq -d
I am trying to extract the number of lines from a file, and then use it in a variable. However, it keeps passing the file name, not the number of lines. I read through this question, but the examples are not working.
for i in $BASE/$TEMPLATE_DIR-$i$PRUNING_METHOD/*.txt
do
NUM_LINES=$(wc -l < $i)
echo $NUM_LINES
UPLOAD_CMD="sh sshpass_setup_verification.sh $EXP_ID-$i$PRUNING_METHOD__$NUM_LINES__features";
echo "$UPLOAD_CMD"
break 1;
done
Prints:
15 #the correct number of lines
sh sshpass_setup_verification.sh di-60sec-max/TemplateUser1.txt #where TemplateUser1.txt is the name of the file
Should print:
15
sh sshpass_setup_verification.sh di-60sec-max__15__features
A summary of what people are telling you in the comments:
for i in "${BASE}/${TEMPLATE_DIR}-${i}${PRUNING_METHOD}"/*.txt
do
num_lines=$(wc -l < "$i")
echo "$num_lines"
upload_cmd="sh sshpass_setup_verification.sh ${EXP_ID}-${i}${PRUNING_METHOD}__${num_lines}__features"
echo "$upload_cmd"
break
done
The key thing here is using double quotes around your parameter expansions and curly braces to disambiguate in situations where characters such as _ could be interpreted as part of a variable name but shouldn't. I've added them in several places where they aren't strictly needed but they do no harm.
I've also changed your variable names to lowercase. This will help you one day when you decide to have a variable called PATH and suddenly all your commands stop working.
Many people have shown how to keep spaces when reading a line in bash. But I have a character based algorithm which need to process each end every character separately - spaces included. Unfortunately I am unable to get bash read to read a single space character from input.
while read -r -n 1 c; do
printf "[%c]" "$c"
done <<< "mark spitz"
printf "[ ]\n"
yields
[m][a][r][k][][s][p][i][t][z][][ ]
I've hacked my way around this, but it would be nice to figure out how to read a single any single character.
Yep, tried setting IFS, etc.
Just set the input field separator(a) so that it doesn't treat space (or any character) as a delimiter, that works just fine:
printf 'mark spitz' | while IFS="" read -r -n 1 c; do
printf "[%c]" "$c"
done
echo
That gives you:
[m][a][r][k][ ][s][p][i][t][z]
You'll notice I've also slightly changed how you're getting the input there, <<< appears to provide a extraneous character at the end and, while it's not important to the input method itself, I though it best to change that to avoid any confusion.
(a) Yes, I'm aware that you said you've tried setting IFS but, since you didn't actually show how you'd tried this, and it appears to work fine the way I do it, I have to assume you may have just done something wrong.
I was given this text file, call stock.txt, the content of the text file is:
pepsi;drinks;3
fries;snacks;6
apple;fruits;9
baron;drinks;7
orange;fruits;2
chips;snacks;8
I will need to use bash-script to come up this output:
Total amount for drinks: 10
Total amount for snacks: 14
Total amount for fruits: 11
Total of everything: 35
My gut tells me I will need to use sed, group, grep and something else.
Where should I start?
I would break the exercise down into steps
Step 1: Read the file one line at a time
while read -r line
do
# do something with $line
done
Step 2: Pattern match (drinks, snacks, fruits) and do some simple arithmetic. This step requires that you tokenized each line which I'll leave an exercise for you to figure out.
if [[ "$line" =~ "drinks" ]]
then
echo "matched drinks"
.
.
.
fi
Pure Bash. A nice application for an associative array:
declare -A category # associative array
IFS=';'
while read name cate price ; do
((category[$cate]+=price))
done < stock.txt
sum=0
for cate in ${!category[#]}; do # loop over the indices
printf "Total amount of %s: %d\n" $cate ${category[$cate]}
((sum+=${category[$cate]}))
done
printf "Total amount of everything: %d\n" $sum
There is a short description here about processing comma separated files in bash here:
http://www.cyberciti.biz/faq/unix-linux-bash-read-comma-separated-cvsfile/
You could do something similar. Just change IFS from comma to semicolon.
Oh yeah, and a general hint for learning bash: man is your friend. Use this command to see manual pages for all (or most) of commands and utilities.
Example: man read shows the manual page for read command. On most systems it will be opened in less, so you should exit the manual by pressing q (may be funny, but it took me a while to figure that out)
The easy way to do this is using a hash table, which is supported directly by bash 4.x and of course can be found in awk and perl. If you don't have a hash table then you need to loop twice: once to collect the unique values of the second column, once to total.
There are many ways to do this. Here's a fun one which doesn't use awk, sed or perl. The only external utilities I've used here are cut, sort and uniq. You could even replace cut with a little more effort. In fact lines 5-9 could have been written more easily with grep, (grep $kind stock.txt) but I avoided that to show off the power of bash.
for kind in $(cut -d\; -f 2 stock.txt | sort | uniq) ; do
total=0
while read d ; do
total=$(( total+d ))
done < <(
while read line ; do
[[ $line =~ $kind ]] && echo $line
done < stock.txt | cut -d\; -f3
)
echo "Total amount for $kind: $total"
done
We lose the strict ordering of your original output here. An exercise for you might be to find a way not to do that.
Discussion:
The first line describes a sub-shell with a simple pipeline using cut. We read the third field from the stock.txt file, with fields delineated by ;, written \; here so the shell does not interpret it. The result is a newline-separated list of values from stock.txt. This is piped to sort, then uniq. This performs our "grouping" step, since the pipeline will output an alphabetic list of items from the second column but will only list each item once no matter how many times it appeared in the input file.
Also on the first line is a typical for loop: For each item resulting from the sub-shell we loop once, storing the value of the item in the variable kind. This is the other half of the grouping step, making sure that each "Total" output line occurs once.
On the second line total is initialized to zero so that it always resets whenever a new group is started.
The third line begins the 'totaling' loop, in which for the current kind we find the sum of its occurrences. here we declare that we will read the variable d in from stdin on each iteration of the loop.
On the fourth line the totaling actually occurs: Using shell arithmatic we add the value in d to the value in total.
Line five ends the while loop and then describes its input. We use shell input redirection via < to specify that the input to the loop, and thus to the read command, comes from a file. We then use process substitution to specify that the file will actually be the results of a command.
On the sixth line the command that will feed the while-read loop begins. It is itself another while-read loop, this time reading into the variable line. On the seventh line the test is performed via a conditional construct. Here we use [[ for its =~ operator, which is a pattern matching operator. We are testing to see whether $line matches our current $kind.
On the eighth line we end the inner while-read loop and specify that its input comes from the stock.txt file, then we pipe the output of the entire loop, which by now is simply all lines matching $kind, to cut and instruct it to show only the third field, which is the numeric field. On line nine we then end the process substitution command, the output of which is a newline-delineated list of numbers from lines which were of the group specified by kind.
Given that the total is now known and the kind is known it is a simple matter to print the results to the screen.
The below answer is OP's. As it was edited in the question itself and OP hasn't come back for 6 years, I am editing out the answer from the question and posting it as wiki here.
My answer, to get the total price, I use this:
...
PRICE=0
IFS=";" # new field separator, the end of line
while read name cate price
do
let PRICE=PRICE+$price
done < stock.txt
echo $PRICE
When I echo, its :35, which is correct. Now I will moving on using awk to get the sub-category result.
Whole Solution:
Thanks guys, I manage to do it myself. Here is my code:
#!/bin/bash
INPUT=stock.txt
PRICE=0
DRINKS=0
SNACKS=0
FRUITS=0
old_IFS=$IFS # save the field separator
IFS=";" # new field separator, the end of line
while read name cate price
do
if [ $cate = "drinks" ]; then
let DRINKS=DRINKS+$price
fi
if [ $cate = "snacks" ]; then
let SNACKS=SNACKS+$price
fi
if [ $cate = "fruits" ]; then
let FRUITS=FRUITS+$price
fi
# Total
let PRICE=PRICE+$price
done < $INPUT
echo -e "Drinks: " $DRINKS
echo -e "Snacks: " $SNACKS
echo -e "Fruits: " $FRUITS
echo -e "Price " $PRICE
IFS=$old_IFS