How to read every 100 lines from a large file [duplicate] - bash

This question already has answers here:
Read n lines at a time using Bash
(17 answers)
Closed 3 years ago.
How do I read every 100 lines in a larger file of around 100000 lines. I want to read every 100 lines in one iteration and make them coma seperated, run some code and next iteration, it should pick from 101 to 200.
I have searched internet, everywhere there is a solution for picking nth line and not n lines.

You can use mapfile and specify count. Remember to check the end condition, as mapfile doesn't set it's return status to nonzero on EOF condition.
# read 100 lines
while while mapfile -t -n2 lines && ((${#lines[#]})); do
# output the lines separated with comma
( IFS=','; echo "${lines[*]}"; )
# run some code
: run some code
done < larger_file

Related

How can I append the same byte to a file a specified number of times? [duplicate]

This question already has answers here:
Bash: repeat character a variable number of times
(3 answers)
Closed 4 months ago.
I have the following code:
yes "$(echo -e "\xff")" | head -n 10 > SomeFile.bin
Which writes 10 times 0xFF and 0x0A (newline) to SomeFile.bin. But my objective is to fill the file with FF.
Is there a way to print only consecutive 0xFF values instead? Without adding a newline every time?
Here is one way:
printf '\xFF%.s' {1..10} >SomeFile.bin

How do I read in a file line by line, and add the value of 5 to every instance of a dollar amount and write it to a new file

I am taking a rather large file which is basically a list of products sold in various quantities and I want to add a fixed number to every existing dollar amount mentioned in the file (so everything with a dollar sign in front of it, to make things even more confusing .) The file contents are very predictable and are arranged as such:
16-point printed 2 side three and a half by two matte finish no round corners turn around 2-4 business days one set
250 $9.40 500 $11.05 750 $13.58 1000 $14.40 2500 $33.25 5000 $43.00 10000 $73.00 15000 $108.00 20000 $140.00 25000 $172.50
and that goes on until forever and a day. All I want to do is add lets say 5 bucks to each dollar amount, and spit out a new file. I am pretty sure that I want to evaluate it one word at a time, but entire lines can be totally skipped, due to quantities/values only appearing every second line. NOt all items have the same number of quantities so a fixed loop can't work.
I have been able to read and regurgitate the file, and I have played with reading single characters at a time. but seeing as this script will be useful now and down the road, I want to do it right and I'm new to BASH.
I'm sure it will be a combination of
#!/bin/sh
while read -r line; do
for word in $line; do
echo -n "'$word'"
done
done < "textfile.txt.bak"
and
n=1
while IFS= read -r "variable$n"; do
n=$((n + 1))
done < textfile.txt
for ((i=1; i<n; i++)); do
var="variable$i"
printf '%s\n' "${!var}"
done
I know I should be searching for the '/$' and reading in the numbers that follow, convert string to a real number, add 5, and then convert back to string, print a $ and the string, rinse wash repeat until next year. I'm pulling my hair out trying to find the best way to approach it in Bash.
PLEASE HELP!
RJM
While possible in bash, I'd recommend to use something else. For example, in Perl it boils down to:
perl -pe 's/\$\K([0-9.]+)/sprintf "%.2f", $1+5/ge if /\$[0-9]/' -- file
-p processes the input line by line, printing eachline after processing.
... if /\$[0-9]/ runs the ... part if the current line contains a dollar sign followed by a digit.
/g does a global replacement, i.e. all the possible occurrences.
/e interprets the replacement as code.
\K forgets what matched so far, in other words it only replaces the number, not the dollar sign.
It's pretty gnarly in bash due to the integer-only arithmetic
while read -ra words; do
for i in "${!words[#]}"; do
if [[ ${words[i]} =~ ^\$([0-9]+)(\.[0-9]+)? ]]; then
printf -v words[i] '$%d%s' $((BASH_REMATCH[1] + 5)) "${BASH_REMATCH[2]}"
fi
done
echo "${words[*]}"
done < file
16-point printed 2 side three and a half by two matte finish no round corners turn around 2-4 business days one set
250 $14.40 500 $16.05 750 $18.58 1000 $19.40 2500 $38.25 5000 $48.00 10000 $78.00 15000 $113.00 20000 $145.00 25000 $177.50
To send to a new file, change the last line to
done < file > new.file
You can also use awk for tnis task.
awk 'NR%2==0{for(i=2;i<=NF;i+=2){split($i,a,"$");$i=sprintf("%s%.2f","$",a[2]+5)}}1' file

Echo to a line above the line where there is a specific string [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 3 years ago.
Improve this question
I am using a for f in $(cat /dir/file) nested loop with a conditional inside it to test if (($f=$z)), z being in the for z in $(cat /dir/file2) loop, where the f loop is nested.
I want to know if its possible to write to the line above the line where the $z string is in file2, if $z=$f.
each line in file2 is a 5 digit number like 07732, if there a line with 07732 in file I want to write that line above the line where that 07732 in file 2 is
What I've got right now is
for z in $(cat /dir/file2)
do
for f in $(cat /dir/file)
do
if (( $f=$z ))
then
(what im asking about)
i=0
break
else
i=1
fi
done
if ((i=1))
then
......
I know I've got syntax errors no need to correct those.
Sorry if I'm explaining it really poorly, please tell me if I need to explain in more detail
Delay the output of the matched line. And, as Charles Duffy pointed out in a commend, don't read lines with for. Also note that you will be reading the entire file2 for each line of file1. If file2 is of any significant size, this could bne very slow. If file2 is pretty small, read it into an array.
Also you should use meaningful variable names. This isn't 1970's BASIC where you are restricted to single characters.
Here is some sample code that incorporates most of that but leaves the inner while loop in place instead of using an array:
while read -r file1_line
do
while read -r -u 3 file2_line
do
if [[ $file2_line == $file1_line ]]
then
echo "some special putput"
i=0
break
else
i=1
fi
done 3< /path/to/file2
echo "NOW I'm echoing $file1_line"
if (( i == 1 ))
then
. . .
fi
done < /path/to/file1
This will echo the lines from file one, but when there's a match it will echo a string before the matched line.
The inner while loop uses an additional file descriptor in order to be able to read from a second file while the outer loop is reading from the STDIN file descriptor. This is so that multiple files can be read independently at the same time. The -u 3 tells read to use file descriptor 3. The 3< redirection puts the contents of file2 on file descriptor 3 so they can be read there.

Shuffling doesn't actually save into array? [duplicate]

This question already has answers here:
How do I set a variable to the output of a command in Bash?
(15 answers)
Copying a Bash array fails
(2 answers)
Closed 3 years ago.
I'm new to shell script and am having an issue with the shuf function.
This is my code
declare -a myarray=( 'A' 'B' 'C' 'D' 'E' 'F' )
myarray = $(shuf -e "${myarray[#]}")
echo "$myarray"
I make an array containing the six characters. I then shuffle them randomly, and print them out. My issue is that if I were to add another line, for example
echo ${myarray[2]}
This doesn't actually print the randomly sorted character in the 3rd position. Instead, it will always print 'C'. How can I actually save the sorted array? Do I need to make another array?
Thank you very much
Arrays in bash are defined with (). Bash is not statically typed, so setting myarray equal to some output of characters will do just that, making it a string you can echo with echo $myarray to see the full output.
You need to wrap your output in parens to make it clear to bash that your new myarray should also be an array:
myarray=($(shuf -e "${myarray[#]}"))

Read range of numbers into a for loop

So, I am building a bash script which iterates through folders named by numbers from 1 to 9. The script depends on getting the folder names by user input. My intention is to use a for loop using read input to get a folder name or a range of folder names and then do some stuff.
Example:
Let's assume I want to make a backup with rsync -a of a certain range of folders. Usually I would do:
for p in {1..7}; do
rsync -a $p/* backup.$p
done
The above would recursively backup all content in the directories 1 2 3 4 5 6 and 7 and put them into folders named as 'backup.{index-number}'. It wouldn't catch folders/files with a leading . but that is not important right now.
Now I have a similar loop in an interactive bash script. I am using select and case statements for this task. One of the options in case is this loop and it shall somehow get a range of numbers from user input. This now becomes a problem.
Problem:
If I use read to get the range then it fails when using {1..7} as input. The input is taken literally and the output is just:
{1..7}
I really would like to know why this happens. Let me use a more descriptive example with a simple echo command.
var={1..7} # fails and just outputs {1..7}
for p in $var; do echo $p;done
read var # Same result as above. Just outputs {1..7}
for p in $var; do echo $p;done
for p in {1..7}; do echo $p;done # works fine and outputs the numbers 1-7 seperated with a newline.
I've found a workaround by storing the numbers in an array. The user can then input folder names seperated by a space character like this: 1 2 3 4 5 6 7
read -a var # In this case the output is similar to the 3rd loop above
for p in ${var[#]}; do echo $p; done
This could be a way to go but when backing up 40 folders ranging from 1-40 then adding all the numbers one-by-one completely makes my script redundant. One could find a solution to one of the millennium problems in the same time.
Is there any way to read a range of numbers like {1..9} or could there be another way to get input from terminal into the script so I can iterate through the range within a for-loop?
This sounds like a question for google but I am obviously using the wrong patterns to get a useful answer. Most of similar looking issues on SO refer to brace and parameter expansion issues but this is not exactly the problem I have. However, to me it feels like the answer to this problem is going in a similar direction. I fail to understand why when a for-loop for assigning {1..7} to a variable works but doing the same like var={1..7} doesn't. Plz help -.-
EDIT: My bash version:
$ echo $BASH_VERSION
4.2.25(1)-release
EDIT2: The versatility of a brace expansion is very important to me. A possible solution should include the ability to define as many ranges as possible. Like I would like to be able to choose between backing up just 1 folder or a fixed range between f.ex 4-22 and even multiple options like folders 1,2,5,6-7
Brace expansion is not performed on the right-hand side of a variable, or on parameter expansion. Use a C-style for loop, with the user inputing the upper end of the range if necessary.
read upper
for ((i=1; i<=$upper; i++)); do
To input both a lower and upper bound separated by whitespace
read lower upper
for (i=$lower; i <= $upper; i++)); do
For an arbitrary set of values, just push the burden to the user to generate the appropriate list; don't try to implement your own parser to process something like 1,2,20-22:
while read p; do
rsync -a $p/* backup.$p
done
The input is one value per line, such as
1
2
20
21
22
Even if the user is using the shell, they can call your script with something like
printf '%s\n' 1 2 20..22 | backup.sh
It's easier for the user to generate the list than it is for you to safely parse a string describing the list.
The evil eval
$ var={1..7}
$ for i in $(eval echo $var); do echo $i; done
this also works,
$ var="1 2 {5..9}"
$ for i in $(eval echo $var); do echo $i; done
1
2
5
6
7
8
9
evil eval was a joke, that is, as long as you know what you're evaluating.
Or, with awk
$ echo "1 2 5-9 22-25" |
awk -v RS=' ' '/-/{split($0,a,"-"); while(a[1]<=a[2]) print a[1]++; next}1'
1
2
5
6
7
8
9
22
23
24
25

Resources