bash script to handle 9000 lines of text by groups of 25 - bash

Question

I have an input file with 9095 lines of text. I need to take this list of filenames and use them as input to an ffmpeg command. I really would like to avoid running it all at once (line by line until end of file).
Is there a way in a bash script to take the first 25 lines, execute the ffmpeg command on them, then continue to the next 25 lines and execute the same command, until the end of the input file?
Input file (example.txt) has list of 9095 filenames to be processed.
I have tried the following:
#!/bin/bash
while mapfile -n 25 convert < example.txt do
while read -r line || [[ -n "$line" ]]; do fn_out="w$line"; (/path/to/file/ffmpeg -report -nostdin -i /path/to/file/watermark.png -i /path/to/file/"$line" -filter_complex "scale2ref[a][b];[b][a]overlay=x=(main_w-overlay_w)/2:y=(main_h-overlay_h)/2" /path/to/file/"$fn_out" -c:v libx265); done < $convert >> script.txt
done
This only gets me the first 25 files. How to account for all 9095 files in example.txt?
Also, is there any way to "pause" the script in between each set of 25 filenames? I am not familiar enough with mapfile and bash scripting to know how to handle this kind of iteration.
Basically, the output needs to be as follows:
Read first 25 lines of example.txt
Run the ffmpeg command on those 25 files (some files are over 2GB. Will take time to do the ffmpeg command.
Pause for about a minute or two. Let server rest/catch up/recover resources.
Get the next 25 names from example.txt
Run the ffmpeg conversion loop again in step 2
Continue until EOF of example.txt
Any direction on this would be greatly appreciated!
Running CentOS 7 with bash 5.0.17.

Answer 1

Assumptions/understandings:
files are to be processed one-at-a-time and serially
after a batch of 25 files have been processed sleep for 60 seconds
One bash approach using a simple counter (and no need for an array):
maxcnt=25
fcnt=0
while read -r fname
do
echo "run ffmpeg against $fname"
((++fcnt % maxcnt == 0)) && sleep 60
done < file.txt
Setup a small demo:
10 filenames in file.txt
process 3 files and then sleep
sleep for 2 seconds (also print message 'sleep 2')
Run the demo:
$ printf "file%s\n" {1..10} > file.txt
$ cat file.txt
file1
file2
file3
file4
file5
file6
file7
file8
file9
file10
$ maxcnt=3; fcnt=0
$ while read -r fname; do echo "run ffmpeg against $fname"; ((++fcnt % maxcnt == 0)) && echo "sleep 2" && sleep 2; done < file.txt; echo "Done."
run ffmpeg against file1
run ffmpeg against file2
run ffmpeg against file3
sleep 2
run ffmpeg against file4
run ffmpeg against file5
run ffmpeg against file6
sleep 2
run ffmpeg against file7
run ffmpeg against file8
run ffmpeg against file9
sleep 2
run ffmpeg against file10
Done.

Answer 2

You need to put the redirection around the entire loop, not just the mapfile line. Just like when you use while read to iterate through a file one line at a time.
while mapfile -tn 25 convert && [[ -n "${convert[#]}" ]]; do
# code that uses $convert array
done < example.txt
Then each time through the loop it will get the next 25 lines of the file.

Answer 3

With mapfile aka readarray, something like:
#!/usr/bin/env bash
num=25
while mapfile -tn "$num" files && (( ${#files[*]} == num )); do
((sum+=${#files[*]}))
for i in "${!files[#]}"; do
echo do_something_with "${files[$i]}"
done
((sum < 9000)) && sleep 60
done < file.txt
A simple test run.
num=25
while mapfile -tn "$num" files && (( ${#files[*]} == num )); do
((sum+=${#files[*]}))
for i in "${!files[#]}"; do
echo do_something_with "${files[$i]}"
done
((sum < 9000)) && echo ===
done < <(printf '%s\n' {1..9000})
Output:
do_something_with 1
do_something_with 2
do_something_with 3
do_something_with 4
do_something_with 5
do_something_with 6
do_something_with 7
do_something_with 8
do_something_with 9
do_something_with 10
do_something_with 11
do_something_with 12
do_something_with 13
do_something_with 14
do_something_with 15
do_something_with 16
do_something_with 17
do_something_with 18
do_something_with 19
do_something_with 20
do_something_with 21
do_something_with 22
do_something_with 23
do_something_with 24
do_something_with 25
===
do_something_with 26
do_something_with 27
do_something_with 28
do_something_with 29
do_something_with 30
do_something_with 31
do_something_with 32
do_something_with 33
do_something_with 34
do_something_with 35
do_something_with 36
do_something_with 37
do_something_with 38
do_something_with 39
do_something_with 40
do_something_with 41
do_something_with 42
do_something_with 43
do_something_with 44
do_something_with 45
do_something_with 46
do_something_with 47
do_something_with 48
do_something_with 49
do_something_with 50
===
do_something_with 51
do_something_with 52
do_something_with 53
do_something_with 54
do_something_with 55
do_something_with 56
do_something_with 57
do_something_with 58
do_something_with 59
do_something_with 60
do_something_with 61
do_something_with 62
do_something_with 63
do_something_with 64
do_something_with 65
do_something_with 66
do_something_with 67
do_something_with 68
do_something_with 69
do_something_with 70
do_something_with 71
do_something_with 72
do_something_with 73
do_something_with 74
do_something_with 75
===
do_something_with 76
do_something_with 77
do_something_with 78
do_something_with 79
do_something_with 80
do_something_with 81
do_something_with 82
do_something_with 83
do_something_with 84
do_something_with 85
do_something_with 86
do_something_with 87
do_something_with 88
do_something_with 89
do_something_with 90
do_something_with 91
do_something_with 92
do_something_with 93
do_something_with 94
do_something_with 95
do_something_with 96
do_something_with 97
do_something_with 98
do_something_with 99
do_something_with 100
===
do_something_with 101
do_something_with 102
do_something_with 103
do_something_with 104
do_something_with 105
do_something_with 106
do_something_with 107
do_something_with 108
do_something_with 109
do_something_with 110
do_something_with 111
do_something_with 112
do_something_with 113
do_something_with 114
do_something_with 115
do_something_with 116
do_something_with 117
do_something_with 118
do_something_with 119
do_something_with 120
do_something_with 121
do_something_with 122
do_something_with 123
do_something_with 124
do_something_with 125
===
do_something_with 126
do_something_with 127
do_something_with 128
do_something_with 129
do_something_with 130
do_something_with 131
do_something_with 132
do_something_with 133
do_something_with 134
do_something_with 135
do_something_with 136
do_something_with 137
do_something_with 138
do_something_with 139
do_something_with 140
do_something_with 141
do_something_with 142
do_something_with 143
do_something_with 144
do_something_with 145
do_something_with 146
do_something_with 147
do_something_with 148
do_something_with 149
do_something_with 150
===
do_something_with 151
do_something_with 152
do_something_with 153
do_something_with 154
do_something_with 155
do_something_with 156
do_something_with 157
do_something_with 158
do_something_with 159
do_something_with 160
do_something_with 161
do_something_with 162
do_something_with 163
do_something_with 164
do_something_with 165
do_something_with 166
do_something_with 167
do_something_with 168
do_something_with 169
do_something_with 170
do_something_with 171
do_something_with 172
do_something_with 173
do_something_with 174
do_something_with 175
===
do_something_with 176
do_something_with 177
do_something_with 178
do_something_with 179
do_something_with 180
do_something_with 181
do_something_with 182
do_something_with 183
do_something_with 184
do_something_with 185
do_something_with 186
do_something_with 187
do_something_with 188
do_something_with 189
do_something_with 190
do_something_with 191
do_something_with 192
do_something_with 193
do_something_with 194
do_something_with 195
do_something_with 196
do_something_with 197
do_something_with 198
do_something_with 199
do_something_with 200
===
do_something_with 201
do_something_with 202
do_something_with 203
do_something_with 204
do_something_with 205
do_something_with 206
do_something_with 207
do_something_with 208
do_something_with 209
do_something_with 210
do_something_with 211
do_something_with 212
do_something_with 213
do_something_with 214
do_something_with 215
do_something_with 216
do_something_with 217
do_something_with 218
do_something_with 219
do_something_with 220
do_something_with 221
do_something_with 222
do_something_with 223
do_something_with 224
do_something_with 225
===
do_something_with 226
do_something_with 227
do_something_with 228
do_something_with 229
do_something_with 230
do_something_with 231
do_something_with 232
do_something_with 233
do_something_with 234
do_something_with 235
do_something_with 236
do_something_with 237
do_something_with 238
do_something_with 239
do_something_with 240
do_something_with 241
do_something_with 242
do_something_with 243
do_something_with 244
do_something_with 245
do_something_with 246
do_something_with 247
do_something_with 248
do_something_with 249
do_something_with 250
===
do_something_with 251
do_something_with 252
do_something_with 253
do_something_with 254
do_something_with 255
do_something_with 256
do_something_with 257
do_something_with 258
do_something_with 259
do_something_with 260
do_something_with 261
do_something_with 262
do_something_with 263
do_something_with 264
do_something_with 265
do_something_with 266
do_something_with 267
do_something_with 268
do_something_with 269
do_something_with 270
do_something_with 271
do_something_with 272
do_something_with 273
do_something_with 274
do_something_with 275
===
do_something_with 276
do_something_with 277
do_something_with 278
do_something_with 279
do_something_with 280
do_something_with 281
do_something_with 282
do_something_with 283
do_something_with 284
do_something_with 285
do_something_with 286
do_something_with 287
do_something_with 288
do_something_with 289
do_something_with 290
do_something_with 291
do_something_with 292
do_something_with 293
do_something_with 294
do_something_with 295
do_something_with 296
do_something_with 297
do_something_with 298
do_something_with 299
do_something_with 300
===
do_something_with 301
do_something_with 302
do_something_with 303
do_something_with 304
do_something_with 305
do_something_with 306
do_something_with 307
do_something_with 308
do_something_with 309
do_something_with 310
do_something_with 311
do_something_with 312
do_something_with 313
do_something_with 314
do_something_with 315
do_something_with 316
do_something_with 317
do_something_with 318
do_something_with 319
do_something_with 320
do_something_with 321
do_something_with 322
do_something_with 323
do_something_with 324
do_something_with 325
===
do_something_with 326
do_something_with 327
do_something_with 328
do_something_with 329
do_something_with 330
do_something_with 331
do_something_with 332
do_something_with 333
do_something_with 334
do_something_with 335
do_something_with 336
do_something_with 337
do_something_with 338
do_something_with 339
do_something_with 340
do_something_with 341
do_something_with 342
do_something_with 343
do_something_with 344
do_something_with 345
do_something_with 346
do_something_with 347
do_something_with 348
do_something_with 349
do_something_with 350
===
do_something_with 351
do_something_with 352
do_something_with 353
do_something_with 354
do_something_with 355
do_something_with 356
do_something_with 357
do_something_with 358
do_something_with 359
do_something_with 360
do_something_with 361
do_something_with 362
do_something_with 363
do_something_with 364
do_something_with 365
do_something_with 366
do_something_with 367
do_something_with 368
do_something_with 369
do_something_with 370
do_something_with 371
do_something_with 372
do_something_with 373
do_something_with 374
do_something_with 375
===
do_something_with 376
do_something_with 377
do_something_with 378
do_something_with 379
do_something_with 380
do_something_with 381
do_something_with 382
do_something_with 383
do_something_with 384
do_something_with 385
do_something_with 386
do_something_with 387
do_something_with 388
do_something_with 389
do_something_with 390
do_something_with 391
do_something_with 392
do_something_with 393
do_something_with 394
do_something_with 395
do_something_with 396
do_something_with 397
do_something_with 398
do_something_with 399
do_something_with 400
===
do_something_with 401
do_something_with 402
do_something_with 403
do_something_with 404
do_something_with 405
do_something_with 406
do_something_with 407
do_something_with 408
do_something_with 409
do_something_with 410
do_something_with 411
do_something_with 412
do_something_with 413
do_something_with 414
do_something_with 415
do_something_with 416
do_something_with 417
do_something_with 418
do_something_with 419
do_something_with 420
do_something_with 421
do_something_with 422
do_something_with 423
do_something_with 424
do_something_with 425
===
do_something_with 426
do_something_with 427
do_something_with 428
do_something_with 429
do_something_with 430
do_something_with 431
do_something_with 432
do_something_with 433
do_something_with 434
do_something_with 435
do_something_with 436
do_something_with 437
do_something_with 438
do_something_with 439
do_something_with 440
do_something_with 441
do_something_with 442
do_something_with 443
do_something_with 444
do_something_with 445
do_something_with 446
do_something_with 447
do_something_with 448
do_something_with 449
do_something_with 450
===
do_something_with 451
do_something_with 452
do_something_with 453
do_something_with 454
do_something_with 455
do_something_with 456
do_something_with 457
do_something_with 458
do_something_with 459
do_something_with 460
do_something_with 461
do_something_with 462
do_something_with 463
do_something_with 464
do_something_with 465
do_something_with 466
do_something_with 467
do_something_with 468
do_something_with 469
do_something_with 470
do_something_with 471
do_something_with 472
do_something_with 473
do_something_with 474
do_something_with 475
===
do_something_with 476
do_something_with 477
do_something_with 478
do_something_with 479
do_something_with 480
do_something_with 481
do_something_with 482
do_something_with 483
do_something_with 484
do_something_with 485
do_something_with 486
do_something_with 487
do_something_with 488
do_something_with 489
do_something_with 490
do_something_with 491
do_something_with 492
do_something_with 493
do_something_with 494
do_something_with 495
do_something_with 496
do_something_with 497
do_something_with 498
do_something_with 499
do_something_with 500
===
do_something_with 501
do_something_with 502
do_something_with 503
do_something_with 504
do_something_with 505
do_something_with 506
do_something_with 507
do_something_with 508
do_something_with 509
do_something_with 510
do_something_with 511
do_something_with 512
do_something_with 513
do_something_with 514
do_something_with 515
do_something_with 516
do_something_with 517
do_something_with 518
do_something_with 519
do_something_with 520
do_something_with 521
do_something_with 522
do_something_with 523
do_something_with 524
do_something_with 525
===
do_something_with 526
do_something_with 527
do_something_with 528
do_something_with 529
do_something_with 530
do_something_with 531
do_something_with 532
do_something_with 533
do_something_with 534
do_something_with 535
do_something_with 536
do_something_with 537
do_something_with 538
do_something_with 539
do_something_with 540
do_something_with 541
do_something_with 542
do_something_with 543
do_something_with 544
do_something_with 545
do_something_with 546
do_something_with 547
do_something_with 548
do_something_with 549
do_something_with 550
===
do_something_with 551
do_something_with 552
do_something_with 553
do_something_with 554
do_something_with 555
do_something_with 556
do_something_with 557
do_something_with 558
do_something_with 559
do_something_with 560
do_something_with 561
do_something_with 562
do_something_with 563
do_something_with 564
do_something_with 565
do_something_with 566
do_something_with 567
do_something_with 568
do_something_with 569
do_something_with 570
do_something_with 571
do_something_with 572
do_something_with 573
do_something_with 574
do_something_with 575
===
do_something_with 576
do_something_with 577
do_something_with 578
do_something_with 579
do_something_with 580
do_something_with 581
do_something_with 582
do_something_with 583
do_something_with 584
do_something_with 585
do_something_with 586
do_something_with 587
do_something_with 588
do_something_with 589
do_something_with 590
do_something_with 591
do_something_with 592
do_something_with 593
do_something_with 594
do_something_with 595
do_something_with 596
do_something_with 597
do_something_with 598
do_something_with 599
do_something_with 600
===
do_something_with 601
do_something_with 602
do_something_with 603
do_something_with 604
do_something_with 605
do_something_with 606
do_something_with 607
do_something_with 608
do_something_with 609
do_something_with 610
do_something_with 611
do_something_with 612
do_something_with 613
do_something_with 614
do_something_with 615
do_something_with 616
do_something_with 617
do_something_with 618
do_something_with 619
do_something_with 620
do_something_with 621
do_something_with 622
do_something_with 623
do_something_with 624
do_something_with 625
===
do_something_with 626
do_something_with 627
do_something_with 628
do_something_with 629
do_something_with 630
do_something_with 631
do_something_with 632
do_something_with 633
do_something_with 634
do_something_with 635
do_something_with 636
do_something_with 637
do_something_with 638
do_something_with 639
do_something_with 640
do_something_with 641
do_something_with 642
do_something_with 643
...
do_something_with 8976
do_something_with 8977
do_something_with 8978
do_something_with 8979
do_something_with 8980
do_something_with 8981
do_something_with 8982
do_something_with 8983
do_something_with 8984
do_something_with 8985
do_something_with 8986
do_something_with 8987
do_something_with 8988
do_something_with 8989
do_something_with 8990
do_something_with 8991
do_something_with 8992
do_something_with 8993
do_something_with 8994
do_something_with 8995
do_something_with 8996
do_something_with 8997
do_something_with 8998
do_something_with 8999
do_something_with 9000
The (( ${#files[*]} == num )) will execute the loop if it is true, if that is not desired just test if the array files has a value (regardless if it is less than 25 elements), change it to (( ${#files[*]} ))
wc -l < file.txt to get the total lines in a file, to save that in a variable you could do, total_lines=$(wc -l < file.txt)
Update:
Instead of using wc -l < file.txt (not to hardcode the total number of lines in a file) to save the number of lines in a variable, something like:
#!/usr/bin/env bash
sum=0
num=25
total_lines=$(wc -l < file.txt)
while mapfile -tn "$num" files && (( ${#files[*]} )); do
((sum+=${#files[*]}))
for i in "${!files[#]}"; do
echo do_something_with "${files[$i]}"
done
if ((sum < total_lines)); then
echo ===
else
printf 'All is done!\n'
fi
done < file.txt
Adapting to #mark-mapuso's modulus approach, something like:
#!/usr/bin/env bash
sum=0
num=25
while mapfile -tn "$num" files && ((${#files[*]})); do
((sum+=${#files[*]}))
for i in "${!files[#]}"; do
echo do_something_with "${files[$i]}"
done
if ((sum % num == 0)); then
sleep 60
else
printf 'All done!\n'
fi
done < file.txt