How do I open / manipulate multiple files in bash? - bash

I have a bash script that take advantage of a local toolbox to perform an operation
my question is fairly simple
I have multiple files that are the same quantities but different time steps i would like to first untar them all, and then use the toolbox to perform some manipulation but i am not sure if i am on the right track.
=============================================
The file is as follows
INPUTS
fname = a very large number or files with same name but numbering
e.g wnd20121.grb
wnd20122.grb
.......
wnd2012100.grb
COMMANDS
> cdo -f nc copy fname ofile(s)
(If this is the ofile(s)=output file how can i store it for sequent use ? Take the ofile (output file) from the command and use it / save it as input to the next, producing a new subsequent numbered output set of ofile(s)2)
>cdo merge ofile(s) ofile2
(then automatically take the ofile(s)2 and input them to the next command and so on, producing always an array of new output files with specific set name I set but different numbering for distinguishing them)
>cdo sellon ofile(s)2 ofile(s)3
------------------------------------
To make my question clearer, I would like to know the way in which I can instruct basically through a bash script the terminal to "grab" multiple files that are usually the same name but have a different numbering to make the separate their recorded time
e.g. file1 file2 ...file n
and then get multiple outputs , with every output corresponding to the number of the file it converted.
e.g. output1 output2 ...outputn
How can I set these parameters so the moment they are generated they are stored for subsequent use in the script, in later commands?

Your question isn't clear, but perhaps the following will help; it demonstrates how to use arrays as argument lists and how to parse command output into an array, line by line:
#!/usr/bin/env bash
# Create the array of input files using pathname expansion.
inFiles=(wnd*.grb)
# Pass the input-files array to another command and read its output
# - line by line - into a new array, `outFiles`.
# The example command here simply prepends 'out' to each element of the
# input-files array and outputs each (modified) element on its own line.
# Note: The assumption is that the filenames have no embedded newlines
# (which is usually true).
IFS=$'\n' read -r -d '' -a outFiles < \
<(printf "%s\n" "${inFiles[#]}" | sed s'/^/out-/')
# Note: If you use bash 4, you could use `readarray -t outFiles < <(...)` instead.
# Output the resulting array.
# This also demonstrates how to use an array as an argument list
# to pass to _any_ command.
printf "%s\n" "${outFiles[#]}"

Related

Replacing string with variable, output to variably named files

I have the template file 12A-r.inp . I want to prepare files from this file whose name will be 16A-r.inp, 20A-r.inp, 24A-r.inp. And I want to change some parameter in those files according to their names. For example, I want to replace the string "12A" in all places in file 12A-r.inp, with 16A in 16A-r.inp, and 20A in 20A-r.inp. I have written the code below for this:
for ((i=12;i<=24;i=i+4))
do
cat 12A-r.inp >> $i\A-r.inp
done
for ((i=12;i<=24;i=i+4))
do
sed -i "s/12A/${i}/g" $i\A-r.inp
done
But the problem is 12A gets replaced by ${i}, not with strings like 16A, 20A etc.
Observations:
In for ((i=12;i<=24;i=i+4)) counts 12,16,20,24. There's no need
to start at 12, since the template is already correct. Worse,
when i=12, this code cat 12A-r.inp >> $i\A-r.inp appends a
copy of the template file onto itself, doubling it, which causes every ensuing
created file to be twice as long as the original template.
The \ in $i\A-r.inp is unnecessary, since A is not a special character.
The cat is unnecessary, sed without -i can do it all.
In sed, s/12A/${i}/g would replace the string "12A", with whatever number $i is, without the "A", unless the variable includes that letter.
The for loop uses a bashism to enumerate i... in this instance there's a simpler equivalent bashism, (see below).
Suggested revision:
for i in {16..24..4}A
do
sed "s/12A/${i}/g" 12A-r.inp > ${i}-r.inp
done
How it works:
$i is set to 16A,20A, and 24A.
sed repeatedly reads in the template, replaces 12A with $i,
prints everything to STDOUT...
which is redirected to the appropriately named file.

Iteratively pass one file from a collection of files as command line parameters in shell script

I have x files: A, B, C... What I need to do is pass each of these files as the first command line argument to a python file and pass the others as the second command line argument until all files have been passed as $1 once. For example, on the first iteration A is $1 and B,C... is $2. On the second iteration, B is $1 and A,C... is $2. I've read about the shift command in shell but am not very sure if it will work in my case (I'm also relatively new to shell scripting). Also, is there a limit to the number of command line arguments I can pass to my python script? I would also like to create a variable to hold the list of file names before iterating through my files. Thank you!
Bash has arrays, and supports array slicing via ${array[#]:start:end} syntax, where start and end are optional indices. That's enough to get the job done.
#!/bin/bash
# Store the master list of file names in an array called $files.
files=("$#")
for ((i = 0; i < ${#files[#]}; ++i)); do
# Store the single item in $file and the rest in an array $others.
file=${files[i]}
others=("${files[#]:0:i}" "${files[#]:i+1}")
# Run command.py. Use ${others[*]} to concatenate all the file names into one long
# string, and override $IFS so they're joined with commas.
(IFS=','; command.py "${files[i]}" "${others[*]}")
done

Execute bash script on multiple input files, ignoring other input variables

I'm using Mac's Automator to perform a bash script on files that are dropped onto a droplet. Inputs are gathered from Automator actions and are passed to a bash script as arguments. The script works with a single input file, but I'm having trouble handling multiple files.
My ultimate goal is to accept several dropped video files, prompt for a number, and extract that number of frames from each video file (using FFmpeg).
I can loop through inputs by using the special variable $#, like so:
for f in "$#"; do
# perform an action
done
However, my script also prompts the user for text input that I don't want included in this loop.
I can access each input file individually by using $1, $2, etc. But I'd like to use a loop instead of referencing each file individually. Also, the quantity of input files is unpredictable and I'm not sure how to distinguish between input files and input text.
How can I loop through only the file inputs without including the text input?
Here is a description of my current workflow:
Get Specified Movies
one.mov
two.mov
Set Value of Variable (accepts input)
source_files
Ask For Text (ignores input)
Enter a Number:
Set Value of Variable (accepts input)
number
Get Value of Variable (ignores input)
source_files
Get Value of Variable (accepts input)
number
Run Shell Script (accepts input, pass "as arguments")
#/bin/bash
for f in "$#"; do
echo $f
done
OUTPUT:
/folder/one.mov
/folder/two.mov
8
I was hoping to have one variable set to the multiple inputs (so I could loop through it) and another variable set to the number, but it doesn't seem to work that way.
How can I loop through each input file without referencing the text input?
Just change the order of the args: get value of variable number first,
then get value of variable source_files,
so that number will be the first parameter,
then:
echo number: $1
for f in "${#:2}"; do
echo file: $f
done

Using bash wildcards with prefix

I am trying to write a bash script that takes a variable number of file names as arguments.
The script is processing those files and creating a temporary file for each of those files.
To access the arguments in a loop I am using
for filename in $*
do
...
generate t_$(filename)
done
After the loop is done, I want to do something like cat t_$* .
But it's not working. So, if the arguments are a b c, it is catting t_a, b and c.
I want to cat the files t_a, t_b and t_c.
Is there anyway to do this without having to save the list of names in another variable?
You can use the Parameter expansion:
cat "${#/#/t_}"
/ means substitute, # means at the beginning.

Open file in bash script

I've got a bash script accepting several files as input which are mixed with various script's options, for example:
bristat -p log1.log -m lo2.log log3.log -u
I created an array where i save all the index where i can find files in the script's call, so in this case it would be an arrat of 3 elements where
arr_pos[0] = 2
arr_pos[1] = 4
arr_pos[3] = 5
Later in the script I must call "head" and "grep" in those files and i tried this way
head -n 1 ${arr_pos[0]}
but i get this error non runtime
head: cannot open `2' for reading: No such file or directory
I tried various parenthesis combinations, but I can't find which one is correct.
The problem here is that ${arr_pos[0]} stores the index in which you have the file name, not the file name itself -- so you can't simply head it. The array storing your arguments is given by $#.
A possible way to access the data you want is:
#! /bin/bash
declare -a arr_pos=(2 4 5)
echo ${#:${arr_pos[0]}:1}
Output:
log1.log
The expansion ${#:${arr_pos[0]}:1} means you're taking the values ranging from the index ${arr_pos[0]} in the array $#, to the element of index ${arr_pos[0]} + 1 in the same array $#.
Another way to do so, as pointed by #flaschenpost, is to eval the index preceded by $, so that you'd be accessing the array of arguments. Although it works very well, it may be risky depending on who is going to run your script -- as they may add commands in the argument line.
Anyway, you may should try to loop through the entire array of arguments by the beginning of the script, hashing the values you find, so that you won't be in trouble while trying to fetch each value later. You may loop, using a for + case ... esac, and store the values in associative arrays.
I think eval is what you need.
#!/bin/bash
arr_pos[0]=2;
arr_pos[1]=4;
arr_pos[2]=5;
eval "cat \$${arr_pos[1]}"
For me that works.

Resources