I have a folder named datafolder which contains five csv files aa.csv ab.csv ac.csv ad.csv ae.csv Each csv file contains data from an excel sheet in the format: date, product type, name, address etc. and I am only interested in the second column which is named product. Basically what I want to happen is for the jobmaster script to count the number of files in datafolder and then to start a map process for each individual file. I have the following scripts:
The jobmaster script runs without problems, however once the map script starts, only the first echo mapping $1 is displaying and the process is stuck in an infinite loop (my guess). When I run the ps command I expect to see 5 map.sh running, however there are none.
I suspect you missed an input redirection in map.sh:
file=$1
echo "mapping $file"
while IFS="," read -r value1 product remainder; do
# ...
done < "$file"
# ^^^^^ provide the standard input to from this file to `read`
Related
I tried to run this code but its repeated the same file and ignored the other files and I want to stored all the directory files into a variable so that, read the file one by one and after addition of hydrogen by reduce program the file contain same file name with H addition and I want to save this output in separate directory how I can do this please help me I am new in this coding field.
I tried to run this code but its repeated the same file and ignored the other files
#!/bin/bash
# A script for reduce program
l = cd ~/hetero
for l in hetero/*.pdb
do
ls | xargs -L 1 -d '\n' reduce *.pdb > ....*H.pdb
done
echo "all done"
Your for-loop defines l as a variable whose value you obtain within the loop with the syntax $l. If you don't use $l inside your loop, you cannot expect loop iterations to depend on it.
csv file with the first row has header file. I need to read this first row and then parse it to see if it has the elements i'm looking for.
First row has 4 elements. 1. HDR 2. today's date 3. From date 4. To date.
here is the code i used to get the first row.
read -r header < "1" -- this game me the first row into header variable.
I tried to read this 'header' variable to further split the row.
read f1 f2 f3 < “$header”
echo "OS is : $f1"
echo "Company is: $f2"
echo "Value is : $f3"
i'm getting no values displayed. I think the reason could be 'header' is not coming in as a string.
I'm new to unix shell scripting. Please help.
read -r header < "1"
No. You are reading from a file named 1 (which probably does not exist, so read is failing) because < is used for redirection. Try typing that very command in a terminal, you'll get an error ("no such file or directory").
read f1 f2 f3 < “$header”
This is wrong. If the header shell variable contains 1 2 3 you are reading from the file named 1 2 3 (a six character file path with two spaces inside) because < is used for redirection
Consider using awk to process files made of lines containing several columns.
Consider starting your (executable) shell script with #!/bin/bash -vx (to get traces) during the debugging phase. Spend a few hours reading some shell scripting tutorial and then the bash reference manual.
Be sure to understand how shells work, notably their globbing. Be aware that every program (e.g. started by some shell) is started by execve(2) (done in some process, e.g. your shell process).
BTW, before the read f1 f2 f3 < “$header” you might (temporarily) add debugging outputs, e.g.
echo "header=" "$header" > /dev/stderr
in your shell script, to understand what is going on.
(You really should spend days in reading more about shells)
I'm working on a program to process requests in bash which are requested by users in a WebInterface. To give users flexibility they can specify several parameters per each job, at the end the request is saved in a file with a specific name, so the bash script could perform the requested task.
This file at the end is filled like this:
ENVIRONMENT="PRO"
INTEGRATION="J050_provisioning"
FILE="*"
DIRECTORY="out"
So the bash script will source this file to perform the needed tasks user requested. And it works great so far, but I see a security issue with this, if user enters malicious data, something like:
SOMEVAR="GONNAHACK $(rm -f some_important_file)"
OTHERVAR="DANGEROUZZZZZZ `while :; do sleep 10 & done`"
This will cause undesirable effects when sourcing the file :). Is there a way to prevent a source file execute any code but variable initializations? Or the only way would be just grep the source file before sourcing it to check it is not dangerous?
Just do not source it. Make it a configuration file composed of name=value lines (without the double quotes), read each name/value pair and assign value to name. In order not to overwrite critical variables like PATH, prefix the name with CONF_ for example.
Crude code:
while IFS='=' read -r conf_name conf_value; do
printf -v "CONF_$conf_name" '%s' "$conf_value" \
|| echo "Invalid configuration name '$conf_name'" >&2
done < your_configuration_file.conf
Test it works:
$ echo "${!CONF_*}"
CONF_DIRECTORY CONF_ENVIRONMENT CONF_FILE CONF_INTEGRATION CONF_OTHERVAR CONF_SOMEVAR
$ printf '%s\n' "$CONF_SOMEVAR"
GONNAHACK $(rm -f some_important_file)
I'm using Mac's Automator to perform a bash script on files that are dropped onto a droplet. Inputs are gathered from Automator actions and are passed to a bash script as arguments. The script works with a single input file, but I'm having trouble handling multiple files.
My ultimate goal is to accept several dropped video files, prompt for a number, and extract that number of frames from each video file (using FFmpeg).
I can loop through inputs by using the special variable $#, like so:
for f in "$#"; do
# perform an action
done
However, my script also prompts the user for text input that I don't want included in this loop.
I can access each input file individually by using $1, $2, etc. But I'd like to use a loop instead of referencing each file individually. Also, the quantity of input files is unpredictable and I'm not sure how to distinguish between input files and input text.
How can I loop through only the file inputs without including the text input?
Here is a description of my current workflow:
Get Specified Movies
one.mov
two.mov
Set Value of Variable (accepts input)
source_files
Ask For Text (ignores input)
Enter a Number:
Set Value of Variable (accepts input)
number
Get Value of Variable (ignores input)
source_files
Get Value of Variable (accepts input)
number
Run Shell Script (accepts input, pass "as arguments")
#/bin/bash
for f in "$#"; do
echo $f
done
OUTPUT:
/folder/one.mov
/folder/two.mov
8
I was hoping to have one variable set to the multiple inputs (so I could loop through it) and another variable set to the number, but it doesn't seem to work that way.
How can I loop through each input file without referencing the text input?
Just change the order of the args: get value of variable number first,
then get value of variable source_files,
so that number will be the first parameter,
then:
echo number: $1
for f in "${#:2}"; do
echo file: $f
done
I Have a Shell script which executes couple of binary files infinitely . My Task is to redirect the output displayed by these binary files into a log file with timestamp (YY-MM-DD). This task is pretty easy , However problem arises when the day changes. This is my problem -- If a particular binary file is in the process of execution (not completed yet) and day changes , the output should displayed should be logged in 2 different files with different timestamps. For Eg: -
while true; do
if (current_date = new_date); then
execute binary >> log.out_$current_date
// If binary is still in the process of execution how to redirect in 2 files ???
else
execute binary >> log.out_$new_date
fi
done
Required Output is :: output of file on current_date to be logged in current log file and output of file on new date to be logged in new file
..... Please Help
Just create a further script that read from the stdout of your binary and check for the date after every single line read
(source of outputdatescript.sh)
#!/bin/bash
while read line; do
# example made with unix data, replace it with your date string generator
date=$(date "+%s")
echo $line >> file.$date
done;
then the command in your main script must be subst from
execute binary >> log.out_$current_date
to
execute binary | outputdatescript.sh