I have very little experience with bash, but am attempting to put together a pipeline that reads in and executes commands line by line from an input file. The input file, called "seeds.txt", is set up like so
program_to_execute -seed_value 4496341759106 -a_bunch_of_arguments c(1,2,3) ; #Short_description_of_run
program_to_execute -seed_value 7502828106749 -a_bunch_of_arguments c(4,5,6) ; #Short_description_of_run
I separated the #Short_descriptions from the commands by a semi-colon (;) since the arguments contain commas (,). When I run the following script I get a "No such file or directory" error
#!/bin/bash
in="${1:-seeds.txt}"
in2=$(cut -d';' -f1 "${in}")
while IFS= read -r SAMP
do
$SAMP
done < $in2
I know that seeds.txt is getting read in fine, so I'm not sure why I'm getting a missing file/directory message. Could anyone point me in the right direction?
Using GNU Parallel it looks like this:
cut -d';' -f1 seeds.txt | parallel
you can try as below with eval...not very safe though, see this for more info
while read line; do eval "$line" ; done < <(cut -d';' -f1 seeds.txt)
Just in case you want to avoid eval
while read -ra line; do command "${line[#]}"; done < <(cut -d';' -f1 seeds.txt)
Note this solution does not work if the program/utility is not an executable within your PATH, e.g. you wan to use a function or an alias. Not sure if the eval solution can do that too. Kudos to the cut solution!
Related
I'm writing a bash script, and I need to take the second field of every line in a file, and save them in another file. I know there are many possible ways to do this, BUT, I tried first using while read line; do, and I got stuck. Now, I really want to know what is happening.
For example, input file would be:
line1 11111
line2 222222
line3 333
line4 4444
(The field separtor is "\t").
This is what I was doing:
inputfile=$1
cat $"inputfile" | while read -r line
do
cut -f2 >> results_file
done
The problem is, the output would be:
222222
333
4444
(skipping the first line)
I´ve alredy tested hundreds of modifications, and tried to used other commands instead of cut(like, sed, grep...). I would appreciate some help, or someone pointing me in the right direction.
Thank you very much!
You are not using the variable $line set by read. Try instead
inputfile=$1
cat "$inputfile" | while read -r line
do
echo "$line" | cut -f2 >> results_file
done
In your original code, the while loop is actually run only once, not four times; try putting echo 'Hello!' in the loop to your original code. You would see the message only once, not four times. I guess, without echo "$line" | part, cut -f2 ... part consumes the pipe away.
That is, your while loop first consumes the first line of the stdin and puts this line in the variable $line, leaving the next three lines for later use. But $line is never used. Instead, the remaining three lines are consumed by the command cut.
All commands within a command group are within the scope of any redirections applied to a command group (or any compound command):
— https://mywiki.wooledge.org/BashGuide/CompoundCommands
The pipe operator creates a subshell environment for each command.
— https://mywiki.wooledge.org/BashGuide/InputAndOutput
We can interpret the quotes as "the stdin to your while loop (i.e., the output of cat "$inputfile") is accessed by cut, unless you sever its access by creating a new subshell e.g., by another pipe echo "$line" | ...."
By the way, you can just use cut -f2 "$inputfile" >> results_file without the while loop.
With respect to your comment Does it mean to use "\t at the end" as a separator - no. You're confusing what was suggested, $'\t' with '\t$'. $'\t' means "the literal tab character generated from the escape sequence \t".
You also said in your comment your real 2nd fields are URLs to be curled. You shouldn't be using a UUOC and cut anyway, here's how to really do this:
while IFS=$'\t' read -r key url; do
val=$(curl "$url" | whatever)
printf '%s\t%s\n' "$key" "$val"
done < "$inputfile" > results_file
Replace whatever with whatever command you use to produce the output you want from the curl output.
I have a file with this structure:
picture1_123.txt
picture2_456.txt
picture3_789.txt
picture4_012.txt
I wanted to get only the first segment of the file name, that is, picture1 to picture4.
I first used the following code:
cat picture | while read -r line; do cut -f1 -d "_"; echo $line; done
This returns the following output:
picture2
picture3
picture4
picture1_123.txt
This error got corrected when I changed the code to the following:
cat picture | while read line; do s=$(echo $line | cut -f1 -d "_"); echo $s; done
picture1
picture2
picture3
picture4
Why in the first:
The lines are printed in a different order than the original file?
no operation is done on picture1_123.txt and picture1 is not printed?
Thank you!
What Was Wrong
Here's what your old code did:
On the first (and only) iteration of the loop, read line read the first line into line.
The cut command read the entire rest of the file, and wrote the results of extracting only the desired field to stdout. It did not inspect, read, or modify the line variable.
Finally, your echo $line wrote the first line in entirety, with nothing being cut.
Because all input had been consumed by cut, nothing remained for the next read line to consume, so the loop never ran a second time.
How To Do It Right
The simple way to do this is to let read separate out your prefix:
while IFS=_ read -r prefix suffix; do
echo "$prefix"
done <picture
...or to just run nothing but cut, and not use any while read loop at all:
cut -f1 -d_ <picture
I have a CSV file with sample entries given below. What I want is to write a Bash script to read the CSV file line by line and put the first entry e.g 005 in one variable and the IP 192.168.10.1 in another variable, that I need to pass to some other script.
005,192.168.10.1
006,192.168.10.109
007,192.168.10.12
008,192.168.10.121
009,192.168.10.123
A more efficient approach, without the need to fork cut each time:
#!/usr/bin/env bash
while IFS=, read -r field1 field2; do
# do something with $field1 and $field2
done < file.csv
The gains can be quite substantial for large files.
Here's how I would do it with GNU tools :
while read line; do
echo $line | cut -d, -f1-2 --output-delimiter=' ' | xargs your_command
done < your_input.csv
while read line; do [...]; done < your_input.csv will read your file line by line.
For each line, we will cut it to its first two fields (separated by commas since it's a CSV) and pass them separated by spaces to xargs which will in turn pass as parameters to your_command.
If this is a very simple csv file with no string literals, etc. you can simply use head and cut:
#!/bin/bash
while read line
do
id_field=$(cut -d',' -f 1 <<<"$line") #here 005 for the first line
ip_field=$(cut -d',' -f 2 <<<"$line") #here 192.168.0.1 for the first line
#do something with $id_field and $ip_field
done < file.csv
The program works as follows: we use cut -d',' to obtain the first and second field of that line. We wrap this around a while read line and use I/O redirection to feed the file to the while loop.
Of course you substitute file.csv with the name of the file you want to process, and you can use other variable names than the ones in this sample.
I've written a template engine script that uses cut to extract certain elements from a file, but I want to use grep in place of the cut. Here is the code I have written:
#!/bin/bash
IFS=# #makes # a delimiter.
while read line
do
dataspace=`echo ${line} | cut -d'=' -f1`
value=`echo ${line} | cut -d"=" -f2`
printf -v $dataspace "$value" #make the value stored in value into the name of a dataspace.
done < 'template.vars' #read template.vars for standard input.
skipflag=false #initialize the skipflag to false
while read line #while it is reading standard input one line at a time
Just came to the conclusion that the code blocks system here does not support bash.
Anyway, since stackoverflow isn't letting me put Bash into codeblocks, I am not putting the entire script since it would look nasty. Based on what is currently high-lighted, how would I go about changing the part using the cut command into a line using the grep command?
As has been noted, you should give more information for a better answer. Going with what you have, I would say that awk is a better option than grep
dataspace=$(awk '$0=$1' FS== <<< "$line")
value=$(awk '$0=$2' FS== <<< "$line")
I have a configuration file that contains lines like "hallo;welt;" and i want to do a grep on this file.
Whenever i try something like grep "$1;$2" my.config or echo "$1;$2 of even line="$1;$2" my script fails with something like:
: command not found95: line 155: =hallo...
How can i tell bash to ignore ; while evaluating "..." blocks?
EDIT: an example of my code.
# find entry
$line=$(grep "$1;$2;" $PERMISSIONSFILE)
# splitt line
reads=$(echo $line | cut -d';' -f3)
writes=$(echo $line | cut -d';' -f4)
admins=$(echo $line | cut -d';' -f5)
# do some stuff on the permissions
# replace old line with new line
nline="$1;$2;$reads;$writes;$admins"
sed -i "s/$line/$nline/g" $TEMPPERM
my script should be called like this: sh script "table" "a.b.*.>"
EDIT: another, simpler example
$test=$(grep "$1;$2;" temp.authorization.config)
the temp file:
table;pattern;read;write;stuff
the call sh test.sh table pattern results in: : command not foundtable;pattern;read;write;stuff
Don't use $ on the left side of an assignment in bash -- if you do it'll substitute the current value of the variable rather than assigning to it. That is, use:
test=$(grep "$1;$2;" temp.authorization.config)
instead of:
$test=$(grep "$1;$2;" temp.authorization.config)
Edit: also, variable expansions should be in double-quotes unless there's a good reason otherwise. For example, use:
reads=$(echo "$line" | cut -d';' -f3)
instead of:
reads=$(echo $line | cut -d';' -f3)
This doesn't matter for semicolons, but does matter for spaces, wildcards, and a few other things.
A ; inside quotes has no meaning at all for bash. However, if $1 contains a doublequote itself, then you'll end up with
grep "something";$2"
which'll be parsed by bash as two separate commands:
grep "something" ; other"
^---command 1----^ ^----command 2---^
Show please show exactly what your script is doing around the spot the error is occurring, and what data you're feeding into it.
Counter-example:
$ cat file.txt
hello;welt;
hello;world;
hell;welt;
$ cat xx.sh
grep "$1;$2" file.txt
$ bash -x xx.sh hello welt
+ grep 'hello;welt' file.txt
hello;welt;
$
You have not yet classified your problem accurately.
If you try to assign the result of grep to a variable (like I do) your example breaks.
Please show what you mean. Using the same data file as before and doing an assignment, this is the output I get:
$ cat xx.sh
grep "$1;$2" file.txt
output=$(grep "$1;$2" file.txt)
echo "$output"
$ bash -x xx.sh hello welt
+ grep 'hello;welt' file.txt
hello;welt;
++ grep 'hello;welt' file.txt
+ output='hello;welt;'
+ echo 'hello;welt;'
hello;welt;
$
Seems to work for me. It also demonstrates why the question needs an explicit, complete, executable, minimal example so that we can see what the questioner is doing that is different from what people answering the question think is happening.
I see you've provided some sample code:
# find entry
$line=$(grep "$1;$2;" $PERMISSIONSFILE)
# splitt line
reads=$(echo $line | cut -d';' -f3)
writes=$(echo $line | cut -d';' -f4)
admins=$(echo $line | cut -d';' -f5)
The line $line=$(grep ...) is wrong. You should omit the $ before line. Although it is syntactically correct, it means 'assign to the variable whose name is stored in $line the result of the grep command'. That is unlikely to be what you had in mind. It is, occasionally, useful. However, those occasions are few and far between, and only for people who know what they're doing and who can document accurately what they're doing.
For safety if nothing else, I would also enclose the $line values in double quotes in the echo lines. It may not strictly be necessary, but it is simple protective programming.
The changes lead to:
# find entry
line=$(grep "$1;$2;" $PERMISSIONSFILE)
# split line
reads=$( echo "$line" | cut -d';' -f3)
writes=$(echo "$line" | cut -d';' -f4)
admins=$(echo "$line" | cut -d';' -f5)
The rest of your script was fine.
It seems like you are trying to read a semicolon-delimited file, identify a line starting with 'table;pattern;' where table is a string you specify and pettern is a regular expression grep will understand. Once the line is identified you wish to replaced the 3rd, 4th and 5th fields with different data and write the updated line back to the file.
Does this sound correct?
If so, try this code
#!/bin/bash
in_table="$1"
in_pattern="$2"
file="$3"
while IFS=';' read -r -d$'\n' tuple pattern reads writes admins ; do
line=$(cut -d: -f1<<<"$tuple")
table=$(cut -d: -f2<<<"$tuple")
# do some stuff with the variables
# e.g., update the values
reads=1
writes=2
admins=12345
# replace the old line with the new line
sed -i'' -n $line'{i\
'"$table;$pattern;$reads;$writes;$admins"'
;d;}' "$file"
done < <(grep -n '^'"${in_table}"';'"${in_pattern}"';' "${file}")
I chose to update by line number here to avoid problems of unknown characters in the left hand of the substitution.