I have this code
TOKEN=$(cat ./config/token)
echo "$TOKEN"
cat > variables.env <<EOF
TOKEN=`echo "$TOKEN"`
EOF
I am trying to get the content of a file and output it in a new file prefixed by some text. The first echo in the console echoes the output I want, keeping the whitespaces and newlines.
However, in the new file the output is just the first line of the original string, while I'd like the same output I can see in the console with the first echo.
Use printf %q (in ksh or bash) to escape content in such a way that it will always evaluate back to its literal value:
printf 'TOKEN=%q\n' "$(<./config/token)" >variables.env
$(<file) is a ksh and bash extension which acts as a more efficient replacement for $(cat file) (as the regular command substitution needs to fork off a subprocess, set up a FIFO, and spawn an external copy of /bin/cat, whereas the $(<file) form simply tells the shell to read the file directly).
This way a taken containing an otherwise-hostile string such as $(rm -rf ~) or content that could simply be expanded as a variable ($$) will be emitted as literal content.
Providing an explicit example of how this behaves:
printf '%s\n' "first line" "second line" >token # write two lines to the file "token"
printf 'TOKEN=%q\n' "$(<token)" >variables.env # write a shell command which assigns those
# two lines to a variable to variables.env
source variables.env # execute variables.env in the current shell
echo "$TOKEN" # emit the value of TOKEN, as given in the current shell
...when run with bash, will emit the exact output:
first line
second line
...after writing the following (with bash 3.2.48; may vary with other releases) to variables.env:
TOKEN=$'first line\nsecond line'
Useless use of echo
This is what you could write:
cat > variables.env <<EOF
TOKEN=${TOKEN}
EOF
you are doing it in a very convoluted way, there are easier methods
sed '1s/./TOKEN=&/' file > newfile
will insert TOKEN= on the first line. This has an additional benefit of not modifying empty files (at least one char should exist in the original file). If that's not intended you can use unconditional insert.
You can do:
echo "TOKEN=" > newfile && cat ./config/token >> newfile
>> appends to a file.
Related
I'm having an issue in something that seems to be a rookie error, but I can't find a way to find a solution.
I have a bash script : log.sh
which is :
#!/bin/bash
echo $1 >> log_out.txt
And with a file made of filenames (taken from the output of "find" which names is filesnames.txt and contains 53 lines of absolute paths) I try :
./log.sh $(cat filenames.txt)
the only output I have in the log_out.txt is the first line.
I need each line to be processed separately as I need to put them in arguments in a pipeline with 2 softwares.
I checked for :
my lines being terminated with /n
using a simple echo without writing to a file
all the sorts of cat filenames.txt or (< filenames.txt) found on internet
I'm sure it's a very dumb thing, but I can't find why I can't iterate more than one line :(
Thanks
It is because your ./log.sh $(cat filenames.txt) is being treated as one argument.
while IFS= read -r line; do
echo "$line";
done < filenames.txt
Edit according to: https://mywiki.wooledge.org/DontReadLinesWithFor
Edit#2:
To preserve leading and trailing whitespace in the result, set IFS to the null string.
You could simplify more and skip using explicit variable and use the default $REPLY
Source: http://wiki.bash-hackers.org/commands/builtin/read
You need to quote the command substitution. Otherwise $1 will just be the first word in the file.
./log.sh "$(cat filenames.txt)"
You should also quote the variable in the script, otherwise all the newlines will be converted to spaces.
echo "$1" >> log_out.txt
If you want to process each word separately, you can leave out the quotes
./log.sh $(cat filenames.txt)
and then use a loop in the script:
#!/bin/bash
for word in "$#"
do
echo "$word"
done >> log_out.txt
Note that this solution only works correctly when the file has one word per line and there are no wildcards in the words. See mywiki.wooledge.org/DontReadLinesWithFor for why this doesn't generalize to more complex lines.
You can iterate with each line.
#!/bin/bash
for i in $*
do
echo $i >> log_out.txt
done
I am trying to write an fgrep statement removing records with a full record match from a file. I can do this on the command line, but not inside a ksh script. The code I am using boils down to these 4 lines of code:
Header='abc def|ghi jkl' #I use the head command to populate this variable
workfile=abc.txt
command="fgrep -Fxv \'$Header\' $workfile" >$outfile
$command
When I echo $command to STDIN the command is exactly what I would type on the command line (with the single quotes) and that works on the command line. When I execute it within the ksh script (file) the single quotes seem not to be recognized because the errors show it is parsing on spaces.
I have tried back ticks, exec, eval, double quotes instead of single quotes, and not using the $command variable. The problem remains.
I can do this on the command line, but not inside a ksh script
Here's a simple, portable, reliable solution using a heredoc.
#!/usr/bin/env ksh
workfile=abc.txt
outfile=out.txt
IFS= read -r Header <<'EOF'
abc def|ghi jul
EOF
IFS= read -r command <<'EOF'
grep -Fxv "$Header" "$workfile" > "$outfile"
EOF
eval "$command"
Explanation :
(Comments can't be added to the script above because they would affect the lines in the heredoc)
IFS= read -r Header <<'EOF' # Line separated literal strings
abc def|ghi jul # Set into the $Header variable
EOF # As if it were a text file
IFS= read -r command <<'EOF' # Command to execute
grep -Fxv "$Header" "$workfile" > "$outfile" # As if it were typed into
EOF # the shell command line
eval "$command" # Execute the command
The above example is the same as having a text file called header.txt, which contains the contents: abc def|ghi jul and typing the following command:
grep -Fxvf header.txt abc.txt
The heredoc addresses the problem of the script operating differently than the command line as a result of quoting/expansions/escaping issues.
A Word of caution regarding eval:
The use of eval in this example is specific. Please see Eval command and security issues for information on how eval can be misused and cause potentially very damaging results.
More Detail / Alternate Example:
For the sake of completeness, clarity, and ability to apply this concept to other situations, some notes about the heredoc and an alternative demonstration:
This implementation of the heredoc in this example is specifically designed with the following criteria:
Literal string assignment of contents, to the variables (using 'EOF')
Use of the eval command to evaluate and execute the referenced variables within the heredoc itself.
File or heredoc ?
One strength of using a heredoc combined with grep -F (fgrep), is the ability to treat a section of the script as if it were a file.
Case for file:
You want to frequently paste "pattern" lines into the file, and remove them as necessary, without having to modify the script file.
Case for heredoc:
You apply the script in an environment where specific files already exist, and you want to match specific exact literal patterns against it.
Example:
Scenario: I have 5 VPS Servers, and I want a script to produce a new fstab file but to ensure it doesn't contain the exact line:
/dev/xvda1 / ext3 errors=remount-ro,noatime,barrier=0 0 1
This scenario fits the type of situation addressed in this question. I could use the boilerplate from the above code in this answer and modify it as following:
#!/usr/bin/env ksh
workfile=/etc/fstab
IFS= read -r Header <<'EOF'
/dev/xvda1 / ext3 errors=remount-ro,noatime,barrier=0 0 1
EOF
IFS= read -r command <<'EOF'
grep -Fxv "$Header" "$workfile"
EOF
eval "$command"
This would give me a new fstab file, without the line contained in the heredoc.
Bash FAQ #50: I'm trying to put a command in a variable, but the complex cases always fail! provides comprehensive guidance - while it is written for Bash, most of it applies to Ksh as well.[1]
If you want to stick with storing your command in a variable (defining a function is the better choice), use an array, which bypasses the quoting issues:
#!/usr/bin/env ksh
Header='abc def|ghi jkl'
workfile=abc.txt
# Store command and arguments as elements of an array
command=( 'fgrep' '-Fxv' "$Header" "$workfile" )
# Invoke the array as a command.
"${command[#]}" > "$outfile"
Note: only a simple command can be stored in an array, and redirections can't be part of it.
[1] The function examples use local to create local variables, which ksh doesn't support. Omit local to make do with shell-global variables instead, or use function <name> {...} syntax with typeset instead of local to declare local variables in ksh.
I intend to read the lines of a short .txt file and assign each line to variables containing the line number in the variable name.
File example.txt looks like this:
Line A
Line B
When I run the following code:
i=1
while read line; do
eval line$i="$line"
echo $line
((i=i+1))
done < example.txt
What I would expect during execution is:
Line A
Line B
and afterwards being able to call
$ echo $line1
Line A
$ echo $line2
Line B
However, the code above results in the error:
-bash: A: command not found
Any ideas for a fix?
Quote-removal happens twice with eval. Your double-quotes are getting removed before eval even runs. I'm not even going to directly answer that part, because there are better ways to do this:
readarray line < example.txt # bash 4
echo "${line[0]}"
Or, to do exactly what you were doing, with a different variable for each line:
i=1
while read line$((i++)); do
:
done < example.txt
Also check out printf -v varname "%s" value for a better / safer way to assign by reference.
Check out the bash-completion code if you want to see some complicated call-by-reference bash shenanigans.
Addressing your comment: if you want to process lines as they come in, but still save previous lines, I'd go with this construct:
lines=()
while read l;do
lines+=( "$l" )
echo "my line is $l"
done < "$infile"
This way you don't have to jump through any syntactic hoops to access the current line (vs. having to declare a reference-variable to line$i, or something.)
Bash arrays are really handy, because you can access a single element by value, or you can do "${#lines[#]}" to get the line count. Beware that unset lines[4] leaves a gap, rather than renumbering lines[5:infinity]. See the "arrays" section in the bash man page. To find the part of the manual that documents $# expansion, and other stuff, search in the manual for ##. The Parameter Expansion section is the first hit for that in the bash 4.3 man page.
eval line$i="$line" tells bash to evaluate the string "line1=Line A", which attempts to invoke a command named A with the environment variable "line1" set to the value of Line. You probably want to do eval "line$i='$line'"
I have a bash script that creates and executes an expect script by stitching together dozens of different files containing pieces of expect code. Those files contain environment variables that need to be expanded. Example:
expect.piece:
send "command\r"
sleep $timeout
send "command argument\r"
script.sh:
#let's try it like this
eval echo $(cat expect.piece)
#or maybe like this
eval "echo \"$(cat expect.piece)\""
output:
send command\r sleep 1 send command argument\r
send commandr
sleep 1
send command argumentr
Desired otput:
send "command\r"
sleep 1
send "command argument\r"
I need a solution without sed string substitution (there is a lot of environment variables) and without modifying original expect script files. I guess it could be done line by line, but is there a more elegant solution?
Default field separator in bash is a space so set input file separator as a new line like IFS=$(echo -e '\n') before executing eval echo $(cat expect.piece) .
Final script would be :
#Storing original field separator in variable OFS
OFS=$IFS
#Setting IFS as new line using echo -e. Unfortunately IFS="\n" does not work in bash
IFS=$(echo -e '\n')
eval echo $(cat expect.piece)
#Resetting the field separator as space
IFS=$OFS
There is one another way which you can put your expect code with -c flag in the shell script as shown below.
script.sh
#Calling the expect.piece file code here
expect -c expect.piece
You can make use of the optional command line values as ,
expect -c "set count 1" myscript.exp
where the variable count will be used in the expect script file myscript.exp.
You can directly give the whole code as
expect -c "
send \"command\r\"
sleep $timeout
send \"command argument\r\"
"
Notice the use of backslash to escape the double quotes wherever needed. Single quotes can also be used. But, if you use double quotes, then only shell substitution can happen.
Please refer here to know more about the -c flag in expect.
It's not clear from your question in what context you want this output. If it's okay to embed the Expect script as a here document, what you want is trivial.
#!/bin/sh
timeout=1
cat <<____HERE
send "command\r"
sleep $timeout
send "command argument\r"
____HERE
(Maybe you can even replace the cat with expect but I'm not familiar enough with Expect to make any recommendations.)
If you need to take the input from a file, and only have a limited set of variables you want expanded, you could do that with sed.
sed "s/\$timeout/$timeout/g" file
If you need a more general solution, you might want to swich to Perl:
perl -pe 's/\$(\w+)/$ENV{$1} || "\$$1" /ge' file
but this requires you to export (or otherwise expose to Perl) the shell environment variables you want exported. (This will just not substitute any undefined variables; change the value after || if you want to change that aspect of the behavior.)
I invented the solution for this problem, it is a kludge, but it works.
expect.piece:
sleep $timeout
send "foo bar\r"
send "$(date)\r"
script.sh:
timeout=1
eval echo $(sed " \
s/\\\/\\\\\\\/g; \
s/\"/\\\\\"/g; \
s/\"/\\\\\`/\"/\\\\\`/g; \
" "expect.piece" | tr '\n' '+') | tr '+' '\n'
output:
sleep 1
send "foo bar\r"
send "Wed Feb 18 03:19:24 2015\r"
First, we need to escape all the backslashes, backticks and quotes in the file, because they will be removed during the evaluation. Then, we need to replace all the newline characters with pluses, in order to make it in a single line. After that, we evaluate that line (the evaluation will "expand" all the environment variables and command substitutions) and replace pluses back to newlines.
How can I write data to a text file automatically by shell scripting in Linux?
I was able to open the file. However, I don't know how to write data to it.
The short answer:
echo "some data for the file" >> fileName
However, echo doesn't deal with end of line characters (EOFs) in an ideal way. So, if you're going to append more than one line, do it with printf:
printf "some data for the file\nAnd a new line" >> fileName
The >> and > operators are very useful for redirecting output of commands, they work with multiple other bash commands.
#!/bin/sh
FILE="/path/to/file"
/bin/cat <<EOM >$FILE
text1
text2 # This comment will be inside of the file.
The keyword EOM can be any text, but it must start the line and be alone.
EOM # This will be also inside of the file, see the space in front of EOM.
EOM # No comments and spaces around here, or it will not work.
text4
EOM
You can redirect the output of a command to a file:
$ cat file > copy_file
or append to it
$ cat file >> copy_file
If you want to write directly the command is echo 'text'
$ echo 'Hello World' > file
#!/bin/bash
cat > FILE.txt <<EOF
info code info
info code info
info code info
EOF
I know this is a damn old question, but as the OP is about scripting, and for the fact that google brought me here, opening file descriptors for reading and writing at the same time should also be mentioned.
#!/bin/bash
# Open file descriptor (fd) 3 for read/write on a text file.
exec 3<> poem.txt
# Let's print some text to fd 3
echo "Roses are red" >&3
echo "Violets are blue" >&3
echo "Poems are cute" >&3
echo "And so are you" >&3
# Close fd 3
exec 3>&-
Then cat the file on terminal
$ cat poem.txt
Roses are red
Violets are blue
Poems are cute
And so are you
This example causes file poem.txt to be open for reading and writing on file descriptor 3. It also shows that *nix boxes know more fd's then just stdin, stdout and stderr (fd 0,1,2). It actually holds a lot. Usually the max number of file descriptors the kernel can allocate can be found in /proc/sys/file-max or /proc/sys/fs/file-max but using any fd above 9 is dangerous as it could conflict with fd's used by the shell internally. So don't bother and only use fd's 0-9. If you need more the 9 file descriptors in a bash script you should use a different language anyways :)
Anyhow, fd's can be used in a lot of interesting ways.
I like this answer:
cat > FILE.txt <<EOF
info code info
...
EOF
but would suggest cat >> FILE.txt << EOF if you want just add something to the end of the file without wiping out what is already exists
Like this:
cat >> FILE.txt <<EOF
info code info
...
EOF
Moving my comment as an answer, as requested by #lycono
If you need to do this with root privileges, do it this way:
sudo sh -c 'echo "some data for the file" >> fileName'
For environments where here documents are unavailable (Makefile, Dockerfile, etc) you can often use printf for a reasonably legible and efficient solution.
printf '%s\n' '#!/bin/sh' '# Second line' \
'# Third line' \
'# Conveniently mix single and double quotes, too' \
"# Generated $(date)" \
'# ^ the date command executes when the file is generated' \
'for file in *; do' \
' echo "Found $file"' \
'done' >outputfile
I thought there were a few perfectly fine answers, but no concise summary of all possibilities; thus:
The core principal behind most answers here is redirection. Two are important redirection operators for writing to files:
Redirecting Output:
echo 'text to completely overwrite contents of myfile' > myfile
Appending Redirected Output
echo 'text to add to end of myfile' >> myfile
Here Documents
Others mentioned, rather than from a fixed input source like echo 'text', you could also interactively write to files via a "Here Document", which are also detailed in the link to the bash manual above. Those answers, e.g.
cat > FILE.txt <<EOF` or `cat >> FILE.txt <<EOF
make use of the same redirection operators, but add another layer via "Here Documents". In the above syntax, you write to the FILE.txt via the output of cat. The writing only takes place after the interactive input is given some specific string, in this case 'EOF', but this could be any string, e.g.:
cat > FILE.txt <<'StopEverything'` or `cat >> FILE.txt <<'StopEverything'
would work just as well. Here Documents also look for various delimiters and other interesting parsing characters, so have a look at the docs for further info on that.
Here Strings
A bit convoluted, and more of an exercise in understanding both redirection and Here Documents syntax, but you could combine Here Document style syntax with standard redirect operators to become a Here String:
Redirecting Output of cat Input
cat > myfile <<<'text to completely overwrite contents of myfile'
Appending Redirected Output of cat Input
cat >> myfile <<<'text to completely overwrite contents of myfile'
This approach works and is the best
cat > (filename) <<EOF
Text1...
Text2...
EOF
Basically the text will search for keyword "EOF" till it terminates writing/appending the file
If you are using variables, you can use
first_var="Hello"
second_var="How are you"
If you want to concat both string and write it to file, then use below
echo "${first_var} - ${second_var}" > ./file_name.txt
Your file_name.txt content will be "Hello - How are you"
Can also use here document and vi, the below script generates a FILE.txt with 3 lines and variable interpolation
VAR=Test
vi FILE.txt <<EOFXX
i
#This is my var in text file
var = $VAR
#Thats end of text file
^[
ZZ
EOFXX
Then file will have 3 lines as below. "i" is to start vi insert mode and similarly to close the file with Esc and ZZ.
#This is my var in text file
var = Test
#Thats end of text file