Is there a bash command like cat except that it takes the filename from stdin rather than the first argument? For example:
echo "/home/root/file.txt" | somecommand
would have the same output as cat /home/root/file.txt
Found the answer:
xargs cat
For example:
echo "/home/root/file.txt" | xargs cat
If the filename that you're expecting doesn't have a space:
#!/bin/bash
read filename
cat $filename
The read builtin shell command reads tokens from stdin into environment variables.
Related
script.sh takes two files as arguments, and it calls a python script that opens them and does further processing on these files:
script.sh file1.txt file2.txt
I would like to remove all lines from file1.txt that start with #.
grep -v '^#' file1.txt
I tried the following but it fails :
script.sh $(grep -v '^#' file1.txt) file2.txt
How can I pipe the output of grep as the first file argument of script.sh?
script.sh <(grep -v '^#' file1.txt) file2.txt
Command substitution as you had it, $(), will insert the output from the grep command verbatim as arguments to script.sh. In contrast, to quote the bash man page:
Process substitution allows a process's input or output to be referred to using a filename. It takes the form of <(list) or >(list). The process list is run asynchronously, and its input or output appears as a filename. This filename is passed as an argument to the current command as the result of the expansion.
and
If the <(list) form is used, the file passed as an argument should be read to obtain the output of list.
So, my understanding is that when you use $(), the output of your grep command is substituted into the final statement with a result something like:
script.sh contents of file1 without comments \n oops newline etc file2.txt
But when you use <(), the grep command is run and the output goes into a named pipe or temporary file or some equivalent, and the name of that "file" is substituted into your statement, so you end up with something along the lines of
script.sh /dev/fd/temp_thing_containing_grep_results file2.txt
In bash I have
grep -vf <(cat myfile <(grep -f myfile otherfile)) otherfile
Given the repetition of myfile, I thought I could pipe it via stdin like so
cat myfile | grep -vf <(cat - <(grep -f - otherfile)) otherfile
However this gives me different results.
So my question is where does the innermost substituted process i.e the grep -f - otherfile, get it's stdin from
A secondary question would be whether there is any advantage to trying to substitute the repeated file name with the same thing passed from stdin
Bash will fork a subshell for process substitution and it will inherit stdin from the current shell.
For your case, the whole right side of | is also running in a subshell so <()'s stdin is the same as this subshell. So <()'s stdin is also from cat myfile.
See the following simpler example:
[STEP 100] # echo $BASH_VERSION
5.0.7(3)-release
[STEP 101] # echo hello | cat <( tr a-z A-Z )
HELLO
[STEP 102] #
I have some strings like:
dimension\u003d1920x1024:format\u003djpg
In a file. I want to decode them so they will look like:
dimension=1920x1024:format=jpg
I know that:
$ echo -e dimension\u003d1920x1024:format\u003djpg
dimensionu003d1920x1024:formatu003djpg
$ echo -e 'dimension\u003d1920x1024:format\u003djpg'
dimension=1920x1024:format=jpg
$ echo -e "dimension\u003d1920x1024:format\u003djpg"
dimension=1920x1024:format=jpg
So I tried this to get what I want:
$ cat file | xargs -L1 echo -e
dimensionu003d1920x1024:formatu003djpg
But as you can see it doesn't work. How can I get this to work? How can I make xargs pass parameters to echo as if they were quoted?
You are actually asking how to convert the sequence \uXXXX into the corresponding Unicode code point. That's quite different from other backslash escapes, or handling backslashes in general. Neither echo -e nor xargs is particularly suited for this task.
Here is one way:
perl -CSD -pe 's/\\u(\X{4})/chr(oct("0x$1"))/ge' <<<"string"
Obscurely, oct("0xff") actually performs hex decoding, because of the "0x" prefix.
Obviously, if your input is the text in a file rather than just a string in the shell, simply pass that as the argument to Perl.
For small files:
Bash:
cat file | echo -e "$(cat -)"
Zsh:
cat file | { echo -e "$(cat -)"; }
For large files in both bash and zsh:
cat file | while read -r LINE; do echo -e "$LINE"; done
(loses spaces at the beginning of the line)
This is a try with ruby where the changes are written to the file
$ cat ./file
dimension\u003d1920x1024:format\u003djpg
dimension=800x600:format\u003djpg
The example above is made a bit more real-world.
$ cat ./script.rb
#!/usr/bin/ruby
contents=File.read("#{ARGV[0]}")
file=File.open("#{ARGV[0]}","w")
if file
file.syswrite(contents.gsub(/\\[uU]\{?([0-9A-F]{4})\}?/i) { $1.hex.chr(Encoding::UTF_8) })
file.close()
else
puts "No file with name #{ARGV[0]} present, Usage script <filename>"
end
$ ./script file
# The changes are written to the file with nothing printed to stdout
$ cat ./file
dimension=1920x1024:format=jpg
dimension=800x600:format=jpg
I want to write a script that gets in its argument a command and executes it while it's running. for example if the script called ex_script, writing
ex_script "cat file1.txt | wc -l"
and the ex_script is:
var=`"${1}"`
echo $var
will assign the number of lines in file1.txt in var and then print it.
But it gives me
./ex_script: line 3: cat file1.txt | wc -l: command not found
How do I write this correctly?
Use eval
var=$(eval "$1")
echo "$var"
I get stuck by this problem:
I wrote a shell script and it gets a large file with many lines from stdin, that's how it is executed:
./script < filename
I want use the file as an input to another operation in the script, however I don't know how to store this file's name in a variable.
It is a script that takes a file from stdin as argument and then do awk operation in this file it self. Say if I write in script:
script:
#!/bin/sh
...
read file
...
awk '...' < "$file"
...
it only reads first line of the input file.
And I find a way to write like this:
Min=-1
while read line; do
n=$(echo $line | awk -F$delim '{print NF}')
if [ $Min -eq -1 ] || [ $n -lt $Min ];then
Min=$n
fi
done
it would take very very long time to wait for processing, it seems awk takes much time.
So how to improve this?
/dev/stdin can be quite useful here.
In fact, it's just a chain of links to your input.
So, writing cat /dev/stdin will give you all input from your file and you can deny using input filename at all.
Now answer to question :) Recursively read links, beginning at /dev/stdin, and you will get filename. Bash code:
r(){
l=`readlink $1`
if [ $? -ne 0 ]
then
echo $1
else
r $l
fi
}
filename=`r /dev/stdin`
echo $filename
UPD:
in Ubuntu I found an option -f to readlink. i.e. readlink -f /dev/stdin gives the same output. This option may absent in some systems.
UPD2:tests (test.sh is code above):
$ ./test.sh <input # that is a file
/home/sfedorov/input
$ ./test.sh <<EOF
> line
> EOF
/tmp/sh-thd-214216298213
$ echo 1 | ./test.sh
pipe:[91219]
$ readlink -f /dev/stdin < input
/home/sfedorov/input
$ readlink -f /dev/stdin << EOF
> line
> EOF
/tmp/sh-thd-3423766239895 (deleted)
$ echo 1 | readlink -f /dev/stdin
/proc/18489/fd/pipe:[92382]
You're overdoing this. The way you invoke your script:
the file contents are the script's standard input
the script receives no argument
But awk already takes input from stdin by default, so all you need to do to make this work is:
not give awk any file name argument, it's going to be the wrapping shell's stdin automatically
not consume any of that input before the wrapping script reaches the awk part. Specifically: no read
If that's all there is to your script, it reduces to the awk invocation, so you might consider doing away with it altogether and just call awk directly. Or make your script directly an awk one instead of a sh one.
Aside: the reason your while read line/multiple awk variant (the one in the question) is slow is because it spawns an awk process for each and every line of the input, and process spawning is order of magnitudes slower than awk processing a single line. The reason why the generate tmpfile/single awk variant (the one in your answer) is still a bit slow is because it's generating the tmpfile line by line, reopening to append every time.
Modify your script to that it takes the input file name as an argument, then read from the file in your script:
$ ./script filename
In script:
filename=$1
awk '...' < "$filename"
If your script just reads from standard input, there is no guarantee that there is a named file providing the input; it could just as easily be reading from a pipe or a network socket.
How about invoking the script differently pipe standard output of YourFilename into
your scriptName as follows (the standard output of the cat filename now becomes standard
input to you script, actually in this case to the awk command
For I have filename Names.data and script showNames.sh execute as follows
cat Names.data | ./showNames.sh
Contents of filename Names.data
Huckleberry Finn
Jack Spratt
Humpty Dumpty
Contents of scrip;t showNames.sh
#!/bin/bash
#whatever awk commands you need
awk "{ print }"
Well I finally find this way to solve my problem, although it will take several seconds.
grep '.*' >> /tmp/tmpfile
Min=$(awk -F$delim 'NF < min || min == "" { min = NF };END {printmin}'</tmp/tmpfile)
Just append each line into a temporary file so that after reading from stdin, the tmpfile is the same as input file.