Bad Substitution error with pdfgrep as variable? - bash

I'm using a bash script to parse information from a PDF and use it to rename the file (with the help of pdfgrep). However, after some working, I'm receiving a "Bad Substitution" error with line 5. Any ideas on how to reformat it?
shopt -s nullglob nocaseglob
for f in *.pdf; do
id1=$(pdfgrep -i "ID #: " "$f" | grep -oE "[M][0-9][0-9]+")
id2=$(pdfgrep -i "Second ID: " "$f" | grep -oE "[V][0-9][0-9]+")
$({ read dobmonth; read dobday; read dobyear; } < (pdfgrep -i "Date Of Birth: " "$f" | grep -oE "[0-9]+"))
# Check id1 is found, else do nothing
if [ ${#id1} ]; then
mv "$f" "${id1}_${id2}_${printf '%02d-%02d-%04d\n' "$dobmonth" "$dobday" "$dobyear"}.pdf"
fi
done

There are several unrelated bugs in this code; a corrected version might look like the following:
#!/usr/bin/env bash
shopt -s nullglob nocaseglob
for f in *.pdf; do
id1=$(pdfgrep -i "ID #: " "$f" | grep -oE "[M][0-9][0-9]+") || continue
id2=$(pdfgrep -i "Second ID: " "$f" | grep -oE "[V][0-9][0-9]+") || continue
{ read dobmonth; read dobday; read dobyear; } < <(pdfgrep -i "Date Of Birth: " "$f" | grep -oE "[0-9]+")
printf -v date '%02d-%02d-%04d' "$dobmonth" "$dobday" "$dobyear"
mv -- "$f" "${id1}_${id2}_${date}.pdf"
done
< (...) isn't meaningful bash syntax. If you want to redirect from a process substitution, you should use the redirection syntax < and the process substitution <(...) separately.
$(...) generates a subshell -- a separate process with its own memory, such that variables assigned in that subprocess aren't exposed to the larger shell as a whole. Consequently, if you want the contents you set with read to be visible, you can't have them be in a subshell.
${printf ...} isn't meaningful syntax. Perhaps you wanted a command substitution? That would be $(printf ...), not ${printf ...}. However, it's more efficient to use printf -v varname 'fmt' ..., which avoids the overhead of forking off a subshell altogether.
Because we put the || continues on the id1=$(... | grep ...) command, we no longer need to test whether id1 is nonempty: The continue will trigger and cause the shell to continue to the next file should the grep fail.

Do what Charles suggests wrt creating the new file name but you might consider a different approach to parsing the PDF file to reduce how many pdfregs and pipes and greps you're doing on each file. I don't have pdfgrep on my system, nor do I know what your input file looks like but if we use this input file:
$ cat file
foo
ID #: M13
foo
Date Of Birth: 05 21 1996
foo
Second ID: V27
foo
and grep -E in place of pdfgrep then here's how I'd get the info from the input file by just reading it once with pdfgrep and parsing that output with awk instead of reading it multiple times with pdfgrep and using multiple pipes and greps to extract the info you need:
$ grep -E -i '(ID #|Second ID|Date Of Birth): ' file |
awk -F': +' '{f[$1]=$2} END{print f["ID #"], f["Second ID"], f["Date Of Birth"]}'
M13 V27 05 21 1996
Given that you can use the same read approach to save the output in variables (or an array). You obviously may need to massage the awk command depending on what your pdfgrep output actually looks like.

Related

mv: Cannot stat - No such file or directory

I have piped the output of ls command into a file. The contents are like so:
[Chihiro]_Grisaia_no_Kajitsu_-_01_[1920x816_Blu-ray_FLAC][D2B961D6].mkv
[Chihiro]_Grisaia_no_Kajitsu_-_02_[1920x816_Blu-ray_FLAC][38F88A81].mkv
[Chihiro]_Grisaia_no_Kajitsu_-_03_[1920x816_Blu-ray_FLAC][410F74F7].mkv
My attempt to rename these episodes according to episode number is as follows:
cat grisaia | while read line;
#get the episode number
do EP=$(echo $line | egrep -o "_([0-9]{2})_" | cut -d "_" -f2)
if [[ $EP ]]
#escape special characters
then line=$(echo $line | sed 's/\[/\\[/g' | sed 's/\]/\\]/g')
mv "$line" "Grisaia_no_Kajitsu_${EP}.mkv"
fi
done
The mv commands exit with code 1 with the following error:
mv: cannot stat
'\[Chihiro\]_Grisaia_no_Kajitsu_-01\[1920x816_Blu-ray_FLAC\]\[D2B961D6\].mkv':
No such file or directory
What I really don't get is that if I copy the file that could not be stat and attempt to stat the file, it works. I can even take the exact same string that is output and execute the mv command individually.
If you surround your variable ($line) with double quotes (") you don't need to escape those special characters. So you have two options there:
Remove the following assignation completely:
then # line=$(echo $line | sed 's/\[/\\[/g' | sed 's/\]/\\]/g')`
or
Remove the double quotes in the following line:
mv $line "Grisaia_no_Kajitsu_${EP}.mkv"
Further considerations
Parsing the output of ls is never a good idea. Think about filenames with spaces. See this document for more information.
The cat here is unnecessary:
cat grisaia | while read line;
...
done
Use this instead to avoid an unnecessary pipe:
while read line;
...
done < grisaia
Why is good to avoid pipes in some scenarios? (answering comment)
Pipes create subshells (which are expensive), and you can also make some mistakes as the following:
last=""
cat grisaia | while read line; do
last=$line
done
echo $last # surprise!! it outputs an empty string
The reason is that $last inside the loop belongs to another subshell.
Now, see the same approach wothout pipes:
while read line; do
last=$line
done < grisaia
echo $last # it works as expected and prints the last line

Terminate tail command after timeout

I'm capturing stdout (log) in a file using tail -f file_name to save a specific string with grep and sed (to exit the tail) :
tail -f log.txt | sed /'INFO'/q | grep 'INFO' > info_file.txt
This works fine, but I want to terminate the command in case it does not find the pattern (INFO) in the log file after some time
I want something like this (which does not work) to exit the script after a timeout (60sec):
tail -f log.txt | sed /'INFO'/q | grep 'INFO' | read -t 60
Any suggestions?
This seems to work for me...
read -t 60 < <(tail -f log.txt | sed /'INFO'/q | grep 'INFO')
Since you only want to capture one line:
#!/bin/bash
IFS= read -r -t 60 line < <(tail -f log.txt | awk '/INFO/ { print; exit; }')
printf '%s\n' "$line" >info_file.txt
For a more general case, where you want to capture more than one line, the following uses no external commands other than tail:
#!/usr/bin/env bash
end_time=$(( SECONDS + 60 ))
while (( SECONDS < end_time )); do
IFS= read -t 1 -r line && [[ $line = *INFO* ]] && printf '%s\n' "$line"
done < <(tail -f log.txt)
A few notes:
SECONDS is a built-in variable in bash which, when read, will retrieve the time in seconds since the shell was started. (It loses this behavior after being the target of any assignment -- avoiding such mishaps is part of why POSIX variable-naming conventions reserving names with lowercase characters for application use are valuable).
(( )) creates an arithmetic context; all content within is treated as integer math.
<( ) is a process substitution; it evaluates to the name of a file-like object (named pipe, /dev/fd reference, or similar) which, when read from, will contain output from the command contained therein. See BashFAQ #24 for a discussion of why this is more suitable than piping to read.
The timeout command, (part of the Debian/Ubuntu "coreutils" package), seems suitable:
timeout 1m tail -f log.txt | grep 'INFO'

Bash: Filter directory when piping from `ls` to `tee`

(background info)
Writing my first bash psuedo-program. The program downloads a bunch of files from the network, stores them in a sub-directory called ./network-files/, then removes all the files it downloaded. It also logs the result to several log files in ./logs/.
I want to log the filenames of each file deleted.
Currently, I'm doing this:
echo -e "$(date -u) >>> Removing files: $(ls -1 "$base_directory"/network-files/* | tr '\n' ' ')" | tee -a $network_files_log $verbose_log $network_log
($base_directory is a variable defining the base directory for the app, $network_files_log etc are variables defining the location of various log files)
This produces some pretty grody and unreadable output:
Tue Jun 21 04:55:46 UTC 2016 >>> Removing files: /home/vagrant/load-simulator/network-files/207822218.png /home/vagrant/load-simulator/network-files/217311040.png /home/vagrant/load-simulator/network-files/442119100.png /home/vagrant/load-simulator/network-files/464324101.png /home/vagrant/load-simulator/network-files/525787337.png /home/vagrant/load-simulator/network-files/581100197.png /home/vagrant/load-simulator/network-files/640387393.png /home/vagrant/load-simulator/network-files/650797708.png /home/vagrant/load-simulator/network-files/827538696.png /home/vagrant/load-simulator/network-files/833069509.png /home/vagrant/load-simulator/network-files/8580204.png /home/vagrant/load-simulator/network-files/858174053.png /home/vagrant/load-simulator/network-files/998266826.png
Any good way to strip out the /home/vagrant/load-simulator/network-files/ part from each of those file paths? I suspect there's something I should be doing with sed or grep, but haven't had any luck so far.
You might also consider using find. Its perfect for walking directories, removing files and using customized printf for output:
find $PWD/x -type f -printf "%f\n" -delete >>$YourLogFile.log
Don't use ls at all; use a glob to populate an array with the desired files. You can then use parameter expansion to shorten each array element.
d=$base_directory/network-files
files=( "$d"/* )
printf '%s Removing files: %s' "$(date -u)" "${files[*]#$d/}" | tee ...
You could do it a couple of ways. To directly answer the question, you could use sed to do it with the substitution command like:
echo -e "$(date -u) >>> Removing files: $(ls -1 "$base_directory"/network-files/* | tr '\n' ' ')" | sed -e "s,$base_directory/network-files/,," | tee -a $network_files_log $verbose_log $network_log
which adds sed -e "s,$base_directory/network-files/,," to the pipeline. It will substitute the string found in base_directory with the empty string, so long as there are not any commas in base_directory. If there are you could try a different separator for the parts of the sed command, like underscore: sed -e "s_$base_directory/network-files__"
Instead though, you could just have the subshell cd to that directory and then the string wouldn't be there in the first place:
echo -e "$(date -u) >>> Removing files: $(cd "$base_directory/network-files/"; ls -1 | tr '\n' ' ')" | tee -a "$network_files_log" "$verbose_log" "$network_log"
Or you could avoid some potential pitfalls with echo and use printf like
{ printf '%s >>>Removing files: '; printf '%s ' "$(cd "$base_directory/network-files"; ls -1)"; printf '\n'; } | tee -a ...
testdata="/home/vagrant/load-simulator/network-files/207822218.png /home/vagrant/load-simulator/network-files/217311040.png"
echo -e $testdata | sed -e 's/\/[^ ]*\///g'
Pipe your output to sed the replace that captured group with nothing.
The regex: \/[^ ]*\/
Start with a /, captured everything that is not a space until it gets to the last /.

Create files using strings which delimited by specific character in BASH

Suppose we have the following command and its related output :
gsettings list-recursively org.gnome.Terminal.ProfilesList | head -n 1 | grep -oP '(?<=\[).*?(?=\])'
Output :
'b1dcc9dd-5262-4d8d-a863-c897e6d979b9', 'ca4b733c-53f2-4a7e-8a47-dce8de182546', '802e8bb8-1b78-4e1b-b97a-538d7e2f9c63', '892cd84f-9718-46ef-be06-eeda0a0550b1', '6a7d836f-b2e8-4a1e-87c9-e64e9692c8a8', '2b9e8848-0b4a-44c7-98c7-3a7e880e9b45', 'b23a4a62-3e25-40ae-844f-00fb1fc244d9'
I need to use gsettings command in a script and create filenames regarding to output ot gessetings command. For example a file name should be
b1dcc9dd-5262-4d8d-a863-c897e6d979b9
the next one :
ca4b733c-53f2-4a7e-8a47-dce8de182546
and so on.
How I can do this?
Another solution... just pipe the output of your command to:
your_command | sed "s/[ ']//g" | xargs -d, touch
You can use process substitution to read your gsettings output and store it in an array :
IFS=', ' read -r -a array < <(gsettings)
for f in "${array[#]}"
do
file=$(echo $f |tr -d "'" ) # removes leading and trailing quotes
touch "$file"
done

Parsing timestamp using sed and embedded command

There's a file with some lines containing some text and either date or time stamp:
...
string1-20141001
string2-1414368000000
string3-1414454400000
...
I want to quickly convert time stamps to dates, like this:
$ date -d #1414368000 +"%Y%m%d"
20141027
and I want to do this dynamically with sed or some similar command line tool. For testing I unsuccessfully use this:
$ echo "something-1414454400000" | sed "s/-\(..........\)...$/-$(date -d #\\1 +'%Y%m%d')/"
date: invalid date '#\\1'
something-
but echoing seems to be working:
$ echo "something-1414454400000" | sed "s/-\(..........\)...$/-$(echo \\1)/"
something-1414454400
so what could be done?
It's interesting what's happening here. Some pointers:
Always single-quote your regex for sed, if possible, when using BASH (etc), especially if using special characters like$. This is why date is being run (with -d #\\1) before sed even gets involved.
Your "working" echo example isn't, actually (I believe): echo \\1 produces \1 (and as above, will do so before sed even gets invoked). This then happens to valid sed replacement syntax, so will substitute your group on the LHS, which is why the output looks about right.
Note that by using -r, you can use easier / more advanced regex syntax.
Hard to say exactly what to do without a bit more context, but to fix the immediate problems, try something like:
echo "something-1414454400000" | sed -re 's/-([0-9]{10,}).+/-$(date -d #\1 +"%Y%m%d")/'
which produces: $(date -d #1414454400) (which you can then pipe to sh)
Or for a more complete solution, you can change the regex to produce a shell command directly, and pipe it:
echo "something-1414454400000" | sed -re 's/(.*-)([0-9]{10,10}).+/echo \1$(date -d #\2 \"+%Y%M%d\")/' | sh
..producing something-20140028
You can do this in BASH:
while read -r p; do
if [[ "$p" =~ ^(.+-)([0-9]{10}).{3}$ ]]; then
echo -n "${BASH_REMATCH[1]}"
date -d "#${BASH_REMATCH[2]}" +"%Y%m%d"
else
echo "$p"
fi
done < file
OUTPUT:
string1-20141001
string2-20141026
string3-20141027
awk -F- 'BEGIN { OFS=FS }
$2 ~ /^[0-9]{13}$/ {
"date -d#" $2/1000 " +%Y%m%d " | getline t; $2=t }1'
Just try this command. I have checked it. It is working on your inputs.
cat file | sed -E "s,(.*)-(.*),\1-`date -d #1414368000 +'%Y%m%d'`,g"

Resources