I need to convert about 12000 TIF files in many directories, and try to write bash-script:
#!/bin/bash
find -name "*.tif" | while read f
do
convert "$f" "${f%.*}.png"
rm -f "$f"
done
Why it say: x.sh: 6: Syntax error: end of file unexpected (expecting "do") and what I should to do?
Great thanks to you all, men, but I was cheated: the computer on which this should be run out works under Windows. I don't know how to work with strings and cycles in DOS, now my script look like:
FOR /R %i IN (*.tif) DO # (set x=%i:tif%png) & (gm convert %i %xtif) & (erase /q /f %i)
%i - one of the .tif files.
%x - filename with .png extension
gm convert - graphics magick utility, work similarly with image magick's convert on linux.
The syntax looks okay, but if it's a problem with EOLs, try adding a semicolon before the do to fix the syntax error (or check the newlines are actually present/encoded as ghostdog74 suggests):
find -name "*.tif" | while read f ; do # ...
Note that the find/read pattern isn't robust. Use can use find's exec capability directly (thanks Philipp for the inline command):
find -name "*.tif" -exec sh -c 'file=$0 && convert "$file" "${file%.tif}.png"' '{}' ';' -delete
Related
I am facing a problem with the following shell script:
#!/bin/bash
searchPattern=".*\/.*\.abc|.*\/.*\.xyz|.*\/.*\.[0-9]{3}"
subFolders=$(find -E * -type d -regex ".*201[0-4][0-1][0-9].*|.*20150[1-6].*" -maxdepth 0 | sed 's/.*/"&"/')
echo "subFolders: $subFolders"
# iterate through subfolders
for thisFolder in $subFolders
do
echo "The current subfolder is: $thisFolder"
find -E $thisFolder -type f -iregex $searchPattern -maxdepth 1 -print0 | xargs -0 7z a -mx=9 -uz1 -x!.DS_Store ${thisFolder}/${thisFolder}_data.7z
done
The idea behind it is to archive filetypes with the ending .abc, .xyz and .000-.999 in one 7z archive per subfolder. However, I can't manage to deal with folders including spaces. When I run the script as shown above I always get the following error:
find: "20130117_test": No such file or directory
If I run the script with the line
subFolders=$(find -E * -type d -regex ".*201[0-4][0-1][0-9].*|.*20150[1-6].*" -maxdepth 0 | sed 's/.*/"&"/')
changed to
subFolders=$(find -E * -type d -regex ".*201[0-4][0-1][0-9].*|.*20150[1-6].*" -maxdepth 0)
the script works like charm, but of course not for folders containing space.
Strangely enough, when I execute the following line directly in shell, it works as expected:
find -E "20130117_test" -type f -iregex ".*\/.*\.abc|.*\/.*\.xyz|.*\/.*\.[0-9]{3}" -maxdepth 1 -print0 | xargs -0 7z a -mx=9 -uz1 -x!.DS_Store "20130117_test"/"20130117_test"_data.7z
I know the issue is somehow related to the storing of a list of folders (in quotes) in the subFolders variable, but I simply cannot find a way to make it work properly.
I hope someone more advanced in shell can help me out here.
In general, you should not use find in an attempt to generate a list of file names. You especially cannot build a quoted list the way you are attempting; there is a difference between quotes in a parameter value and quotes around a parameter expansion. Here, especially, you can just use simple patterns:
shopt -s nullglob
subFolders=(
*201[0-4][0-1][0-9]*
*20150[1-6]*
)
for thisFolder in "${subFolders[#]}"; do
echo "The current subfolder is: $thisFolder"
to_archive=(
*/*.abc
*/*.xyz
*/*.[0-9][0-9][0-9]
)
7z a -mx9 -uz1 -x!.DS_Store "$thisFolder/$thisFolder_data.7z" "${to_archive[#]}"
done
Combining the input from gniourf_gniourf and chepner I was able to produce the following code, which does exactly what I want.
#!/bin/bash
shopt -s nullglob
find -E "$PWD" -type d -maxdepth 1 -regex ".*201[0-5][0-1][0-9].*" -print0 | while IFS="" read -r -d "" thisFolder ; do
echo "The current folder is: $thisFolder"
to_archive=( "$thisFolder"/*.[Aa][Bb][Cc] "$thisFolder"/*.[Xx][Yy][Zz] "$thisFolder"/*.[0-9][0-9][0-9] )
if [ ${#to_archive[#]} != 0 ]
then
7z a -mx=9 -uz1 -x!.DS_Store "$thisFolder"/"${thisFolder##*/}"_data.7z "${to_archive[#]}" && rm "${to_archive[#]}"
fi
done
shopt -s nullglob leads to ignorance towards non-matching characters
find... searches for directories matching the regex pattern and streams each matching folder to the while loop using the null separator.
inside the while loop I can safely quote the $thisFolder variable expansion and therefore deal with possible spaces.
using absolute paths instead of relative paths instructs 7z to create no folders inside the archive
I can convert an image like this:
convert -resize 50% foo.jpg foo_50.jpg
How can I automate such a command to convert all the images in a folder?
You can assume every image has .jpg extension.
A solution easily adaptable to automate the conversion of all the images inside the subdirectories of the working directory is preferable.
You can use a for loop with pattern expansion:
for img in */*.jpg ; do
convert -resize 50% "$img" "${img%.jpg}"_50.jpg
done
${variable%pattern} removes the pattern from the right side of the $variable.
You can use find -exec:
find -type f -name '*.jpg' -exec \
bash -c 'convert -resize 50% "$0" "${0%.jpg}"_50.jpg' {} \;
find -type f -name '*.jpg' finds all .jpg files (including those in subdirectories) and hands it to the command after -exec, where it can be referenced using {}.
Because we want to use parameter expansion, we can't use -exec convert -resize directly; we have to call bash -c and supply {} as a positional parameter to it ($0 inside the command). \; marks the end of the -exec command.
You can also try this (less elegant) one-liner using ls+awk:
ls *.jpg | awk -F '.' '{print "convert -resize 50% "$0" "$1"_50.jpg"}' | sh
this assumes that all the .jpg files are in the current directory. before running this, try to remove the | sh and see what is printed on the screen.
So,
I've been trying to convert a large java source tree from cp1252 to UTF-8 in Windows, using tips and trix I've found online, specificly here. Problem is, I'm on Windows; I don't do VB; Cygwin's iconv doesn't take the -o switch.
The line I first tried to use is:
find . -type f -print -exec iconv -f cp1252 -t utf-8 {} > {}.converted \; -exec mv {}.converted {} \;
This creates a file {}.converted in the working directory and the second -exec fails for obvious reasons.
Putting quotes around the iconv expression:
find . -type f -print -exec 'iconv -f cp1252 -t utf-8 {} > {}.converted' \; -exec mv {}.converted {} \;
resulsts in the folowing error:
find: `iconv -f cp1252 -t utf-8 ./java/dv/framework/activity/model/ActivitiesMediaViewImpl.java > ./java/dv/framework/activity/model/ActivitiesMediaViewImpl.java.converted': No such file or directory
though executing the individual expressions by hand works perfectly.
I've experimented with random quoting but nothing seems to work, what am I missing? Why won't it work..?
Thanx in advance,
Lars
for f in `find . -type f`; do
iconv -f cp1252 -t utf-8 $f > $f.converted
mv $f.converted $f
done
Allright, once again answering my own question (this is starting to become a bad habit...)
Allthough there is nothing wrong with Neevek's solution, the perfectionist in me wants to get the find -exec expression right. Wrapping the iconv statement in a sh -c '...' does the trick:
find . -type f -print -exec sh -c 'iconv -f cp1252 -t utf-8 {} > {}.converted' \; -exec mv {}.converted {} \;
Still, the underlying question of why there is a problem using i/o redirection in find -exec statements remains unresolved...
I haven't used Cygwin very much but there's a "native" windows version of Iconv that I use all the time. Here's an excerpt from a batch file that i use to convert all the files in a sub-dir from HP-ROMAN8 encoding to UTF-8 encoding -- putting the result './temp" under the originals:
#set dir=original
#set ICONV="C:\Program Files (x86)\iconv-1.9.2.win32\bin\iconv"
if EXIST .\%dir%\temp (
erase .\%dir%\temp*.* /Q
#if ERRORLEVEL 1 (#echo Unable to erase all files from the "temp" sub-directory
#goto THE_END
)
) else (
mkdir .\%dir%\temp
#if ERRORLEVEL 1 (#echo Unable to create the "temp" sub-directory
#goto THE_END
)
)
for %%f IN (./%dir%/*.xml) do (
%ICONV% -f HP-ROMAN8 -t UTF-8 "./%dir%/%%f" > "./%dir%/temp/%%f"
if ERRORLEVEL 1 (goto ICONV_ERROR)
)
The error in the first try is that the redirection operator '>' ist evaluated by the shell before find starts.
The error in the second try is that the text between the single quotes is interpreted as the name of a command that is to be executed by find, but that doesn't exist.
In your working solution the first command to be executed by find is a subshell, and the options are enclosed in single quotes, so they are not interpreted by the outer shell but by the subshell.
Ok, I got a directory (for instance, named '/photos') in which there are different directories
(like '/photos/wedding', '/photos/birthday', '/photos/graduation', etc...) which have .jpg files in them. Unfortunately, some of jpeg files are broken. I need to find a way how to determine, which files are broken.
I found out, that there is tool named imagemagic, which can help a lot. If you use it like this:
identify -format '%f' whatever.jpg
it prints the name of the file only if file is valid, if it is not it prints something like "identify: Not a JPEG file: starts with 0x69 0x75 `whatever.jpg' # jpeg.c/EmitMessage/232.".
So the correct solution should be find all files ending with ".jpg", apply to them "identify", and if the result is just the name of the file - don't do anything, and if the result is different from the name of the file - then save the name of the file somethere (like in a file "errors.txt").
Any ideas how I can probably do that?
The short-short version:
find . -iname "*.jpg" -exec jpeginfo -c {} \; | grep -E "WARNING|ERROR"
You might not need the same find options, but jpeginfo was the solution that worked for me:
find . -type f -iname "*.jpg" -o -iname "*.jpeg"| xargs jpeginfo -c | grep -E "WARNING|ERROR" | cut -d " " -f 1
as a script (as requested in this question)
#!/bin/sh
find . -type f \
\( -iname "*.jpg" \
-o -iname "*.jpeg" \) \
-exec jpeginfo -c {} \; | \
grep -E "WARNING|ERROR" | \
cut -d " " -f 1
I was clued into jpeginfo for this by http://www.commandlinefu.com/commands/view/2352/find-corrupted-jpeg-image-files and this explained mixing find -o OR with -exec
One problem with identify -format is that it doesn't actually verify that the file is not corrupt, it just makes sure that it's really a jpeg.
To actually test it you need something to convert it. But the convert that comes with ImageMagick seems to silently ignore non-fatal errors in the jpeg (such as being truncated.)
One thing that works is this:
djpeg -fast -grayscale -onepass file.jpg > /dev/null
If it returns an error code, the file has a problem. If not, it's good.
There are other programs that could be used as well.
You can put this into bash script file or run directly:
find -name "*.jpg" -type f |xargs --no-run-if-empty identify -format '%f' 1>ok.txt 2>errors.txt
In case identify is missing, here is how to install it in Ubuntu:
sudo apt install imagemagick --no-install-recommends
This script will print out the names of the bad files:
#!/bin/bash
find /photos -name '*.jpg' | while read FILE; do
if [[ $(identify -format '%f' "$FILE" 2>/dev/null) != $FILE ]]; then
echo "$FILE"
fi
done
You could run it as is or as ./badjpegs > errors.txt to save the output to a file.
To break it down, the find command finds *.jpg files in /photos or any of its subdirectories. These file names are piped to a while loop, which reads them in one at a time into the variable $FILE. Inside the loop, we grab the output of identify using the $(...) operator and check if it matches the file name. If not, the file is bad and we print the file name.
It may be possible to simplify this. Most UNIX commands indicate success or failure in their exit code. If the identify command does this, then you could simplify the script to:
#!/bin/bash
find /photos -name '*.jpg' | while read FILE; do
if ! identify "$FILE" &> /dev/null; then
echo "$FILE"
fi
done
Here the condition is simplified to if ! identify; then which means, "did identify fail?"
Is there any way of batch renaming files in sub directories?
For example:
Rename *.html to *.htm in a folder which has directories and sub directories.
Windows command prompt: (If inside a batch file, change %x to %%x)
for /r %x in (*.html) do ren "%x" *.htm
This also works for renaming the middle of the files
for /r %x in (website*.html) do ren "%x" site*.htm
find . -regex ".*html$" | while read line;
do
A=`basename ${line} | sed 's/html$/htm/g'`;
B=`dirname ${line}`;
mv ${line} "${B}/${A}";
done
In python
import os
target_dir = "."
for path, dirs, files in os.walk(target_dir):
for file in files:
filename, ext = os.path.splitext(file)
new_file = filename + ".htm"
if ext == '.html':
old_filepath = os.path.join(path, file)
new_filepath = os.path.join(path, new_file)
os.rename(old_filepath, new_filepath)
If you have forfiles (it comes with Windows XP and 2003 and newer stuff I think) you can run:
forfiles /S /M *.HTM /C "cmd /c ren #file *.HTML"
In Bash, you could do the following:
for x in $(find . -name \*.html); do
mv $x $(echo "$x" | sed 's/\.html$/.htm/')
done
In bash use command rename :)
rename 's/\.htm$/.html/' *.htm
# or
find . -name '*.txt' -print0 | xargs -0 rename 's/.txt$/.xml/'
#Obs1: Above I use regex \. --> literal '.' and $ --> end of line
#Obs2: Use find -maxdepht 'value' for determine how recursive is
#Obs3: Use -print0 to avoid 'names spaces asdfa' crash!
I'm sure there's a more elegant way, but here's the first thing that popped in my head:
for f in $(find . -type f -name '*.html'); do
mv $f $(echo "$f" | sed 's/html$/htm/')
done
On Linux, you may use the 'rename' command to rename files in batch.
AWK on Linux. For the first directory this is your answer... Extrapolate by recursively calling awk on dir_path perhaps by writing another awk which writes this exact awk below... and so on.
ls dir_path/. | awk -F"." '{print "mv file_name/"$0" dir_path/"$1".new_extension"}' |csh
On Unix, you can use rnm:
rnm -rs '/\.html$/.htm/' -fo -dp -1 *
Or
rnm -ns '/n/.htm' -ss '\.html$' -fo -dp -1 *
Explanation:
-ns : name string (new name). /n/ is a name string rule that expands to the filename without the extension.
-ss : search string (regex). Searches for files with match.
-rs : replace string of the form /search_regex/replace_part/modifier
-fo : file only mode
-dp : depth of directory (-1 means unlimited).
there is pretty powerfull forfiles command:
forfiles /? gives u hint of what is possible with the command.
in this case it can be used like:
forfiles /S /M *.html /C "cmd /c rename #file #fname.htm"