I'm trying to translate a set of .tif images to one multiband .tif following this tutorial. I'm not getting any errors but every time I run the script it generates a three-band image when I have a lot more input .tifs than that. I have tried the -r flag and it didn't change anything.
I have:
wdir="/Users/<mydir>"
creoleLA="$wdir/data/s2l2a_tiffs/T15RVP"
dhaka="$wdir/data/s2l2a_tiffs/T45QZG"
easthouston="$wdir/data/s2l2a_tiffs/T15RTN"
kolkata="$wdir/data/s2l2a_tiffs/T45QXF"
murcia="$wdir/data/s2l2a_tiffs/T30SXH"
wuhan="$wdir/data/s2l2a_tiffs/T50RKU"
#=============================================
# Main script
#=============================================
read -p 'ROI: ' roi
indir=${!roi}
outdir="$wdir/data/s2l2a_tiffs"
ls ${indir}*"_B"*.tif > "$outdir/btif_${roi}.txt"
gdalbuildvrt -separate -overwrite -input_file_list "$outdir/btif_${roi}.txt" "$outdir/S2L2A_${roi}.vrt"
gdal_translate -strict -ot uint16 "$outdir/S2L2A_${roi}.vrt" "$outdir/S2L2A_${roi}_mb.tif"
rm "$outdir/S2L2A_${roi}.vrt"
rm "$outdir/btif_${roi}.txt"
Where "$outdir/btif_${roi}.txt" is a textile of the GeoTIFF file paths like this:
/Users/<mydir>/data/s2l2a_tiffs/T15RVP_20200831T164849_B01_60m.tif
/Users/<mydir>/data/s2l2a_tiffs/T15RVP_20200831T164849_B02_60m.tif
...
I am processing Sentinel-2 imagery and using OSX 11.6.
Related
I have a large image that I am using imagemagick to convert into tiles for use in a Google Earth KML as explained here
instructions on image pyramid construction
The idea is to chop up the images into 4 pieces, then 16, then 64, etc.
To keep things simple, I made the image canvas 4096x4096 so that dividing it in will produce equal size files. The basic command is very simple. For example:
convert large.png -crop 512x512 tiles.png
The issue is the convert command creates file names sequentially, while google needs a format of row column. For instance if there were four files output, the file names should be:
tiles00.png
tiles01.png
tiles10.png
tiles11.png
I brute forced renaming scripts for up to 64 files, but before doing the 256 file case, I'd like to know if there is a simpler way to generate the file names. I'm using linux.
Here is one way in Imagemagick 6 using for loops.
lena.png
The lena.png image is 256x256. I choose 128x128 size tiles. So there will be a total of 2 rows and 2 columns for four output images.
infile="lena.png"
tx=128
ty=128
ncols=`convert -ping "$infile" -format "%[fx:floor(w/$tx)]" info:`
nrows=`convert -ping "$infile" -format "%[fx:floor(h/$ty)]" info:`
for ((j=0; j<nrows; j++)); do
offy=$((j*ty))
for ((i=0; i<ncols; i++)); do
offx=$((i*tx))
convert lena.png -crop ${tx}x${ty}+${offx}+${offy} +repage lena_tile${j}${i}.png
done
done
lena_tile00
lena_tile01
lena_tile10
lena_tile11
An alternate, more compact way is to use -set filename command with fx calculations to name the files in the image chain.
infile="lena.png"
tx=128
ty=128
ncols=`convert -ping "$infile" -format "%[fx:floor(w/$tx)]" info:`
nrows=`convert -ping "$infile" -format "%[fx:floor(h/$ty)]" info:`
convert "$infile" -crop ${tx}x${ty} -set filename:row_col "%[fx:floor(t/$nrows)]%[fx:mod(t,$ncols)]" "lena_tile%[filename:row_col].png"
See:
https://imagemagick.org/Usage/basics/#set
https://imagemagick.org/script/fx.php
Are you trying to make your own in order to learn about the process? If not, existing tools like dzsave can build complete pyramids for you very quickly in a single command. For example:
$ vipsheader wtc.jpg
wtc.jpg: 10000x10000 uchar, 3 bands, srgb, jpegload
$ /usr/bin/time -f %M:%e vips dzsave wtc.jpg x --layout google
211224:1.52
$ ls -R x | wc
2404 2316 15186
So that's making a google-style pyramid of 2400 tiles in directory x from a 10,000 x 10,000 pixel JPG image. It takes about 1.5s and 210mb of ram.
There's a chapter in the manual introducing dzsave:
http://libvips.github.io/libvips/API/current/Making-image-pyramids.md.html
I know that image resizing on the command line is something ImageMagick and similar could do unfortunately I do only have very basic bash scripting abilities so I wonder if this is even possible:
check all directories and subdirectories for all files that are an image
check width and height of the image
if any of both exceeds X amount of pixels resize it to X while keeping aspect ratio.
replace old file with new file (old file shall be removed/deleted)
Thank you for any input.
Implementation might be not so trivial even for advanced users. As a one-liner:
find \ # 1
~/Downloads \ # 2
-type f \ # 3
-exec file \{\} \; \ # 4
| awk -F: '{if ($2 ~/image/) print $1}' \ # 5
| while IFS= read -r file_path; do \ # 6
mogrify -resize 1024x1024\> "$file_path"; \ # 7
done # 8
Lines 1-4 are an invocation of the find command:
Specify a directory to scan.
Specify you need files only.
Per each found item run file command. Example outputs per file:
/Downloads/391A6 625.png: PNG image data, 1024 x 810, 8-bit/color RGB, interlaced
/Downloads/STRUCTURED NODES IN UML 2.0 ACTIVITES.pdf: PDF document, version 1.4
Note how file names are delimited from their info by : and info about PNG contains image word. This also will be true for other image formats.
Use awk to filter only those files which have image word in their info. This gives us image files only. Here, -F: specifies that the delimiter is :. This gives us the variable $1 to contain the original file name and $2 for the file info. We search image word in file info and print file name if it's present.
This one is a bit tricky. Lines 6-8 read the output of awk line by line and invoke the mogrify command to resize images. Here we do not use piping and xargs, as if file paths contain spaces or other characters which must be escaped,
we will get xargs unterminated quote errors and it's a pain to handle that.
Invoke the mogrify command of ImageMagic. Unlike convert, which is also ImageMagic's command, mogrify changes files in-place without creating new ones. Here, 1024x1024\> tells to resize image to have max size of 1024x1024. The \> part tells to preserve aspect ratio, so that the final image will have the biggest side of 1024px. Other side will be smaller than that, unless the original image is square. Pay attention to the ;, as it's needed inside loops.
Note, it's safe to run mogrify several times over the same file: if a file's size already corresponds to your target dimensions, it will not be resized again. However, it will change file's modification time, though.
Additionally, you may need not only to resize images, but to compress them as well. Please, refer to my gist to see how this can be done: https://gist.github.com/oblalex/79fa3f85f05924017d25004496493adb
If your goal is just to reduce big images in size, e.g. bigger than 300K, you may:
find /path/to/dir -type f -size +300k
and as before combine it with mogrify -strip -interlace Plane -format jpg -quality 85 -define jpeg:extent=300KB "$FILE_PATH"
In such case new jpg files will be created for non-jpg originals and originals will need to be removed. Refer to the gist to see how this can be done.
You can do that with a bash unix shell script looping over your directories. You must identify all the file formats you want such as jpg and png, etc. Then for each directory, loop over each file of the given list of formats. Then use ImageMagick to resize the files.
cd
dirlist="path2/directory1 path2/directory2 ...."
for dir in $dirlist; do
cd "$dir"
imglist=`ls | grep -i ".jpg\|.png"`
for img in $imglist; do
convert $img -resize "200x200>" $img
done
done
See https://www.imagemagick.org/script/command-line-processing.php#geometry
How can I pipe an image into exiv2 or imagemagick, strip the EXIF tag, and pipe it out to stdout for more manipulation?
I'm hoping for something like:
exiv2 rm - - | md5sum
which would output an image supplied via stdin and calcualte its md5sum.
Alternatively, is there a faster way to do this?
Using exiv2
I was not able to find a way to get exiv2 to output to stdout -- it only wants to overwrite the existing file. You could use a small bash script to make a temporary file and get the md5 hash of that.
image.sh:
#!/bin/bash
cat <&0 > tmp.jpg # Take input on stdin and dump it to temp file.
exiv2 rm tmp.jpg # Remove EXIF tags in place.
md5sum tmp.jpg # md5 hash of stripped file.
rm tmp.jpg # Remove temp file.
You would use it like this:
cat image.jpg | image.sh
Using ImageMagick
You can do this using ImageMagick instead by using the convert command:
cat image.jpg | convert -strip - - | md5sum
Caveat:
I found that stripping an image of EXIF tags using convert resulted in a smaller file-size than using exiv2. I don't know why this is and what exactly is done differently by these two commands.
From man exiv2:
rm Delete image metadata from the files.
From man convert:
-strip strip image of all profiles and comments
Using exiftool
ExifTool by Phil Harvey
You could use exiftool (I got the idea from https://stackoverflow.com/a/2654314/3565972):
cat image.jpg | exiftool -all= - -out - | md5sum
This too, for some reason, produces a slightly different image size from the other two.
Conclusion
Needless to say, all three methods (exiv2, convert, exiftool) produce outputs with different md5 hashes. Not sure why this is. But perhaps if you pick a method and stick to it, it will be consistent enough for your needs.
I tested with NEF file. Seems only
exiv2 rm
works best. exiftool and convert can't remove all metadata from .nef FILE.
Notice that the output file of exiv2 rm can no longer be displayed by most image viewers. But I only need the MD5 hash keeps same after I update any metadata of the .NEF file. It works perfect for me.
I have a bunch of png files named as 1.png, 2.png, etc. and I want to create an animated gif image from them all. I haven't been successful in finding a solution for a terminal command that will convert these png files into a single animated gif.
Can someone post some commands that I can try? I have tried "convert" commands but my terminal always says convert is not found even though I have installed ImageMagik.
convert *.png screens.gif
This answer suggested installing convert with brew install ImageMagick.
ImageMagick's convert command works perfectly for this but you'll want to list the filenames in the correct order. Using *.png will jumble frames if the digits don't have the leading zeros because the ordering is alphabetical:
1.png 10.png 11.png 2.png 3.png ...
If you use zsh you can simply use a glob qualifier:
convert *.png(n) out.gif
Otherwise you can sort the ls output
convert $(ls *.png | sort -V) out.gif
If your filenames have leading zeros go ahead and use *.png. Note that the default delay between frames is small, so depending on your use case the frame rate might be too quick. To change that use the -delay option, for example:
convert -delay 50 *.png out.gif
This will set FPS to 100/50 = 2 frames per second.
I'm working on a shell script to convert MPO stereographic 3D images into standard JPEG images. A MPO file is just two JPEG images, concatenated together.
As such, you can split out the JPEG files by finding the byte offset of the second JPEG's magic number header (0xFFD8FFE1). I've done this manually using hexdump/xxd, grep, head, and tail.
The problem here is grep: what can I use to search a binary directly for a specific magic number, and get back a byte offset? Or should I not use a shell script for this at all? Thanks.
You can do this using bbe (http://bbe-.sourceforge.net/) which is a sed like program for binary files:
In order to extract the first JPEG use:
bbe -b '/\xFF\xD8\xFF\xE1/:' -e 'D 2' -o first_jpeg mpo_file
And for the second one:
bbe -b '/\xFF\xD8\xFF\xE1/:' -e 'D 1' -o second_jpeg mpo_file
Note that this will not work if the JPEG's magic number occurs somewhere else in the MPO file.
I think that Bart is on to your biggest problem.. If that binary sequence repeats during the process, you will get partial JPEGs.
I did a quick test by concatenating some JPEGs and then extracting them with awk (please note that the magic number in my files ended in 0xE0 and not 0xE1):
# for i in *.jpg ; do cat $i ; done > test.mpo
# awk 'BEGIN {RS="\xFF\xD8\xFF\xE0"; FILENUM=-1} {FILENUM++; if (FILENUM == 0) {next}; FILENAME="image0"FILENUM".jpg"; printf "%s",RS$0 > FILENAME;}' test.mpo
# file image0*.jpg
image01.jpg: JPEG image data, JFIF standard 1.01
image010.jpg: JPEG image data, JFIF standard 1.01
image011.jpg: JPEG image data, JFIF standard 1.01
This seemed to work ok for me, but the above mentioned issues are still unhandled and very real.
I've found a much better explanation of MPO file structure (and how to process it correctly) at http://www.davidglover.org/2010/09/using-the-fuji-finepix-real-3d-w3-camera-on-a-mac-or-linuxunix.html
Edit, October 2019:
Since the blog entry now 404s, here is the script that I wrote based on it. I haven't used it in many years.
#!/usr/bin/env bash
# Script to convert 3D MPO files, as used in the Fuji FinePix series of 3D cameras, into standard JPEG files.
# Based on work by David Glover, posted at http://www.davidglover.org/2010/09/using-the-fuji-finepix-real-3d-w3-camera-on-a-mac-or-linuxunix.html
# This script requires exiftool and ImageMagick.
FULLNAME="$1"
FILENAME="$(basename $FULLNAME)"
DIRNAME="$(dirname $FULLNAME)"
BASENAME="${FILENAME%.*}"
# Create output directories
mkdir -p "$DIRNAME"/stereoscopic-rl/
mkdir -p "$DIRNAME"/stereoscopic-mpo/
mkdir -p "$DIRNAME"/stereoscopic-anaglyph/
mkdir -p "$DIRNAME"/monoscopic-l/
mkdir -p "$DIRNAME"/monoscopic-r/
# Create separate left and right images
exiftool -trailer:all= "$FULLNAME" -o "$DIRNAME"/monoscopic-l/"$BASENAME"-left.jpg
exiftool "$FULLNAME" -mpimage2 -b > "$DIRNAME"/monoscopic-r/"$BASENAME"-right.jpg
# Move the MPO file to its new home
mv "$FULLNAME" "$DIRNAME"/stereoscopic-mpo/
# Determine parallax value and create cropped images for stereo generation
# 36 is only appropriate for 4:3 or 3:2 images
parallax=$(exiftool -b -Parallax "$DIRNAME"/monoscopic-r/"$BASENAME"-right.jpg)
parallax=$(echo "$parallax"*36+0.5 | bc | cut -d . -f 1)
# The above pipeline can't deal with a parallax of zero
# In theory, this fix doesn't cover values between zero and -1
# TODO improve the calculation
if [ ! $parallax ]; then
parallax=0
fi
echo $parallax
if [ $parallax -ge 0 ]; then
convert "$DIRNAME"/monoscopic-l/"$BASENAME"-left.jpg -crop +"$parallax"+0 "$DIRNAME"/monoscopic-l/"$BASENAME"-left-cropped.jpg
convert "$DIRNAME"/monoscopic-r/"$BASENAME"-right.jpg -crop -"$parallax"+0 "$DIRNAME"/monoscopic-r/"$BASENAME"-right-cropped.jpg
else
convert "$DIRNAME"/monoscopic-l/"$BASENAME"-left.jpg -crop -"$((-1*$parallax))"+0 "$DIRNAME"/monoscopic-l/"$BASENAME"-left-cropped.jpg
convert "$DIRNAME"/monoscopic-r/"$BASENAME"-right.jpg -crop +"$((-1*$parallax))"+0 "$DIRNAME"/monoscopic-r/"$BASENAME"-right-cropped.jpg
fi
# Create stereoscopic images for cross-eye (right-left) and anaglyph (red-cyan) viewing
convert "$DIRNAME"/monoscopic-r/"$BASENAME"-right-cropped.jpg "$DIRNAME"/monoscopic-l/"$BASENAME"-left-cropped.jpg +append "$DIRNAME"/stereoscopic-rl/"$BASENAME"-stereoscopic-rl.jpg
composite -stereo 0 "$DIRNAME"/monoscopic-r/"$BASENAME"-right-cropped.jpg "$DIRNAME"/monoscopic-l/"$BASENAME"-left-cropped.jpg "$DIRNAME"/stereoscopic-anaglyph/"$BASENAME"-stereoscopic-anaglyph.jpg
# Clean up separated parallax-corrected images
rm "$DIRNAME"/monoscopic-l/"$BASENAME"-left-cropped.jpg
rm "$DIRNAME"/monoscopic-r/"$BASENAME"-right-cropped.jpg
exit 0
I think a very simple home brew approach will be your best bet. The code for doing this would be very small, depending on all the special cases of your binary file format.
Use mmap to get a convenient view of your file in memory.
Start scanning, and save the byte-offset in a variable, say start.
Scan until you reach your delimiter, saving the ending offset, in say end.
Create a new file
Memory-map the new file
Copy the byte-range from start to end into the new file.
Close the new file and start scanning again.
FFE1 is not part of the some jpeg "magic number", it's the APP1 marker. And it's not guaranteed to come right after the SOI marker FFD8. Also, you should be careful that some jpeg images embed a thumbnail jpeg in an EXIF block. That will most likely also contain an APP1 marker.