I am trying to open NASA's Clementine LIDAR data of the Lunar surface (Link: https://pds-geosciences.wustl.edu/missions/clementine/lidar.html). The data is saved as a *.tab file which is really an ASCII file. When I look for the data I need (x,y,z,r,g,b) I can only find data points x,y,z but not r,g,b.
Main Question If my goal is to open this file in CloudCompare and develop a mesh/dem file from this, do I need r,g,b data?
Side Questions If so, how do you recommend I get it from data that is from the 90s? Or atleast, how would I go about opening this.
As far as I know, there is no need to have R,G,B in CloudCompare, but you definitely will have to prepare the data in the format that is readable by CloudCompare. That shouldn't be a problem using simple Python scripts. If you further help, let me know.
You do not need the RGB data to open it on CloudCompare, just the point coordinates. There is no way of retrieving the radiometric information of this point cloud unless you have some imagery of the same epoch. Then you could associate the RGB to the point cloud by using collinearity equations, for instance.
Related
I started to work on a PNG encoding/decoding library for learning purposes so I want to implement every part of it by hand.
I got pretty long with it but now I'm a bit stuck. Here are the things I succesfully implemented already:
I can load a PNG binary and go through its bytes
I can read the signature and the IHDR chunk for metadata
I can read the IDAT chunks and concatenate the image data into a buffer
I can read and interpret the zlib headers from the above mentioned image data
And here is where I got stuck. I vaguely know the steps from here which are:
Extract the zlib compressed data according to its headers
Figure out the filtering methods used and "undo" them to get the raw data
If everything went correctly, now I have raw RGB data in the form of [<R of 1st pixel>, <G of 1st pixel>, <B of 1st pixel>, <R of 2nd pixel>, <G of 2nd pixel>, etc...]
My questions are:
Is there any easy-to-understand implementation (maybe with examples) or guide on the zlib extraction as I found the official specifications hard to understand
Can there be multiple filtering methods used in the same file? How to figure these out? How to figure out the "borders" of these differently filtered parts?
Is my understanding of the how the final data will look like correct? What about the alpha channel or when a palette is used?
Yes. You can look at puff.c, which is an inflate implementation written with the express purpose of being a guide to how to decode a deflate stream.
Each line of the image can use a different filter, which is specified in the first byte of the decompressed line.
Yes, if you get it all right, then you will have a sequence of pixels, where each pixel is a grayscale value, G, that with an alpha channel, GA, RGB (red-green-blue, in that order), or RGBA.
I am wondering how to find and replace certain values of data in H2O flow. For example, if a line of data is '0.003 8.938 0.005 9.999' I would like to find all of the '9.999' and replace with NaN, so sort of like sed?
Is there a way to do this directly in the Flow interface?
Thanks
At this point in time, Flow is primarily a modeling tool and not a data munging tool, so there is not currently a way to do this. Sed is an efficient tool for find/replace, so that's what I'd recommend. Once you edit the file, you should import the file into the H2O cluster for modeling. Or you can do more advanced munging on the data as an H2OFrame using the R or Python API for H2O.
I am trying to automate the cleanup process of a large amount of scanned films. I have all the images in 48-bit RGBI TIFF files (RGB + Infrared), and I can use the infrared channel to create masks for dust removal. I wonder if there is any decent open source implementation of in-painting that I can use to achieve this (all the other software I use for batch processing are open source libraries I access through Ruby interfaces).
My first choice was ImageMagick, but I couldn't find any advanced in-painting option in it (maybe I am wrong, though). I have heard this can be done with MagickWand libraries, but I haven't been able to find a concrete example yet.
I have also had a look at OpenCV, but it seems that OpenCV's in-paint method accept only 8-bit-per-channel images, while I must preserve the 16.
Is there any other library, or even an interesting code snippet I am not aware of? Any help is appreciated.
Samples:
Full Picture
IR Channel
Dust and scratch mask
What I want to remove automatically
What I consider too large to remove with no user intervention
You can also download the original TIFF file here. It contains two alpha channels. One is the original IR channel, and the other one is the IR channel already prepared for dust removal.
I have had an attempt at this, and can go some way to achieving some of your objectives... I can read in your 16-bit image, detect the dust pixels using the IR channel data, and replace them, and write out the result without any alpha channel and all the while preserving your 16-bit data.
The part that is lacking is the replacement algorithm - I have just propagated the next pixel from above. You, or someone cleverer than me on Stack Overflow, may be able to implement a better algorithm but this may be a start.
It is in Perl, but I guess it could be readily converted to another language. Here is the code:
#!/usr/bin/perl
use strict;
use warnings;
use Image::Magick;
# Open the input image
my $image = Image::Magick->new;
$image->ReadImage("pa.tiff");
my $v=0;
# Get its width and height
my ($width,$height)=$image->Get('width','height');
# Create output image of matching size
my $out= $image->Clone();
# Remove alpha channel from output image
$out->Set(alpha=>'off');
# Load Red, Green, Blue and Alpha channels of input image into arrays, values normalised to 1.0
my (#R,#G,#B,#A);
for my $y (0..($height-1)){
my $j=0;
my #RGBA=$image->GetPixels(map=>'RGBA',height=>1,width=>$width,x=>0,y=>$y,normalize=>1);
for my $x (0..($width-1)){
$R[$x][$y]=$RGBA[$j++];
$G[$x][$y]=$RGBA[$j++];
$B[$x][$y]=$RGBA[$j++];
$A[$x][$y]=$RGBA[$j++];
}
}
# Now process image
my ($d,$r,$s,#colours);
for my $y (0..($height-1)){
for my $x (0..($width-1)){
# See if IR channel says this is dust, and if so, replace with pixel above
if($A[$x][$y]<0.01){
$colours[0]=$R[$x][$y-1];
$colours[1]=$G[$x][$y-1];
$colours[2]=$B[$x][$y-1];
$R[$x][$y]=$R[$x][$y-1];
$G[$x][$y]=$G[$x][$y-1];
$B[$x][$y]=$B[$x][$y-1];
$out->SetPixel(x=>$x,y=>$y,color=>\#colours);
}
}
}
$out->write(filename=>'out.tif',compression=>'lzw');
The result looks like this, but I had to make it a JPEG just to fit on SO:
I cannot comment, so I write an answer.
I suggest using G'Mic with the filter "inpaint".
You should load the image, take the IR image and convert it to b/w, then tell the filter inpaint to fill the areas marked in the IR image.
OpenCV has a good algorithm for image inpaiting, which is basically what you were searching for.
https://docs.opencv.org/3.3.1/df/d3d/tutorial_py_inpainting.html
If that will not help then only Neural Networks algorithms
I have a binary file that starts off with some data. After this data, the file has a JPEG image embedded in it. After the image, the file continues with some other data. I wish to use the image as an OpenGL texture.
At the moment, the only method I know of creating an OpenGL texture with Magick is to read the image file with Magick, write it into a blob, and upload the blob.data() to opengl (from this link: http://www.imagemagick.org/pipermail/magick-developers/2005-July/002276.html).
I am trying to use Magick++, but it only allows me to specify a filename, not a C-style filehandle or filestream...Are my only options the following? :
Save the JPEG image portion in the binary file as a separate temporary file and get Magick++ to read that image. I don't wish to do this as writing to disk will slow my program down.
Read the image portion into an array, create a Blob with the array as its data, and then read the Blob to obtain an image. I don't wish to do this either because after I get the image, I will need to again write the image data to another blob, and the entire code becomes unnecessarily long.
Switch to another library like DevIL, which offers support for what I want. Unfortunately, DevIL is not as feature rich as Magick.
I also looked into the core C API for Magick, where I can specify a filehandle, but the documentation says that the filehandle is closed by Magick after the image is read, which is definitely not good for my program (it is going to be pretty ugly to get the rest of my program to reopen the binary file to continue its processing...
If there is a way to provide Magick with custom I/O routines, or better still, a cleaner way of using Magick with OpenGL, please enlighten me!
The next release of GraphicsMagick does not close the input file handle after the image is read. You can try the latest development snapshot.
You could consider using mmap() (memory mapped file) to access the data and treat it as an in-memory BLOB using Magick++. The main issue with this is you might not know how long the data was in case you need to access data following the embedded JPEG image data.
It is trivial to add FILE* support to Magick++. The only reason I did not do so was for philosophical reasons (C++ vs C).
I'm trying to open an image file and store a list of pixels by color in a variable/array so I can output them one by one.
Image type: Could be BMP, JPG, GIF or PNG. Any of them is fine and only one needs to be supported.
Color Output: RGB or Hex.
I've looked at a couple libraries (RMagick, Quick_Magick, Mini_Magick, etc) and they all seem like overkill. Heroku also has some sort of difficulties with ImageMagick and my tests don't run. My application is in Sinatra.
Any suggestions?
You can use Rmagick's each_pixel method for this. each_pixel receives a block. For each pixel, the block is passed the pixel, the column number and the row number of the pixel. It iterates over the pixels from left-to-right and top-to-bottom.
So something like:
pixels = []
img.each_pixel do |pixel, c, r|
pixels.push(pixel)
end
# pixels now contains each individual pixel of img
I think Chunky PNG should do it for you. It's pure ruby, reasonably lightweight, memory efficient, and provides access to pixel data as well as image metadata.
If you are only opening the file to display the bytes, and don't need to manipulate it as an image, then it's a simple process of opening the file like any other, reading X number of bytes, then iterating over them. Something like:
File.open('path/to/image.file', 'rb') do |fi|
byte_block = fi.read(1024)
byte_block.each_byte do |b|
puts b.asc
end
end
That will merely output bytes as decimal. You'll want to look at the byte values and build up RGB values to determine colors, so maybe using each_slice(3) and reading in multiples of 3 bytes will help.
Various image formats contain differing header and trailing blocks used to store information about the image, data format and EXIF information for the capturing device, depending on the type. Probably going with a something that is uncompressed would be good if you are going to read a file and output the bytes directly, such as uncompressed TIFF. Once you've decided on that you can jump into the file to skip headers if you want, or just read those too to see or learn what's in them. Wikipedia's Image file formats page is a good jumping off place for more info on the various formats available.
If you only want to see the image data then one of the high-level libraries will help as they have interfaces to grab particular sections of the image. But, actually accessing the bytes isn't hard, nor is it to jump around.
If you want to learn more about the EXIF block, used to describe a lot of different vendor's Jpeg and TIFF formats ExifTool can be handy. It's written in Perl so you can look at how the code works. The docs nicely show the header blocks and fields, and you can read/write values using the app.
I'm in the process of testing a new router so I haven't had a chance to test that code, but it should be close. I'll check it in a bit and update the answer if that didn't work.