Image extraction with pdfimages and mupdf/mutool works fine so far.
Images in PDFs produced with FreePDF are always sliced, so one image results in multiple image files.
Is there a trick to avoid this? How can I use the results of pdfshow?
Are there coordinates to know the position and height and width to
cut/crop the image after converting the PDF to a PNG or JPEG?
The most likely reason why your images are "sliced" after extracting them is this: they were "sliced" already before extracting them -- as their way of living within the PDF file itself.
Don't ask me why some PDF generating software does this.
MS Powerpoint is infamous for this -- background images showing some gradient often get sliced up into tens of thousands of 1x1, 1x2 or 1x8 pixels and similarly-sized mini images inside the PDF.
Update
1. Identify the scope of the problem
The image fragments of the sample PDF can be identified with the pdfimages -list command (this requires a recent version of pdfimages based on the Poppler fork, not the xpdf one!):
pdfimages -list so-28023312-test1.pdf
page num type width height color comp bpc enc interp objectID x-ppi y-ppi size ratio
------------------------------------------------------------------------------------------
1 0 image 271 271 rgb 3 8 jpeg no 18 0 163 163 26.7K 12%
1 1 image 271 271 rgb 3 8 jpeg no 19 0 163 163 21.7K 10%
1 2 image 271 271 rgb 3 8 jpeg no 30 0 163 163 22.9K 11%
1 3 image 271 271 rgb 3 8 jpeg no 31 0 163 163 21.8K 10%
1 4 image 132 271 rgb 3 8 jpeg no 32 0 162 163 9895B 9.2%
1 5 image 271 271 rgb 3 8 jpeg no 33 0 163 163 22.5K 10%
1 6 image 271 271 rgb 3 8 jpeg no 34 0 163 163 16.5K 7.7%
1 7 image 271 271 rgb 3 8 jpeg no 35 0 163 163 16.9K 7.9%
1 8 image 271 271 rgb 3 8 jpeg no 36 0 163 163 20.3K 9.4%
1 9 image 132 271 rgb 3 8 jpeg no 37 0 162 163 14.5K 14%
1 10 image 271 271 rgb 3 8 jpeg no 20 0 163 163 17.1K 8.0%
1 11 image 271 271 rgb 3 8 image no 21 0 163 163 107K 50%
1 12 image 271 271 rgb 3 8 image no 22 0 163 163 96.7K 45%
1 13 image 271 271 rgb 3 8 image no 23 0 163 163 119K 56%
1 14 image 132 271 rgb 3 8 jpeg no 24 0 162 163 10.7K 10%
1 15 image 271 99 rgb 3 8 jpeg no 25 0 163 161 7789B 9.7%
1 16 image 271 99 rgb 3 8 jpeg no 26 0 163 161 6456B 8.0%
1 17 image 271 99 rgb 3 8 jpeg no 27 0 163 161 7202B 8.9%
1 18 image 271 99 rgb 3 8 jpeg no 28 0 163 161 8241B 10%
1 19 image 132 99 rgb 3 8 jpeg no 29 0 162 161 5905B 15%
Because there are only 20 different fragments on 1 page, it is easy to...
...first extract them all and convert them to JPEGs, and
...then stitch them together again.
2. Extract the fragments as JPEGs
The following command will extract the fragments and try to save them as JPEGs (-j) 28023312:
pdfimages so-28023312-test1.pdf 28023312
There are 3 images which came out as PPM. Use ImageMagick's convert to make JPEGs from them (not strictly required, but it simplifies the 'stitching' command line:
for i in 11 12 13; do
convert 28023312-0${i}.ppm 28023312-0${i}.jpg
done
Here are the first three fragments, 280233312-000.jpg, 280233312-001.jpg and 280233312-002.jpg:
3. Stitch the 20 fragments together again
ImageMagick can stitch the 20 images together again. When looking at the PDF page as well as at the 20 JPEGs it is easy to determine the order they need to be put together:
convert \
\( 28023312-0{00,01,02,03,04}.jpg +append \) \
\( 28023312-0{05,06,07,08,09}.jpg +append \) \
\( 28023312-0{10,11,12,13,14}.jpg +append \) \
\( 28023312-0{15,16,17,18,19}.jpg +append \) \
-append \
complete.jpg
Dissecting the command:
The +append image operator appends all listed images in a horizontal order.
The \( ... \) lines indicate an 'aside' processing of the resprective part of the image stack (which needs to be separated by the escaped parentheses). The result of this horizontal appending operation will then replace the individual fragments inside the current image stack.
The final -append image operator appends the current images vertically.
Here is the resulting JPEG, fully stitched together again:
Could this be automated?
In theory we could automate this process. For this we would have to analyse the PDF source code. However, that is rather difficult, because the content stream may be compressed.
In order to uncompress all or most of the content streams and to get a nicer representation of the PDF file structure, we could use mutool clean -d, podofouncompress or qpdf --qdf.
I prefer qpdf, the 'structural, content-preserving PDF file transformer'. Here is the command:
qpdf --qdf --object-streams=disable so-28023312-test1.pdf qdf.pdf
The resulting PDF file, qdf.pdf is more easy to analyse, because most (but not all) previously binary sections are now in ASCII. When you search for the occurrences of Do inside this file, you will see where images are inserted (however, I cannot give you a complete PDF analysing tutorial here, sorry...).
The following command prints all lines where Do occurs, plus the preceding line (-B 1):
grep -a -B 1 " Do" qdf.pdf
1002 0 0 1002 236 5776.67 cm
/Im0 Do
--
1001 0 0 1002 1237 5776.67 cm
/Im1 Do
--
120.12 0 0 120.24 268.44 693.2004 cm
/Im2 Do
--
[...skipping 15 other output segments...]
--
1002 0 0 369 3237 3406.67 cm
/Im18 Do
--
490 0 0 369 4238 3406.67 cm
/Im19 Do
--
1 0 0 1 204.9037018 508.5130005 cm
/Fm0 Do
All the /ImNN Do lines insert images (the /Fm0 Do line refers to a form object not an image).
The preceding lines, for example 490 0 0 369 4238 3406.67 cm set up the current transformation matrix. From this line alone, one can sometimes conclude the position of the image and its size. In the case of this file, it is not enough -- the contents of more preceding lines would be required in order to determine the current 'drawing position'.
FreePDF uses Ghostscript and creates a 'virtual printer'. When you 'print to PDF' what actually happens is that your application prints to the Windows print pipeline, which sends the graphics primitives to the Windows PostScript printer driver, which sends the PostScript to the Port Monitor. The FreePDF Port Monitor stores this PostScript program on disk. When the output is complete, it starts up Ghostscript which interprets the PostScript and produces a PDF file.
Now, unless you are using a spectacularly old version of Ghostscript (which is possible, you should check!) this will take whatever was in the input and put it in the output. It won't slice up images.
Which means that, as Kurt and David said above, that the real reason for the problem is that the PostScript program has sliced up images in it, before Ghostscript ever saw it.
Now I know that's not generally the case, but it depends heavily on what PostScript printer driver you have installed, how its configured, what version of Windows you are using and what the application driving the printer is.
As David rightly says, Microsoft Office applications have a bad habit of drawing certain kinds of patterns this way (to get a 'translucent effect' they use a pattern where the cell is an imagemask, the 'white' pixels are transparent).
Also if you have large photographs (for example) and the PostScript printer is configured with minimal memory, the driver might split up the image in order not to exhaust the printer's memory. Obviously that's a configuration problem because on a desktop PC you would have to be using monster images to overwhelm Ghostscript.
So basically, we need a lot more information from you before we can answer this fully, but the principle is that the damage was done before it got to FreePDF. The version of Ghostscript used to create the PDF file will be in the PDF file metadata, unless FreePDF chose to erase/overwrite it.
Finally, as Kurt pointed out, you should post a link to the PDF file, and ideally the application file and intermediate PostScript file which was used to produce the PDF.
Related
Can anyone confirm the following behaviour on MacOS 10.13 High Sierra with GhostScript? I don't get the problem when using 10.12 Sierra.
When I create PDFs in GhostScript, it always leaves the Title, Author and other metadata blank.
I know that you can set the metadata with PDFmarks within the PostScript file itself (or a secondary merged PS file that just contains PDFmarks), but that requires manually setting the field for each file.
Currently, my PDFs fail PDF-X validation until I manually add the metadata.
My PostScript does contain DSC comments, and the GS documentation implies that this should be picked up, as ParseDSCCommentsForDocInfo is true by default.
/usr/local/bin/gs \
-dPDFX \
-dNOPAUSE \
-dBATCH \
-sDEVICE=pdfwrite \
-sOutputFile="$filename" \
-dProcessColorModel=/DeviceCMYK \
-dCompatibilityLevel=1.4 \
"$f" \
/Library/PostScript/PDFX_def.ps
The source of the PostScript does not seem to be a factor: Adobe apps like InDesign, MacOS's cgpdftops, all behave similarly.
Here's a simple test PS file. Distiller takes the comments and uses them for DocInfo; GhostScript doesn't on MacOS 10.13.
%!PS-Adobe-3.0
%%Title: (MyTitle.file)
%%Creator: (MyApp: cgpdftops CUPS filter)
%%CreationDate: (Saturday, February 03 2018 10:19:01 GMT)
%%For: (User Me)
%%BoundingBox: 36 36 576 756
%%Pages: 1
%%LanguageLevel: 2
%%EndComments
%%BeginSetup
% this is where fonts would be embedded
%%EndSetup
%%Page: (1) 1
%%BeginPageSetup
% this is where page-specific features would be specified
%%EndPageSetup
% Draw a black box around the page
0 setgray
1 setlinewidth
36 36 540 720 rectstroke
% Draw a two inch blue circle in the middle of the page
0 0 1 setrgbcolor
306 396 144 0 360 arc closepath fill
% Draw two half inch yellow circles for eyes
1 1 0 setrgbcolor
252 432 36 0 360 arc closepath fill
360 432 36 0 360 arc closepath fill
% Draw the smile
1 setlinecap
18 setlinewidth
306 396 99 200 340 arc stroke
% Print it!
showpage
%%EOF
As of MacOS Mojave 10.14.3 and GhostScript version 9.26, this now fixed. I can only presume it was a bug in GhostScript or MacOS.
I'm using Otool to look into a compiled library (.a) and I want to see what the file size of each component in the binary is. I see that
otool -l [lib.a]
will show me this information but there is also a LOT of other information I do not need. Is there a way I can just see the file size and not everything else? I can't seem to find it if there is.
The size command does that, e.g.,
size lib.a
will show the size of each object stored in the lib.a archive. For example:
$ size libasprintf.a
text data bss dec hex filename
0 0 0 0 0 lib-asprintf.o (ex libasprintf.a)
639 8 1 648 288 autosprintf.o (ex libasprintf.a)
on most systems. OS X format is a little different:
$ size libl.a
__TEXT __DATA __OBJC others dec hex
86 0 0 32 118 76 libl.a(libmain.o)
75 0 0 32 107 6b libl.a(libyywrap.o)
Oddly (though "everyone" implements it), I do not see size on the POSIX site. OS X has a manual page for it.
Need to recruit the help of any budding bioinformaticians that are lurking in the shadows here.
I am currently in the process of formatting some .fasta files for use in a set of grouping programs but I cannot for the life of me get them to work. First things first, all the files have to have a 3 or 4 character name such as the following:
PP41.fasta
PP59.fasta
PPBD.fasta
...etc...
The files must have headers for each gene sequence that look like so: >xxxx|yyyyyyyyyy where xxxx is the same 3 or 4 letter 'taxon' identifier as the file names I put above and yyyyyyy is a numerical identifier for each of the proteins within each of the taxons (the pipe symbol can also be replaced with an _ as below). I then cat all of these in to one file which has a header that looks correct like so:
>PP49_00001
MIENFNENNDMSDMFWEVEKGTGEVINLVPNTSNTVQPVVLMRLGLFVPTLKSTKRGHQG
EMSSMDATAELRQLAIVKTEGYENIHITGARLDMDNDFKTWVGIIHSFAKHKVIGDAVTL
SFVDFIKLCGIPSSRSSKRLRERLGASLRRIATNTLSFSSQNKSYHTHLVQSAYYDMVKD
TVTIQADPKIFELYQFDRKVLLQLRAINELGRKESAQALYTYIESLPPSPAPISLARLRA
RLNLRSRVTTQNAIVRKAMEQLKGIGYLDYTEIKRGSSVYFIVHARRPKLKALKSSKSSF
KRKKETQEESILTELTREELELLEIIRAEKIIKVTRNHRRKKQTLLTFAEDESQ*
>PP49_00002
MQNDIILPINKLHGLKLLNSLELSDIELGELLSLEGDIKQVSTGNNGIVVHRIDMSEIGS
FLIIDSGESRFVIKAS*
Next step is to construct a blast database which I do as follows, using the formatdb tool of NCBI Blast:
formatdb -i allproteins.fasta -p T -o T
This produces a set of files for the database. Next I conduct an all-vs-all BLAST of the concatenated proteins against the database that I made of them like so, which outputs a tabular file which I suspect is where my issues are beginning to arise:
blastall -p blastp -d allproteins.fasta -i allproteins.fasta -a 6 -F '0 S' -v 100000 -b 100000 -e 1e-5 -m 8 -o plasmid_allvall_blastout
These files have 12 columns and look like the below. It appears correct to me, but my supervisor suspects the error is in the blast file - I don't know what I'm doing wrong however.
PP49_00001 PP51_00025 100.00 354 0 0 1 354 1 354 0.0 552
PP49_00001 PP49_00001 100.00 354 0 0 1 354 1 354 0.0 552
PP49_00001 PPTI_00026 90.28 288 28 0 1 288 1 288 3e-172 476
PP49_00001 PPNP_00026 90.28 288 28 0 1 288 1 288 3e-172 476
PP49_00001 PPKC_00016 89.93 288 29 0 1 288 1 288 2e-170 472
PP49_00001 PPBD_00021 89.93 288 29 0 1 288 1 288 2e-170 472
PP49_00001 PPJN_00003 91.14 79 7 0 145 223 2 80 8e-47 147
PP49_00002 PPTI_00024 100.00 76 0 0 1 76 1 76 3e-50 146
PP49_00002 PPNP_00024 100.00 76 0 0 1 76 1 76 3e-50 146
PP49_00002 PPKC_00018 100.00 76 0 0 1 76 1 76 3e-50 146
SO, this is where the problems really begin. I now pass the above file to a program called orthAgogue which analyses the paired sequences I have above using parameters laid out in the manual (still no idea if I'm doing anything wrong) - all I know is the several output files that are produced are all just nonsense/empty.
Command looks like so:
orthAgogue -i plasmid_allvsall_blastout -t 0 -p 1 -e 5 -O .
Any and all ideas welcome! (Hope I've covered everything - sorry about the long post!)
EDIT Never did manage to find a solution to this. Had to use an alternative piece of software. If admins wish to close this please do, unless it is worth having open for someone else (though I suspect its a pretty niche issue).
Discovered this issue (of orthAgogue) first today:
though my reply may be old, I hope it may help future users;
issue is due to a missing parameter: seems like you forgot to specify the separator: -s '_', ie, the following set of command-line parameters should do the trick*:
orthAgogue -i plasmid_allvsall_blastout -t 0 -p 1 -e 5 -O -s '_'
(* Under the assumption that your input-file is a tabular-seperated file of columns.)
A brief update after comment made by Joe:
In brief, the problem described in the intiail error report (by Joe) is (in most cases) not a bug. Instead it is one of the core properties of the Inparanoid algorithm which orthAgogue implements: if your ortholog-result-file is empty (though constructed), this (in most cases) implies that there are no reciprocal best match between a protein-pair from two different taxa/species.
One (of many) explanations for this could be that your blastp-scores are too similar, a case where I would suggest a combined tree-based/homology clustering as in TREEFAM.
Therefore, when I receive your data, I'll send it to one of the biologists I'm working with, with goal of identifying the tool proper for your data: hope my last comment makes your day ;)
Ole Kristian Ekseth, developer of orthAgogue
I'm using tesseract v 3.02
I have the following image
Download Image (right click -- save link as)
I would like to get the text from it. I'm using tesseract for the purpose.
When writing this command:
tesseract cropped.png tess -psm 7
The result I get is "suackea I 30 10193020 NL 3 e 1 64 :23 23% 34% 120". While the end is ok, the beginning is incorrect. The expected result is:
"Strackea III €0.10/€0.20 NL 6 6 1 €4 €23 23% 34% 120"
I have tried to do some transformation using imageMagick before tesseract to get an image with text written in black on a white background:
convert cropped.png -fuzz 28000 -fill black -opaque white cropped.png
convert cropped.png -fuzz 25000 -fill white -opaque rgb(118,118,118) cropped.png
The resulting image is
Download Image (right click -- save link as)
tesseract cropped.png tess -psm 7
but the result is the same.
What transformation or other command line tool would you use to recognize the text correctly ?)
The font the text is written in is Microsof Sans Serif
what I want is to add eps file into temporary ps file which has text written, then I convert my ps file to eps file using ghostscript, but when I see my eps file in AI outline mode, I see extra square around my eps file which is size box, which should not be there, It should be part of single compound box
Ghostscript version is 9.05 and before I include eps into ps I need to resize it. So resized eps file shows page border into outline mode. Which is actually not there, but when it goes to machine it will cut out that path which should not be case.
Alright, I think I have some understanding of what you're doing and where the trouble may be creeping in. As I commented, you're running the file through ghostscript multiple times. Each time it has to interpret the postscript code and construct an internal display-list representation and then recreate appropriate postscript code on the output end. So it's the clone of a clone of a clone problem. Any little hiccup can cause cascade failures.
So, enough of being preachy. The alternative is to manipulate the eps file as text.
So, if we want the image to be scaled to fill a 500x500 square, we will be guided by the number in the BoundingBox comment. I'll quote this silly file from the linked question as an example:
%!PS-Adobe-2.0 EPSF-2.0
%%BoundingBox: 72 700 127 708 %<-- modify this
%%HiResBoundingBox: 72.000000 700.000000 127.000000 707.500000 %<-- delete this
%%EndComments
% EPSF created by ps2eps 1.68
%%BeginProlog
save
countdictstack
mark
newpath
/showpage {} def
/setpagedevice {pop} def
%%EndProlog
%%Page 1 1 %<-- insert translate and scale after this line
/Times-Roman findfont
11 scalefont setfont
72 700 moveto
(This is a test)show
%%Trailer
cleartomark
countdictstack
exch sub { end } repeat
restore
%%EOF
So, the BoundingBox was %%BoundingBox: 72 700 127 708 and it needs to be 0 0 500 500 or rather (to preserve the aspect ratio) 0 0 x 500 or 0 0 500 y where x or y (whichever it happens to be) is < 500. The existing size is 127-72 x 708 - 700 = 55 x 8. So our scaling factor is 500/55. But we also want to translate the lower-left corner to the origin, and its simplest to do that first, so the scaling doesn't affect the interpretation of the numbers.
So, to take 72 700 127 708 to 0 0 500 y, first we add -72 -700 translate to the file, and modify the bounding box to 0 0 55 8, and delete that silly HiRes line: we don't really need it.
Then, we add 500 55 div dup scale (let the interpreter do the math, hee hee). So the maximum x will now be 500, but, oh, what to put for the y? A quick calculation yields 72!
So, this awk program will modify an eps file to be 500 points wide, with y scaled appropriately.
/%%BoundingBox: ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*)/{x=$2;y=$3;w=$4-x;h=$5-y;print $1,0,0,500,(500/w)*h}
!/%%BoundingBox:/&&!/%%HiRes/{print}
/%%Page /{print -x,-y,"translate"; print 500,w,"div dup scale"}
Usage:
$ awk -f epsscale.awk etest.eps