I've come across a bottleneck that occurs when writing the final png file from a WMS request. I'm rendering a geotiff (tiled, LZW compressed, with external overviews, 90 MB file), reprojecting the layer from native CRS to EPSG:4326. When requesting an image of the same size in native CRS, the request takes around 100 ms, when reprojected, the request takes on average around 7000 ms.
I had a look in logs with GEOSERVER_DEVELOPER_LOGGING options and both requests (native/reprojected) are taking similar amount of time except when writing the final png output image, where the request for reprojected image gets stucked:
2022-10-18 14:12:02,059 DEBUG [org.geoserver.wms.map] - Writing png image ...
2022-10-18 14:12:09,139 DEBUG [org.geoserver.wms.map] - Writing png image ... done!
Using these log messages, I was able to look up following function in java:
public void formatImageOutputStream(
RenderedImage image, OutputStream outStream, WMSMapContent mapContent)
As I'm not sure what this function does, could anyone advise me where to look next? The server is a virtual machine with Windows Server running in HyperV, multiple cores and plenty of memory allocated to java. The issue seems to be affecting all layers, however same rasters and layers are working fine in other Geoserver instances that I run on physical machines.
Thanks a lot.
Geoserver version: 2.20.2
Container: Tomcat 9.0 (jre1.8.0_321)
Related
I am working in processing trying to load images using these file formats and the website the data is from (http://yann.lecun.com/exdb/mnist/) says that "Users of Intel processors and other low-endian machines must flip the bytes of the header." Is there a way to do this in processing? Would it be easier in a different language, like python, or with a different program?
The files are loading fine but the images appear to be very zoomed-in. (maybe?)
Thanks
I have a batch processing script in GIMP that, for every image file in a directory, involves importing the image, importing a background image as a layer, doing some edits, and exporting the image. The edits take no time at all but the gimp-file-load, gimp-file-load-layer, and gimp-file-save steps take a combined total of 3-4 seconds for a 69x96 .tga image and so the batch process will take the better part of a day to handle thousands of files.
Is there a faster way to import/export these images GIMP? Maybe I can eliminate the background import step by keeping the background image open until the batch process is complete. But then what would I use in place of
(gimp-file-load-layer 1 image background)
to add the background image as a layer? I don't know of any procedures that can transfer data between two images, open in GIMP or not, without using clipboard (which I'm already using to transfer alpha channel data) or file-load.
Not really an answer but too long for a comment:
Using two 200x200 TGA files (filled with plasma):
import time
times=[]
times.append(time.time())
image = pdb.gimp_file_load("/tmp/TGA/TGA-200x200-in.tga","/tmp/TGA/TGA-200x200-in.tga")
times.append(time.time())
layer = pdb.gimp_file_load_layer(image, "/tmp/TGA/TGA-200x200-in2.tga")
times.append(time.time())
pdb.gimp_image_add_layer(image, layer, 0)
times.append(time.time())
layerOut = pdb.gimp_image_flatten(image)
times.append(time.time())
pdb.file_tga_save(image,layerOut,"/tmp/TGA/TGA-200x200-out.tga","/tmp/TGA/TGA-200x200-out.tga", 1, 0)
times.append(time.time())
print "Steps:", [ "%5.1fms" % ((e-s)*1000) for s,e in zip(times[:-1],times[1:])]
print "Total: %5.1fms" % ((times[-1]-times[0])*1000)
Yields:
Steps: [' 97.7ms', '106.3ms', ' 20.6ms', ' 22.2ms', '102.6ms']
Total: 349.4ms
So this is 10 times faster for me. Tried variations (using file-save instead of file-tga-save for instance) without any significant changes in running time.
Yes, this is Python but AFAIK this ends up running the same code in Gimp (otherwise you have a solution...). So IMHO you have an I/O bottleneck.
Measurements on Core i5-9400H 2.50GHz with SSD, running Linux and an ext4 file system (which could be another solution...).
I would like to set 4 additional properties on 8000 images using the Google Earth Engine command line tool. Properties are unique per image. I was using Python 2.7 and the subprocess.call(cmd) or subprocess.check_output(cmd) methods. Both are very slow (takes 9 seconds per image i.e. 20 hours total). Therefore I tried to send the commands without waiting for a response using supprocess.Popen(). It causes my PC to crash due to the amount of tasks (CPU close to 100%)
I was looking for ways to have my PC use say 80% of its CPU or even better, scale down CPU use if I am using other things. I found the os.nice() and nice argument for subprocess.Popen(["nice",20]) but struggle to use them in my code.
this is an example of a command I send using the subprocess method:
earthengine asset set -p parameter=DomWN_month users/rutgerhofste/PCRGlobWB20V04/demand/global_historical_PDomWN_month/global_historical_PDomWN_month_millionm3_5min_1960_2014I000Y1960M01
Can any one please let me know will jmeter support performance testing for file uploads more than 10gb files. The way the files are uploading is through chunking in JAVA. I cannot do the file upload for more than 10 GB because int allows max size of 2^31. In the http sampler i am declaring the file size as one one chunk
for eg: file size is 444,641,856 bytes, I am declaring the whole in one chunk instead of dividing it into chunks of 5mb each.
The developers are not willing to change the code and also if I give the result using one chunk size its not a valid performance test.
Can anyone suggest will jmeter allowing chunking mechanism ..... and also is there a solution for file uploading for more than 10Gb Files
Theoretically JMeter doesn't have 2GB limitation (especially HTTPClient implementations) so given you configured it properly you shouldn't face errors.
However if you don't have as much RAM as 10GB x number of virtual users you might want to try HTTP Raw Request sampler available via JMeter Plugins.
References:
https://groups.google.com/forum/#!topic/jmeter-plugins/VDqXDNDCr6w%5B1-25%5D
http://jmeter.512774.n5.nabble.com/fileupload-test-with-JMeter-td4267154.html
I have a VS 2005 application using C++ . It basically importing a large XML of around 9 GB into the application . After running for more than 18 hrs it gave an exception 0xc0000006 In page error. THe virtual memory consumed is 2.6 GB (I have set the 3GB) flag.
Does any one have a clue as to what caused this error and what could be the solution
Instead of loading the whole file into the memory you can use SAX parsers to load only a part of the file to the memory.
9Gb seems overly large to read in. I would say that even 3Gb is too large in one go.
Is your OS 64bit?
What is the maximum pagefile size set to?
How much RAM do you have?
Were you running this in debug or release mode?
I would suggest that you try to reading the XML in smaller chunks.
Why are you trying to read in such a large file in one go?
I would imagine that your application took so long to run before failing as it started to copy the file into virtual memory, which is basically a large file on the hard disk. Thus the OS is reading the XML from the disk and writing it back onto a different area of disk.
**Edit - added text below **
Having had a quick peek at Expat XML parser it does look as if you're running into problems with stack or event handling, most likely you are adding too much to the stack.
Do you really need 3Gb of data on the stack? At a guess I would say that you are trying to process a XML database file, but I can't imagine that you have a table row that is so large.
I think that really you should use it to search for key areas and discard what is not wanted.
I know nothing other than what I have just read about Expat XML Parser but would suggest that you are not using it in the most efficient manner.