I have a huge PHP File ( approx 2 MB). As its a third party file so can't change it. Now I want to generate documentation for it using automatic documentation generators.
I tried Doxygen, Apigen and PHP Documenter. but each of it hangs or exhaust the memory.
Is there a way to generate documentation for this file.
You could try increasing the memory limit setting in your php.ini
If this fails, you could split the file up into several new ones, and later "stitch" the output of the generator together.
Related
I'm creating my custom binary file extension.
I use the RIFF standard for encoding data. And it seems to work pretty well.
But there are some additional requirements:
Binary files could be large up to 500 MB.
Real-time saving data into the binary file in intervals when data on the application has changed.
Application could run on the browser.
The problem I face is when I want to save data it needs to read everything from memory and rewrite the whole binary file.
This won't be a problem when data is small. But when it's getting larger, the Real-time saving feature seems to be unscalable.
So main requirement of this binary file could be:
Able to partially read the binary file (Cause file is huge)
Able to partially write changed data into the file without rewriting the whole file.
Streaming protocol like .m3u8 is not an option, We can't split it into chunks and point it using separate URLs.
Any guidance on how to design a binary file system that scales in this scenario?
There is an answer from a random user that has been deleted here.
It seems great to me.
You can claim your answer back and I'll delete this one.
He said:
If we design the file to be support addition then we able to add whatever data we want without needing to rewrite the whole file.
This idea gives me a very great starting point.
So I can append more and more changes at the end of the file.
Then obsolete old chunks of data in the middle of the file.
I can then reuse these obsolete data slots later if I want to.
The downside is that I need to clean up the obsolete slot when I have a chance to rewrite the whole file.
I'm working on a Perl/CGI script which reads an 8MB file with over 100k lines and displays it in chunks of 100 lines (using pagination).
Which one of the following will be faster
Storing the entire input file into an array and extracting 100 lines for each page (using array slicing)
my #extract = #main_content[101..200];
or
For each page, using the sed command to extract any 100 lines that the user wants to view.
sed -n '101,200'p filename
If you really want performance then don't use CGI, try using something that keeps a persistent copy of the data in memory between requests. 8mb is tiny these days but loading for every request would not be sensible nor would scanning the whole file. Modperl was the older way of doing this , it was a perl interpreter embedded in the webserver , the newer way is to use catalyst or dancer, instructions for those are outside the scope of this reply. You could get away with using CGI if this was only to be use occasionally and was password protected to limit use.
Using the following code in Matlab:
nc_file_list = {'http://data.nodc.noaa.gov/thredds/dodsC/ghrsst/L2P/MODIS_A/JPL/2015/287/20151014-MODIS_A-JPL-L2P-A2015287235500.L2_LAC_GHRSST_D-v01.nc.bz2'};
temp.sl = ncreadatt(nc_file_list,'/','northernmost_latitude');
I try to get a single attribute from a netcdf file on a THREDDS OPeNDAP server. Ive been told this should be very quick as the netcdf philosophy is build around accessing small parts of data in big data sets.
The total size of the netcdf file is around 20 Mb. Running this code takes 17 seconds (internet speed is 5 Mb/s).
I need to process 19,000 files so I want this netcdf attribute reading to go alot quicker. Is there a way to read the attribute of the link given above in under 1 second?
The file is bz2-compressed, so the whole thing must be decompressed before the NetCDF library can perform random access operations on it. There's no way to avoid that.
You can use the THREDDS DAS Service as is explained by this answer:
I need to do an integrity check for a single big file. I have read the SHA code for Android, but it will need one another file for the result digest. Is there another method using a single file?
I need a simple and quick method. Can I merge the two files into a single file?
The file is binary and the file name is fixed. I can get the file size using fstat. My problem is that I can only have one single file. Maybe I should use CRC, but it would be very slow because it is a large file.
My object is to ensure the file on the SD card is not corrupt. I write it on a PC and read it on an embedded platform. The file is around 200 MB.
You have to store the hash somehow, no way around it.
You can try writing it to the file itself (at the beginning or end) and skip it when performing the integrity check. This can work for things like XML files, but not for images or binaries.
You can also put the hash in the filename, or just keep a database of all your hashes.
It really all depends on what your program does and how it's set up.
A part of my program uses WinInet's caching function (e.g. FindFirstUrlCacheEntry, FindNextUrlCacheEntry) to go through the system cache and delete files that meet certain criteria.
The problem is that when a large file is found in the cache, FindNextUrlCacheEntry fails with ERROR_INSUFFICIENT_BUFFER, and requests an unreasonable buffer size to continue (over 10MB), which I fail to allocate on that system.
I need a way to either:
- Skip large files (somehow get to the next entry)
- Get the cache entry of large files without allocating a large buffer
I noticed the "Retrieve" cache functions, but they all require URLs - and I can't even get the URL of my entry...
Any suggestions?
Thanks,
Guypo
Turns out it was my bug, WinInet doesn't actually attempt to read the full file.
Still, a way to skip files could have been useful...