When outputting a text file, it randomly (it seems) misses characters. I have this running on over 3000 sites daily, and roughly 20 sites have it occur on a semi-regular basis. The file will range from 300kb to 5 meg! And the position it occurs is not the same, and regenerating the file once the problem is noticed, it will not miss the characters the second time.
The file is created with some incredibly simple vb6 code:
intfile = FreeFile
Open wFileName For Output As #intfile
Print #intfile, wContents
Close #intfile
If anyone has any ideas as to what could be causing it, that would be awesome, I'm completely stumped.
Thank you in advance for any advice!
Related
I have a VB script that does row processing on a small Excel file (35 columns, 2000 rows, no hidden content). It takes about 3 seconds to process the file when applied to an .xls file. If I save the .xls as a .xlsx file, the same script takes about 30 to 60 seconds, completely freezing Excel several times in the process (although it finishes and the results are correct).
I suspect some loop parts of the code are responsible, but I need to find out which. I have found scripts like https://stackoverflow.com/a/3506143/5064264 which basically amount to timing all parts of the script by hand - similar to printf-debugging, which is not very elegant and also consumes much time for large scripts.
Is there a better alternative to semi-manual profiling, like performance analyzers in most IDEs, or exist any other approaches to this problem? I do not want to post the code as it is rather long and I also want to solve it for other scripts in the future.
yes - kind of. I use this to test how fast things are running and experiment with different approaches to get better timings. I stole this from somewhere here on SO.
sub thetimingstuff()
Dim StartTime As Double
Dim SecondsElapsed As Double
StartTime = Timer
'your code goes here
SecondsElapsed = Round(Timer - StartTime, 2)
MsgBox "This code ran successfully in " & SecondsElapsed & " seconds", vbInformation
end sub
Another EDIT
you could bury debug.print lines sporadically through your code to see how much each chunk takes to execute.
The error in my case was as Comintern's comment suggested - a method was copying whole columns into a dictionary to reorder them, instead of just the cells with values in them. To preserve this for future readers, I am copying his/her excellent comment here, thanks again!
An xls file has a maximum of 65,536 rows. An xlsx file has a maximum of 1,048,576 rows. I'd start by searching for .Rows. My guess is that you have some code iterates the entire worksheet instead of only rows with data in them.
As for generic profiling or how to get there, I used a large amount of breakpoints in the editor (set/remove with left-click in the line number column of the code editor):
I first set them every 20 lines apart in the main method, ran the program with the large dataset, then after each press of continue (F5) manually measured if it was slow or not.
Then I removed the breakpoints in the fast regions and added additional ones in the slow regions to further narrow it down into methods and then into lines inside the methods.
At the end, I could verify if by commenting out the responsible line of code and fix it.
I have a small problem in my code. I have programmed my program to find word inside a .txt file and now I want to know how to make SAPI speaking from 5000 characters from the place the word is found inside the .txt file.
If you have the position of the word in the file it shouldn't be a challenge to read the next 5000 characters into a buffer that can then be fed to the speech engine. It's possible that a hard 5000 character cutoff will leave a word broken at the end so you may want to seek ahead past that point to the next blank space or whatever to make the end smoother.
Without knowing specifics like language and platform and which SAPI you are using it is impossible to give more detailed advice.
I'm trying to download a small zip file (1159 bytes) and pipe it through funzip. This works great with larger files fro that server. However three small files give me an error:
Broken pipe, closing control connection.
I use the following code:
wget -O - --ftp-user=username --ftp-password=secret ftp://server/small-file.zip | funzip
Also downloading the file directly works good, only the piping to funzip doesn't work. I suspect the file is too small.
Anyone knows how to fix this?
Edit: Size doesn't seem to matter (don't let the girls tell you otherwise :)), even files of 400 bytes are not giving errors
Ok, if nobody can answer it, I'll answer it myself
I found there are two solutions, one is limiting the download rate for wget
--limit-rate=1000
This works for the files of around 1kb but now sometimes larger files seem to suffer from the same error. It also slows down the whole process.
Now I just pipe the download through a script that sleeps 1 second at the end. This seems to solve it.
I have a vb script that processes text files on a server, however, very occasionally without any apparent reason, the script gets stuck in a loop.
Files are uploaded by clients via ftp (smallish about 2 - 10 kb), a .Net service that is watching the ftp folder kicks off an executable (written in VB6), the executable moves the file from the ftp folder to a folder where it is processed. At this point the exe kicks off the vb script. 98% of the time the script runs without issue.
When the infinite loop occurs I kill the exe process. The odd thing is that when I re-kick off the process manually (by copying it back to the the folder that's being watched) the exact same file it goes through without issue.
In short the script opens the file into a text stream then it loops through the TextStream object until its AtEndOfStream property is true. Within the loop it creates a new file with some additional information added, now whenever the infinate loop situation occurs the temp file doesnt contain any of the sourcefile data, just the extra data added by the script, for example, the following code works 98% of the time:
Do While ts.AtEndOfStream <> True
sOriginalRow = ts.ReadLine
sUpdatedRow = sOriginalRow & ",Extra_Data"
NewFile.WriteLine sUpdatedRow
Loop
So if the source file contains:
LineALineBLineC
The new file is created conbtaining:
LineA,Extra_DataLineB,Extra_DataLineC,Extra_Data
but when the problem occurs and the new file is instantly populated with thousands of rows of
,Extra_Data,Extra_Data,Extra_Data,Extra_Data,Extra_Data...
its as though the AtEndOfStream property never becomes true.
My initial thought is that the source file is getting corrupted somehow but when these source files are reprocessed they are fine. Another thought is that the textstream object is getting incorrectly created, and not picking up the new line character, maybe because the file is locked by another process or something.
The only way I have found to replicate the behaviour is to comment out the ReadLine part of the code, which in effect prevents the current line number from incrementing. e.g.
sOriginalRow = "" 'ts.ReadLine
Can anyone offer any suggestions?
Well, as you say, the only thing I can offer are suggestions, not real answers:
Maybe your stream bumps against an ObjectDisposedException (not catchable in VBScript), so the EndOfStream condition gets <> True but not False.
For the sake of the experiment, you could try to change Do While ts.AtEndOfStream <> True to Do While Not ts.AtEndOfStream or Do Until ts.AtEndOfStream but my bio-logical circuits tell me that is probably not going to work.
There is another problem described where the stdIn and stdErr are conflicting with each other causing a hang in particular situations.
Could you also please answer the comment of Tmdean. On Error Resume Next could create a real mess: If ts.AtEndOfStream returns Null or garbage (because ts was destroyed for example) it will not become True and will cause a loop in Foreverland.
I am running a batch job that has been going for many many hours, and the log file it is generating is increasing in size very fast and I am worried about disk space.
Is there any way through the command line, or otherwise, that I could hollow out that text file (set its contents back to nothing) with the utility still having a handle on the file?
I do not wish to stop the job and am only looking to free up disk space via this file.
Im on Vista, 64 bit.
Thanks for the help,
Well, it depends on how the job actually works. If it's a good little boy and it pipes it's log info out to stdout or stderr, you could redirect the output to a program that you write, which could then write the contents out to disk and manage the sizes.
If you have access to the job's code, you could essentially tell it to close the file after each write (hopefully it's an append) operation, and then you would have a timeslice in which you could actually wipe the file.
If you don't have either one, it's going to be a bit tough. If someone has an open handle to the file, there's not much you can do, IMO, without asking the developer of the application to find you a better solution, or just plain clearing out disk space.
Depends on how it is writing the log file. You can not just delete the start of the file, because the file handle has a offset of where to write next. It will still be writing at 100mb into the file even though you just deleted the first 50mb.
You could try renaming the file and hoping it just creates a new one. This is usually how rolling logs work.
You can use a rolling log class, which will wrap the regular file class but silently seek back to the beginning of the file when the file reaches a maximum designated size.
It is a very simple wrap, either write it yourself or try finding an implementation online.