Fox: Visual FoxPro 6 / SP5 & Visual FoxPro 9 / SP2
OS: Windows 7 Professional & Windows 10 Professional
I have a legacy application that imports data from a text file of wage information. Each data line is SSN, Name, Wage. As you can expect, these wage files can get quite large, a few hundred MB sometimes.
The data from the wage file is imported into a memo field of a FoxPro table.
append memo xmemo.xmemo from (m.drive_file)
This works as it should, even with files nearing 400MB.
Then there's some code that verifies and checks the new data in the memo field for things that may need to be stripped out. (linefeeds and carriage returns). The code that does this uses the OCCURS() function.
m.nchr10 = occurs(chr(10), xmemo.xmemo) && count for line feeds
m.nchr13 = occurs(chr(13), xmemo.xmemo) && count for carriage returns
Within the last week, the above two lines have failed with the error "String is too long to fit". Smaller wage files (about 60MB or so), will work fine. But anything larger than that will throw the error. I can even reproduce the error from the command line with
?occurs("x", xmemo.xmemo) && show a count of 'x' characters.
Background:
This application has been in production for over 20 years and has taken large files without issue all of that time.
I can reproduce the error in Fox 6 and Fox 9.
At this point, I'm stumped. Anyone have any idea what the issue is? Thanks in advance to anyone that can offer some help.
First....thanks to everyone that's replied so far. I'm sorry if this seems confusing. It's confusing to me as well.
Second, I was wrong about the working file size. The memo field can accept data up to 2GB. But if you use a string operation against any memo field holding more than about 16MB worth of data, you will get the 'String is too long to fit' error. This is the part that's new and confusing. It's CLEARLY in the documentation as such, but like I said...this has worked without error for a very long time with very large files.
As for why it's not working:
There HAS BEEN very recent updates to my Win 7 and Win 10 PC's that I use to develop. The timing does coincide with when this stopped working correctly.
The error occurs regardless of the contents of the memo field (thank you, Tamar).
No changes have been made to the import code. I'm sure of this, as I'm the only FoxPro Dev here. I'm not the original Dev for the app, but it's been under my umbrella and mine alone for about 6 years and I've never needed to work with the 'import' module.
The data isn't funky. It's just alpha and numeric text. We deal with wages and they come to us in files that are formatted like below. The Fox app takes the entire file of data, brings it into the memo field, checks it for anything that needs to be stripped out, then processes the file line by line.
E2021000000000 9999999999 99 12212021119999
S999999999LASTNAME FIRSTNAME X47 00000000000000 12212021119999
S999999999LASTNAME FIRSTNAME X47 00000000000000 12212021119999
S999999999LASTNAME FIRSTNAME X47 00000000000000 12212021119999
S999999999LASTNAME FIRSTNAME X47 00000000000000 12212021119999
S999999999LASTNAME FIRSTNAME X47 00000000000000 12212021119999
S999999999LASTNAME FIRSTNAME X47 00000000000000 12212021119999
S999999999LASTNAME FIRSTNAME X47 00000000000000 12212021119999
S999999999LASTNAME FIRSTNAME X47 00000000000000 12212021119999
S999999999LASTNAME FIRSTNAME X47 00000000000000 12212021119999
F99999999
At this point, I've spent way too much time trying to figure it out. I’m going to chalk it up the Windows updates or just…magic. My solution, at least until the app can be truly replaced, is to take any large files (15MB+) from my Users, split them up into 15-16MB chunks, and then let them get processed individually. As you can imagine, this takes a chunk of my time, but this is wage data and it's time sensitive.
Thanks again, to anyone that spent any brain power on this.
For each memo, I would copy it to a file (COPY MEMO MemoFieldName TO FileName),
Fopen() it and use fgets() to process the file line by line
Related
I am developing a USB HID device using an STMicro microcontroller. I started with STMicro's HID example which works fine. I am using C++ on Windows 7 64-bit for the PC side. I have an application that works with my device. There is one thing I can't figure out, however.
The example firmware only allowed sending and receiving 2 bytes at a time, which is determined by a HIDP_CAPS.OutputReportByteLength and InputReportByteLength. I would like to send more data than this at once, but I can't figure out how to increase the report lengths. I successfully changed the endpoint wMaxPacketSize, the VID and PID, and a few other things, but I can't figure out how Windows is calculating the in and out report lengths. There doesn't seem to be any fields in my report or device descriptions that indicate this length, but I can't imagine where else it might be coming from.
Can anyone tell me how Windows determines the HIDP_CAPS.OutputReportByteLength and HIDP_CAPS.InputReportByteLength?
How can I increase these lengths?
I figured it out. I thought I would post here in case anyone else needs to know. I'm not entirely sure I really understand it all, so if I made a mistake, someone please correct me.
I had to change the report description in my firmware. I had several usages. Windows gets the report description and figures out which usage requires the longest length and uses that length. On one of my input reports I made the following changes (the input report is just an array of bytes in firmware):
0x27, 0xFF, 0xFF, 0xFF, 0xFF, //Logical maximum is 4 bytes long, and has a value of 0xFFFFFFFF
0x95, 0x01, //There is one report
0x75, 0x20, //There are 32 bits per report
I did something similar for the output, but there is no report number field (0x95).
Windows now tells me I can send and receive 5 bytes, which I believe means the end point plus report number times report size.
I have A zebra RZ600 printer and I'm having trouble writing the EPC memory of the new DogBone inlay with the monza 4D chip.
My problem is that impinj has changed the bit function to comply with the Gen2V2 specification as described here.
I've tried to use Bartender 10.1V and Zebradesiner pro V2.5, both are latest and both don't have a solution for this issue.
Side note, I'm printing tags for sporting events so usually I'm writing incrementing hex values to the EPC and print the decimal value on the tag.
Thanks,
Idan.
How i solved it:
print to file the zpl code of the printing job.
change the ^FD1000^FS to ^FD1400^FS (1400 is for 8 bytes EPC or 3000 to 3400 for 24 bytes. it depends on the size of the new EPC data length).
write a script to change the printing detail and the data to write and duplicate it to X number of tags (crate one big file with a zpl code for each tag)
send the file to the printer manualy with bartender
AT+CPMS? query to a SIM returns
+CPMS: "SM",0,60, "SM,0,60,"SM",0,60
OK
The total storage is 60 messages maximum, and 0 are currently stored.
All messages are stored on the SIM.
An existing legacy application fails when trying to populate all 60 message boxes because message box 55 is the last successful write location.
When trying to populate the spaces 56-60, the command returns CMS Error 322 which is a memory full error.
The text in each SMS is simply "P H" which is 3 ASCII bytes, so each text doesn't use an excessive amount of memory.
Has anyone an explanation why the spaces 56-60 are unavailable, although the CPMS? query shows them as available? Thanks.
I suddenly am getting what appears to be a Unicode issue in Visual Studio after a Windows 7 restart. Does anyone have an idea about how to resolve this? I've been messing around with virus scanners and .cspoj files (where the errors are located) for the last few hours to no avail.
Error 1 The build stopped unexpectedly because of an internal failure.
System.Text.EncoderFallbackException: Unable to translate Unicode character \uD97C at index 1321 to specified code page.
at System.Text.EncoderExceptionFallbackBuffer.Fallback(Char charUnknown, Int32 index)
at System.Text.EncoderFallbackBuffer.InternalFallback(Char ch, Char*& chars)
at System.Text.UTF8Encoding.GetByteCount(Char* chars, Int32 count, EncoderNLS baseEncoder)
at System.Text.UTF8Encoding.GetByteCount(String chars)
at System.IO.BinaryWriter.Write(String value)
at Microsoft.Build.BackEnd.NodePacketTranslator.NodePacketWriteTranslator.TranslateDictionary(Dictionary`2& dictionary, IEqualityComparer`1 comparer)
at Microsoft.Build.Execution.BuildParameters.Microsoft.Build.BackEnd.INodePacketTranslatable.Translate(INodePacketTranslator translator)
at Microsoft.Build.BackEnd.NodePacketTranslator.NodePacketWriteTranslator.Translate[T](T& value, NodePacketValueFactory`1 factory)
at Microsoft.Build.BackEnd.NodeConfiguration.Translate(INodePacketTranslator translator)
at Microsoft.Build.BackEnd.NodeProviderOutOfProcBase.NodeContext.SendData(INodePacket packet)
at Microsoft.Build.BackEnd.NodeProviderOutOfProc.CreateNode(Int32 nodeId, INodePacketFactory factory, NodeConfiguration configuration)
at Microsoft.Build.BackEnd.NodeManager.AttemptCreateNode(INodeProvider nodeProvider, NodeConfiguration nodeConfiguration)
at Microsoft.Build.BackEnd.NodeManager.CreateNode(NodeConfiguration configuration, NodeAffinity nodeAffinity)
at Microsoft.Build.Execution.BuildManager.PerformSchedulingActions(IEnumerable`1 responses)
at Microsoft.Build.Execution.BuildManager.HandleNewRequest(Int32 node, BuildRequestBlocker blocker)
at Microsoft.Build.Execution.BuildManager.IssueRequestToScheduler(BuildSubmission submission, Boolean allowMainThreadBuild, BuildRequestBlocker blocker)
AND THE ANSWER IS:
http://www.hanselman.com/blog/CSIVisualStudioUnableToTranslateUnicodeCharacterAtIndexXToSpecifiedCodePage.aspx
NB. Hans was the closest to working out what had happened... so I awarded the points to him
Well, the message is pretty accurate. \uD97C a utf-16 surrogate, surrogates must always appear in a pair to encode a character whose value is larger than \uFFFF. The exception message says that the second surrogate of the pair does not occur in the string.
Seeing this occur in a build is very bad news, such characters should never appear in project files. You don't write them in an ancient dead Middle-Eastern language or an obscure native American language with a couple of thousand people that still know how to speak it :). The only reasonable explanation is that the file(s) on your disk are scrambled all to hell. You'll need to get your machine fixed, replacing the disk should be high on your list of priorities right now.
Before anyone says it, I know this isn't the way it should be done, but it's the way it was done and I'm trying to support it without rewriting it all.
I can assure you this isn't the worst bit by far.
The problem occurs when the application reads an entire file into a string variable.
Normally this work OK because the files are small, but one user created a file of 107MB and that falls over.
intFreeFile = FreeFile
Open strFilename For Binary Access Read As intFreeFile
ReadFile = String(LOF(intFreeFile), " ")
Get intFreeFile, , ReadFile
Close intFreeFile
Now, it doesn't fall over at the line
ReadFile = String(LOF(intFreeFile), " ")
but on the
Get intFreeFile, , ReadFile
So what's going on here, surely the String has done the memory allocation so why would it complain about running out of memory on the Get?
Usually reading a file involves some buffering, which takes space. I'm guessing here, but I'd look at the space needed for byte to character conversion. VB6 strings are 16 bit, but (binary) files are 8 bit. You'll need 107MB for the file content, plus 214 MB for the converted results. The string allocation only reserves the 214 MB.
You do not need that "GET" call, just remove it, you are already putting the file into a string, so there is no need to use the GET call.
ReadFile = Input(LOF(intFreeFile), intFreeFile)
I got the same error . And we just checked the taskmanager showing 100% resource usage . we found out one of the update application was taking too much ram memory and we just killed it.
this solved the issue for me. One more thing was we gone in to config settings.
START->RUN->MSCONFIG
and go to startup tab and uncheck the application that looks like a updater application or some odd application that you dont use.