We have multiple applications developed in Visual Foxpro 8.0 running in a data center on Windows 2008 R2 on VMware. We also have a Citrix farm on the same network where users run yet another VFP 8.0 application in Citrix sessions. All applications share the same set of data tables located on a file server (also Windows 2008 R2 VM). Virtual hosts are connected by 10Gb LAN (managed switch).
Since mid-July we started seeing random 1104 "Error reading file..." errors on multiple different applications on multiple servers. All of them reference different files on the file server.
The problem started mid-July and it frequency gradually increased. Earlier it was most frequent in the afternoons by 3 pm, now it happens from early morning till late afternoon. It affects EDI servers (these run batch jobs in unattended mode) and Citrix servers and a variety of applications. It occurs when a VFP application (any of them) tries to open a database container file or individual tables most often with USE command but some times executing a SQL Select statement, or when loading a VFP form that opens tables in DataEnvironment
We caught a moment when the same exact error happened on two different servers running different applications at the same exact moment (up to a second). We also saw two different applications running on the same computer erroring out at the same moment.
We replaced the file server with a new virtual machine with no relief (we since changed it back to the old file server ).
We disabled the antivirus.
We updated VMware on all hosts to the latest version.
Sysinternals Process Monitor displays "INVALID_NETWORK_RESPONSE" event when the error occurs.
We captured traffic on both the server side and client side when the error occurred and had it analyzed by a network analysis specialist. He observed a peculiar pattern, where client OS starts retrieving the file in question from the file server AFTER VFP application had thrown an error. It seems that VFP application requests a file from OS, then it either gets an abnormal response or just times out and only after that the OS sends packets requesting the file. Again, this happens sporadically.
OpLocks and SMB2 have been disabled on all computers both on the server and client side of the equation for many years and everything was running smoothly until now...
Any advice would be greatly appreciated.
My first piece of advice would be to re-enable OpLocks and SMB2. There is no reason to mess with either of those items as things stand today and you are losing a huge amount of performance running at SMB1 level.
In my experience these issues have almost always been caused by one of the following.
Antivirus/antimalware software.
Replication or online backup software like MozyPro.
The Windows Search indexing service.
You should consider installing the Windows 7 / Server 2008 R2 Enterprise Hotfix Rollup if you haven't already.
That problem mostly related by SMB2!
Some Antivirus Software!
Windows updates! If you use VFP apps by DBF/DBC file. Do not update your system/OS. That is my personal suggestion. Windows Server 2012+ or Windows 10+ prorbably would big problems at near future.
And the point high probably is:
What is your I/O request per secs? if your IO request bigger than 1000~2000 per secs for a dbf file that is a bottle neck; and your storage device is HDD -> you need to switch/update your HDD to SSD. I suggest m.2 pro series SSD.
Related
I have a problem that i have a difficult time explaining, which makes any online search very hard. Here is my dilema.
I'm migrating a VM. The purpose of this machine is to compile send out daily/weekly/monthly reports. I know there are other ways (like Power BI) but this is the situation we are in right now. The older machine has win10 pro and office 365 installed while the new has win10 enterprise version and office 2016 installed. This machine runs 24/7 in the background running specific tasks (via system scheduler app) at given times, that is it's a Virtual machine and has done so without issues since it was created. The reason for the migration is because we need to domain change and bring the machine under a new corporate policy and we don't want to do this on a live server.
We've set it the VM's the same way, same programs and same settings. Everything seams to be running smooth expect for this one thing, and here is the problem i have a hard time to explain or figure out:
MS Access will update the tables and the computer will run the tasks as set but it will not export the data to pdf unless i have a remote desktop connection open. Will not export the pdf's otherwise. MS Access uses a autoexec macro where the pdf export is set with ExportWithFormatting. This works without issues on the old server.
We thought this to be a permission or user specific issue at first but even re-creating the tasks did not work and changing paths. Otherwise also i expect we would have problems with tables updating, specially since it works when you have an active remote desktop conn running.
I'm lost and therefore hoping this community will be able to help or guide me to a solution.
I believe that we found the reason for this. It was caused by windows easy print and the printer drivers of the machine. It worked for some reason differently between the servers. after reinstalling the printer drivers and a few restarts it started working. It exports now from access again.
This is at least solved.
I've a Perl script that forks in order to use Win32::OLE to extract PNGs of slides from a Powerpoint presentation. I'm in the process of migrating it from my old server (Windows Server 2003) to a new server (Windows Server 2012). The script works fine on the old server (which has Microsoft Office 2010) but on the new server (with Microsoft Office 2016), the script bombs with an error about there not being enough storage available:
Win32::OLE(0.1712) error 0x8007000e: "Not enough storage is available to complete this operation"
I've found quite a few references to this error code, but none particularly useful. Whilst this Microsoft support article says what the problem is and what to do to fix it, I have no idea why this is suddenly an issue now I'm moving to a different server and it doesn't seem to apply to my situation.
The new server has loads of spare disk space and RAM, so there should be plenty to go around. (The old server has far less of both, but that still works.)
Could this be something to do with the new server being 64-bit?
Could it be because of the differing versions of Office?
Could Apache or Perl be configured differently by default than when installed on the old server? (It's a new installation of each, but from the same source as on the old server, and I can't find any configuration that limits memory.)
One interesting point that leads me to believe that it's something to do with the script running via Apache (v2.4.27) is that if I run it from the command line (requiring a minor modification), it works fine. Apache runs as a service and I've tried running it as the same user for which it works on the command line, and that has no impact.
I've run out of places to look and things to try now, so any help would be appreciated.
UPDATE (21 August): Since nobody seems to have any ideas, I refactored my code slightly so that it gets called via a scheduled task. That works fine, supporting my theory that it's something to do with it being run via Apache. I'll keep this question open, partly in case a solution is presented and partly in case my workaround can help anyone else who has a similar problem.
Tomcat is running a webapp under Windows. After a few days (under very low load), the exception mentioned in the title starts to appear in the logs, no new connections can be established from that point on, the only fix is then to reboot the server.
Environment:
Latest Tomcat 6
Windows Server 2008 R2
JDK 6 update 30
SQL Server 2008
Kerberos authentication
Evidence collected so far:
netstat shows no excessive amount of connections
ProcessExplorer shows no excessive amount of open file handles
system main memory usage is average
JVM heap usage is average
restarting Tomcat does not solve the problem
Open questions:
if we were leaking connections, shouldn't they show up in netstat?
shouldn't a restart of the appserver resolve the problem, because the OS should free all process resources?
is there a way to trace the problem to its origin? E.g. installing monitoring software, maybe something similar to lsof etc.?
I'm out of ideas, any hints appreciated!
The reason we got this error is a bug in Windows Server 2008 R2 / Windows 7. The kernel leaks loopback sockets due to a race condition on machines with more than one core, this patch fixes the issue:
http://support.microsoft.com/kb/2577795
I was running Alfresco Community 4.0d on Windows 7 64 bit and had the same symptoms and errors.
The problem was fixed with Microsoft's patch: "Kernel sockets leak on a multiprocessor computer that is running Windows Server 2008 R2 or Windows 7" (http://support.microsoft.com/kb/2577795) (ie. Buddy Casino's answer (see below)).
Another observation I'd like to add is that Windows connections (Internet Explorer, Remote Desktop etc) would work again about 5-10 mins after the Alfresco services were shutdown.
Alfresco is an excellent product and I was afraid I would have to scrap it. Fortunately stackoverflow came to the rescue !
Thanks again to Buddy Casino's answer.
Boo to the person who down-voted the Question.
We are seeing the same thing on a similar setup, W2008R2, Tomcat 6.0.29, Java 1.6.0.25. Restarting tomcat does not help, but restarting the server itself does, at least for a while. After the last time we started shutting down individual services and believe we have it narrowed down to either an instance of Alfresco that is also running on the server or the Backup Exec Agent services. After those services (four in total) were stopped, the applications in Tomcat started working again, although we were still seeing the buffer/connections error in the stdout log which was strange. Will need to wait for the problem to return before confirming which are the culprit, which could be anywhere from a few days to a week or more.
Any chance you are running either Alfresco or BE on your server?
I'm having a problem with vfp9 on Windows 7 64-bit. I've found that create sql view is taking 5-6 seconds. These happen instantaneous in XP. When my app starts up, I'm doing a few of these, so in Win 7, my app is taking 30+ seconds longer to start up than in XP. My views look like this:
create sql view MyView remote connection MyConn as select * from MyTable
I've also found that calling dbsetprop is adding another 1-2 seconds in Win 7. Again its instantaneous in XP.
dbsetprop('MyView.MyPk', 'Field', 'KeyField', .T.)
dbsetprop('MyView.MyPk', 'Field', 'Updatable', .T.)
Once created, the views work as they should. No slowness on with platform.
Does anyone have any ideas about what I could try or any info on what is/could be causing this?
Thank you in advance.
I don't know why as I haven't worked with Windows 7 yet with VFP... However, what I would check within VFP and try changing some settings to see if it helps.
From the VFP/IDE menu, go to Tools, then Options. On the multi-tab form, click on the "Remote Data" tab.
I don't know if/what its trying to do, but maybe for testing, make sure the "Records to fetch at a time" is NOT set to "All" (checkbox).
I would also look into SQLSETPROP() function to see if any of those settings might help.
I can't reproduce this on Windows 7 64 bit, either with VFP9 RTM or VFP9 SP2. I don't have a database of any size to work with but on the sample database Northwind the commands you list seem to work instantaneously.
A couple of questions:
Is this reproducible on any machine running Windows 7?
Where is your database? Is it on the local machine, a local network, or the internet?
There seems to be more scope for speed problems with Windows 7 and Visual FoxPro (and similar) applications, and I think this is down to the different network stack in Windows 7, immature network card drivers, an increased susceptibility to cabling and network switch problems, or any combination of these.
Ensure that all your Windows 7 boxes are on SP1 (and any Server 2008 boxes with shared DBF files also), as this fixes a file corruption issue that affected Visual FoxPro indexes.
Ensure that your network card drivers are 100% up to date. This can make a big difference.
One thing that I have seen which can give a massive improvement to the speed of networked Visual FoxPro applications is the network card driver Interrupt Moderation setting. This is present on Intel, Broadcom and many other NICs, although with possibly slightly different names.
I have personally seen situations where disabling this has changed a networked VFP application from taking 30 seconds to start to about 6 seconds.
Found the solution.
Write caching was being disabled on the drive by the raid controller software included with the machine.
Write caching was enabled under Device Manager > Disk Drive > Properties > Polices. However the software was overriding this setting.
It can be reproduced without the raid software by unchecking it in Windows 7 Polices.
The LAN which has about a half dozen windows xp professional pcs and one windows 7 professional pc.
A jet/access '97 database file is acting as the database.
The method of acccess is via dao (DAO350.dll) and the front end app is written in vb6.
When an instance is created it immediately opens a global database object which it keeps open for the duration of its lifetime.
The windows 7 machine was acting as the fileserver for the last few months without any glitches.
Within the last week what's happened is that instances of the app will work for a while (say 30 mins) on the xp machines and then will fail on database operations, reporting connection errors (eg disk or network error or unable to find such and such a table.
Instances on the windows 7 machine work normally.
Moving the database file to one of the xp machines has the effect that the app works fine on ALL the xp machines but the error occurs on the windows 7 machine instead.
Just before the problem became apparent a newer version of the app was installed.
Uninstalling and installing the previous version did not solve the problem.
No other network changes that I know of were made although I am not entirely sure about this as the hardware guy did apparently visit about the same time the problems arose, perhaps even to do something concerning online backing up of data. (There is data storage on more than one computer) Apparently he did not go near the win 7 machine.
Finally I know not very much about networks so please forgive me if the information I provide here is superfluous or deficient.
I have tried turning off antivirus on the win 7 machine, restarting etc but nothing seems to work.
It is planned to move our database from jet to sql server express in the future.
I need some suggestions as to the possible causes of this so that I can investigate it further. Any suggestions would be gretly appreciated
UPDATE 08/02/2011
The issue has been resolved by the hardware guy who visited the client today. The problem was that on this particular LAN the IP addresses were allocated dynamically except for the Win 7 machine which had a static IP address.
The static address happened to lie within the range from which the dynamic addresses were being selected. This wasn't a problem until last week when a dynamic address was generated that matched the static one and gave rise to the problems I described above.
Thanks to everyone for their input and thanks for not closing the question.
Having smart knowledgeable people to call on is a great help when you're under pressure from an unhappy customer and the gaps in your own knowledge mean that you can't confidently state that your software is definitely not to blame.
I'd try:
Validate that same DAO and ODBC-drivers is used on both xp- and vista machines.
Is LAN single broadcast domain? If not, rewire. (If routers required make
sure WINS is working)
Upgrade to ms-sql. It could be just a day of well worth work, ;-)
regards,
//t