I'm using Spring-Integration (File) 5.4.6 to listen for changes on a directory. As watcher, I'm using the setUseWatchService of the FileReadingMessageSource.
If a new file is available it will be picked up and processed. The service is working well on the local filesystem also with a large number of files. But if the service is listening on a network drive, the drive is mounted over SMB, not all files are picked up. There is no error message ... I assume that the filesystem events are not always triggered.
Can anybody confirm this? I've done a lot of tests ... everything is working well on the local filesystem ... the problem occurs only on a network drive. The service is running in a Windows 10 VM.
I'm not aware how network mounted drive works, but here is a Spring Integration extension for SMB protocol: https://github.com/spring-projects/spring-integration-extensions/tree/main/spring-integration-smb. Probably that one will help you somehow to implement your solution without relying on the filesystem simulation via network maunting.
Related
I have installed a service which purpose is to listen incoming files in a mapped network drive. For now, it does not respond to any new incoming file in the mapped network drive.
The service is just a jar file which can be run using java -jar. When I run using java -jar, it works.
Does anyone have an idea what to do to make it work in a service way?
Thanks!!
Your service is most likely running under a different user account (probably the Local System Account) so it won't see the mapped drive. Try running it under the same account as the logged-in user.
I have NiFi instance 1.3.0 running on my local Windows. I want to copy a file from a one network drive to HDFS.
I think getFile processor reads files inside share directory
//server/shared/path/to/file.
Normally my shared drive prompts for credentials before access folder.
I tried getFile processor for access shared drive without credentials it worked.
But it not worked if credentials asked for network drive.
Can anyone suggest way to access network drive folder with credentials and move it into HDFS using NiFi processors?
Let's say I have a shared folder on my machine, for example : \MyMachine\foo.
If I try to access it on the "MyMachine" computer, will it be smart enough to know that it is on the same machine and won't try to use the network adapter and be super slow? I tested it by disabling my network adapter and it was still working, but it might have been because I disabled it that it checked locally instead of using the network adapter.
My concern for not wanting to use the network adapter is speed. Large amount of data needs to be acceded sometime by network path if used outside, but if it is getting acceded locally it has to be quickly.
Every UNC is resolved down to it's core protocol address. Assuming a standard network of TCP/IP, your UNC will be resolved to local machine and the IP will be the same as your loopback IP. The network adapter won't be needed in this case so nothing will be sent over the network. In short, communication will be via the loopback address which does not require network.
Extracted from:
https://serverfault.com/questions/411389/does-traffic-from-accessing-locally-shared-file-via-unc-path-still-go-through-th
It wont, because the changes are taking place in your computer.
However if someone else is trying to access that directory then your computer must send the directory's information to the other computer, thus using the network.
I setup a Failover FTP using a script service/application on our 3 node cluster. I have followed the following guide which seems to be fairly complete: http://support.microsoft.com/kb/974603
However the FTP site I've added which is linked to the storage for that service will not start. I get the following error: 0x80070490 Element Not Found. I think it may be related to this kbb, but I'm not sure: http://support.microsoft.com/kb/2720218
Failing over/moving the service around the 3 nodes seems to work fine (except the FTP doesn't start, and starting it manually fails). The IP, computer name, and 2 mount points for storage get moved successfully. The only way I can get it to start is to go into IIS on the owning node, remove the FTP site and set it up again. As soon as I fail it over to another node however, I'm back to the error.
I believe it has something to do with IIS not seeing the storage despite it being available. I've made the storage a prerequisite for the script so the storage must be online before the script tries to start the FTP site. Nevertheless, it doesn't work.
Summary: Windows 2008 R2 Cluster FTP Server is set to broadcast on the service IP. It's root directory is the a root drive of assigned storage in the cluster service. The other storage is a MP mounted underneath this drive. FTP site works fine on initial setup but fails when failing over with Element Not Found error. Seems to be related to disk not being available despite it existing as if you go to of of the other nodes without the disks, the FTP site in IIS has the red 'X' on it and attempting to start it gives the same error.
This was my fault for not setting up Offline files. Once I completed that it worked. Offline files requires two server restarts and I didn't want to go through that process without testing how the Clustered FTP would work (this Cluster is in Production use). Unfortunately, once the share hosting the IIS shared configuration goes offline it will NOT come back online until you recycle the Microsoft FTP Service (which is why offline files is required). I could have modified the script to perform a recycle in the StartFTPSVC function (instead of just checking if it was started and if not starting it).
I have a requirement where data on SQL server needs to be synced to WP7 devices through USB.
I initially thought of writing a WCF Lib., but it'll go online & sync even if device is connected to USB in local network (Where it's easy & cheaper to get data over local network rather than go to internet & get it).
I have coded isolated storage to save the data synced, but i need an efficient way of getting data from local network, while it's connected to local network.
My questions are:
Is there a way to request data from local network when connected to local network in WCF?
Does someone have a C# code i can use to write synchronization routines to usb over local network? (I would love to have the code, if anyone has this)
Is there a better way of achieving what I'm trying to do?
I'm hoping for some quick answers for this problem, any help is greatly appreciated.
Thanks.
My recommendation would be for your WCF API to have an DNS name pointing to an internal IP address, which is not available outside of the local network and in the routing it will get resolved through USB and another DNS name pointing to an external IP Address which is only available outside of the local network. Your application would try to connect using the local IP address first, which should go through the local network and if it fails it should retry the with external option.
You can use NetworkInterface.NetworkInterfaceType to detect a connection of type Ethernet.
Implementing a full sync protocol is complicated and not something I would recommend attempting unless you really have to.
If you do go down this route I'd recommend using item based message queues and then syncing the queues (and replaying as necessary) rather than trying to sync database tables. It'll depend on the requirements of your application and nature of the data though.
Further to both Murven and Matt's answers, I'd recommend looking into the Microsoft Sync Framework, which supports WP7 as a client, commonly uses WCF on the server, and has great support for a wide range of edge cases.
You can then, as per Matt's answer, watch for connectivity changes and, on USB connection, start a sync to a local IP.
When it comes to targeting mango, you can implement it as a ResourceIntensiveTask and it will run when USB is connected, even if the app isn't running.