I currently have the following problem and can't decide which way to go:
I have a local directory with subfolders and files and want to mirror and sync that with a remote directory on a server. The problem is that I don't have any direct access to the server itself. The only access point I have is a bunch of REST API calls such as: uploading a file, downloading a file, getting metadata of a file (including creation and change date) and getting a file/directory list
I have already spent some time to look for possible programs/implementations but none of those have really convinced me. Here are some of the possiblities I considered so far:
Use a Powershell or Python script and manually check each file and folder for changes. Schedule a task to call the script every x minutes/hours
Use the Microsoft Sync Framework (MSF) and implement a custom SyncProvider which handles the REST calls and translates it into MSF format. Here I can't really tell if it's feasable at all and how complex it would be
Use tools like Syncthing or similar, but I couldn't find something that supports a remote sync directory only accessible via REST calls, but as there are quite a lot of tool I might have missed some that do
I'm working under Windows 10 so the solution should run on Windows and preferably not require too many addition resources.
Furthermore the solution should be somewhat resilient to errors as the REST API calls seem to have a tendency to fail sometimes (roughly 1 in 10 calls fails)
Any ideas and suggestions are welcome :)
Related
I am looking for a simple way to send messages between a Winforms Application and a Windows Service. The service will be run under LocalSystem so will be able to install updates to my Winforms App. The app is run in very locked down environments where port will be blocked and the file system is not reliable enough to use it for logging. I have tried using Named Pipes but i could not get this to work. I want to keep it simple so was thinking of trying Memory Mapped Files?
I only want to pass simple strings back and forth between the app and service, e.g.
APP-> Service [Please download this file http... and place it here C:\Program Files...]
Service->APP [0% downloaded]
Service->APP [1% downloaded]
etc..
Service->APP [Update Complete/Failed]
I cant seem to find a good example of how this can be achieved? Is memory mapped files the best way to go? If so, where do i start?! I have been reading through this Post but i cannot seem to make sense of it, its been a long day! I want everything to be in memory, unlike in this example. Can anyone help?
we have a need to regularly provide large files to clients on a daily or weekly basis. Currently our process is this:
Internal process creates the file and places it in a specific folder
Our client connects via SFTP and downloads the file
This work well when the files are small. As they get bigger (50-100 GB in size), we keep getting network interruptions and internal disk space related issues.
What I'd like to see is the following:
Our internal process creates the file.
This file is copied to an intermediary service (similar to something like FileDropper).
Our client will download the file from this intermediary service.
I'd like to know if other people had similar issues and what possible solutions are in place. File Dropper works great for non-business related files but obviously I won't be putting client data on there. We also have an Office 365 subscription. I tried to see what I could use with that but I haven't found anything yet that would help solve this.
Any hints, suggestions or feedback is much appreciated!
Consider Amazon S3.
I have used it several times in the past and it is very reliable both for processing a lot of files and for processing large files
I need to deploy an application onto some Windows machines for purposes of data collection from a group of people (i.e. the application will be used to gather responses to a series of survey questions). The process is interactive, alternating between displays of text and images with specific timing requirements. I have put together a prototype application using HTML and JavaScript that implements the survey. However, there are some unique constraints on the deployment environment that have me stuck:
While the machine is Internet-connected, the client requires that the survey application must run fully local to the PC that it runs on. Therefore, sending the survey results to a remote server is not permissible. Obviously, saving to a local file from a Web browser is typically not permitted for security reasons.
Installation of applications onto the machines that will run the survey is not permitted.
The configuration of the machines is not known specifically a priori, but I can assume some recent version of Windows with IE8+.
The "no remote access" requirement was a late comer, and has thrown a wrench into the plan of just writing a simple Web application that could post results to an HTTP server. I'm now looking for the easiest way forward. Two main approaches come to mind:
Use a GUI framework that provides a control that can display HTML/JavaScript; running a full-blown application on the PC would allow me to save the results to the filesystem. I've never done this, but it seems like in this day and age it shouldn't be too difficult. This would allow me to reuse much of my existing prototype implementation, but I would need some way of transferring the results (which would be stored in a JavaScript data structure) outside of the Web control to where the rest of the application could access it.
Reimplement the entire application using some GUI framework (I've used PyQt successfully before, although not on Windows). This approach is obviously less desirable than #1 due to the lack of reuse. However, it may be necessary if #1 isn't feasible.
Any recommendations for the best way to go? Ideally, I'm looking for a solution that can be run in a "portable" manner from a USB thumbdrive or similar.
Have you looked at HTML Applications (HTA)? They work in IE5+ and can use Windows Scripting Host to write to local drives and UNC shares...
Maybe you can use a portable web server with a scripting language on the server side. http://code.google.com/p/mongoose/ Mongoose, for example, you can run PHP, CGI, etc. .. scripts. Then, simply create a script to save a file to your hard drive. And let the rest of the application in the same manner.
Use a script to start the web server, and perhaps a portable web browser like K-Meleon to start the application http://kmeleon.sourceforge.net/ This is highly configurable. Or start the system explorer to your localhost URL.
The only problem may be that the user has to modify the firewall for the first time you run the server?
I'm building a website (hosted on GoDaddy) that will display graphs and charts about data obtained via another website's API. The other website updates its data only once a day, so I only need to make an API call once a day to get up-to-date info. Here's my plan:
A Ruby script on my hosting server's filesystem does the API call, and saves relevant info to .txt files on that local filesystem.
Another Ruby script generates a whole bunch of PHP files based on a bunch of different slices and dices of the data in those .txt files. (The PHP files are what process user requests)
Both of the above scripts are run as cron jobs, once per day.
To rephrase my question: Where do server-side scripts, which are agnostic toward user requests, get placed on the website's file system, specifically if a user is never supposed to see them or even really know they exist? (Or am I mistaken in assuming a setup like /ruby/script.rb is bad practice because people could directly access that file via a web browser?)
Or is my whole approach to this fundamentally flawed? I should clarify that I'm doing this project to teach myself Ruby and learn more about web development.
Under /srv if placed by the sysadmin, in /usr/share/<appname> if packaged up.
When they're for your use only it's up to you. For commands that run on the command-line, I usually put them in ~/bin. Shared files used by them go in ~/lib, or you can create a subdirectory of bin for the lib files: ~/bin/lib.
Files/apps under development usually go in a ~/dev or ~/development directory.
We have an Oracle 10g forms application running on a Solaris OAS server, with the forms displaying in IE. Part of the application involves uploading and downloading files (Word docs and PDFs, mainly) from the PC to the OAS server, using Oracle's webutil utility.
The problem is with large files (anything over 25Megs or so), it takes a long time, sometimes many minutes. Uploading seems to work, even with large files. Downloading large files, though, will cause it to error out part way through the download.
I've been testing with a 189Meg file in our development system. Using WEBUTIL_FILE_TRANSFER.Client_To_DB (or Client_To_DB_with_Progress), the download would error out after about 24Megs. I switched to WEBUTIL_FILE_TRANSFER.URL_To_Client_With_Progress, and finally got the entire file to download, but it took 22 minutes. Doing without the progress bar got it down to 18 minutes, but that's still too long.
I can display files in the browser, and my test file displayed in about 5 seconds, but many files need to be downloaded for editing and then re-uploaded.
Any thoughts on how to accomplish this uploading and downloading faster? At this point, I'm open to almost any idea, whether it uses webutil or not. Solutions that are at least somewhat native to Oracle are preferred, but I'm opn to suggestions.
Thanks,
AndyDan
This may be totally out to lunch, but since you're looking for any thoughts that might help, here are mine.
First of all, I'm assuming that the actual editing of the files happens outside the browser, and that you're just looking for a better way to get the files back and forth.
In that case, one option I've used in the past is just to route around the web application using Apache, or any other vanilla web server you like. For downloading, create a unique file session token, remember it in the web application, and place a copy of the file, named with the token (e.g. <unique token>.doc), in a download directory visible to Apache. Then provide a link to the file that will be served via Apache.
For upload, you have a couple of options. One is to use the mechanism you've got, then when a file is uploaded, you just have to match on the token in the name to patch the file back into your archive. Alternately, you could create a very simple file upload form separate from your application that will upload the file to a temp directory via Apache, then route the user back into your application and provide the token in the URL HTTP GET-style or else in a cookie.
Before you go to all that trouble, you'll want to make sure that your vanilla web server will provide better upload and download speed and reliability than your current solution, but it should.
As an aside, I don't know whether the application server you're using provides HTTP compression, but if it does, you should make sure it's enabled and working. This is probably the best single thing you can do to increase transfer speed of large files, assuming they're fairly compressible. If your application server doesn't support it, then most any vanilla web server will.
I hope that helps.
I ended up using CLIENT_HOST to call an FTP command to download the files. My 189MB test file took 20-22 minutes to download using WEBUTIL_FILE_TRANSFER.URL_To_Client_With_Progress, and only about 20 seconds using FTP. It's not the best solution because it leaves the FTP password exposed on the PC temporarily, but only for as long as the download takes, and even then the user would have to know where to find it.
So, we're implementing this for now, and looking for a more secure but still performant long term solution.