I have a large file on window azure and I want to download and save it on my disk. The maximum time for each link on window azure is 60 minutes. If I dowload directly base on link, maybe it isn't enough time. How to download it?
Nathan, your question isn't very clear, but I suspect you are referring to the time allowed for a Shared Access Signature, and being concerned that the client might not download the file within the time allowed?
There are 2 scenarios here:
Once a storage transaction (ie. download file) which uses a SAS
begins, then the transfer will be able to continue past the
expiration of the SAS. It is only new requests which are
authenticated using the SAS and which will fail if they are
attempted past the expiration time on the SAS.
If the client has to resume the download (or is downloading in
blocks), then the client has to be smart enough to detect the failed
authentication after the SAS expires and then re-request a new SAS
from the issuer.
try using a download accelerator like flashgot or something similar ...
One option would be to download the file in pieces and reassemble it once you have the pieces. There are a couple of ways to do that.
If the blob was uploaded in multiple blocks, then you could download each block individually. This is supported directly in the client libraries, so if you can do this it's probably easier. You can also download the blocks in parallel to reduce the total time it takes to download.
You could use HTTP Range headers to get certain byte ranges. I don't believe this is supported in the clients, so you'd probably have to code it yourself. But it will work even if the blob was not uploaded in blocks. I think this could also be done in parallel, but I'm not sure.
Related
I'm having multiple downloads in a swf, downloading external data.
Can I limit the download speed of one download thread or prioritize one?
Flash will download them in the order you ask for them. If you need one first, then ask for that one first.
When running within a browser it is actually the browser that makes the download on behalf of the client container (the client merely requests that the browser does the download). A lot of browsers have restrictions on concurrent file downloads anyway, especially in the same domain, and that is not something you will be able to circumvent with the in-built classes.
My guess is that you could roll your own (or use an existing client as a base) using Socket instead of URLLoader. This would be a large amount of work though - the browser does a lot of free work for you (SSL, cookies from the current session, gzip, keep-alives etc).
I have a ton of video in this app I'm designing, and I want to come in under the 20meg limit for the initial download, but I would love to have the thing work for folks when they don't have access to WIFI/3G. I want to have the video stream in and download the first time it's viewed, but then be saved and play from the local file every time afterward.
Anybody done something like this before? Thanks!
A basic way for caching could be done as follows: When you want to see a video, first see if it is in the cache.If it is there already then load it from the cache. Otherwise download it and save it in the cache. You can use the video URI (or its hash) as the file-name when you save it in the storage. Implementing File operations won't be a big problem as there are lots of resources out there to help you. You might want to create a utility class for handling all the caching operations.
I'm tempted to include a ton of binary files right into an Azure role package so that it can become several hundred megabytes in size. Before I do that I have to ask - is there a limit to how big an Azure role package can be?
The limit I believe is 600 MB (http://social.msdn.microsoft.com/Forums/pl-PL/windowsazuremanagement/thread/65620bdb-3439-4ce0-8f17-561914d325ac). However as suggested by Travis, you should look into storing items like images, videos etc. in blob storage because even if you change a single image, you will need to redeploy your application.
Another thing you would need to consider with large packages is timeout errors when deploying your code through Windows Azure portal when you are specifying the package file source as a file on your computer. An alternative to that would be to upload this package file in blob storage before hand and specify the blob URL when creating a deployment through Windows Azure portal.
Hope this helps.
Thanks
Gaurav
A few hundred megabytes is not an issue on a role package for Azure. There are limitations for storage account sizes, which your packages needs to get dumped into if it is large enough of about 100gigs. Outside of the huge amount of time it would take to upload the package and spin up a new instance, it should be acceptable.
For large binary files, I would consider using a storage account instead of inside your deployment package. The larger the deployment package, the longer it takes to spin up a new instance - which already takes like 20 minutes now. Since the link indicates most of this storage need is for related COM data it might not be an easy option for you.
Possibly related size issue: if all these files you are putting into your upload payload appear within an IIS site, they will need to be processed by the IISConfigurator. If you have "too many of these" (not a hard and fast number, sorry, but more than 10,000), you may get timeout errors. The issue is discussed here.
I'm researching solutions for a potential client. They're requesting the ability to download a large amount of MP3's (1000+) from their online catalog.
I've researched/tested building a zip containing all MP3s using ZipArchive but ran into obvious memory leak issues that have ruled that solution out.
I'm now trying to think out of the box.
One idea was to create an FTP queue or a Torrent type download link for them. Is there anything out there that can pull something like this off?
Any help or suggested direction would be greatly appreciated! Thanks!!
Edit: Here is the overall process/goal that we're trying to achieve.
The client creates music for TV/Flim placement. They maintain a online catalog AND a local copy they send to potential buyers. The online catalog and the offline catalog need to mirror each other. Problem being, they have multiple offices that will have to update their local copy with the new files added to the online catalog from many different locations
Example: East Coast User updates catalog with 100 new files. West Coast User needs to update the offline catalog with the new files retrieved from the online catalog.
We had hoped to create custom zip's of the files each user needed to update their catalog based on the user's download history that we'd maintain in MySQL. We were testing ZipArchive but we couldn't seem to build Zips over 175 MEG (give or take). We're in the process of testing ZipStreaming but are having some issues.
I hope this clears up the overall goal and problems we are facing.
GNU wget?
It can download recursive. Just give wget a list of all files on the server, e.G.
http://www.example.org/filelist.html which contains links like file1.mp3, file2.mp3 etc (apache normally generates such an index file automatically wenn a directory without index.html/php in it gets called.
http://linux.die.net/man/1/wget
Frankly speaking, I can't identify the actual problem/question from your post. If you are looking for minimizing network load, then you need to remember that MP3 files are not compressed well because they are already compressed (not as well as possible, but well). If you are looking for a transport, than any file transfer protocol will do (FTP, SFTP, HTTP, WebDAV).
If you need flexibility and features, I'd recommend SFTP: this is a protocol for remote file system access, so besides "get file" operation it has plenty of useful operations including machine-readable directory listing (not always available in FTP and not available in standard HTTP), built-in ZLib compression, built-in possibility to resume file transfers and more bonuses. HTTP also has ZLib compression, but this one is not always available.
Update: your approach doesn't care about what is really available on the client and you are going to prepare ZIP files based on your (possibly incorrect) knowledge of the client already has.
If the client and server are both applications that you develop, then you should use RSync protocol or something similar to update data online (not using any ZIP files) and download the files that are missing on the client. If direct communication between the client and the server is not possible, you can make the client send his state to the server and the server will prepare an individual package after that. As for ZIP functionality - it's needed only when you use batch update (no real-time communication between the client and the server). I don't know what technology you are using but if your only problem is with ZIP component, you can use something else for data packing - either different ZIP component (for .NET and VCL we have ZIP component) or some other packing solution (for example, our SolFS product doesn't have size limits). Unfortunately I am not aware of RSync-like implementation available as a component.
We have an Oracle 10g forms application running on a Solaris OAS server, with the forms displaying in IE. Part of the application involves uploading and downloading files (Word docs and PDFs, mainly) from the PC to the OAS server, using Oracle's webutil utility.
The problem is with large files (anything over 25Megs or so), it takes a long time, sometimes many minutes. Uploading seems to work, even with large files. Downloading large files, though, will cause it to error out part way through the download.
I've been testing with a 189Meg file in our development system. Using WEBUTIL_FILE_TRANSFER.Client_To_DB (or Client_To_DB_with_Progress), the download would error out after about 24Megs. I switched to WEBUTIL_FILE_TRANSFER.URL_To_Client_With_Progress, and finally got the entire file to download, but it took 22 minutes. Doing without the progress bar got it down to 18 minutes, but that's still too long.
I can display files in the browser, and my test file displayed in about 5 seconds, but many files need to be downloaded for editing and then re-uploaded.
Any thoughts on how to accomplish this uploading and downloading faster? At this point, I'm open to almost any idea, whether it uses webutil or not. Solutions that are at least somewhat native to Oracle are preferred, but I'm opn to suggestions.
Thanks,
AndyDan
This may be totally out to lunch, but since you're looking for any thoughts that might help, here are mine.
First of all, I'm assuming that the actual editing of the files happens outside the browser, and that you're just looking for a better way to get the files back and forth.
In that case, one option I've used in the past is just to route around the web application using Apache, or any other vanilla web server you like. For downloading, create a unique file session token, remember it in the web application, and place a copy of the file, named with the token (e.g. <unique token>.doc), in a download directory visible to Apache. Then provide a link to the file that will be served via Apache.
For upload, you have a couple of options. One is to use the mechanism you've got, then when a file is uploaded, you just have to match on the token in the name to patch the file back into your archive. Alternately, you could create a very simple file upload form separate from your application that will upload the file to a temp directory via Apache, then route the user back into your application and provide the token in the URL HTTP GET-style or else in a cookie.
Before you go to all that trouble, you'll want to make sure that your vanilla web server will provide better upload and download speed and reliability than your current solution, but it should.
As an aside, I don't know whether the application server you're using provides HTTP compression, but if it does, you should make sure it's enabled and working. This is probably the best single thing you can do to increase transfer speed of large files, assuming they're fairly compressible. If your application server doesn't support it, then most any vanilla web server will.
I hope that helps.
I ended up using CLIENT_HOST to call an FTP command to download the files. My 189MB test file took 20-22 minutes to download using WEBUTIL_FILE_TRANSFER.URL_To_Client_With_Progress, and only about 20 seconds using FTP. It's not the best solution because it leaves the FTP password exposed on the PC temporarily, but only for as long as the download takes, and even then the user would have to know where to find it.
So, we're implementing this for now, and looking for a more secure but still performant long term solution.