We have found various solutions around upload the file to google cloud bucket from local system. However I am wondering if is there a way we can upload file to bucket using the public URL or link.
https://googleapis.dev/ruby/google-cloud-storage/latest/index.html
I want to upload a file from remote url to GCS bucket via ruby code. Any suggestion here would be really appreciated.
Your code sits between the remote URL and the Google Cloud Storage (GCS) Bucket.
You've 2 alternatives:
(As you describe) Download the file behind the remote URL to a file system accessible to your code and then upload it to GCS;
Stream the file from the remote location into memory (you'll need to write this) and then (using GCS client library) stream the file into a GCS object.
You tagged question with ruby-on-rails-3
Old rails versions use uploaders like carrierwave
It's possible to use it to upload files to GCS
You can upload not only local files using this gem but also from remote URL, just use special attribute
Related
Am somewhat new to using Amazon S3 for image storage and just wondering the easiest way to import about about 5,000 (currently publicly available) Image URLs into S3 so that the end result is hosting of the images on our S3 Account (and resulting New URLs of the same images).
Am wondering if there is a need to first download and save each image on my computer, and then to import the resulting Images to S3 (which would of course result in new URLs for those images)? Or is there an easier to accomplish this.
Thanks for any suggestions.
James
The AWS CLI for S3 doesn’t support copying a file from a remote URL to S3, but you can copy from a local filestream to S3.
Happily, cURL outputs the contents of URLs as streams by default, so the following command will stream a remote URL to a location in S3 via the local machine:
# The "-" means "stream from stdin".
curl https://example.com/some/file.ext | aws s3 cp - s3://bucket/path/file.ext
Caveat: while this doesn’t require a local temporary file, it does ship all of the file bytes through the machine where the command is running as a relay to S3.
Add whatever other options for the S3 cp command that you need, such as --acl.
Threaded multipart transfers with chunk management will probably be more performant but this is a quick solution in a pinch if you have access to the CLI.
S3 is a “dumb” object storage, meaning that it cannot do any data processing itself, including but not limited to downloading files. So yes, you’ll have to dowload your objects first and then upload them to S3.
I already have PlayFramework app runing , but I am in process of migrating it to Heroku. Because on Heroku I can not use local filesystem like I did in my app. I am forced to use Amazon S3 ,but I do not know how to rewrite creating of thumbnails. For that I am using :
https://github.com/coobird/thumbnailator
Thumbnails.of(picture.getFile()).size(300,300).toFile(new File("public/images/data", thumb));
The problem is that I can not do this at heroku,because file wont be saved.
How do I create thumbnails then? If I do not want to use another service which will generate thumbnails for me and save them to s3 somehow...
Honestly if I would know how many different services would I need for simple page with java then I would stay at php forever...
In Heroku (as in many PaaS) there's no persistent filesystem. However, you do have access to temp directory.
So you could save it into a temp file
File temp = File.createTempFile("prefix", "suffix").getAbsolutePath;
Thumbnails.of(picture.getFile()).size(300,300).toFile(temp, thumb));
Then take that file and upload it to S3.
If you strictly insist on not using S3 for storing binary files, then you could base64 the file content and save it into the DB. (see some pros/cons for such approach here)
Our company is using Ruby 2.1.3 with AWS SDK V1 for uploading files on S3. I need to stream files directly from a private external bucket to one of our personal bucket (without actually downloading them locally). I can't find any good documentation on the subject.
The copy_from method provided by the SDK, I think, does not permit streaming from a private external bucket to one of our bucket.
We have tried using open-uri to stream the download and stream the upload to s3 but the file was always downloaded fully first and then uploaded (is it supposed to be like that?).
Any help is welcomed!
Thank you.
The V1 SDK doesn't allow you to transfer between buckets directly. You can do what open-uri does and download the file and then upload to the new bucket.
If you want a solution that can still work in Ruby I suggest using the AWS CLI. You can add a line like this to your code:
`aws s3 cp s3://frombucket/ s3://tobucket/`
The backticks allow you to execute system commands in your ruby script. Alternatively you could upgrade to the V2 SDK and use the object.copy_to command and copy between buckets. Hope this helps!
I want to try AppHarbor, but I have an application which stores uploaded files in certain place on a filesystem. Is it compatible with AppHarbor? Can I store files in the file system and access them later?
(what kind of path can I expect, like c:\blabla something or what?)
Thank you.
You can store files on the local filesystem, but the application directory is wiped on each new deployment so it's not recommended to rely on for file storage.
Instead we recommend that you use a cloud storage service such as Amazon S3, Google Cloud Storage or similar. There are .NET libraries for both services.
We recently wrote a blog post about uploading files directly to S3 and GCS from the browser that you might want to read.
If you are using a background worker, you need to 'Enable File System Write Access' in the settings of you application.
Then, you are permitted access to write to: Path.GetTempPath()
Sourced from this support question: http://support.appharbor.com/discussions/problems/5868-create-directory-in-background-worker
I have an app in Heroku, I need simple file storage for uploaded images for this I
used send_data using attachment_fu plugin.
After that I have used the tmp/ directory to write this file and want to display on browser, But these files are not displayed in browser.
How can I display these images on browser?
What is the alternate solution to store and retrieve images?
Thanks!
You cannot store uploaded files on Heroku.
You must use an alternative strategy. A couple alternative strategies:
Store uploaded files in your database on Heroku. Because database systems are not specifically optimized for storing files, you should test out how well your application performs under load before you use this strategy.
Store uploaded files in an external storage service. Amazon S3 is a popular cloud storage service that anyone can use, and it has pay-as-you-go pricing just like Heroku. (In fact, Heroku itself runs on top of another of Amazon's cloud services, EC2.)