I want to write a file in gcs with a stream object but I've only found the "create_file" function that creates a new file object by providing a path to a local file to upload and the path to store it with in the bucket.
Is there any function to create a file in gcs from a stream?
Fuse over GCS
You could try gcsfuse which layers a user-space fs over a bucket but it is only beta s/ware at present. There's a nice section on limitations which you should read first.
I use fog to access GCS but that is a thin layer which doesn't try to impose any additional semantics into the bucket/object model.
Warning, if your problem really requires a standard file-system underneath any possible solution then GCS is not a good fit.
The ability to provide an IO object instead of a File object has only recently been possible. It was added in PR 1335, and will be included in the next release.
Until then, quickest way is to write the stream to a tempfile and upload that. For more see Issue 305.
Related
My goal is to download a large zip file (15 GB) and extract it to Google Cloud using Laravel Storage (https://laravel.com/docs/8.x/filesystem) and https://github.com/spatie/laravel-google-cloud-storage.
My "wish" is to sort of stream the file to Cloud Storage, so I do not need to store the file locally on my server (because it is running in multiple instances, and I want to have the disk size as small as possible).
Currently, there does not seem to be a way to do this without having to save the zip file on the server. Which is not ideal in my situation.
Another idea is to use a Google Cloud Function (eg with Python) to download, extract and store the file. However, it seems like Google Cloud Functions are limited to a max timeout of 9 mins (540 seconds). I don't think that will be enough time to download and extract 15GB...
Any ideas on how to approach this?
You should be able to use streams for uploading big files. Here’s the example code to achieve it:
$disk = Storage::disk('gcs');
$disk->put($destFile, fopen($sourceZipFile, 'r+'));
I'm creating a web application using flask that takes 3 input from the user: name, picture, grades.
I want to store these information temporary depending on the user's session.
and as a beginner I read that sessions are not for storing files, what other secure way you recommend me to use?
I would recommend to write the files to disk.
If this is really temporary, e.g. you have a two-step-sign-up-form, you could write the files to temporary files or into a temporary directory.
Please see the excellent documentation at https://docs.python.org/3/library/tempfile.html
Maybe this should not be this temporary? It sounds like a user picture is something more permanent.
Then I would recommend e.g. to create a directory for each user and store the files there.
This is done with standard Python io, e.g with the open function.
More info about reading and writing files also can be found in the official Python documentation:
https://docs.python.org/3/tutorial/inputoutput.html#reading-and-writing-files
I have a third-party executable that takes a directory path as an argument and in turn looks there for a collection of .db files. I have said collection of files stored in a Google Cloud Storage bucket and would like to stream the content of those files into some local named pipes that can be used as input to the executable.
I'm writing an application to perform the above in Go and am using the "cloud.google.com/go/storage" package to work with cloud storage objects.
As a note, I need all pipes/files to be available for reading at the time I run the executable.
What is the best way to go about this? I'm looking to essentially used the named pipe as a proxy of sorts to make remote files look local to this executable. Possible?
I saved a file to storage using:
$request->file('avatar')->store('avatars');
Which saved it to:
storage/app/avatars/avatar.png
How can I get the path to this file/folder (not the URL)? What is the correct way to do this using Laravel's Filesystem?
There is no correct way to do this; because it should not be done. The Storage is an opaque system to talk to different storage systems; as such there is no api to get the backing file path. As an example, that wouldn't work with Amazon S3. The only path your application knows about is the string you send to the Storage facade to work with the file, there are no guarantees that this string is used to generate the filename when the storage system stores the file.
There are some hacks you can use that works for the local disk, but those are not available for the Storage system in general. Using these solutions means that you'll limit yourself to only use the local disk; this will cause you troubles when you need to scale out and add another server. You'll then have two servers with two separate local disks, with separate content.
The correct way to work with the files, that will work for all configurations, is to get the file content (Storage::get), do the modifications (including storing them in a temporary file) and then write back the new file content (Storage::set).
If you're really sure that you will only ever use the local filesystem, use the File facade instead of the Storage facade. I'm unable to find any documentation for this, only the interface it exposes.
Reference: https://github.com/laravel/framework/issues/13610
Try this
storage_path('app/avatars/avatar.png');
you can only get the storage folder path from laravel function, you can give nested folder name after it, it will bind the base url as well
storage_path(folder1/folder2/.../file.png);
I would like to modify a Spotlight metadata attribute of a file within my application (i.e. not in a Spotlight importer) but I can't find any API for doing so. Is it possible? Pointers to the relevant docs would be ideal.
In case it's helpful, here's my use case:
I want to store a reference to a file
system path in a Core Data store.
Ideally, I should be able to find the
file even if is moved, potentially
across mounted volumes. My understanding is that
an archived FSRef or AliasRecord will not do the trick because they
are not invariant wrt to moves across mounted
volumes. So my plan was to store a URL
and also to add a UUID (also stored in
the data store) to the file's
Spotlight metadata so that I could
perform a Spotlight query for that
UUID if the URL no longer pointed to
the file when the app goes back to
look for the file.
After further research, using Spotlight is not the best solution for the use case. The AliasRecord is a better persistent storage for a file. It automatically tracks moves/renames/etc. You can read more about AliasRecords here. Chris Hansen has written an Objective-C wrapper for AliasRecords, BDAlias. It's currently available from the rentzsch.com SVN.