I need upload some big files(about 1Gb) into google drive.
I using google-api-client(ruby) version 0.5.0:
media = Google::APIClient::UploadIO.new(file_name, mimeType, original_name)
result = client.execute!(
:api_method => client.service.files.insert,
:body_object => file,
:media => media,
:parameters => {
'uploadType' => 'resumable',
'alt' => 'json'})
I expected that my client split big file on parts and upload these parts on drive.
But I see in logs, that client sending only ONE BIG chunk to drive.
Here is small log example:
Content-Length: "132447559"
Content-Range: "bytes 0-132447558/132447559"
How can I upload big files by chunks with google-api-client?
The intended usage is to try and upload the file in a single chunk. Overall, it's more efficient/faster that way. But there are cases where chunking is preferable, so if you need to chunk the upload for whatever reason, just set the chunk_size property:
media = Google::APIClient::UploadIO.new(file_name, mimeType, original_name)
media.chunk_size = 1000000 # 1mb chunks
result = client.execute!(....)
I'm using the API version 0.7.1, even though I know we're supposed to be using version 0.9 now, because the older version matches the Ruby examples on Google's documentation.
I had to do resort to uploading in chunks because I was getting errors in httpclient library complaining about the file size being too large to convert to integer!
Unfortunately, using #stevebazyl did not work for me as it only uploads the first chunk and then throws a TransmissionError. This seems to be in the google-api-ruby-client code, specifically, Google::APIClient class in the execute! method. It doesn't seem to be handling an HTTP status of 308, which is what a resumable upload returns when it needs the next chunk. I did this to the code:
when 200...300, 308
result
(See api_client.rb)
And use the #send_all method in the ResumableUpload class just like the sample code in the docs and it worked for me. So in addition to #stevebazyl code, I have:
media = Google::APIClient::UploadIO.new(opts[:file], 'video/*')
media.chunk_size = 499200000
videos_insert_response = client.execute!(
:api_method => youtube.videos.insert,
:body_object => body,
:media => media,
:parameters => {
:uploadType => 'resumable',
:part => body.keys.join(',')
}
)
videos_insert_response.resumable_upload.send_all(client)
Related
I'm prototyping a Rails application to upload documents to FSCrawler (running the REST interface), to incorporate into an Elasticsearch index. Using their example, this works:
response = `curl -F "file=##{params[:document][:upload].tempfile.path}" "http://127.0.0.1:8080/fscrawler/_upload?debug=true"`
The file gets uploaded, and the content gets indexed. This is an example of what I get:
"{\n \"ok\" : true,\n \"filename\" : \"RackMultipart20200130-91061-16swulg.pdf\",\n \"url\" : \"http://127.0.0.1:9200/local/_doc/d661edecf3e28572676e97a6f0d1d\",\n \"doc\" : {\n \"content\" : \"\\n \\n \\n\\nBasically, what you need to know is that Dante is all IP-based, and makes use of common IT standards. Each Dante device behaves \\n\\nmuch like any other network device you would already find on your network. \\n\\nIn order to make integration into an existing network easy, here are some of the things that Dante does: \\n\\n▪ Dante...
When I run curl at the command line, I get EVERYTHING, like the "filename" being properly set. If I use it as above, in the Rails controller, as you can see, the filename is set to the Tempfile's filename. That's not a workable solution. Trying to use params[:document][:upload].tempfile (without .path) or just params[:document][:upload] both fail entirely.
I'm trying to do this "the right way," but every incarnation of using a proper HTTP client to do this fails. I can't figure out how to invoke an HTTP POST that will submit a file to FSCrawler the way curl (on the command line) does it.
In this example, I'm just trying to send the file by using the Tempfile file object. For some reason, FSCrawler gives me the error in the comment, and get a little metadata, but no content is indexed:
## Failed to extract [100000] characters of text for ...
## org.apache.tika.exception.ZeroByteFileException: InputStream must have > 0 bytes
uri = URI("http://127.0.0.1:8080/fscrawler/_upload?debug=true")
request = Net::HTTP::Post.new(uri)
form_data = [['file', params[:document][:upload].tempfile,
{ filename: params[:document][:upload].original_filename,
content_type: params[:document][:upload].content_type }]]
request.set_form form_data, 'multipart/form-data'
response = Net::HTTP.start(uri.hostname, uri.port) do |http|
http.request(request)
end
If I change the above to use params[:document][:upload].tempfile.path, then I don't get the error about the InputStream, but I also (still) do not get any content indexed. This is an example of what I get:
{"_index":"local","_type":"_doc","_id":"72c9ecf2a83440994eb87d28786e6","_version":3,"_seq_no":26,"_primary_term":1,"found":true,"_source":{"content":"/var/folders/bn/pcc1h8p16tl534pw__fdz2sw0000gn/T/RackMultipart20200130-91061-134tcxn.pdf\n","meta":{},"file":{"extension":"pdf","content_type":"text/plain; charset=ISO-8859-1","indexing_date":"2020-01-30T15:33:45.481+0000","filename":"Similarity in Postgres and Rails using Trigrams · pganalyze.pdf"},"path":{"virtual":"Similarity in Postgres and Rails using Trigrams · pganalyze.pdf","real":"Similarity in Postgres and Rails using Trigrams · pganalyze.pdf"}}}
If I try to use RestClient, and I try send the file by referencing the actual path to the Tempfile, then I get this error message, and I get nothing:
## Unsupported media type
response = RestClient.post 'http://127.0.0.1:8080/fscrawler/_upload?debug=true',
file: params[:document][:upload].tempfile.path,
content_type: params[:document][:upload].content_type
If I try to .read() the file, and submit that, then I break the FSCrawler form:
## Internal server error
request = RestClient::Request.new(
:method => :post,
:url => 'http://127.0.0.1:8080/fscrawler/_upload?debug=true',
:payload => {
:multipart => true,
:file => File.read(params[:document][:upload].tempfile),
:content_type => params[:document][:upload].content_type
})
response = request.execute
Obviously, I've been trying this every way I can, but I can't replicate whatever curl is doing with any known Ruby-based HTTP clients. I'm utterly lost as to how to get Ruby to submit data to FSCrawler in a way that will get the document contents indexed properly. I've been at this far longer than I care to admit. What am I missing here?
I finally tried Faraday, and, based on this answer, came up with the following:
connection = Faraday.new('http://127.0.0.1:8080') do |f|
f.request :multipart
f.request :url_encoded
f.adapter :net_http
end
file = Faraday::UploadIO.new(
params[:document][:upload].tempfile.path,
params[:document][:upload].content_type,
params[:document][:upload].original_filename
)
payload = { :file => file }
response = connection.post('/fscrawler/_upload', payload)
Using Fiddler helped me to see the results of my attempts, as I got closer and closer to the curl request. This snippet posts the request almost exactly as curl does. To route this call through the proxy, I just needed to add , proxy: 'http://localhost:8866' to the end of the connection setup.
I am developing a web player of sorts, I'm using the php framework Laravel to handle the data of the playlist. I create an array of the playlist with all the necessary information. With this array I make a howl instance of the playlist object when it needs to be played.
This works fluidly on Firefox & Chrome, both on desktop as on mobile. However I'm encountering issues when testing on Safari or iOS browsers.
What happens: The audio plays normally however at around 1-2 minutes into the song it loops back on itself to about 20-45secs ago. This creates a really annoying song where it's just repeating the same part of the song until it ends. Which it does. Because despite this looping back the app still continues ticking up the seconds of the song. (sound.seek() keeps ticking up.)
Looking at the network tab I've noticed something odd, whereas the other browsers only fetch the audio source once, Safari does this multiple times. This is about the only tangible change I've noticed.
Since I don't have 10 rep image goes here: https://imgur.com/Y48J52g
What the oddest part is that a locally hosted version doesn't have issues either. So is this a webserver issue? Browser? I'm at a loss.
The onloaderror and onplayerror events also don't fire either, so no issues there as far as I know.
Instancing the howl:
sound = data.howl = new Howl({
src: ['./get-audio' + data.file],
html5: true,
//After this I instance all onX functions (onplay, onend, etc)
...
sound.play()
Then whenever I need the next song I unload this howl instance and create the next one.
Most of my code is adjusted from the HowlerJS example of their 'player' in case you'd like to delve deeper in the code itself.
How the audio gets served:
public function getAudio($map, $name)
{
$fileName = $map.'/'.$name;
$file = Storage::disk('local')->get($fileName);
$filesize = Storage::disk('local')->size($fileName);
$size = $filesize;
$length = $size;
$start = 0;
$end = $size - 1;
return response($file)
->withHeaders([
'Accept-Ranges' => "bytes",
'Accept-Encoding' => "gzip, deflate",
'Pragma' => 'public',
'Expires' => '0',
'Cache-Control' => 'must-revalidate',
'Content-Transfer-Encoding' => 'binary',
'Content-Disposition' => ' inline; filename='.$name,
'Content-Length' => $filesize,
'Content-Type' => "audio/mpeg",
'Connection' => "Keep-Alive",
'Content-Range' => 'bytes 0-'.$end .'/'.$size,
'X-Pad' => 'avoid browser bug',
'Etag' => $name,
]);
}
So I'm not sure why Safari/iOS has an issue with the hosted version whilst locally it does work.
This is my first question on this site, so if you'd like some more information let me know.
I found out the issue.
Namely Safari thought I was serving an audio stream rather than just an mp3 file, causing it to continuously send requests. I solved this by serving my audio like this:
$path = storage_path().DIRECTORY_SEPARATOR."app".DIRECTORY_SEPARATOR."songs".DIRECTORY_SEPARATOR.$name;
$response = new BinaryFileResponse($path);
BinaryFileResponse::trustXSendfileTypeHeader();
return $response;
I am using CodeIgniter to upload some files through a form. I know how to translate "regular" form errors (required, valid_email, etc) but I don't know how to do it with file errors (file is not allowed, file is too big, etc).
Which keys should I use in
$this->form_validation->set_message('KEY', 'TRANSLATION');
?
Thanks
METHOD 1
By default, language files are typically stored in system/language directory. Alternately you can create a file called upload_lang.php inside your application/language folder and store them there.
$lang['upload_userfile_not_set'] = "Unable to find a post variable called userfile.";
$lang['upload_file_exceeds_limit'] = "The uploaded file exceeds the maximum allowed size in your PHP configuration file.";
$lang['upload_file_exceeds_form_limit'] = "The uploaded file exceeds the maximum size allowed by the submission form.";
$lang['upload_file_partial'] = "The file was only partially uploaded.";
$lang['upload_no_temp_directory'] = "The temporary folder is missing.";
$lang['upload_unable_to_write_file'] = "The file could not be written to disk.";
$lang['upload_stopped_by_extension'] = "The file upload was stopped by extension.";
$lang['upload_no_file_selected'] = "You did not select a file to upload.";
$lang['upload_invalid_filetype'] = "The filetype you are attempting to upload is not allowed.";
$lang['upload_invalid_filesize'] = "The file you are attempting to upload is larger than the permitted size.";
$lang['upload_invalid_dimensions'] = "The image you are attempting to upload exceedes the maximum height or width.";
$lang['upload_destination_error'] = "A problem was encountered while attempting to move the uploaded file to the final destination.";
$lang['upload_no_filepath'] = "The upload path does not appear to be valid.";
$lang['upload_no_file_types'] = "You have not specified any allowed file types.";
$lang['upload_bad_filename'] = "The file name you submitted already exists on the server.";
$lang['upload_not_writable'] = "The upload destination folder does not appear to be writable.";
METHOD 2
You can use Flashdata after upload process.
if ($this->upload->do_upload())
{
$this->session->set_flashdata('success', 'Yep! Upload complete');
redirect('go_back_home');
}
else
{
$this->session->set_flashdata('error', 'Uh, upload not complete! The problem is..'); // Manually check
redirect('go_back_form');
}
You can also check the problem with $this->upload->data(). This is a helper function that returns an array containing all of the data related to the file you uploaded. Here is the array prototype :
Array
(
[file_name] => mypic.jpg
[file_type] => image/jpeg
[file_path] => /path/to/your/upload/
[full_path] => /path/to/your/upload/jpg.jpg
[raw_name] => mypic
[orig_name] => mypic.jpg
[client_name] => mypic.jpg
[file_ext] => .jpg
[file_size] => 22.2
[is_image] => 1
[image_width] => 800
[image_height] => 600
[image_type] => jpeg
[image_size_str] => width="800" height="200"
)
Try this:
lang:your_translation_key
A real example:
$this->form_validation->set_rules('name', 'lang:web_name', 'required|trim|xss_clean|min_length[2]|max_length[100]');
Has anybody succeeded in sending standalone attachments using the standalone attachment API (if possible gziped) to couchDB from ruby?
I know there are a few CURL examples, but my tries with Typhoeus haven't been successful so far. It usually just stops and waits for > 1 minute after the first few documents.
CouchRest doesn't seem to support it, neither do any of the other libraries I've looked at
edit: clarification
I'm NOT looking for regular Base64 encoded attachments. CouchRest does that pretty fine.
Got it working with typhoeus
Typhoeus::Request.put("http://127.0.0.1:5984/db/document/my_attachment_name?rev=#{rev}", "content-type" => "text/html", "Content-Encoding" => "gzip", "Accept-Encoding" => "gzip", :body => my_html_body)
this will store the "my_html_body" string into couchDB as a gziped standalone attachment
For a binary standalone attachment, I just used IO.read("/path/to/my/file") to give a string to the put method as the :body. It looks like it is working but I don't know if it is the right way to do it.
It looks like this:
res = Typhoeus::Request.get("http://localhost:5984/_uuids")
uuid = JSON.parse(res.body)["uuids"].first
doc = {}
doc["name"] = name
...
res = Typhoeus::Request.put("http://localhost:5984/products/#{uuid}", :body => JSON.generate(doc))
res = Typhoeus::Request.put("http://localhost:5984/products/#{uuid}/image.jpg?rev=#{rev}", :headers => {"Content-Type" => "image/jpeg" }, :body => IO.read("output/images/#{image}"))
I simply want to upload an image to a server with POST. As simple as this task sounds, there seems to be no simple solution in Ruby.
In my application I am using WWW::Mechanize for most things so I wanted to use it for this too, and had a source like this:
f = File.new(filename, File::RDWR)
reply = agent.post(
'http://rest-test.heroku.com',
{
:pict => f,
:function => 'picture2',
:username => #username,
:password => #password,
:pict_to => 0,
:pict_type => 0
}
)
f.close
This results in a totally garbage-ready file on the server that looks scrambled all over:
alt text http://imagehub.org/f/1tk8/garbage.png
My next step was to downgrade WWW::Mechanize to version 0.8.5. This worked until I tried to run it, which failed with an error like "Module not found in hpricot_scan.so". Using the Dependency Walker tool I could find out that hpricot_scan.so needed msvcrt-ruby18.dll. Yet after I put that .dll into my Ruby/bin-folder it gave me an empty error box from where on I couldn't debug very much further. So the problem here is that Mechanize 0.8.5 has a dependency on Hpricot instead of Nokogiri (which works flawlessly).
The next idea was to use a different gem, so I tried using Net::HTTP. After short research I could find out that there is no native support for multipart forms in Net::HTTP and instead you have to build a class that encodes etc. for you. The most helpful I could find was the Multipart-class by Stanislav Vitvitskiy. This class looked good so far, but it does not do what I need, because I don't want to post only files, I also want to post normal data, and that is not possible with his class.
My last attempt was to use RestClient. This looked promising, as there have been examples on how to upload files. Yet I can't get it to post the form as multipart.
f = File.new(filename, File::RDWR)
reply = RestClient.post(
'http://rest-test.heroku.com',
:pict => f,
:function => 'picture2',
:username => #username,
:password => #password,
:pict_to => 0,
:pict_type => 0
)
f.close
I am using http://rest-test.heroku.com which sends back the request to debug if it is sent correctly, and I always get this back:
POST http://rest-test.heroku.com/ with a 101 byte payload,
content type application/x-www-form-urlencoded
{
"pict" => "#<File:0x30d30c4>",
"username" => "s1kx",
"pict_to" => "0",
"function" => "picture2",
"pict_type" => "0",
"password" => "password"
}
This clearly shows that it does not use multipart/form-data as content-type but the standard application/x-www-form-urlencoded, although it definitely sees that pict is a file.
How can I upload a file in Ruby to a multipart form without implementing the whole encoding and data aligning myself?
Long problem, short answer: I was missing the binary mode for reading the image under Windows.
f = File.new(filename, File::RDWR)
had to be
f = File.new(filename, "rb")
Another method is to use Bash and Curl. I used this method when I wanted to test multiple file uploads.
bash_command = 'curl -v -F "file=#texas.png,texas_reversed.png"
http://localhost:9292/fog_upload/upload'
command_result = `#{bash_command}` # the backticks are important <br/>
puts command_result