I want to upload data binary directly to GCP storage, without writing the file to disk. Below is the code snippet I have created to get to the state that I am going to be at.
require 'google/cloud/storage'
bucket_name = '-----'
data = File.open('image_block.jpg', 'rb') {|file| file.read }
storage = Google::Cloud::Storage.new("project_id": "maybe-i-will-tell-u")
bucket = storage.bucket bucket_name, skip_lookup: true
Now I want to directly put this data into a file on gcs, without having to write a file to disk.
Is there an efficient way we can do that?
I tried the following code
to_send = StringIO.new(data).read
bucket.create_file to_send, "image_inder_11111.jpg"
but this throws an error saying
/google/cloud/storage/bucket.rb:2898:in `file?': path name contains null byte (ArgumentError)
from /home/inder/.gem/gems/google-cloud-storage-1.36.1/lib/google/cloud/storage/bucket.rb:2898:in `ensure_io_or_file_exists!'
from /home/inder/.gem/gems/google-cloud-storage-1.36.1/lib/google/cloud/storage/bucket.rb:1566:in `create_file'
from champa.rb:14:in `<main>'
As suggested by #stefan, It should be to_send = StringIO.new(data), i.e. without .read (which would return a string again)
Related
I checked the whole azure-storage-blob gem and didn't find any way to get the URI for a blob. Is there some way to construct it correctly and in a generic way that will work for any other blob in any region?
I used S3 SDK before and I'm well grounded in S3 but new to Azure.
There is a protected method called blob_uri that looks like this:
def blob_uri(container_name, blob_name, query = {}, options = {})
if container_name.nil? || container_name.empty?
path = blob_name
else
path = ::File.join(container_name, blob_name)
end
options = { encode: true }.merge(options)
generate_uri(path, query, options)
end
So you could take the short cut of:
blob_client = Azure::Storage::Blob::BlobService.create(storage_account_name: 'XXX' , storage_access_key: 'XXX')
blob_client.send(:blob_uri, container_name,blob_name)
However, the actual URI is simply:
https://[storage_account_name].blob.core.windows.net/container/[container[s]]/[blob file name]
So since you have to know the blob name and the container to access to blob.
File.join(blob_client.host,container,blob_name)
Is the URI to the blob
I'm having difficulty to Encrypt large files (bigger than available memory) using GPGME in Ruby.
#!/usr/bin/ruby
require 'gpgme'
def gpgfile(localfile)
crypto = GPGME::Crypto.new
filebasename = File.basename(localfile)
filecripted = crypto.encrypt File.read(localfile), :recipients => "info#address.com", :always_trust => true
File.open("#{localfile}.gpg", 'w') { |file| file.write(filecripted) }
end
gpgpfile("/home/largefile.data")
In this case I got an error of memory allocation:
"read: failed to allocate memory (NoMemoryError)"
Someone can explain me how to read the source file chunk by chunk (of 100Mb for example) and write them passing by the crypting?
The most obvious problem is that you're reading the entire file into memory with File.read(localfile). The Crypto#encrypt method will take an IO object as its input, so instead of File.read(localfile) (which returns the contents of the file as a string) you can pass it a File object. Likewise, you can give an IO object as the :output option, letting you write the output directly to a file instead of in memory:
def gpgfile(localfile)
infile = File.open(localfile, 'r')
outfile = File.open("#{localfile}.gpg", 'w')
crypto = GPGME::Crypto.new
crypto.encrypt(infile, recipients: "info#address.com",
output: outfile,
always_trust: true)
ensure
infile.close
outfile.close
end
I've never used ruby-gpgme, so I'm not 100% sure this will solve your problem since it depends a bit on what ruby-gpgme does behind the scenes, but from the docs and the source I've peeked at it seems like a sanely-built gem so I'm guessing this will do the trick.
Been messing around with Kickbox's api for email verification. I'm trying to have the program only display the result object in the returned JSON.
Here's the code:
require "kickbox"
require 'httparty'
require 'json'
client = Kickbox::Client.new('ac748asdfwef2fbf0e8177786233a6906cd3dcaa')
kickbox = client.kickbox()
response = kickbox.verify("test#easdfwf.com")
file = File.read(response)
json = JSON.parse(file)
json['result']
I'm getting an error verify.rb:10:in read': no implicit conversion of Kickbox::HttpClient::Response into String (TypeError)
from verify.rb:10:in'
Here's a sample response:
{
"result":"undeliverable",
"reason":"rejected_email",
"role":false,
"free":false,
"disposable":false,
"accept_all":false,
"did_you_mean":"bill.lumbergh#gmail.com",
"sendex":0,
"email":"bill.lumbergh#gamil.com",
"user":"bill.lumbergh",
"domain":"gamil.com",
"success":true,
"message":null
}
You are getting this error:
read': no implicit conversion of Kickbox::HttpClient::Response into String (TypeError)
Because, in this line:
file = File.read(response)
Your response is a Kickbox::HttpClient::Response type object, but the File.read is expecting a String object instead (possibly a file name with path).
I'm not sure what you are trying to do, but this: file = File.read(response) is wrong. You can't do this and that's why you are getting the mentioned error.
If you really want to use file, then you can write the response to a file and then read the response back from the file and use that:
f = File.new('response.txt', 'w+') # creating a file in read/write mode
f.write(response) # writing the response into that file
file_content = File.read('response.txt') # reading the response back from the file
So, the issue is not about Accessing a 3rd party API JSON object in ruby, but you are trying to use File.read in a wrong way.
You can get the response from the API by doing this:
client = Kickbox::Client.new('YOUR_API_KEY')
kickbox = client.kickbox()
response = kickbox.verify("test#easdfwf.com")
Then, you can play with the response e.g. can do a puts response.inspect or puts response.body.inspect and see what's inside that object.
And, from there you can extract your required outputs only.
I'm capturing/creating user entered text into files from my app, attempting to temporarily store them in my Heroku tmp directory, then upload them to a cloud service such as Google Drive.
In using Tempfile I can successfully upload, but when using File.open I get the following error when attempting to upload:
ArgumentError (wrong number of arguments (1 for 0))
The error is on the call:
#client.upload_file_by_folder_id(save_path, #folder_id)
Where #client is a session with the cloud service, save_path is the location of the attached file for upload and #folder_id is the folder they should go into.
When I use Tempfile.new I am successful in doing so:
tempfile = Tempfile.new([final_filename, '.txt'], Rails.root.join('tmp','text-temp'))
tempfile.binmode
tempfile.write msgbody
tempfile.close
save_path = tempfile.path
upload_file = #client.upload_file_by_folder_id(save_path, #folder_id)
tempfile.unlink
File.open code is:
path = 'tmp/text-temp'
filename = "#{final_filename}.txt"
save_path = Rails.root.join(path, filename)
File.open(save_path, 'wb') do |file|
file.write(msgbody)
file.close
end
upload_file = #client.upload_file_by_folder_id(save_path, #folder_id)
File.delete(save_path)
Could it be that the File.path is a string, and Tempfile.path is the full path (but not as a string)? When I put out each, they look identical.
I'd like to use File as I don't want to change the filename of the existing attachments I'm uploading, whereas Tempfile appends to the filename.
Any and all assistance is greatly appreciated. Thanks!
In order for it to work using File, I needed to set the save_path to a string:
save_path.to_s
I am trying to save compressed strings to a file and load them later for use in the game. I kept getting "in 'finish': buffer error" errors when loading the data back up for use. I came up with this:
require "zlib"
def deflate(string)
zipper = Zlib::Deflate.new
data = zipper.deflate(string, Zlib::FINISH)
end
def inflate(string)
zstream = Zlib::Inflate.new
buf = zstream.inflate(string)
zstream.finish
zstream.close
buf
end
setting = ["nothing","nada","nope"]
taggedskills = ["nothing","nada","nope","nuhuh"]
File.open('testzip.txt','wb') do |w|
w.write(deflate("hello world")+"\n")
w.write(deflate("goodbye world")+"\n")
w.write(deflate("etc")+"\n")
w.write(deflate("etc")+"\n")
w.write(deflate("Setting: name "+setting[0]+" set"+(setting[1].class == String ? "str" : "num")+" "+setting[1].to_s)+"\n")
w.write(deflate("Taggedskill: "+taggedskills[0]+" "+taggedskills[1]+" "+taggedskills[2]+" "+taggedskills[3])+"\n")
w.write(deflate("etc")+"\n")
end
File.open('testzip.txt','rb') do |file|
file.each do |line|
p inflate(line)
end
end
It was throwing errors at the "Taggedskill:" point. I don't know what it is, but trying to change it to "Skilltag:", "Skillt:", etc. continues to throw a buffer error, while things like "Setting:" or "Thing:" work fine, while changing the setting line to "Taggedskill:" continues to work fine. What is going on here?
In testzip.txt, you are storing newline separated binary blobs. However, binary blobs may contain newlines by themselves, so when you open testzip.txt and split it by line, you may end up splitting one binary blob that inflate would understand, into two binary blobs that it does not understand.
Try to run wc -l testzip.txt after you get the error. You'll see the file contains one more line, than the number of lines you are putting in.
What you need to do, is compress the whole file at once, not line by line.