Can someone tell me what I am doing wrong with trying to upload an image to blob storage? Below is my code.
print(type(img['image'])) #Output is <class 'bytes'>
connection_string = get_blob_connection_string()
blob_service_client = BlobServiceClient.from_connection_string(connection_string)
blob_client = blob_service_client.get_blob_client(container="images", blob=img['id'])
exists = blob_client.exists()
if (exists == False):
result = blob_client.upload_blob(img['image'], blob_type="blockblob")
print(result)
When inserting the blob, it throws the error
quote_from_bytes() expected bytes
This error makes no sense, I gave it bytes. What am I missing?
After reproducing from my end, I have received the same issue. You are receiving this error because of incompatible type of the file (i.e., file format).
After changing the below line to the correct format I could able to achieve your requirement.
blob_client = blob_service_client.get_blob_client(container="images", blob=img['id'])
Below is the correct format
blob_client=blob_service_client.get_blob_client(container='container', blob='<LOCAL FILE PATH>');
Below is the complete code that worked for me
from azure.storage.blob import BlobServiceClient
from PIL import Image
connection_string = "<CONNECTION STRING>"
blob_service_client = BlobServiceClient.from_connection_string(connection_string)
blob_client=blob_service_client.get_blob_client(container='container', blob='<LOCALPATH>');
with open(file='<PATH IN YOUR STORAGE ACCOUNT WITH FILE NAME>', mode="rb") as data:
blob_client.upload_blob(data)
RESULTS:
Related
I successfully built logic app where whenever a blob is added in container-one, it gets copied to container-2. However it fails when any blobs larger than 50 MB (default size) is uploaded.
Could you please guide.
Blobs are added via rest api.
Below is the flow,
Currently, the maximum file size with disabled chunking is 50MB. One of the workarounds is to use Azure functions in order to transfer the files from one container to another.
Below is the sample Python Code that worked for me when I'm trying to transfer files from One container to Another
from azure.storage.blob import BlobClient, BlobServiceClient
from azure.storage.blob import ResourceTypes, AccountSasPermissions
from azure.storage.blob import generate_account_sas
from datetime import datetime,timedelta
connection_string = '<Your Connection String>'
account_key = '<Your Account Key>'
source_container_name = 'container1'
blob_name = 'samplepdf.pdf'
destination_container_name = 'container2'
# Create client
client = BlobServiceClient.from_connection_string(connection_string)
# Create sas token for blob
sas_token = generate_account_sas(
account_name = client.account_name,
account_key = account_key,
resource_types = ResourceTypes(object=True),
permission= AccountSasPermissions(read=True),
expiry = datetime.utcnow() + timedelta(hours=4)
)
# Create blob client for source blob
source_blob = BlobClient(
client.url,
container_name = source_container_name,
blob_name = blob_name,
credential = sas_token
)
# Create new blob and start copy operation
new_blob = client.get_blob_client(destination_container_name, blob_name)
new_blob.start_copy_from_url(source_blob.url)
RESULT:
REFERENCES:
General Limits
How to copy a blob from one container to another container using Azure Blob storage SDK
I want to upload data binary directly to GCP storage, without writing the file to disk. Below is the code snippet I have created to get to the state that I am going to be at.
require 'google/cloud/storage'
bucket_name = '-----'
data = File.open('image_block.jpg', 'rb') {|file| file.read }
storage = Google::Cloud::Storage.new("project_id": "maybe-i-will-tell-u")
bucket = storage.bucket bucket_name, skip_lookup: true
Now I want to directly put this data into a file on gcs, without having to write a file to disk.
Is there an efficient way we can do that?
I tried the following code
to_send = StringIO.new(data).read
bucket.create_file to_send, "image_inder_11111.jpg"
but this throws an error saying
/google/cloud/storage/bucket.rb:2898:in `file?': path name contains null byte (ArgumentError)
from /home/inder/.gem/gems/google-cloud-storage-1.36.1/lib/google/cloud/storage/bucket.rb:2898:in `ensure_io_or_file_exists!'
from /home/inder/.gem/gems/google-cloud-storage-1.36.1/lib/google/cloud/storage/bucket.rb:1566:in `create_file'
from champa.rb:14:in `<main>'
As suggested by #stefan, It should be to_send = StringIO.new(data), i.e. without .read (which would return a string again)
I have been trying to upload an image in chunks with client side streaming using grpcurl. The service is working without error except that at the server, image data received is 0 bytes.
The command I am using is:
grpcurl -proto image_service.proto -v -d # -plaintext localhost:3010 imageservice.ImageService.UploadImage < out
This link mentions that the chunk data should be base64 encode and so the contents of my out file are:
{"chunk_data": "<base64 encoded image data>"}
This is exactly what I am trying to achieve, but using grpcurl.
Please tell what is wrong in my command and what is the best way to achieve streaming via grpcurl.
I have 2 more questions:
Does gRPC handles the splitting of data into chunks?
How can I first send a meta-data chunk (ImageInfo type) and then the actual image data via grpcurl?
Here is my proto file:
syntax = "proto3";
package imageservice;
import "google/protobuf/wrappers.proto";
option go_package = "...";
service ImageService {
rpc UploadImage(stream UploadImageRequest) returns (UploadImageResponse) {}
}
message UploadImageRequest {
oneof data {
ImageInfo info = 1;
bytes chunk_data = 3;
};
}
message ImageInfo {
string unique_id = 1;
string image_type = 2;
}
message UploadImageResponse {
string url = 1;
}
Interesting question. I've not tried streaming messages with (the excellent) grpcurl.
The documentation does not explain how to do this but this issue shows how to stream using stdin.
I recommend you try it that way first to ensure that works for you.
If it does, then bundling various messages into a file (out) should also work.
Your follow-on questions suggest you're doing this incorrectly.
chunk_data is the result of having split the file into chunks; i.e. each of these base64-encoded strings should be a subset of your overall image file (i.e. a chunk).
your first message should be { "info": "...." }, subsequent messages will be { "chunk_data": "<base64-encoded chunk>" } until EOF.
I have a webpage where the user can upload a PDF file and send it using AJAX to my Flask application. Here is the ajax code:
var formData = new FormData();
formData.append('attachment', document.getElementById("attachment").files[0]);
$.ajax({
type: 'POST',
url: 'process_award_storage',
contentType: false,
processData: false,
data: formData
})
I do not think that this has any problems because I can print out the content and title of file in the python code. Then my flask model is defined as such, using LargeBinary for the PDF attachment:
class AwardStore(db.Model):
__tablename__ = 'awards_store'
id = db.Column(db.Integer, primary_key=True)
...
file = db.Column(db.LargeBinary, nullable=True) #I am sure this is the right file type for PDF saving
Lastly, here is how the AJAX file is saved to the database:
award = AwardStore(name=name, file=request.files['attachment'].read())
db.session.add(award)
db.session.commit()
I can see using my MySQL Workbench that it is saved. However, when I try to download the BLOB from there to the Desktop and open it, it says "Failed to load PDF document". The same happens when I use flask's send_file. It seems like I did everything as I saw online, but something is wrong.
Maybe these warnings are related?:
C:\Users\msolonko\Desktop\NICKFI~1\CACHTM~1\APP_DE~1\virt\lib\site-packages\pymysql\cursors.py:170: Warning: (1300, "Invalid utf8 character string: 'C4E5F2'")
result = self._query(query)
C:\Users\msolonko\Desktop\NICKFI~1\CACHTM~1\APP_DE~1\virt\lib\site-packages\pymysql\cursors.py:170: Warning: (1265, "Data truncated for column 'file' at row 1")
result = self._query(query)
I tried googling them and did not find anything. I appreciate any assistance.
EDIT:
I noticed that small files are typically uploaded and displayed properly. The issue is with files with sizes > ~50kB. The database I am using the AWS RDS, Is there a setting I can change somewhere to enable greater sizes?
The LargeBinary accepts a length field also.
which converts to mysql's BLOB whose default length is 65535 bytes (64kb).
Try increasing the length and set it to
a MEDIUMBLOB for 16777215 bytes (16 MB)
a LONGBLOB for 4294967295 bytes (4 GB).
Hope this will help
I am trying to send a jpeg image file through a ZeroMQ connection with pyzmq, but the output is 3 times the size of the input, and no longer a valid jpeg. I load the image and send with...
f = open("test1.jpg",'rb')
strng = f.read()
socket.send(strng)
f.close()
I receive and save with...
message = socket.recv()
f = open("test2.jpg", 'w')
f.write(str(message))
f.close()
I am new to zmq, and I could not find any info on sending images. Has anyone sent images through ZeroMQ, or have any ideas on how to find the problem?
Before sending the file you can "base64" encode it and decode it when received.
Sending:
import base64
f = open("test1.jpg",'rb')
bytes = bytearray(f.read())
strng = base64.b64encode(bytes)
socket.send(strng)
f.close()
Receiving:
import base64
message = socket.recv()
f = open("test2.jpg", 'wb')
ba = bytearray(base64.b64decode(message))
f.write(ba)
f.close()
You can try imagezmq. It's specially built for transporting images using PyZMQ messaging.
Sender
import socket
import imagezmq
sender = imagezmq.ImageSender(connect_to='tcp://receiver_name:5555')
sender_name = socket.gethostname() # send your hostname with each image
image = open("test1.jpg",'rb')
sender.send_image(sender_name, image)
Receiver
import imagezmq
image_hub = imagezmq.ImageHub()
sender_name, image = image_hub.recv_image()
image_hub.send_reply(b'OK')
Zero-copy string manipulations on C-strings
( from enter link description here )
Bytes and Strings
Note
If you are using Python >= 2.6, to prepare your PyZMQ code for Python3 you should use the b'message' syntax to ensure all your string literal messages will still be bytes after you make the upgrade.
The most cumbersome part of PyZMQ compatibility from a user’s perspective is the fact that, since ØMQ uses C-strings, and would like to do so without copying, we must use the Py3k bytes object