I am trying to upload a .mp4 file, selected from the user's iOS or Android device, to my Azure Media Services account.
This code works for small files ( less than ~95MB):
public static async Task<string> UploadBlob(string blobContainerSasUri, string blobName, byte[] blobContent, string path)
{
string responseString;
int contentLength = blobContent.Length;
string queryString = (new Uri(blobContainerSasUri)).Query;
string blobContainerUri = blobContainerSasUri.Split('?')[0];
string requestUri = string.Format(System.Globalization.CultureInfo.InvariantCulture, "{0}/{1}{2}", blobContainerUri, blobName, queryString);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(requestUri);
request.Method = "PUT";
request.AllowWriteStreamBuffering = false;
request.Headers.Add("x-ms-blob-type", "BlockBlob");
request.ContentLength = contentLength;
request.Timeout = Int32.MaxValue;
request.KeepAlive = true;
int bufferLength = 1048576; //upload 1MB at time, useful for a simple progress bar.
Stream requestStream = request.GetRequestStream();
requestStream.WriteTimeout = Int32.MaxValue;
ProgressViewModel progressViewModel = App.Locator.GetProgressBar(App.Locator.MainViewModel.currentModuleItemId);
MyVideosPage myVideosPage = App.Locator.GetVideosPage(App.Locator.MainViewModel.currentModuleItemId);
FileStream fileStream = new FileStream(path, FileMode.Open, FileAccess.Read);
int nRead = 0;
int currentPos = 0;
while ((nRead = fileStream.Read(blobContent, currentPos, bufferLength)) > 0)
{
await requestStream.WriteAsync(blobContent, currentPos, nRead);
currentPos += nRead;
}
fileStream.Close();
requestStream.Close();
HttpWebResponse objHttpWebResponse = null;
try
{
// this is where it fails for large files
objHttpWebResponse = (HttpWebResponse)request.GetResponse();
Stream responseStream = objHttpWebResponse.GetResponseStream();
StreamReader stream = new StreamReader(responseStream);
responseString = stream.ReadToEnd();
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
finally
{
if (objHttpWebResponse != null)
objHttpWebResponse.Close();
}
return responseString;
}
An exception is thrown after this line is called:
(HttpWebResponse)request.GetResponse();
The exception message is "The request body is too large and exceeds the maximum permissible limit."
The exception StatusCode is "RequestEntityTooLarge".
How can I upload large files? Is this a problem with HttpWebRequest, or Azure Media Services?
Azure Storage supports one shot upload (aka PutBlob API) up to 256MB if you are using the new REST API versions. But since you are not specifying the REST API version, you're defaulting to a very old version where the maximum supported size of one shot upload is 100MB.
Use x-ms-version: 2018-03-28 header to be able to upload up to 256MB in one HTTP request.
If you have to deal with larger files, you will need to use block & commit upload. You will need to use PutBlock API to stage blocks from the source file. Blocks can be up to 100MB each. Then you need to commit all the blocks using the PutBlockList API. If you don't have to deal with this logic yourself, simply use the Azure Storage SDK for .NET (supports Xamarin) and use the uploadFromFile method. It is simple, and resilient.
Related
I am hoping there is someone here who is familiar with a Real Estate data standard known as RETS. The National Association of Realtors provides a dll for interfacing with their services called libRETS, but it is not being supported like it once was and recent events have prompted us to create our own as a replacement. For logistics reasons, we can't do this in Core and are using the current C#.Net 4.7.2.
There are 2 or 3 different "security levels" for connecting to a RETS Server, with the method being a per case basis from one MLS to the next. We can successfully connect to those who only require a login and password, but are hitting a wall on those who also require what is called a UserAgent and UserAgentPassword, which must passed somehow using Md5 encryption. The server is returning:
The remote server returned an error: (401) Unauthorized.
private WebResponse GetLoginBasicResponse()//*** THIS ONE WORKS ***
{
try
{
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
var request = (HttpWebRequest)WebRequest.Create(new Uri(_cred.loginUri));
request.Method = "GET";
request.Headers.Add("RETS-Version", _retsVersion);
request.Credentials = new NetworkCredential(_login, _password);
return request.GetResponse();
}
catch (Exception ex)
{
string ignore = ex.Message;
return null;
}
}
private WebResponse GetLoginWithUserAgentResponse()//*** THIS ONE DOES NOT WORK ***
{
try
{
// ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
var request = (HttpWebRequest)WebRequest.Create(new Uri(_cred.loginUri));
request.Method = "GET";
request.Headers.Add("RETS-Version", _retsVersion);
if (!string.IsNullOrEmpty(_cred.userAgent))
{
request.UserAgent = Md5(_cred.userAgent + ":" + _cred.userAgentPassword);
//request.Headers.Add("RETS-UA-Authorization", "Digest " + Md5(_cred.userAgent + ":" + _cred.userAgentPassword));
}
request.Credentials = new NetworkCredential(_login, _password);
return request.GetResponse();
}
catch (Exception ex)
{
string ignore = ex.Message;
return null;
}
}
public string Md5(string input) //*** Borrowed this from from .NET Core Project and presume it works
{
// Use input string to calculate MD5 hash
using (System.Security.Cryptography.MD5 md5 = System.Security.Cryptography.MD5.Create())
{
byte[] inputBytes = Encoding.ASCII.GetBytes(input);
byte[] hashBytes = md5.ComputeHash(inputBytes);
// Convert the byte array to hexadecimal string
StringBuilder sb = new StringBuilder();
for (int i = 0; i < hashBytes.Length; i++)
{
sb.Append(hashBytes[i].ToString("X2"));
}
return sb.ToString();
}
}
Page 20 of this document describes how to build the UA header: https://www.ranww.org/documents/resources/rets_1_8.pdf
There’s a few other fields you need to include.
We were not able to solve the issue in .NET but found a .NET Core project in GitHub that we are using instead. https://github.com/CrestApps/RetsConnector
This case can be closed
Not seeing an option to "Mark as Answer". Have tried both MS Edge and Google Chrome
I've searched all documents in google drive api and I can't able to find how to upload a file to folderid using REST APi. Can anyone please help me on this?
public void UploadFiletoDrive()
{
var gmodel = GetAccessToken();
WebRequest request = WebRequest.Create("https://www.googleapis.com/upload/drive/v3/files/?uploadType=media");
request.Method = "POST";
request.Headers["Authorization"] = "Bearer " + gmodel.access_token;
request.ContentType = "image/jpeg";
Stream dataStream = request.GetRequestStream();
FileStream filestream = new FileStream(#"C:\Users\Developer\Downloads\unnamed (2).jpg", FileMode.Open, FileAccess.Read);
byte[] buffer = new byte[4096];
int bytesRead = 0;
while ((bytesRead = filestream.Read(buffer, 0, buffer.Length)) != 0)
{
dataStream.Write(buffer, 0, bytesRead);
}
filestream.Close();
dataStream.Close();
WebResponse response = request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string responseFromServer = reader.ReadToEnd();
reader.Close();
response.Close();
}
It seems you've missed the Work with Folders docs.
Inserting a file in a folder using Java:
String folderId = "0BwwA4oUTeiV1TGRPeTVjaWRDY1E";
File fileMetadata = new File();
fileMetadata.setName("photo.jpg");
fileMetadata.setParents(Collections.singletonList(folderId));
java.io.File filePath = new java.io.File("files/photo.jpg");
FileContent mediaContent = new FileContent("image/jpeg", filePath);
File file = driveService.files().create(fileMetadata, mediaContent)
.setFields("id, parents")
.execute();
System.out.println("File ID: " + file.getId());
Implementation for other languages are also included like PHP, Python, NodeJS.
Also, check this SO thread for additional reference.
body.setParents(Arrays.asList(new ParentReference().setId(folderId)));
Your sample code is doing a media upload, ie. no metadata, You should be using a multipart upload so you can specify both metadata such as parent folder id and content.
Uploading a file to google drive using REST API has following steps.
Get parent folderID using list API
Create file with parent="folder ID" using create api and get "fileId" in response
upload file to "fileId
Following is javascript code to upload file using REST API
const url = 'https://www.googleapis.com/upload/drive/v3/files/' + fileId + '?uploadType=media';
if(self.fetch){
// console.log("Fetch found, Using fetch");
var setHeaders = new Headers();
setHeaders.append('Authorization', 'Bearer ' + authToken.access_token);
setHeaders.append('Content-Type', mime);
var setOptions = {
method: 'PATCH',
headers: setHeaders,
body: blob
};
fetch(url,setOptions)
.then(response => { if(response.ok){
// console.log("save to google using fetch");
}
else{
// console.log("Response wast not ok");
}
})
.catch(error => {
// console.log("There is an error " + error.message);
});
}
I am using .NET to list the payments from my square account.
I am able to get a list of the payments, but to get the description field I have to go one level deeper and make http end point calls for each payment. This is time consuming.
Question: Can anyone provide me with a sample in Visual C# or Java to make batch calls for retrieving payments (using multiple payment id's)?
Your help is greatly appreciated.
Thanks,
Prashant
#Andrew - Here's what I am using, I am just not sure how to add the headers for batch payments retrieval.
string res = string.Empty;
string qs = string.Empty;
foreach (string s in parameters.Keys)
{
if (qs == string.Empty)
qs = "?";
else
qs += "&";
qs += s + "=" + parameters[s];
}
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(_connectUrl + "/" + command + qs); ///
request.Proxy = null;
request.Headers.Add("Authorization", "Bearer " + _accessToken);// ");
request.ContentType = "application/json";
request.Method = method; // "GET";
try { HttpWebResponse responseGet = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(responseGet.GetResponseStream());
StringBuilder output = new StringBuilder();
output.Append(reader.ReadToEnd());
responseGet.Close();
request = null;
return output.ToString();
}
catch (Exception exp)
Looks like I've been able to answer my own query.
We need to be able to send the following POST to the HTTP Endpoint
{"requests":[{"method":"GET","relative_path":"/v1/me/payments/<payment_id>","access_token":"XXXX","request_id":"1"},{"method":"GET","relative_path":"/v1/me/payments/<payment_id>","access_token":"XXXX","request_id":"2"}]}
the following code in .NET achieves the above
//Convert the body of request into a byte array
byte[] byteArray = Encoding.UTF8.GetBytes(body);
//Set the length
request.ContentLength = byteArray.Length;
//Write the body to the request by using a datastream
//This line never returns....
Stream datastream = request.GetRequestStream();
datastream.Write(byteArray, 0, byteArray.Length);
datastream.Close();
And that's all there is to it.
Hope this helps anyone is is set out to use the batch mode.
Thanks
When I use the the ObjectContent object to create the HttpContent to send a request via HttpClient to a Web API service I am getting the following error:
Cannot write more bytes to the buffer than the configured maximum buffer size: 65536
The following code is being used to send the request. The Card object has about 15 properties.
var client = new HttpClient();
var content = new ObjectContent<IEnumerable<Card>>(cards, "application/xml");
MessageBox.Show(content.ReadAsString()); //This line gives me the same error.
var response = client.Post("http://localhost:9767/api/cards", content);
How do I change the configured size to something greater than 65,536?
Since the problem resides in the ReadAsString extension method I would suggest that you create your own extension method to solve the maximum buffer size issue.
Here’s an example of a ReadAsLargeString extension method that maybe solves the problem.
public static string ReadAsLargeString(this HttpContent content)
{
var bufferedContent = new MemoryStream();
content.CopyTo(bufferedContent);
if (bufferedContent.Length == 0)
{
return string.Empty;
}
Encoding encoding = DefaultStringEncoding;
if ((content.Headers.ContentType != null) && (content.Headers.ContentType.CharSet != null))
{
encoding = Encoding.GetEncoding(content.Headers.ContentType.CharSet);
}
return encoding.GetString(bufferedContent.GetBuffer(), 0, (int)bufferedContent.Length);
}
There is a thread about this. Try using HttpCompletionOption.ResponseContentRead:
var message = new HttpRequestMessage(HttpMethod.Post, "http://localhost:9767/api/cards");
message.Content = content;
var client = new HttpClient();
client.Send(message, HttpCompletionOption.ResponseContentRead);
Try this for the client:
HttpClient client = new HttpClient("http://localhost:52046/");
// enable support for content up to 10 MB size
HttpClientChannel channel = new HttpClientChannel() {
MaxRequestContentBufferSize = 1024 * 1024 * 10
};
client.Channel = channel;
On the Server (snippet is based preview 4 but you should get the clue):
public class CustomServiceHostFactory : HttpConfigurableServiceHostFactory {
public override ServiceHostBase CreateServiceHost(string constructorString, Uri[] baseAddresses) {
var host = base.CreateServiceHost(constructorString, baseAddresses);
foreach (HttpEndpoint endpoint in host.Description.Endpoints) {
endpoint.TransferMode = TransferMode.Streamed;
endpoint.MaxReceivedMessageSize = 1024 * 1024 * 10;
}
return host;
}
}
I am using C# and a console app and I am using this script to download files from a remote server. There area a couple of things I want to add. First, when it writes to a file, it doesn't take into consideration a newline. This seems to run a certain amount of bytes and then goes to a newline. I would like it to keep the same format as the file it is reading from. Second, there are multiple .jpg files on the server that I need to download. How can I use this script to download multiple, .jpg files
public static int DownLoadFiles(String remoteUrl, String localFile)
{
int bytesProcessed = 0;
// Assign values to these objects here so that they can
// be referenced in the finally block
StreamReader remoteStream = null;
StreamWriter localStream = null;
WebResponse response = null;
// Use a try/catch/finally block as both the WebRequest and Stream
// classes throw exceptions upon error
try
{
// Create a request for the specified remote file name
WebRequest request = WebRequest.Create(remoteUrl);
request.PreAuthenticate = true;
NetworkCredential credentials = new NetworkCredential("id", "pass");
request.Credentials = credentials;
if (request != null)
{
// Send the request to the server and retrieve the
// WebResponse object
response = request.GetResponse();
if (response != null)
{
// Once the WebResponse object has been retrieved,
// get the stream object associated with the response's data
remoteStream = new StreamReader(response.GetResponseStream());
// Create the local file
localStream = new StreamWriter(File.Create(localFile));
// Allocate a 1k buffer
char[] buffer = new char[1024];
int bytesRead;
// Simple do/while loop to read from stream until
// no bytes are returned
do
{
// Read data (up to 1k) from the stream
bytesRead = remoteStream.Read(buffer, 0, buffer.Length);
// Write the data to the local file
localStream.WriteLine(buffer, 0, bytesRead);
// Increment total bytes processed
bytesProcessed += bytesRead;
} while (bytesRead > 0);
}
}
}
catch (Exception e)
{
Console.WriteLine(e.Message);
}
finally
{
// Close the response and streams objects here
// to make sure they're closed even if an exception
// is thrown at some point
if (response != null) response.Close();
if (remoteStream != null) remoteStream.Close();
if (localStream != null) localStream.Close();
}
// Return total bytes processed to caller.
return bytesProcessed;
Why don't you use WebClient.DownloadData or WebClient.DownloadFile instead?
WebClient client = new WebClient();
client.Credentials = new NetworkCredentials("id", "pass");
client.DownloadFile(remoteUrl, localFile);
By the way the correct way to copy a stream to another is not what you did. You shouldn't read into char[] at all, as you might run into encoding and end of line issues as you are downloading a binary file. The WriteLine method call is problematic too. The right way to copy contents of a stream to another is:
void CopyStream(Stream destination, Stream source) {
int count;
byte[] buffer = new byte[BUFFER_SIZE];
while( (count = source.Read(buffer, 0, buffer.Length)) > 0)
destination.Write(buffer, 0, count);
}
The WebClient class is much easier to use and I suggest using that instead.
The reason you're getting spurious newlines in the result file is because StreamWriter.WriteLine() puts them there. Try using StreamWriter.Write() instead.
Regarding downloading multiple files, can't you just run the function several times, passing it the URLs of the different files you need?