I'm using the export Google Drive API to retrieve a Google Doc as Pdf: https://developers.google.com/drive/v3/reference/files/export
I'm having the following problem: for documents bigger than a certain size (I don't know exactly the threshold, but it happens even with relatively small files around 1,5 MB) the API return a 200 response code with a blank result (normally it should contains the pdf data as byte stream), as you can see in the following screenshot:
I can successfully export the file via GoogleDrive/GoogleDoc UI with the "File -> Download as.. -> Pdf" command, despite it takes a bit of time.
Here is the file used for test (1.180 KB exported from Google Doc), I shared it so you can access to try export:
https://docs.google.com/document/d/18Cz7kHfEiDLeTWHyyoOi6U4kFQDMeg0D-CCJzILMMCk/edit?usp=sharing
Here is the (Java) code I'm using to perform the operation:
#Override
public GoogleDriveDocumentContent downloadFileContentAsPDF(String executionGoogleUser, String fileId) {
GoogleDriveDocumentContent documentContent = new GoogleDriveDocumentContent();
String conversionMimeType = "application/pdf";
try {
getLogger().info("GDrive APIs - Downloading file content in PDF format ...");
InputStream gDriveFileData = getDriveService(executionGoogleUser).files()
.export(fileId, conversionMimeType)
.executeMediaAsInputStream();
getLogger().info("GDrive APIs - File content as PDF format downloaded.");
documentContent.setFileName(null);
documentContent.setMimeType(conversionMimeType);
documentContent.setData(gDriveFileData);
} catch (IOException e) {
throw new RuntimeException(e);
}
return documentContent;
}
Does anyone has the same issue and know how to solve it?
The goal is to generate a pdf from a Google Doc.
Thanks
I think you should try using media downloadeder you will have to alter it for Google drive rather than storage service.
{
// Create the service using the client credentials.
var storageService = new StorageService(new BaseClientService.Initializer()
{
HttpClientInitializer = credential,
ApplicationName = "APP_NAME_HERE"
});
// Get the client request object for the bucket and desired object.
var getRequest = storageService.Objects.Get("BUCKET_HERE", "OBJECT_HERE");
using (var fileStream = new System.IO.FileStream(
"FILE_PATH_HERE",
System.IO.FileMode.Create,
System.IO.FileAccess.Write))
{
// Add a handler which will be notified on progress changes.
// It will notify on each chunk download and when the
// download is completed or failed.
getRequest.MediaDownloader.ProgressChanged += Download_ProgressChanged;
getRequest.Download(fileStream);
}
}
static void Download_ProgressChanged(IDownloadProgress progress)
{
Console.WriteLine(progress.Status + " " + progress.BytesDownloaded);
}
Code ripped from here
Related
I created a Blazor Server app that would allow end users to upload large excel files that would be consumed in downstream logic.
I use a the standard .NET core 5 InputFile component to upload the excel file to the app, within the app, I read the stream async, copy it into a memory stream and then use ExcelDataReader to convert it into dataset.
The challenge I see is that the upload takes a long time specifically when App is deployed to Azure. To dig a bit deeper into what exactly was consuming the time, I track progress of the StreamCopy operation:
The following code handles my upload:
private async Task OnInputFileChange(InputFileChangeEventArgs e)
{
this.StateHasChanged();
IReadOnlyList<IBrowserFile> selectedFiles;
selectedFiles = e.GetMultipleFiles();
foreach (var file in selectedFiles)
{
DataSet ds = new DataSet();
{
bool filesuccesfullRead = false;
//allowing a 100MB file at once
var timer = new Timer(new TimerCallback(_ =>
{
if (fileTemplateData.uploadProgressInfo.percentage <= 100)
{
// Note that the following line is necessary because otherwise
// Blazor would not recognize the state change and not refresh the UI
InvokeAsync(() =>
{
StateHasChanged();
});
}
}), null, 1000, 1000);
using (Stream stream = file.OpenReadStream(104857600))
using (MemoryStream ms = new MemoryStream())
{
fileTemplateData.uploadProgressInfo = new GlobalDataClass.CopyProgressInfo();
await ExtensionsGeneric.CopyToAsync(stream, ms, 128000, fileTemplateData.uploadProgressInfo);
System.Text.Encoding.RegisterProvider(System.Text.CodePagesEncodingProvider.Instance);
try
{
using (var reader = ExcelReaderFactory.CreateReader(ms))
{
ds = reader.AsDataSet(new ExcelDataSetConfiguration()
{
ConfigureDataTable = _ => new ExcelDataTableConfiguration()
{
UseHeaderRow = false
}
});
filesuccesfullRead = true;
}
}
catch (Exception ex)
{
Message = "Unable to read provided file(s) with exception " + ex.ToString();
}
stream.Close();
ms.Close();
}
}
ds.Dispose();
ds = null;
}
fileTemplateData.fileloading = false;
this.StateHasChanged();
}
Here is the CopyToAsync Function which is same as regular stream copy but provides progress tracking:
public static async Task CopyToAsync(this Stream fromStream, Stream destination, int bufferSize, GlobalDataClass.CopyProgressInfo progressInfo)
{
var buffer = new byte[bufferSize];
int count;
progressInfo.TotalLengthinBytes = fromStream.Length;
while ((count = await fromStream.ReadAsync(buffer, 0, buffer.Length)) != 0)
{
progressInfo.BytesTransfered += count;
progressInfo.percentage = Math.Round((((double)progressInfo.BytesTransfered /(double) progressInfo.TotalLengthinBytes) * 100), 1);
await destination.WriteAsync(buffer, 0, count);
}
}
public class CopyProgressInfo
{
public long BytesTransfered { get; set; }
public long TotalLengthinBytes { get; set; }
public double percentage { get; set; }
public DateTime LastProgressUpdateVisualized = new DateTime();
}
Now Let me put the question:
Using this code, I achieve a fair upload speed when the app is running on a local host(A 75MB file with tonnes of data would upload in around 18 seconds). When the app is deployed to an Azure App service plan, the same file would take more than 10 minutes to upload, which makes me feel something is seriously wrong. Using progress tracking, I was able to confirm that the time is being consumed by the CopytoAsync function and not the logic after that.
Here's what I have investigated:
I checked my internet upload speed on two seprate connections with a stable upload bandwidth of more than 25Mbps, so this is not an issue.
I upgraded the app service plan to a higher tier momentarily to see if upload bandwidth was somehow linked with Azure App Service plan tier, even increasing it to a powerful P3V2 tier made no difference.
To see if the specific Datacenter where my App service sits in was offering poor upload performance from my part of the world, I checked average upload speed using https://www.azurespeed.com/Azure/UploadLargeFile and a 75Mb file would upload in around 38 seconds to Azure West Europe Datacenter. So I donot see if the connectivity is the problem here.
With all that is mentioned above, what could be causing such a poor file upload speed when uploading the file onto a Deployed Blazor Server Web App.
I don't see such performance impact. I upload to azure blob storage though.
My implementation summary:
razor component called imageUpload.razor that contains
public async Task HandleFileSelected(InputFileChangeEventArgs e)
and calls a service like:
await hService.UploadImgToAzureAsync(imageFile.OpenReadStream(), fileName);
service that contains the following:
public async Task UploadImgToAzureAsync(Stream fileStream, string fileName)
{
return await ImageHelper.UploadImageToStorage(fileStream, fileName);
}
ImageHelper calls AzureStorage.cs
AzureStorage.cs that handles calls UploadFromStreamAsync
I finally managed to improve the upload performance, unfortunately Blazor's built in InputFile component doesn't seem to be designed very well for large file uploads specially when the app has been deployed. I used Tewr's upload file component with a larger buffer size(128000) and that has significantly improved performance(3X reduction). Tewr's sample code is available here:
https://github.com/Tewr/BlazorFileReader/blob/master/src/Demo/Blazor.FileReader.Demo.Common/IndexCommon.razor
I have a .net core 2.1 api application that will download a file from a remote location based on the file name. Here is the code:
static public class FileDownloadAsync
{
static public async Task DownloadFile(string filename)
{
//File name is 1GB.zip for testing
Stopwatch stopwatch = new Stopwatch();
stopwatch.Start();
using (HttpClient client = new HttpClient())
{
string url = #"http://speedtest.tele2.net/" + filename;
using (HttpResponseMessage response = await client.GetAsync(url, HttpCompletionOption.ResponseHeadersRead))
using (Stream readFrom = await response.Content.ReadAsStreamAsync())
{
string tempFile = $"D:\\Test\\{filename}";
using (Stream writeTo = File.Open(tempFile, FileMode.Create))
{
await readFrom.CopyToAsync(writeTo);
}
}
stopwatch.Stop();
Debug.Print(stopwatch.Elapsed.ToString());
}
}
}
This is working great, it will pull a 1 gig file down in about 50 seconds. Well within the required download time. I have hard coded a test file to download in this code for testing as well as storage location--these values will ultimately come from a config file when moved into production. Here is the API endpoint that calls this function:
[HttpGet("{fileName}")]
public async Task<string> GetFile(string fileName)
{
await FileDownloadAsync.DownloadFile(fileName);
return "Done";
}
So getting the file from a remote location down to the local server is not a problem. I need some help/guidance on re-posting this file to another API. Once the file is downloaded, there is some work done on the file to prepare it for upload (the files are all MP4 files), and once that work is done, I need to post it to another API for more proprietary processing. Here is the API end point data I have:
POST: /batch/requests Allocates resources to start new batch transcription. Use this method to request[work] on the input
audio data. Upon the accepted request, the response provides
information about the associated request ID and processing status.
Headers: Authorization: Authorization token
Accept: application/json
Content-Type: Indicates the audio format. The value must be:
audio/x-wav;codec=pcm;bit=16;rate=8000;channels=1
audio/x-wav;codec=pcm;bit=16;rate=16000;channels=1
audio/x-raw;codec=pcm;bit=16;rate=8000;channels=1
audio/x-raw;codec=pcm;bit=16;rate=16000;channels=1
video/mp4
Content-Length (optional): The size of the input voice file. Not
required if a chunked transfer is used.
Query string parameters (required):
profileId: one of supported (see GET profiles) customerId: the id of
the customer. A string of minimum 1 and up to 250 alphanumeric, dot
(.) and dash (-) characters.
So I will set the Content-Type to video/MP4 for processing. Note that if the input size is not used if a chunked transfer is used.
Right now, I am more concerned with just posting (streaming) the file in a non-chunked format while we await for more information on what they consider "chunking" a file.
So I am looking for help on steaming the file from disk to the endpoint. Everything I am running across for .net core API is creating the API to download the file from a POST like a Razor page or Angular page--I already have that. I just need some help on "re-posting" to another API.
Thanks
Using the HttpClient you open a stream to the file, create a content stream, set the necessary headers and post to the endpoint
Stream file = File.Open(filepath, FileMode.Open);
var content = new StreamContent(file);
content.Headers.ContentType = new MediaTypeHeaderValue("video/MP4");
client.DefaultRequestHeaders.Add("Authorization", "token here");
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json");
using (HttpResponseMessage response = await client.PostAsync(url, content)) {
//...
}
How to find URL of google spreadsheet using spreadsheet ID using google java client? I don't want to build a string. I want to make call to google REST API and find out.
Files: list with search parm using the mime type for a sheet
mimeType = 'application/vnd.google-apps.spreadsheet'
Code from the documentation.
/**
* Retrieve a list of File resources.
*
* #param service Drive API service instance.
* #return List of File resources.
*/
private static List<File> retrieveAllFiles(Drive service) throws IOException {
List<File> result = new ArrayList<File>();
Files.List request = service.files().list();
do {
try {
FileList files = request.execute();
result.addAll(files.getItems());
request.setPageToken(files.getNextPageToken());
} catch (IOException e) {
System.out.println("An error occurred: " + e);
request.setPageToken(null);
}
} while (request.getPageToken() != null &&
request.getPageToken().length() > 0);
return result;
}
response contains:
Remember unless the user has access to the files the link wont work for them
"selfLink": "https://www.googleapis.com/drive/v2/files/1-0ReBjBqKh_Q9r1BDsC_BB9JgkeLoFPkDIXFcXiqQZ",
"alternateLink": "https://docs.google.com/spreadsheets/d/1-0ReBjBqKh_Q9r1BDsC_BB9JgkeLoFPkDIXFcXiqQZ/edit?usp=drivesdk",
"embedLink": "https://docs.google.com/spreadsheets/d/1-0ReBjBqKh_Q9r1BDsC_BB9JgkeLoFPkDIXFcXiqQZ/htmlembed",
I am using loopback Storage component REST API in Xamarin to finish a file uploading job. However, it does not work and does not return any exceptions to me.
Here is my code:
library using: RestSharp.portable
public async Task addFiles(string name, byte[] file)
{
try
{
var client = new RestClient(App.StrongLoopAPI);
var request = new RestRequest("containers/container1/upload", HttpMethod.Post);
request.AddHeader("cache-control", "no-cache");
request.AddHeader("content-type", "multipart/form-data");
request.AddFile("file", file, name + ".jpg", System.Net.Http.Headers.MediaTypeHeaderValue.Parse("multipart/form-data"));
var res = await client.Execute(request);
}
catch (Exception ex)
{
//return null;
}
}
Does my function have any problems?
You're setting the Content Type (Mime Type) wrong.
AddFile accepts as the last parameter the Content Type (for example image/jpeg for a JPG image), where you're using multipart/form-data.
There are different ways to figure out the Content Type of a file, see here:
Get MIME type from filename extension
This should fix your issue.
My script fetches xml via httpConnection and saves to persistent store. No problems there.
Then I loop through the saved data to compose a list of image url's to fetch via queue.
Each of these requests calls the httpConnection thread as so
...
public synchronized void run()
{
HttpConnection connection = (HttpConnection)Connector.open("http://www.somedomain.com/image1.jpg");
connection.setRequestMethod("GET");
String contentType = connection.getHeaderField("Content-type");
InputStream responseData = connection.openInputStream();
connection.close();
outputFinal(responseData, contentType);
}
public synchronized void outputFinal(InputStream result, String contentType) throws SAXException, ParserConfigurationException, IOException
{
if(contentType.startsWith("text/"))
{
// bunch of xml save code that works fine
}
else if(contentType.equals("image/png") || contentType.equals("image/jpeg") || contentType.equals("image/gif"))
{
// how to save images here?
}
else
{
//default
}
}
What I can't find any good documentation on is how one would take the response data and save it to an image stored on the device.
Maybe I just overlooked something very obvious. Any help is very appreciated.
Thanks
I tried following this advise and found the same thing I always find when looking up BB specific issues: nothing.
The problem is that every example or post assumes you know everything about the platform.
Here's a simple question: What line of code writes the read output stream to the blackberry device? What path? How do I retrieve it later?
I have this code, which I do not know if it does anything because I don't know where it is supposedly writing to or if that's even what it is doing at all:
** filename is determined on a loop based on the url called.
FileOutputStream fos = null;
try
{
fos = new FileOutputStream( File.FILESYSTEM_PATRIOT, filename );
byte [] buffer = new byte [262144];
int byteRead;
while ((byteRead = result.read (buffer ))!=- 1)
{
fos.write (buffer, 0, byteRead);
}
fos.flush();
fos.close();
}
catch(IOException ieo)
{
}
finally
{
if(fos != null)
{
fos.close();
}
}
The idea is that I have some 600 images pulled from a server. I need to loop the xml and save each image to the device so that when an entity is called, I can pull the associated image - entity_id.png - from the internal storage.
The documentation from RIM does not specify this, nor does it make it easy to begin figuring it out.
This issue does not seem to be addressed on this forum, or others I have searched.
Thanks
You'll need to use the Java FileOutputStream to do the writing. You'll also want to close the connection after reading the data from the InputStream (move outputFinal above your call to close). You can find all kinds of examples regarding FileOutputStream easily.
See here for more. Note that in order to use the FileOutputStream your application must be signed.