I want to design a web project, when users upload file to hadoop hdfs,users can see their upload status via web. Is there any simple java api?
Can anyone help?
Right now I only know the way,how to use api to upload file to hdfs.
public synchronized static void upload(FileSystem fs, String local,
String remote) {
// Path home = fs.getHomeDirectory();
Path workDir = fs.getWorkingDirectory();
Path dst = new Path(workDir + "/" + remote);
Path src = new Path(local);
try {
fs.copyFromLocalFile(false, true, src, dst);
log.info("upload " + local + " to " + remote + " successed. ");
} catch (Exception e) {
log.error("upload " + local + " to " + remote + " failed :"
+ ExceptionUtils.getFullStackTrace(e));
}
}
You might find FileStatus class useful. Through this class you can get almost all the necessary information which your user might wanna know, like getting the access time of the file, modification time of the file, length of the file etc etc.
And, since you are talking about a web app, I would recommend you to have a look at the WebHDFS REST API. It allows you to submit a GET HTTP request which will send a FileStatus JSON object in response. You could then easily use this JSON object to present the status to your user through a web page.
HTH
Related
I have few files stored in a database, these files were fetched from a FTP server and processed.
I am trying to move these files to archive directory on FTP server and to do that I am using producerTemplate.
This is what I have done so far.
try {
DefaultCamelContext camelContext = new DefaultCamelContext();
ProducerTemplate template = camelContext.createProducerTemplate();
template.request("ftp://xxxxx/include=fileFromDb.xml&move=archive/fileFromDb.xml", outExchange -> {
String body = outExchange.getIn().getBody(String.class);
});
} catch (Exception ex) {
// update the status in database to indicate archive failed
}
But this is failing with following error and can't get this resolved.
WARN o.a.c.c.f.remote.RemoteFileProducer - Writing file failed with: Cannot write null body to file: archive/ID-192-168-1-113-tpgi-com-au-1627667653835-1-2
I have tried implementing solutions from other answers but they didn't worked for either.
If I write a custom route like below, it works fine but because I need perform post move actions I prefere using producer template.
from("ftp://xxxxx/include=fileFromDb.xml&move=archive/fileFromDb.xml")
.log("file moved successfully").
If I write a custom route like below, it works fine but because I need perform post move actions I prefere using producer template.
You could use onCompletion to call another route to handle the post move actions.
String sftpURI = "sftp:localhost:2222/upload" +
"?username="+username+"&password="+password
+ "&move=done";
String sftpPostMoveURI = "sftp:localhost:2222/upload/done" +
"?username="+username+"&password="+password
+ "&fileName=${headers.CamelFileName}";
from(sftpURI)
.routeId("ReadFileUsingSFTPRoute")
.onCompletion().onCompleteOnly()
.setBody().constant("")
.to("direct:postMoveActions")
.end()
.log("file ${headers.CamelFileName} moved");
from("direct:postMoveActions")
.routeId("PostMoveActionsRoute")
.log("doing post move actions!")
.pollEnrich().simple(sftpPostMoveURI).timeout(5000)
.log("File: ${body}");
I am at learning phase of web api and I want to upload image to server using web api in mvc dot net. I have tried much but not getting solution.
In controller I have done this.
string img = username + "_" + labTestId + fb;
var element2 = image;
MemoryStream ms = new MemoryStream(Convert.FromBase64String(element2));
byte[] imagesbytes = ms.ToArray();
string folder_name = "lab-orders";
folder_name = folder_name.ToLower();
string SaveLocation = ServerConfig.BlobPath + folder_name + "/" + fb+imagesbytes;
BlobUploader.UploadTo(folder_name,img, ms);
// var filePath = HttpContext.Current.Server.MapPath("~/Userimage/" + postedFile.FileName + extension);
var filePath = HttpContext.Current.Server.MapPath(SaveLocation + postedFile.FileName + extension);
postedFile.SaveAs(filePath);
How can I upload image to server using web api?
It's not that simple.
In this tutorial You have step by step everything You have to do, to upload image.
In brief:
Create Blob Storage Account, and get access keys.
Save keys in Your web config.
Install WindowsAzureStorage from nuget packege manager
Create blob storage client, get or create blob container
create CloudBlockBlob and upload it.
Edit
Change link to web api tutorial insted of mvc tutorial
I'm using the export Google Drive API to retrieve a Google Doc as Pdf: https://developers.google.com/drive/v3/reference/files/export
I'm having the following problem: for documents bigger than a certain size (I don't know exactly the threshold, but it happens even with relatively small files around 1,5 MB) the API return a 200 response code with a blank result (normally it should contains the pdf data as byte stream), as you can see in the following screenshot:
I can successfully export the file via GoogleDrive/GoogleDoc UI with the "File -> Download as.. -> Pdf" command, despite it takes a bit of time.
Here is the file used for test (1.180 KB exported from Google Doc), I shared it so you can access to try export:
https://docs.google.com/document/d/18Cz7kHfEiDLeTWHyyoOi6U4kFQDMeg0D-CCJzILMMCk/edit?usp=sharing
Here is the (Java) code I'm using to perform the operation:
#Override
public GoogleDriveDocumentContent downloadFileContentAsPDF(String executionGoogleUser, String fileId) {
GoogleDriveDocumentContent documentContent = new GoogleDriveDocumentContent();
String conversionMimeType = "application/pdf";
try {
getLogger().info("GDrive APIs - Downloading file content in PDF format ...");
InputStream gDriveFileData = getDriveService(executionGoogleUser).files()
.export(fileId, conversionMimeType)
.executeMediaAsInputStream();
getLogger().info("GDrive APIs - File content as PDF format downloaded.");
documentContent.setFileName(null);
documentContent.setMimeType(conversionMimeType);
documentContent.setData(gDriveFileData);
} catch (IOException e) {
throw new RuntimeException(e);
}
return documentContent;
}
Does anyone has the same issue and know how to solve it?
The goal is to generate a pdf from a Google Doc.
Thanks
I think you should try using media downloadeder you will have to alter it for Google drive rather than storage service.
{
// Create the service using the client credentials.
var storageService = new StorageService(new BaseClientService.Initializer()
{
HttpClientInitializer = credential,
ApplicationName = "APP_NAME_HERE"
});
// Get the client request object for the bucket and desired object.
var getRequest = storageService.Objects.Get("BUCKET_HERE", "OBJECT_HERE");
using (var fileStream = new System.IO.FileStream(
"FILE_PATH_HERE",
System.IO.FileMode.Create,
System.IO.FileAccess.Write))
{
// Add a handler which will be notified on progress changes.
// It will notify on each chunk download and when the
// download is completed or failed.
getRequest.MediaDownloader.ProgressChanged += Download_ProgressChanged;
getRequest.Download(fileStream);
}
}
static void Download_ProgressChanged(IDownloadProgress progress)
{
Console.WriteLine(progress.Status + " " + progress.BytesDownloaded);
}
Code ripped from here
I want to upload a file to a Google drive folder, if the file name is same i want to skip. With the code below i able upload file to Google Drive. But how to get the filename inside a folder?
File newFile = GoogleDriveHelper.uploadFile(service, dir, directoryId);
I'm referring to google REST document when I pass in all the param, it will return the Id in Google Drive.
Thx for Advice!
Check here pass the fileId to this function. The fileId can be get at here
public static void printFile(DriveService service,String fileId)
{
try
{
File file = service.Files.Get(fileId).Execute();
Console.WriteLine("Title: " + file.Title);
}
catch (Exception e)
{
Console.WriteLine("An error occurred: " + e.Message);
}
}
I am using the Background Transfer to upload Photographs to my Web Service. As the Photograph uploads can consume significant time and memory, I thought it might be a nice idea to use the background transfer request to accomplish this. After the photo is uploaded, I want to obtain the Id of the uploaded photo and then use it for post-processing. However, it turns out I can't do that in a background transfer request.
Per my understanding, Background Transfer works using the following logic ONLY:
You have to obtain the file you want to upload and then save/copy it to your app's Isolated Storage under the folder: shared/transfers. This is extremely important. Apparently, using file in a different location didn't work for me. Maybe it isn't the shared/transfers as much as it is a 'relative' path. But I would stick to the same conventions.
After you have saved the file in that location, your background request can be created based on that. It doesn't look like you can pass POST CONTENT other than the file contents, so any other parameters like file name, mime type etc. will need to be passed as QUERY String parameters only. I can understand this, but it would've been nice if I could pass both as POST Content. I don't think HTTP has a limitation on how this works.
Here is some code for creating a request using Hammock:
string url = App.ZineServiceAuthority + "articles/save-blob?ContainerName={0}&MimeType={1}&ZineId={2}&Notes={3}&IsPrivate={4}&FileName={5}";
url = String.Format(url, userId, "image/jpg", ZineId, txtStatus.Text, true, UploadFileName);
var btr = new BackgroundTransferRequest(new Uri(url, UriKind.Absolute));
btr.TransferPreferences = TransferPreferences.AllowCellularAndBattery;
btr.Method = "POST";
btr.Headers.Add("token", IsolatedStorageHelper.GetTravzineToken());
btr.UploadLocation = new Uri(#"/shared\transfers/" + UploadFileName, UriKind.Relative);
btr.TransferStatusChanged += new EventHandler<BackgroundTransferEventArgs>(btr_TransferStatusChanged);
btr.TransferProgressChanged += new EventHandler<BackgroundTransferEventArgs>(btr_TransferProgressChanged);
BackgroundTransferService.Add(btr);
In my case, I am literally passing all the necessary parameters using the query string. On a successful save, my Web Service returns back the Id of the Photo I just uploaded. However:
There is NO way (or at least I know of) to obtain and evaluate the RESPONSE. The Background Transfer Request Event handlers do not expose a RESPONSE.
Here are my event handlers:
void btr_TransferProgressChanged(object sender, BackgroundTransferEventArgs e)
{
bool isUploading = e.Request.TotalBytesToSend > 0 ? true : false;
lblStatus.Text = isUploading ? "Uploading" + e.Request.BytesSent.ToString() + " sent" : "Done";
}
void btr_TransferStatusChanged(object sender, BackgroundTransferEventArgs e)
{
if (e.Request.TransferStatus == TransferStatus.Completed)
{
using (IsolatedStorageFile iso =
IsolatedStorageFile.GetUserStoreForApplication())
{
if (iso.FileExists(e.Request.UploadLocation.OriginalString))
iso.DeleteFile(e.Request.UploadLocation.OriginalString);
}
BackgroundTransferService.Remove(e.Request);
if (null != e.Request.TransferError)
{
MessageBox.Show(e.Request.TransferError.Message);
}
else
{
lblStatus.Text = "Done baby done";
}
}
}
So now my question is, how does anyone do any sort of POST Processing in such scenarios?
Can anyone please tell me the line of thought behind designing such an inflexible class?
Any thoughts on how I could get around this issue would be appreciated.
Also, does anyone have any working examples of a homegrown BackgroundTransfer?
Haven't tried it but why not set a download location like this:
btr.DownloadLocation = "myDownloadFile.html";
btr.UploadLocation = "myUploadFile.jpg";
...
If the request is completed read the file "myDownloadFile.html" where your response has been stored and delete it afterwards.