We need to call a third party lib, ideally at the time of onUploadChunk callback.
As shown in the documentation (http://docs.fineuploader.com/branch/master/api/events.html#uploadChunk), we can have some parameters in order to identify the chunk and do stuff with the javascript slice method.
But, the question is : how to give back updated chunk into the fineuploader upload process ?
Thanks a lot for help.
You cannot modify the chunks created by Fine Uploader, nor should you be able to as it may change the size of the chunk, the expected total number of chunks, and require adjustments of internal state and sent parameters. If you'd like to modify any files, you have two options:
Modify the file before it is sent to to Fine Uploader
Modify the file before the file upload begins. You can cancel the original file, and then submit the changed versions via the addFiles API method.
Related
I have a workflow where I am getting json files as a response of rest api. I am getting approximately 100k files in a session. total size of all the files is 15GB. I have to save each file to file system, which i am doing. at the end of the process I have to wait for all the files to be present before I send a success message.
Once I save the file in FS, I am calling notify+wait. but I dont need 15 gb data in flowfile anymore. So to release some space, I thought of using either replaceText or ModifyByte to clear content. so notify+wait runs smoothly. Total wait for this process is 3 hrs.
But process is taking too long in both (replaceText or ModifyByte) case.
Can you suggest, fastest way to clear flowfile data.I do not need any attributes too. so is thr a way I can abandon old flowfile and generate kb flowfile, midway?
what i want is something like generateflowfile, but in middle, so for each of my existing flowfile, i can drop old one, and generate blank flowfile for notify and wait.
Thanks
NiFi's Content Repository and FlowFile Repository are based on a copy-on-write mechanism, so if you don't change the contents or metadata, then you are not necessarily "keeping" the 15GB across those processors.
Having said that, if all you need is the existence of such flow files on disk (but not contents or metadata), try ExecuteScript with the following Groovy script:
def flowFiles = session.get(1000)
flowFiles.each {
session.transfer(session.create(), REL_SUCCESS)
}
session.remove(flowFiles)
This script will grab up to 1000 flow files at a time, and for each one, send an empty flow file downstream. It then removes all the original incoming flow files.
Note that this (i.e. your use case) will "break" the provenance/lineage chain, so if something goes wrong in your flow, you won't be able to tell which flow files came from which parent flow files, etc. This limitation is one reason why you don't see a full processor that performs this kind of function.
In case you need to keep the attributes, lineage and metadata you can use the following code (grabs only 1 flowfile at a time). The only thing that changes is the UUID, but otherwise everything is kept - except the content of course.
f = session.get()
session.transfer(session.create(f), REL_SUCCESS)
session.remove(f)
I'm building an application that builds a pdf file and returns it to the client whenever it receives a request.
Since some of these pdf files might take some time to generate, I would like to periodically send some sort of status update back to client while it is running.
When it's finished building the pdf file, it should be returned to the client as well.
Something akin to:
func buildReport(writer http.ResponseWriter, request *http.Request){
//build pdf build pdf file
for { //for example purposes only
writer.Write([]byte("building. Please wait."))
}
pdf.OutputFileAndClose("report.pdf")
//set header to pdf so that the client knows it's a PDF
writer.Header().Set("Content-Type", "application/pdf")
http.ServeFile(writer, request, "report.pdf")
}
func main() {
http.HandleFunc("/", buildReport)
http.ListenAndServe(":8081", nil)
}
Setting the header might not work, as the writer can only have one header.
TL;DR is that it cannot be implemented that way. You need to
An API that requests the PDF creation. That queues PDF creation job in a task queue (so that too many PDF creation requests won't blow the HTTP server worker pool)
Provide an API that allows you to check where are you with the PDF rendering (I am assuming that the job can provide interim stats). This is going to be polled by the client on a regular basis.
An API to pull the PDF once it is ready.
Hope this helps and best of luck with your project.
This is by no means comprehensive, but a reasonable example of how you might construct your API (which needs to be asynchronous, as the previous respondent pointed out) can be found here: https://www.adayinthelifeof.nl/2011/06/02/asynchronous-operations-in-rest/
The job queue model is a pretty common one. I would recommend you also write a basic API binding library (you'd want this for your own testing purposes in any case) so that your users can understand how you intend them to use the API, and in writing it, you'll get a better sense of how asynchronous REST interactions feel from the end user side.
Contrary to what others have said, what you want is in fact
directly possible but requires fullfillment of the two preconditions:
HTTP/1.1 and above.
You'll be sending custom content to the clients — not PDF data
directly, — and they're prepared to accept and parse it.
You can then employ the so-called "chunked" payload encoding specifically
invented to handle "streamed" downloads where the server does not know how
many bytes it's about to send.
So you may invent some creative kind of payload where you first periodically
stream a "no op" / "progress" marker and then the actual payload.
Say, while the file is being prepared you periodically send a line of text
reading "PROCESSING" + LF then, when a result is ready you send
a line of text "READY" SIZE + LF where SIZE is the size, in bytes,
of the immediately following PDF document. After the document is streamed,
the server signals the end of data.
Hence the stream would look like
PROCESSING
PROCESSING
…
PROCESSING
READY 8388608
%PDF-1.3
…
%%EOF
The clients have to be able to parse this information from the stream
they're receiving and have a simple FSM in place to switch from state to
state as they fetch your stream.
The server has to make sure it flushes the stream after each "informational" line otherwise the whole thing would not be "interactive".
If you have a good idea about the overall state of the processing of the
document, each "status update" line could include the percentage of the work done, like in "PROCESSINGNN" + LF.
I'm trying to learn how to use Parse and while it's very simple, it's also... not? Perhaps I'm just missing something, but it seems like Parse requires a lot of client-side code, and even sending multiple requests for a single request. For example, in my application I have a small photo gallery that each user has. The images are stored on Parse and obtained from parse when needed.
I want to make sure that a user can not store any more than 15 images in their gallery at a time, I also want these images to be ordered by an index.
Currently it seems like the only viable option is to perform the following steps on the client:
Execute a query/request to get the amount of pictures stored.
If the amount is less than 15, then execute a request to upload the picture.
Once the picture is uploaded, execute a request that stores an object linking the user that uploaded the PFFile.
This is a total of 3 or? 6 requests just to upload a file, depending on if a "response" is considered a request by parse too. This also does not provide any way to order the pictures in the gallery. Would I have to create a custom field called "index" and set that to the number of photos received in the first query + 1?
It's worse than you think: to create the picture you must create a file, save it, then save a reference to the file in an object and save that, too.
But it's also better than you think: this sort of network usage is expected in a connected app, and some of it can be mitigated with additional logic on the server ("cloud code" in parse parlance).
First, in your app, consider a simple data model where _User has an array of images (represented, say, by an "UserImage" custom class). If you keep this relationship as an array of pointers on user, than a user's images can be fetched eagerly, when the app starts, so you'll know the image count as a fact along with the user. The UserImage object will have a file reference in it, so you can optionally fetch the image data and just hold the lighter metadata with the current user.
Ordering is a more ephemeral idea. One doesn't order objects as they are saved, but rather as they are retrieved. Queries can be ordered according to any attribute, and even more to the point, since you're retrieving all 15 images, you should consider ordering them for presentation a function of the UI, not the data.
Finally, parse limits your app not by transaction count, but by transaction rate, with a free limit low enough to serve plenty of users.
I'm using the jquery version of FineUploader v5.3, and chunk all my uploaded files. Thanks for the help yesterday, and this is working nicely now.
Here's my current issue:
On the server I currently look at the qquuid within the Request and create a folder as necessary to house the temporary chunk files that are coming in. This was originally designed when only 1 file was incoming, but now it's unnecessary code on the second and subsequent chunks (folder already exists, but index order of chunks is unknown). Is there a way to determine the qquuid that will be used for the file within an event BEFORE the actual post so I can do the needed infrastructure? I tried using the OnSubmitted event, but all that I can get from here is the ID (chunk index) or the name. Please let me know if you require any clarification. Thanks!
v3.5 is incredibly old - about 2.5 years old to be exact. Looking at the API doc from that time, I see that a getUuid method existed. If you pass in the file ID, you will be able to get a handle on the file's UUID. You can make this call in your onSubmitted callback handler.
I have page, on which i am holding base64 representation of some images(around 1mb each ), now i am posting this data via ajax to the server(contentType is default - url-encoded). This works fine if i have one or two images to be sent, but if i have more than 2mb of request data, the server doesn't accept that, and request parameters doesn't have anything, so i increased the maxPostSize in my tomcat, and it started accepting more data as well, but i am a bit apprehensive if this would create memory issues, especially if i have lot of images ?
Also i tried changing the contentType to multipart/formdata, but it errors out, "saying the request was rejected because no multipart boundary was found".
EDIT
I think i should elaborate more, actually requirement is something like this - User clicks on an upload link, he should be able to upload the file and then he should be able to see a thumbnail of the image on the page(all this without refreshing the page). I tried following approaches for this.
Reading the file using file reader, showing the thumbnail and then explicitly triggering the upload, when user clicks on save, simple but not cross browser, doesn't work in IE
Allowed user to upload the file send the base64 version of the image from the server and when user clicks on save, send the base64 string back to the server and convert it back to byte array and save to the db.
Now, i have a screen where all the records are by default editable, so clicking on save means, sending the image strings for all the records to the server, which will ofcourse create memory issues.
"3". Not implemented yet, but thinking of first saving the other fields(the non image fields) and then explicitly saving the images one by one(looks okay, but number of requests will be high)
Waiting for someone to suggest a 4th approach, hope i have explained enough
Disclaimer ... Not done anything like this, but...
Why not send each image separately? :)