Post large amount of data with Wicket.ajax.post - ajax

I'm trying to use the Behavior and JavaScript FileReader as can be seen in this question: Wicket Drag and drop functionality for adding an image
If I post more than 200k, I get an error Form too large. So perhaps I am missing some multipart stuff.
What is the proper way to call Wicket.ajax.post() with a large amount of data?
I tried setting mp to true, but then it started complaining that I do not have a form id. Does it need a form?
Btw. I use Jetty, but that has no problems using a regular file upload using forms.

This error comes from jetty Request implementation.
If you look at sources of the Request#extractFormParameters method, you will see next:
if (_context != null)
{
maxFormContentSize = _context.getContextHandler().getMaxFormContentSize();
maxFormKeys = _context.getContextHandler().getMaxFormKeys();
}
if (maxFormContentSize < 0)
{
Object obj = _channel.getServer().getAttribute("org.eclipse.jetty.server.Request.maxFormContentSize");
if (obj == null)
maxFormContentSize = 200000;
else if
...
}
So, in fact, you can really set your context values as pikand proposed as 0, or set server config as follows:
<Configure id="Server" class="org.eclipse.jetty.server.Server">
<Call name="setAttribute">
<Arg>org.eclipse.jetty.server.Request.maxFormContentSize</Arg>
<Arg>-1<!-- or 0, or any lt 0 --></Arg>
</Call>
...
</Configure>
Your exception is thrown a bit later according to this code:
if (contentLength > maxFormContentSize && maxFormContentSize > 0)
{
throw new IllegalStateException("Form too large: " + contentLength + " > " + maxFormContentSize);
}
So, you can see, that maxFormContentSize could be <= 0 not to throw this exception.
I think, there is no need to update something via ajax. But in fact it is better to limit data size, not to allow users put down your server.
Other application servers have their own settings, for most of them you should set maxPostSize value to zero, to disable this restriction.
Also, wicket Form component has it's own maxSize property, you can set it with Form#setMaxSize. The problem is that Form transmittes this value as Bytes value to FileUploadBase class, which has next javadoc:
The maximum size permitted for the complete request, as opposed to
fileSizeMax. A value of -1 indicates no maximum.
And actually this parameter is set via fileUpload.setSizeMax(maxSize.bytes());, and Bytes can't hold negative value. But I think you can try to set it as 0 and check if it works. By default, Form#getSizeMax() method checks:
return getApplication().getApplicationSettings().getDefaultMaximumUploadSize();
Which returns Bytes.MAX, which is equals to 8388608 terabytes. I think, this is about to be "no limit" value :)
Additionaly, as I know - you don't need to set Form id, to allow using multipart parameter. Only if you updating your form via ajax, you have to set Form.setOutputMarkupId(true). But actually, Form creates id by itself in renderHead method if it is multipart:
// register some metadata so we can later properly handle multipart ajax posts for
// embedded forms
registerJavaScriptNamespaces(response);
response
.render(JavaScriptHeaderItem.forScript("Wicket.Forms[\"" + getMarkupId()
+ "\"]={multipart:true};", Form.class.getName() + '.' + getMarkupId()
+ ".metadata"));
Note, that getMarkupId() method creates markup id if does not exists.

There is a form size limit in Jetty defaulting to 200k.
Add a jetty-web.xml in your webapp/WEB-INF folder. There you can set a form size limit of the needed size.
<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Mort Bay Consulting//DTD Configure//EN"
"http://jetty.mortbay.org/configure.dtd">
<Configure id="WebAppContext" class="org.eclipse.jetty.webapp.WebAppContext">
<Set name="maxFormContentSize" type="int">900000</Set>
<Set name="maxFormKeys">5000</Set>
</Configure>

Related

Classic ASP XMLHttp Send very slow

I've inherited a classic asp project and as part of the upgrade process we're moving a lot of the business logic to a REST API (WebApi 2.2)
The authorization endpoint for the api is written, and the asp site can call it, but it's very slow compared with calling directly via Postman.
(I'm a C# coder not a VBScript one so the below code may be offensive)
Asp Code:
' Send a prebuilt HTTP request and handle the response
' Returns true if the request returns a 200 response, False otherwise
' Response body is placed in Response
' ErrorMessage is set to return status text if an error code is returned
Function HandleRequest(ByRef objRequest, strBody)
set profiler = Server.CreateObject("Softwing.Profiler")
HandleRequest = False
' Add auth token if we have it
If Not m_accessToken&"" = "" Then
objRequest.SetRequestHeader "Authorization", "Bearer " & m_accessToken
End If
' Originating IP for proxy forwarding
If Not m_clientIp&"" = "" Then
objRequest.SetRequestHeader "X-Forwarded-For", m_clientIp
End If
On Error Resume Next
If (strBody&"" = "") Then
objRequest.Send()
Else
profiler.ProfileStart()
objRequest.Send(strBody)
flSendRequest = profiler.ProfileStop()
End If
If Err.Number = 0 Then
Dim jsonResponse
If (objRequest.ResponseText&"" <> "") Then
profiler.ProfileStart()
set jsonResponse = JSON.parse(objRequest.ResponseText)
flJson = profiler.ProfileStop()
set m_Response = jsonResponse
End If
If objRequest.Status = 200 Then
HandleRequest = True
m_errorMessage = ""
Else
m_errorMessage = objRequest.statusText
End If
Else
m_errorMessage = "Unable to connect to Api server"
End If
On Error GoTo 0
End Function
You can see there's some profiling code in there.
The following post request takes 392ms
POST localhost:5000/oauth/token
Content-Type application/x-www-form-urlencoded
client_id:ABCDEF0-ABCD-ABCD-ABCD-ABCDEF-ABCDEF01234
client_secret:aBcDeF0123456789aBcDeF0123456789=
username:demo
password:demo
grant_type:password
If I issue the same request direct to the Api via Postman it takes 30ms.
That's more than 13x slower.
What gives?
Edit
Raw result from Softwing Profiler:
flJson 10.9583865754112
flSendRequest 392.282022557137
So after a lengthy-ish discussion with the #J-Tolley it looks as though the issue is with the Softwing.Profiler documentation which states;
all results are given in milliseconds
even though earlier in the page it states;
has a ten milliseconds resolution
Have not used the Softwing.Profiler component alone before and would recommend anyone using in a Classic ASP environment to implement it using the SlTiming class library provided by 4GuysFromRolla.
In that article it even warns anyone using the Softwing.Profiler ProfileStop() method to;
Be aware that Softwing.Profiler's ProfileStop method returns a value in ticks (tenths of milliseconds).

Simulate session using WithServer

I am trying to port tests from using FakeRequest to using WithServer.
In order to simulate a session with FakeRequest, it is possible to use WithSession("key", "value") as suggested in this post: Testing controller with fake session
However when using WithServer, the test now looks like:
"render the users page" in WithServer {
val users = await(WS.url("http://localhost:" + port + "/users").get)
users.status must equalTo(OK)
users.body must contain("Users")
}
Since there is no WithSession(..) method available, I tried instead WithHeaders(..) (does that even make sense?), to no avail.
Any ideas?
Thanks
So I found this question, which is relatively old:
Add values to Session during testing (FakeRequest, FakeApplication)
The first answer to that question seems to have been a pull request to add .WithSession(...) to FakeRequest, but it was not applicable to WS.url
The second answer seems to give me what I need:
Create cookie:
val sessionCookie = Session.encodeAsCookie(Session(Map("key" -> "value")))
Create and execute request:
val users = await(WS.url("http://localhost:" + port + "/users")
.withHeaders(play.api.http.HeaderNames.COOKIE -> Cookies.encodeCookieHeader(Seq(sessionCookie))).get())
users.status must equalTo(OK)
users.body must contain("Users")
Finally, the assertions will pass properly, instead of redirecting me to the login page
Note: I am using Play 2.4, so I use Cookies.encodeCookieHeader, because Cookies.encode is deprecated

Worklight 5.0.6 : Ajax request exception: Form too large while sending large data to data adapter

My question is relatively the same than the one posted on developerworks forum (forum is read only due to migration) wich is :
I have a http adapter that interfaces with external web services. Part
of the payload is audio, and images. We're hitting a form size limit.
Please see attached exception at end of this post. I've read on
previous posts that jetty configurations need to be adjusted to
accommodate the larger payload. We want to control this size limit at
the server side application layer, and thought that of creating a
jetty-web.xml to define the max form size:
400000
In Worklight is this the proper approach to resolve this issue?
If this is the proper approach can you provide details whether the
jetty-web.xml should be placed under server/conf, or does it need to
be under WEB-INF of the application war?
If the file needs to be placed under WEB-INF can you explain how to
accomplish this file being placed under WEB-INF during the WL project
build.
Thanks E: Ajax request exception: Form too large802600>200000
2013-02-06 11:39:48 FWLSE0117E: Error code: 1, error description:
INTERNAL_ERROR, error message: FWLSE0069E: An internal error occurred
during gadget request Form too large802600>200000, User Identity
{wl_authenticityRealm=null, GersServiceAdapterRealm=(name:USAEMP4,
loginModule:GersServiceAdapterLoginModule),
wl_remoteDisableRealm=(name:NullLoginModule,
loginModule:NullLoginModule), SampleAppRealm=null,
wl_antiXSRFRealm=(name:antiXSRF, loginModule:WLAntiXSRFLoginModule),
wl_deviceAutoProvisioningRealm=null, WorklightConsole=null,
wl_deviceNoProvisioningRealm=(name:device,
loginModule:WLDeviceNoProvisioningLoginModule),
myserver=(name:3e857b6a-d2f6-40d1-8c9c-10ca1b96c8df,
loginModule:WeakDummy),
wl_anonymousUserRealm=(name:3e857b6a-d2f6-40d1-8c9c-10ca1b96c8df,
loginModule:WeakDummy)}.
I have exactly the same problem :
I send a large amount of data to a worklight adapter and my application fail with the following error message into the log :
2013-08-21 09:48:17] FWLSE0020E: Ajax request exception: Form too large202534>200000
[2013-08-21 09:48:18] FWLSE0117E: Error code: 1, error description: INTERNAL_ERROR, error message: FWLSE0069E: An internal error occurred during gadget request Form too large202534>200000, User Identity {wl_authenticityRealm=null, wl_remoteDisableRealm=(name:null, loginModule:NullLoginModule), SampleAppRealm=null, wl_antiXSRFRealm=(name:b2isf3704k2fl8hovpa6lv9mig, loginModule:WLAntiXSRFLoginModule), wl_deviceAutoProvisioningRealm=null, WorklightConsole=null, wl_deviceNoProvisioningRealm=(name:40a24da9-0a32-464a-8dec-2ab402c683ae, loginModule:WLDeviceNoProvisioningLoginModule), myserver=(name:2b1a7864-37c4-47f0-9f5c-49621b6915b5, loginModule:WeakDummy), wl_anonymousUserRealm=(name:2b1a7864-37c4-47f0-9f5c-49621b6915b5, loginModule:WeakDummy)}.
This occurs on calling an adapter procedure by calling WL.Client.invokeProcedure(...) and before the first line of the called procedure... If I try to log the start of the called procedure I have nothing written in my debug log...
I can give you my source code :
This one is called by a dhtml user event (onclick) :
// Construct the param to pass to the WL adapter insert procedure
var paramObject = {
QCDART: machine, // machine is a javascript variable as long int
QTITRE: title, // title is a javascript variable as string(255)
QDESC: desc, // desc is a javascript variable as string(255)
QHODAT: todayDateDb2IntFormat, // todayDateDb2IntFormat is a javascript variable as long int
QACTIF: active, // active is a javascript variable as int
SSRCFIC: currentPdfFileDataBase64, // currentPdfFileDataBase64 is a javascript variable as base64 encoded string from a binary file > 150 ko approx.
SMIMFIC: 'application/pdf',
SSIZFIC: currentPdfFileSize // currentPdfFileSize is a javascript variable as long int
};
// Construct adapter invocation data
var invocationData = {
adapter : 'IseriesDB2Backend', // adapter name
procedure : 'addModeleReleves', // procedure name
parameters : [paramObject] // parameters if any
};
WL.Client.invokeProcedure(invocationData, {
timeout: 60000,
onSuccess: function() {
// Notify success
alert('OK');
}, // invokeProcedure success callback
onFailure: function(invocationResult) {
alert('ERROR');
} // invokeProcedure failure callback
});
This one is my adapter source code :
var addModeleReleveStatement = WL.Server.createSQLStatement("select QCDDOC from FINAL TABLE (insert into ERIHACFICH.DOCENTQ (QCDART, QTITRE, QDESC, QHODAT, QACTIF) values (?, ?, ?, ?, ?))");
function addModeleReleves(params) {
WL.Logger.debug('Starting adapter procedure...');
var modeleReleveResult = WL.Server.invokeSQLStatement({
preparedStatement : addModeleReleveStatement,
parameters : [params.QCDART, params.QTITRE, params.QDESC, params.QHODAT, params.QACTIF]
});
if(modeleReleveResult.isSuccessful) {
WL.Logger.debug('Success !');
}
WL.Logger.debug('Adapter procedure ended !');
// Return result (with the last id inside)
return modeleReleveResult;
}
if the javascript varable called currentPdfFileDataBase64 is small, all is working normally but if it exceeds approximatively 200000 chars length it fails...
Last, I ca say that the problem occurs in development environment (WL Studio 5.0.6 + WL Server 5.0.6), I didn't test it on the production environment based on SLES + Websphere application server 7 + worklight.
Thanks for any help
I understand you are using the test server provided by Worklight.
It looks like this is a Jetty limitation so you could try any of these:
1) Set the system property org.eclipse.jetty.server.Request.maxFormContentSize to a bigger value (i.e. adding -Dorg.eclipse.jetty.server.Request.maxFormContentSize=25000000) to the end of eclipse.ini before launching Worklight.
or
2) Instead, set this other system property -Dorg.mortbay.jetty.Request.maxFormContentSize=25000000 to the same place.
Another way to solve the problem was to use WL Studio version 6 that doesn't use Jetty anymore as test environment

EWS. How to change DateTimeCreate property via EWS Proxy Classes

I write client application that uses Exchange Web Services Proxy Classes in order to connect to Exchange Web Services. Sometimes, I need create ItemType object and make it looks like as received letter. Therefore I need set up such properties of ItemType as DateTimeSent, DateTimeCreate, DateTimeReceived, but they haven’t public set assessTherefore I need set up such properties of ItemType as DateTimeSent, DateTimeCreate, DateTimeReceived, but they haven’t public set assessor.
I found resolve for some of them via MAPI properties:
ItemType newItem = xmlParser.LoadItem(); //info for newItem takes from xml
newItem.ExtendedProperty = new ExtendedPropertyType[1];
PathToExtendedFieldType q = new PathToExtendedFieldType();
q.PropertyTag = "3590"; //DeliveryTime
q.PropertyType = MapiPropertyTypeType.SystemTime;
newItem.ExtendedProperty[0] = new ExtendedPropertyType();
newItem.ExtendedProperty[0].ExtendedFieldURI = q;
newItem.ExtendedProperty[0].Item = new System.DateTime(2014, 5, 5, 5, 5, 5).ToString("yyyy-MM-ddTHH:mm:ssZ");
Well, it works for DateTimeSent and DateTimeReceived, but not for DateTimeCreate. ES dont give any errors, but DateTimeCreate doesnt change. I tried to UpdateItem with DateTimeCreate propery, but there was no result (update another properties runs fine).
P.S. MAPI ID for CreationTime: 0x3007.
Can someone help me with this problem?
I finally found a solution for this.
Source: https://social.msdn.microsoft.com/Forums/en-US/40a29c69-96d3-488b-8f0e-911dd5f04086/setting-a-emailmessage-datetimesent-and-isdraft?forum=exchangesvrdevelopment
You have to set 3 Extended MAPI properties PR_MESSAGE_FLAGS, PR_MESSAGE_DELIVERY_TIME, and PR_CLIENT_SUBMIT_TIME. Make sure when setting the Time you use UTC time.
For example:
EmailMessage emUploadEmail = new EmailMessage(service);
emUploadEmail.MimeContent = new MimeContent("us-ascii", bdBinaryData1);
// PR_CLIENT_SUBMIT_TIME
emUploadEmail.SetExtendedProperty(new ExtendedPropertyDefinition(57,MapiPropertyType.SystemTime), DateTime.Now.ToUniversalTime().ToString("yyyy-MM-ddTHH:mm:ssZ"));
// PR_MESSAGE_DELIVERY_TIME
emUploadEmail.SetExtendedProperty(new ExtendedPropertyDefinition(3590, MapiPropertyType.SystemTime), DateTime.Now.ToUniversalTime().ToString("yyyy-MM-ddTHH:mm:ssZ"));
// PR_MESSAGE_FLAGS
emUploadEmail.SetExtendedProperty(new ExtendedPropertyDefinition(3591,MapiPropertyType.Integer),"1");
emUploadEmail.Save(WellKnownFolderName.Inbox);
Create and last modified dates are read-only and cannot be set. The store provider updates these properties internally.

Sending An HTTP Request using Intersystems Cache

I have the following Business Process defined within a Production on an Intersystems Cache Installation
/// Makes a call to Merlin based on the message sent to it from the pre-processor
Class sgh.Process.MerlinProcessor Extends Ens.BusinessProcess [ ClassType = persistent, ProcedureBlock ]
{
Property WorkingDirectory As %String;
Property WebServer As %String;
Property CacheServer As %String;
Property Port As %String;
Property Location As %String;
Parameter SETTINGS = "WorkingDirectory,WebServer,Location,Port,CacheServer";
Method OnRequest(pRequest As sgh.Message.MerlinTransmissionRequest, Output pResponse As Ens.Response) As %Status
{
Set tSC=$$$OK
Do ##class(sgh.Utils.Debug).LogDebugMsg("Packaging an HTTP request for Saved form "_pRequest.DateTimeSaved)
Set dateTimeSaved = pRequest.DateTimeSaved
Set patientId = pRequest.PatientId
Set latestDateTimeSaved = pRequest.LatestDateTimeSaved
Set formName = pRequest.FormName
Set formId = pRequest.FormId
Set episodeNumber = pRequest.EpisodeNumber
Set sentElectronically = pRequest.SentElectronically
Set styleSheet = pRequest.PrintName
Do ##class(sgh.Utils.Debug).LogDebugMsg("Creating HTTP Request Class")
set HTTPReq = ##class(%Net.HttpRequest).%New()
Set HTTPReq.Server = ..WebServer
Set HTTPReq.Port = ..Port
do HTTPReq.InsertParam("DateTimeSaved",dateTimeSaved)
do HTTPReq.InsertParam("HospitalNumber",patientId)
do HTTPReq.InsertParam("Episode",episodeNumber)
do HTTPReq.InsertParam("Stylesheet",styleSheet)
do HTTPReq.InsertParam("Server",..CacheServer)
Set Status = HTTPReq.Post(..Location,0) Quit:$$$ISERR(tSC)
Do ##class(sgh.Utils.Debug).LogDebugMsg("Sent the following request: "_Status)
Quit tSC
}
}
The thing is when I check the debug value (which is defined as a global) all I get is the number '1' - I have no idea therefore if the request has succeeded or even what is wrong (if it has not)
What do I need to do to find out
A) What is the actual web call being made?
B) What the response is?
There is a really slick way to get the answer the two questions you've asked, regardless of where you're using the code. Check the documentation out on the %Net.HttpRequest object here: http://docs.intersystems.com/ens20102/csp/docbook/DocBook.UI.Page.cls?KEY=GNET_http and the class reference here: http://docs.intersystems.com/ens20102/csp/documatic/%25CSP.Documatic.cls?APP=1&LIBRARY=ENSLIB&CLASSNAME=%25Net.HttpRequest
The class reference for the Post method has a parameter called test, that will do what you're looking for. Here's the excerpt:
method Post(location As %String = "", test As %Integer = 0, reset As %Boolean = 1) as %Status
Issue the Http 'post' request, this is used to send data to the web server such as the results of a form, or upload a file. If this completes correctly the response to this request will be in the HttpResponse. The location is the url to request, e.g. '/test.html'. This can contain parameters which are assumed to be already URL escaped, e.g. '/test.html?PARAM=%25VALUE' sets PARAM to %VALUE. If test is 1 then instead of connecting to a remote machine it will just output what it would have send to the web server to the current device, if test is 2 then it will output the response to the current device after the Post. This can be used to check that it will send what you are expecting. This calls Reset automatically after reading the response, except in test=1 mode or if reset=0.
I recommend moving this code to a test routine to view the output properly in terminal. It would look something like this:
// To view the REQUEST you are sending
Set sc = request.Post("/someserver/servlet/webmethod",1)
// To view the RESPONSE you are receiving
Set sc = request.Post("/someserver/servlet/webmethod",2)
// You could also do something like this to parse your RESPONSE stream
Write request.HttpResponse.Data.Read()
I believe the answer you want to A) is in the Server and Location properties of your %Net.HttpRequest object (e.g., HTTPReq.Server and HTTPReq.Location).
For B), the response information should be in the %Net.HttpResponse object stored in the HttpResponse property (e.g. HTTPReq.HttpResponse) after your call is completed.
I hope this helps!
-Derek
(edited for formatting)
From that code sample it looks like you're using Ensemble, not straight-up Cache.
In that case you should be doing this HTTP call in a Business Operation that uses the HTTP Outbound Adapter, not in your Business Process.
See this link for more info on HTTP Adapters:
http://docs.intersystems.com/ens20102/csp/docbook/DocBook.UI.Page.cls?KEY=EHTP
You should also look into how to use the Ensemble Message Browser. That should help with your logging needs.

Resources