I have the following Business Process defined within a Production on an Intersystems Cache Installation
/// Makes a call to Merlin based on the message sent to it from the pre-processor
Class sgh.Process.MerlinProcessor Extends Ens.BusinessProcess [ ClassType = persistent, ProcedureBlock ]
{
Property WorkingDirectory As %String;
Property WebServer As %String;
Property CacheServer As %String;
Property Port As %String;
Property Location As %String;
Parameter SETTINGS = "WorkingDirectory,WebServer,Location,Port,CacheServer";
Method OnRequest(pRequest As sgh.Message.MerlinTransmissionRequest, Output pResponse As Ens.Response) As %Status
{
Set tSC=$$$OK
Do ##class(sgh.Utils.Debug).LogDebugMsg("Packaging an HTTP request for Saved form "_pRequest.DateTimeSaved)
Set dateTimeSaved = pRequest.DateTimeSaved
Set patientId = pRequest.PatientId
Set latestDateTimeSaved = pRequest.LatestDateTimeSaved
Set formName = pRequest.FormName
Set formId = pRequest.FormId
Set episodeNumber = pRequest.EpisodeNumber
Set sentElectronically = pRequest.SentElectronically
Set styleSheet = pRequest.PrintName
Do ##class(sgh.Utils.Debug).LogDebugMsg("Creating HTTP Request Class")
set HTTPReq = ##class(%Net.HttpRequest).%New()
Set HTTPReq.Server = ..WebServer
Set HTTPReq.Port = ..Port
do HTTPReq.InsertParam("DateTimeSaved",dateTimeSaved)
do HTTPReq.InsertParam("HospitalNumber",patientId)
do HTTPReq.InsertParam("Episode",episodeNumber)
do HTTPReq.InsertParam("Stylesheet",styleSheet)
do HTTPReq.InsertParam("Server",..CacheServer)
Set Status = HTTPReq.Post(..Location,0) Quit:$$$ISERR(tSC)
Do ##class(sgh.Utils.Debug).LogDebugMsg("Sent the following request: "_Status)
Quit tSC
}
}
The thing is when I check the debug value (which is defined as a global) all I get is the number '1' - I have no idea therefore if the request has succeeded or even what is wrong (if it has not)
What do I need to do to find out
A) What is the actual web call being made?
B) What the response is?
There is a really slick way to get the answer the two questions you've asked, regardless of where you're using the code. Check the documentation out on the %Net.HttpRequest object here: http://docs.intersystems.com/ens20102/csp/docbook/DocBook.UI.Page.cls?KEY=GNET_http and the class reference here: http://docs.intersystems.com/ens20102/csp/documatic/%25CSP.Documatic.cls?APP=1&LIBRARY=ENSLIB&CLASSNAME=%25Net.HttpRequest
The class reference for the Post method has a parameter called test, that will do what you're looking for. Here's the excerpt:
method Post(location As %String = "", test As %Integer = 0, reset As %Boolean = 1) as %Status
Issue the Http 'post' request, this is used to send data to the web server such as the results of a form, or upload a file. If this completes correctly the response to this request will be in the HttpResponse. The location is the url to request, e.g. '/test.html'. This can contain parameters which are assumed to be already URL escaped, e.g. '/test.html?PARAM=%25VALUE' sets PARAM to %VALUE. If test is 1 then instead of connecting to a remote machine it will just output what it would have send to the web server to the current device, if test is 2 then it will output the response to the current device after the Post. This can be used to check that it will send what you are expecting. This calls Reset automatically after reading the response, except in test=1 mode or if reset=0.
I recommend moving this code to a test routine to view the output properly in terminal. It would look something like this:
// To view the REQUEST you are sending
Set sc = request.Post("/someserver/servlet/webmethod",1)
// To view the RESPONSE you are receiving
Set sc = request.Post("/someserver/servlet/webmethod",2)
// You could also do something like this to parse your RESPONSE stream
Write request.HttpResponse.Data.Read()
I believe the answer you want to A) is in the Server and Location properties of your %Net.HttpRequest object (e.g., HTTPReq.Server and HTTPReq.Location).
For B), the response information should be in the %Net.HttpResponse object stored in the HttpResponse property (e.g. HTTPReq.HttpResponse) after your call is completed.
I hope this helps!
-Derek
(edited for formatting)
From that code sample it looks like you're using Ensemble, not straight-up Cache.
In that case you should be doing this HTTP call in a Business Operation that uses the HTTP Outbound Adapter, not in your Business Process.
See this link for more info on HTTP Adapters:
http://docs.intersystems.com/ens20102/csp/docbook/DocBook.UI.Page.cls?KEY=EHTP
You should also look into how to use the Ensemble Message Browser. That should help with your logging needs.
Related
Regarding: “Sending response back to the out/reply queue.”
There is a requirement to send the response back to a different queue (reply queue).
While sending the response, we have to use the correlation and message id from the request message and pass it to the reply queue as header. I suspect the format of correlation/message id is wrong.
While reading the message, the correlation id and message id format are as below:
MessageId = “ID:616365323063633033343361313165646139306638346264”
CorrelationId = “ID:36626161303030305f322020202020202020202020202020”
While sending the back to out/reply queue, we are passing these ids as below:
ITextMessage txtReplyMessage = sessionOut.CreateTextMessage();
txtReplyMessage.JMSMessageID = “616365323063633033343361313165646139306638346264”;
txtReplyMessage.JMSCorrelationID = “36626161303030305f322020202020202020202020202020”;
txtReplyMessage.Text = sentMessage.Contents;
txtReplyMessage.JMSDeliveryMode = DeliveryMode.NonPersistent;
txtReplyMessage.JMSPriority = sentMessage.Priority;
messagePoducerOut.Send(txtReplyMessage);
Please note:
With the XMS.NET library, we need to pass the correlation and message id in string format as per shown above
With MQ API’s (which we were using earlier) passing the correlation and message ids we use to send in bytes format like below:
MQMessage queueMessage = new MQMessage();
string[] parms = document.name.Split('-');
queueMessage.MessageId = StringToByte(parms[1]);
queueMessage.CorrelationId = StringToByte(parms[2]);
queueMessage.CharacterSet = 1208;
queueMessage.Encoding = MQC.MQENC_NATIVE;
queueMessage.Persistence = 0; // Do not persist the replay message.
queueMessage.Format = "MQSTR ";
queueMessage.WriteString(document.contents);
queueOut.Put(queueMessage);
queueManagerOut.Commit();
Please help to troubleshoot the problem.
Troubleshooting is a bit difficult because you haven’t clearly specified the trouble (is there an exception, or is the message just not be correlated successfully?).
In your code you have missed to add the “ID:” prefix. However, to address the requirements, you should not need to bother too much about what is in this field, because you simply need to copy one value to the other:
txtReplyMessage.JMSCorrelationID = txtRequestMessage.JMSMessageID
A bit unclear what the issue is. Are you able to run the provided examples in the MQ tools/examples? This approach uses tmp queues(AMQ.*) as JMSReplyTo
Start the "server" application first.
Request/Response Client: "SimpleRequestor"
Request/Response Server: "SimpleRequestorServer"
You can find the exmaples at the default install location(win):
"C:\Program Files\IBM\MQ\tools\dotnet\samples\cs\xms\simple\wmq"
The "SimpleMessageSelector" will show how to use the selector pattern.
Note the format on the selector: "JMSCorrelationID = '00010203040506070809'"
IBM MQ SELECTOR
I'm trying to use SimpleHttpOperator for consuming a RESTful API. But, As the name suggests, it only supporting HTTP protocol where I need to consume a HTTPS URI. so, now, I have to use either "requests" object from Python or handle the invocation from within the application code. But, It may not be a standard way. so, I'm looking for any other options available to consume HTTPS URI from within Airflow. Thanks.
I dove into this and am pretty sure that this behavior is a bug in airflow. I have created a ticket for it here:
https://issues.apache.org/jira/browse/AIRFLOW-2910
For now, the best you can do is override SimpleHttpOperator as well as HttpHook in order to change the way that HttpHook.get_conn works (to accept https). I may end up doing this, and if I do I'll post some code.
Update:
Operator override:
from airflow.operators.http_operator import SimpleHttpOperator
from airflow.exceptions import AirflowException
from operators.https_support.https_hook import HttpsHook
class HttpsOperator(SimpleHttpOperator):
def execute(self, context):
http = HttpsHook(self.method, http_conn_id=self.http_conn_id)
self.log.info("Calling HTTP method")
response = http.run(self.endpoint,
self.data,
self.headers,
self.extra_options)
if self.response_check:
if not self.response_check(response):
raise AirflowException("Response check returned False.")
if self.xcom_push_flag:
return response.text
Hook override
from airflow.hooks.http_hook import HttpHook
import requests
class HttpsHook(HttpHook):
def get_conn(self, headers):
"""
Returns http session for use with requests. Supports https.
"""
conn = self.get_connection(self.http_conn_id)
session = requests.Session()
if "://" in conn.host:
self.base_url = conn.host
elif conn.schema:
self.base_url = conn.schema + "://" + conn.host
elif conn.conn_type: # https support
self.base_url = conn.conn_type + "://" + conn.host
else:
# schema defaults to HTTP
self.base_url = "http://" + conn.host
if conn.port:
self.base_url = self.base_url + ":" + str(conn.port) + "/"
if conn.login:
session.auth = (conn.login, conn.password)
if headers:
session.headers.update(headers)
return session
Usage:
Drop-in replacement for SimpleHttpOperator.
This is a couple of months old now, but for what it is worth I did not have any issue with making an HTTPS call on Airflow 1.10.2.
In my initial test I was making a request for templates from sendgrid, so the connection was set up like this:
Conn Id : sendgrid_templates_test
Conn Type : HTTP
Host : https://api.sendgrid.com/
Extra : { "authorization": "Bearer [my token]"}
and then in the dag code:
get_templates = SimpleHttpOperator(
task_id='get_templates',
method='GET',
endpoint='/v3/templates',
http_conn_id = 'sendgrid_templates_test',
trigger_rule="all_done",
xcom_push=True
dag=dag,
)
and that worked. Also notice that my request happens after a Branch Operator, so I needed to set the trigger rule appropriately (to "all_done" to make sure it fires even when one of the branches is skipped), which has nothing to do with the question, but I just wanted to point it out.
Now to be clear, I did get an Insecure Request warning as I did not have certificate verification enabled. But you can see the resulting logs below
[2019-02-21 16:15:01,333] {http_operator.py:89} INFO - Calling HTTP method
[2019-02-21 16:15:01,336] {logging_mixin.py:95} INFO - [2019-02-21 16:15:01,335] {base_hook.py:83} INFO - Using connection to: id: sendgrid_templates_test. Host: https://api.sendgrid.com/, Port: None, Schema: None, Login: None, Password: XXXXXXXX, extra: {'authorization': 'Bearer [my token]'}
[2019-02-21 16:15:01,338] {logging_mixin.py:95} INFO - [2019-02-21 16:15:01,337] {http_hook.py:126} INFO - Sending 'GET' to url: https://api.sendgrid.com//v3/templates
[2019-02-21 16:15:01,956] {logging_mixin.py:95} WARNING - /home/csconnell/.pyenv/versions/airflow/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
[2019-02-21 16:15:05,242] {logging_mixin.py:95} INFO - [2019-02-21 16:15:05,241] {jobs.py:2527} INFO - Task exited with return code 0
I was having the same problem with HTTP/HTTPS when trying to set the connections using environment variables (although it works when i set the connection on the UI).
I've checked the issue #melchoir55 opened (https://issues.apache.org/jira/browse/AIRFLOW-2910) and you don't need to make a custom operator for that, the problem is not that HttpHook or HttpOperator can't use HTTPS, the problem is the way get_hook parse the connection string when dealing with HTTP, it actually understand that the first part (http:// or https://) is the connection type.
In summary, you don't need a custom operator, you can just set the connection in your env as the following:
AIRFLOW_CONN_HTTP_EXAMPLE=http://https%3a%2f%2fexample.com/
Instead of:
AIRFLOW_CONN_HTTP_EXAMPLE=https://example.com/
Or set the connection on the UI.
It is not a intuitive way to set up a connection but I think they are working on a better way to parse connections for Ariflow 2.0.
In Airflow 2.x you can use https URLs by passing https for schema value while setting up your connection and can still use SimpleHttpOperator like shown below.
my_api = SimpleHttpOperator(
task_id="my_api",
http_conn_id="YOUR_CONN_ID",
method="POST",
endpoint="/base-path/end-point",
data=get_data,
headers={"Content-Type": "application/json"},
)
Instead of implementing HttpsHook, we could just put one line of codes into HttpsOperator(SimpleHttpOperator)#above as follows
...
self.extra_options['verify'] = True
response = http.run(self.endpoint,
self.data,
self.headers,
self.extra_options)
...
in Airflow 2, the problem is been resolved.
just check out that :
host name in Connection UI Form, don't end up with /
'endpoint' parameter of SimpleHttpOperator starts with /
I am using Airflow 2.1.0,and the following setting works for https API
In connection UI, setting host name as usual, no need to specify 'https' in schema field, don't forget to set login account and password if your API server request ones.
Connection UI Setting
When constructing your task, add extra_options parameter in SimpleHttpOperator, and put your CA_bundle certification file path as the value for key verify, if you don't have a certification file, then use false to skip verification.
Task definition
Reference: here
I'm trying to query my Server 2012 Essentials R2 server to determine the most recent Client Backup time for a given Device, so I can display nag screens at signon for forgetful users. (They're on laptops, so I can't depend on the machine being available during the automatic window.)
The closest thing in the way of documentation I've been able to find is this: (https://msdn.microsoft.com/en-us/library/jj713757.aspx)
GET services/builtin/DeviceManagement.svc/devices/index/{index}/count/{count}
But it requires a preceding call to get the token: (https://msdn.microsoft.com/en-us/library/jj713753.aspx)
GET https://www.contoso.com/services/builtin/session.svc/login HTTP/1.1
Accept: application/xml
Host: servername
Authorization: Basic VXNlcjpQYXNzd29yZCE=
AppName: Sample App Name
AppPublisher: publisher
AppVersion: 1.0
Does anyone know what the values for those last three headers should be—or how to discover them—for a standard WSE 2012 R2 installation? The documentation provides no assistance here.
Or if someone knows a better way to accomplish this, please let me know.
OK, I got it working. The code is below.
As it turns out, the value of the AppName header is irrelevant—it can be any string, but it can't be empty.
I already knew it couldn't be empty from a look at the WSE source in Wssg.WebApi.Framework in the GAC, but the code is decoupled to the point that it's next to impossible to find out what process picks up the the RemoteConnectionClientInfo object once it gets dropped into the HTTP session.
The part that was misleading me was—go figure—the documentation itself.
There's a bang (!) after the password on the Authentication page, suggesting that it should trail the actual password prior to encoding. This was why I was getting an authentication error, which in turn I was (mistakenly) attributing to the statement in the documentation: "Add Appname, Apppublisher, and Appversion values in HTTP header fields. These values are also required to log on."
So once I cleared all that up, I sailed right in.
And there are other errors in the documentation. On the Devices page we are told that the Host header should be set to the domain name, and that a Content-Length header should be added.
These are both incorrect. The Host header should be the server's hostname and there should be no Content-Length header (that's a response header, not a request header).
AND...! After all this, I find that the Device info returned doesn't contain the most recent backup time. I'll have to dig further for that. But at least now I can connect.
So Microsoft's incomplete, inaccurate and sloppy documentation has cost me a day's work. Hopefully somebody else can use this and avoid the pain I went through.
Module Main
Public Sub Main()
Dim aCredentials() As Byte
Dim _
oAuthenticateUri,
oDeviceListUri As Uri
Dim _
sCanary,
sCookie,
sDevices As String
aCredentials = Encoding.ASCII.GetBytes($"{USERNAME}:{PASSWORD}")
Using oClient As New HttpClient
oAuthenticateUri = New Uri($"https://{HOST}/services/builtin/session.svc/login")
oDeviceListUri = New Uri($"https://{HOST}/services/builtin/devicemanagement.svc/devices/index/0/count/99")
oClient.DefaultRequestHeaders.Accept.Add(New MediaTypeWithQualityHeaderValue("application/xml"))
oClient.DefaultRequestHeaders.Authorization = New AuthenticationHeaderValue("Basic", Convert.ToBase64String(aCredentials))
oClient.DefaultRequestHeaders.Host = HOST
oClient.DefaultRequestHeaders.Add("AppPublisher", String.Empty)
oClient.DefaultRequestHeaders.Add("AppVersion", String.Empty)
oClient.DefaultRequestHeaders.Add("AppName", "None")
Using oAuthenticateResponse As HttpResponseMessage = oClient.GetAsync(oAuthenticateUri).Result
If oAuthenticateResponse.IsSuccessStatusCode Then
sCanary = oAuthenticateResponse.Headers.Single(Function(Pair) Pair.Key = CANARY_HEADER).Value(0)
sCookie = Split(oAuthenticateResponse.Headers.Single(Function(Pair) Pair.Key = COOKIE_HEADER).Value(0), ";")(0)
oClient.DefaultRequestHeaders.Clear()
oClient.DefaultRequestHeaders.Host = HOST
oClient.DefaultRequestHeaders.Add(CANARY_HEADER, sCanary)
oClient.DefaultRequestHeaders.Add(COOKIE_HEADER, sCookie)
Using oDeviceListResponse As HttpResponseMessage = oClient.GetAsync(oDeviceListUri).Result
If oDeviceListResponse.IsSuccessStatusCode Then
sDevices = oDeviceListResponse.Content.ReadAsStringAsync.Result
Else
Console.WriteLine("{0} ({1})", oDeviceListResponse.StatusCode, oDeviceListResponse.ReasonPhrase)
End If
End Using
Else
Console.WriteLine("{0} ({1})", oAuthenticateResponse.StatusCode, oAuthenticateResponse.ReasonPhrase)
End If
End Using
End Using
End Sub
Private Const CANARY_HEADER As String = "Canary"
Private Const COOKIE_HEADER As String = "Set-Cookie"
Private Const USERNAME As String = "domain.admin"
Private Const PASSWORD As String = "admin.password"
Private Const HOST As String = "server"
End Module
I am trying to port tests from using FakeRequest to using WithServer.
In order to simulate a session with FakeRequest, it is possible to use WithSession("key", "value") as suggested in this post: Testing controller with fake session
However when using WithServer, the test now looks like:
"render the users page" in WithServer {
val users = await(WS.url("http://localhost:" + port + "/users").get)
users.status must equalTo(OK)
users.body must contain("Users")
}
Since there is no WithSession(..) method available, I tried instead WithHeaders(..) (does that even make sense?), to no avail.
Any ideas?
Thanks
So I found this question, which is relatively old:
Add values to Session during testing (FakeRequest, FakeApplication)
The first answer to that question seems to have been a pull request to add .WithSession(...) to FakeRequest, but it was not applicable to WS.url
The second answer seems to give me what I need:
Create cookie:
val sessionCookie = Session.encodeAsCookie(Session(Map("key" -> "value")))
Create and execute request:
val users = await(WS.url("http://localhost:" + port + "/users")
.withHeaders(play.api.http.HeaderNames.COOKIE -> Cookies.encodeCookieHeader(Seq(sessionCookie))).get())
users.status must equalTo(OK)
users.body must contain("Users")
Finally, the assertions will pass properly, instead of redirecting me to the login page
Note: I am using Play 2.4, so I use Cookies.encodeCookieHeader, because Cookies.encode is deprecated
I am doing a video crawler in ruby. In there I have to log in to a page by enabling cookies and download pages. For that I am using the CURL library in ruby. I can successfully log in, but I can't download the pages inside that with curl. How can I fix this or download the pages otherwise?
My code is
curl = Curl::Easy.new(1st url)
curl.follow_location = true
curl.enable_cookies = true
curl.cookiefile = "cookie.txt"
curl.cookiejar = "cookie.txt"
curl.http_post(1st url,field)
curl.perform
curl = Curl::Easy.perform(2nd url)
curl.follow_location = true
curl.enable_cookies = true
curl.cookiefile = "cookie.txt"
curl.cookiejar = "cookie.txt"
curl.http_get
code = curl.body_str
What I've seen in writing my own similar "post-then-get" script is that ruby/Curb (I'm using version 0.7.15 with ruby 1.8) seems to ignore the cookiejar/cookiefile fields of a Curl::Easy object. If I set either of those fields and the http_post completes successfully, no cookiejar or cookiefile file is created. Also, curl.cookies will still be nil after your curl.http_post, however, the cookies ARE set within the curl object. I promise :)
I think where you're going wrong is here:
curl = Curl::Easy.perform(2nd url)
The curb documentation states that this creates a new object. That new object doesn't have any of your existing cookies set. If you change your code to look like the following, I believe it should work. I've also removed the curl.perform for the first url since curl.http_post already implicitly does the "perform". You were basically http_post'ing twice before trying your http_get.
curl = Curl::Easy.new(1st url)
curl.follow_location = true
curl.enable_cookies = true
curl.http_post(1st url,field)
curl.url = 2nd url
curl.http_get
code = curl.body_str
If this still doesn't seem to be working for you, you can verify if the cookie is getting set by adding
curl.verbose = true
Before
curl.http_post
Your Curl::Easy object will dump all the headers that it gets in the response from the server to $stdout, and somewhere in there you should see a line stating that it added/set a cookie. I don't have any example output right now but I'll try to post a follow-up soon.
HTTPClient automatically enables cookies, as does Mechanize.
From the HTTPClient docs:
clnt = HTTPClient.new
clnt.get_content(url1) # receives Cookies.
clnt.get_content(url2) # sends Cookies if needed.
Posting a form is easy too:
body = { 'keyword' => 'ruby', 'lang' => 'en' }
res = clnt.post(uri, body)
Mechanize makes this sort of thing really simple (It will handle storing the cookies, among other things).