Related
I am implementing integration tests for Spring Boot. One of my endpoint gives back all the users, even the deleted ones. Here deleted means the deleted boolean attribute is set to true on AppUser Entity (~soft delete).
The endpoint works perfectly, I have manual tested with Postman (you can see below the response). The code:
#Test
void testGetAllAppUsers_whenAuthorizedOnlyActiveFalse_gotDeletedUser() throws Exception {
String token = jwtProvider.generateToken(TestUtils.ACCOUNT_MANAGER_USERNAME);
String url = "/api/account/user";
mvc.perform(get(url)
.header("Authorization", token)
.param("only-active", "false"))
.andExpect(status().isOk())
.andExpect(jsonPath("$[*].deleted").value(Matchers.arrayContaining(true)));
}
the response:
...
Body = [{"id":7,"username":"ablabadmin#account.com","fullName":"Alpha Beta Lab Admin","deleted":false,"labsAsUser":[],"labsAsAdmin":[],"roles":[],"managedLabs":[]},{"id":11,"username":"ablabdeleteduser#account.com","fullName":"Alpha Beta Lab Deleted User","deleted":true,"labsAsUser":[],"labsAsAdmin":[],"roles":[],"managedLabs":[]},{"id":4,"username":"ablabman#account.com","fullName":"Alpha Beta Lab Manager","deleted":false,"labsAsUser":[],"labsAsAdmin":[],"roles":[],"managedLabs":[{"id":1,"key":"alab","name":"Alpha Lab","deleted":false,"createdAt":"2021-06-09","updatedAt":null,"labManagerUsernames":["alabman#account.com","ablabman#account.com"]},{"id":2,"key":"blab","name":"Beta Lab","deleted":false,"createdAt":"2021-06-09","updatedAt":null,"labManagerUsernames":["blabman#account.com","ablabman#account.com"]}]},{"id":10,"username":"ablabuser#account.com","fullName":"Alpha Beta Lab User","deleted":false,"labsAsUser":[],"labsAsAdmin":[],"roles":[],"managedLabs":[]},{"id":5,"username":"alabadmin#account.com","fullName":"Alpha Lab Admin","deleted":false,"labsAsUser":[],"labsAsAdmin":[],"roles":[],"managedLabs":[]},{"id":2,"username":"alabman#account.com","fullName":"Alpha Lab Manager","deleted":false,"labsAsUser":[],"labsAsAdmin":[],"roles":[],"managedLabs":[{"id":1,"key":"alab","name":"Alpha Lab","deleted":false,"createdAt":"2021-06-09","updatedAt":null,"labManagerUsernames":["alabman#account.com","ablabman#account.com"]}]},{"id":8,"username":"alabuser#account.com","fullName":"Alpha Lab User","deleted":false,"labsAsUser":[],"labsAsAdmin":[],"roles":[],"managedLabs":[]},{"id":1,"username":"aman#account.com","fullName":"Account Manager","deleted":false,"labsAsUser":[],"labsAsAdmin":[],"roles":[{"id":1,"role":"ACCOUNT_MANAGER"}],"managedLabs":[]},{"id":6,"username":"blabadmin#account.com","fullName":"Beta Lab Admin","deleted":false,"labsAsUser":[],"labsAsAdmin":[],"roles":[],"managedLabs":[]},{"id":3,"username":"blabman#account.com","fullName":"Beta Lab Manager","deleted":false,"labsAsUser":[],"labsAsAdmin":[],"roles":[],"managedLabs":[{"id":2,"key":"blab","name":"Beta Lab","deleted":false,"createdAt":"2021-06-09","updatedAt":null,"labManagerUsernames":["blabman#account.com","ablabman#account.com"]}]},{"id":9,"username":"blabuser#account.com","fullName":"Beta Lab User","deleted":false,"labsAsUser":[],"labsAsAdmin":[],"roles":[],"managedLabs":[]}]
Forwarded URL = null
Redirected URL = null
Cookies = []
java.lang.AssertionError: JSON path "$[*].deleted"
Expected: [<true>]
but: was a net.minidev.json.JSONArray (<[false,true,false,false,false,false,false,false,false,false,false]>)
Expected :[<true>]
Actual :a net.minidev.json.JSONArray (<[false,true,false,false,false,false,false,false,false,false,false]>)
...
I have tried out many other arguments: "true", false I have tried out contains method, and I ave tried out with "$[*].username" regex with an existing username, but every time I got AssertionError.
Can you please help me? There is definitely something I am missing. Also any advice is welcomed, since I am relatively new in Spring integration testing. Thanks in advance!
arrayContainingInAnyOrder(true,false) should work in this case.
But,
The test case would not have any value, even if it works with arrayContainingInAnyOrder(true,false).
As you can see, that [*].deleted returns value of deleted for all objects present in the response. The test must verify specific value.
For Example, If you are expecting that first user should have true then assert that $.[?(#.id ==1)].deleted is true.
I'm trying to use SimpleHttpOperator for consuming a RESTful API. But, As the name suggests, it only supporting HTTP protocol where I need to consume a HTTPS URI. so, now, I have to use either "requests" object from Python or handle the invocation from within the application code. But, It may not be a standard way. so, I'm looking for any other options available to consume HTTPS URI from within Airflow. Thanks.
I dove into this and am pretty sure that this behavior is a bug in airflow. I have created a ticket for it here:
https://issues.apache.org/jira/browse/AIRFLOW-2910
For now, the best you can do is override SimpleHttpOperator as well as HttpHook in order to change the way that HttpHook.get_conn works (to accept https). I may end up doing this, and if I do I'll post some code.
Update:
Operator override:
from airflow.operators.http_operator import SimpleHttpOperator
from airflow.exceptions import AirflowException
from operators.https_support.https_hook import HttpsHook
class HttpsOperator(SimpleHttpOperator):
def execute(self, context):
http = HttpsHook(self.method, http_conn_id=self.http_conn_id)
self.log.info("Calling HTTP method")
response = http.run(self.endpoint,
self.data,
self.headers,
self.extra_options)
if self.response_check:
if not self.response_check(response):
raise AirflowException("Response check returned False.")
if self.xcom_push_flag:
return response.text
Hook override
from airflow.hooks.http_hook import HttpHook
import requests
class HttpsHook(HttpHook):
def get_conn(self, headers):
"""
Returns http session for use with requests. Supports https.
"""
conn = self.get_connection(self.http_conn_id)
session = requests.Session()
if "://" in conn.host:
self.base_url = conn.host
elif conn.schema:
self.base_url = conn.schema + "://" + conn.host
elif conn.conn_type: # https support
self.base_url = conn.conn_type + "://" + conn.host
else:
# schema defaults to HTTP
self.base_url = "http://" + conn.host
if conn.port:
self.base_url = self.base_url + ":" + str(conn.port) + "/"
if conn.login:
session.auth = (conn.login, conn.password)
if headers:
session.headers.update(headers)
return session
Usage:
Drop-in replacement for SimpleHttpOperator.
This is a couple of months old now, but for what it is worth I did not have any issue with making an HTTPS call on Airflow 1.10.2.
In my initial test I was making a request for templates from sendgrid, so the connection was set up like this:
Conn Id : sendgrid_templates_test
Conn Type : HTTP
Host : https://api.sendgrid.com/
Extra : { "authorization": "Bearer [my token]"}
and then in the dag code:
get_templates = SimpleHttpOperator(
task_id='get_templates',
method='GET',
endpoint='/v3/templates',
http_conn_id = 'sendgrid_templates_test',
trigger_rule="all_done",
xcom_push=True
dag=dag,
)
and that worked. Also notice that my request happens after a Branch Operator, so I needed to set the trigger rule appropriately (to "all_done" to make sure it fires even when one of the branches is skipped), which has nothing to do with the question, but I just wanted to point it out.
Now to be clear, I did get an Insecure Request warning as I did not have certificate verification enabled. But you can see the resulting logs below
[2019-02-21 16:15:01,333] {http_operator.py:89} INFO - Calling HTTP method
[2019-02-21 16:15:01,336] {logging_mixin.py:95} INFO - [2019-02-21 16:15:01,335] {base_hook.py:83} INFO - Using connection to: id: sendgrid_templates_test. Host: https://api.sendgrid.com/, Port: None, Schema: None, Login: None, Password: XXXXXXXX, extra: {'authorization': 'Bearer [my token]'}
[2019-02-21 16:15:01,338] {logging_mixin.py:95} INFO - [2019-02-21 16:15:01,337] {http_hook.py:126} INFO - Sending 'GET' to url: https://api.sendgrid.com//v3/templates
[2019-02-21 16:15:01,956] {logging_mixin.py:95} WARNING - /home/csconnell/.pyenv/versions/airflow/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
[2019-02-21 16:15:05,242] {logging_mixin.py:95} INFO - [2019-02-21 16:15:05,241] {jobs.py:2527} INFO - Task exited with return code 0
I was having the same problem with HTTP/HTTPS when trying to set the connections using environment variables (although it works when i set the connection on the UI).
I've checked the issue #melchoir55 opened (https://issues.apache.org/jira/browse/AIRFLOW-2910) and you don't need to make a custom operator for that, the problem is not that HttpHook or HttpOperator can't use HTTPS, the problem is the way get_hook parse the connection string when dealing with HTTP, it actually understand that the first part (http:// or https://) is the connection type.
In summary, you don't need a custom operator, you can just set the connection in your env as the following:
AIRFLOW_CONN_HTTP_EXAMPLE=http://https%3a%2f%2fexample.com/
Instead of:
AIRFLOW_CONN_HTTP_EXAMPLE=https://example.com/
Or set the connection on the UI.
It is not a intuitive way to set up a connection but I think they are working on a better way to parse connections for Ariflow 2.0.
In Airflow 2.x you can use https URLs by passing https for schema value while setting up your connection and can still use SimpleHttpOperator like shown below.
my_api = SimpleHttpOperator(
task_id="my_api",
http_conn_id="YOUR_CONN_ID",
method="POST",
endpoint="/base-path/end-point",
data=get_data,
headers={"Content-Type": "application/json"},
)
Instead of implementing HttpsHook, we could just put one line of codes into HttpsOperator(SimpleHttpOperator)#above as follows
...
self.extra_options['verify'] = True
response = http.run(self.endpoint,
self.data,
self.headers,
self.extra_options)
...
in Airflow 2, the problem is been resolved.
just check out that :
host name in Connection UI Form, don't end up with /
'endpoint' parameter of SimpleHttpOperator starts with /
I am using Airflow 2.1.0，and the following setting works for https API
In connection UI, setting host name as usual, no need to specify 'https' in schema field, don't forget to set login account and password if your API server request ones.
Connection UI Setting
When constructing your task, add extra_options parameter in SimpleHttpOperator, and put your CA_bundle certification file path as the value for key verify, if you don't have a certification file, then use false to skip verification.
Task definition
Reference: here
I'm trying to query my Server 2012 Essentials R2 server to determine the most recent Client Backup time for a given Device, so I can display nag screens at signon for forgetful users. (They're on laptops, so I can't depend on the machine being available during the automatic window.)
The closest thing in the way of documentation I've been able to find is this: (https://msdn.microsoft.com/en-us/library/jj713757.aspx)
GET services/builtin/DeviceManagement.svc/devices/index/{index}/count/{count}
But it requires a preceding call to get the token: (https://msdn.microsoft.com/en-us/library/jj713753.aspx)
GET https://www.contoso.com/services/builtin/session.svc/login HTTP/1.1
Accept: application/xml
Host: servername
Authorization: Basic VXNlcjpQYXNzd29yZCE=
AppName: Sample App Name
AppPublisher: publisher
AppVersion: 1.0
Does anyone know what the values for those last three headers should be—or how to discover them—for a standard WSE 2012 R2 installation? The documentation provides no assistance here.
Or if someone knows a better way to accomplish this, please let me know.
OK, I got it working. The code is below.
As it turns out, the value of the AppName header is irrelevant—it can be any string, but it can't be empty.
I already knew it couldn't be empty from a look at the WSE source in Wssg.WebApi.Framework in the GAC, but the code is decoupled to the point that it's next to impossible to find out what process picks up the the RemoteConnectionClientInfo object once it gets dropped into the HTTP session.
The part that was misleading me was—go figure—the documentation itself.
There's a bang (!) after the password on the Authentication page, suggesting that it should trail the actual password prior to encoding. This was why I was getting an authentication error, which in turn I was (mistakenly) attributing to the statement in the documentation: "Add Appname, Apppublisher, and Appversion values in HTTP header fields. These values are also required to log on."
So once I cleared all that up, I sailed right in.
And there are other errors in the documentation. On the Devices page we are told that the Host header should be set to the domain name, and that a Content-Length header should be added.
These are both incorrect. The Host header should be the server's hostname and there should be no Content-Length header (that's a response header, not a request header).
AND...! After all this, I find that the Device info returned doesn't contain the most recent backup time. I'll have to dig further for that. But at least now I can connect.
So Microsoft's incomplete, inaccurate and sloppy documentation has cost me a day's work. Hopefully somebody else can use this and avoid the pain I went through.
Module Main
Public Sub Main()
Dim aCredentials() As Byte
Dim _
oAuthenticateUri,
oDeviceListUri As Uri
Dim _
sCanary,
sCookie,
sDevices As String
aCredentials = Encoding.ASCII.GetBytes($"{USERNAME}:{PASSWORD}")
Using oClient As New HttpClient
oAuthenticateUri = New Uri($"https://{HOST}/services/builtin/session.svc/login")
oDeviceListUri = New Uri($"https://{HOST}/services/builtin/devicemanagement.svc/devices/index/0/count/99")
oClient.DefaultRequestHeaders.Accept.Add(New MediaTypeWithQualityHeaderValue("application/xml"))
oClient.DefaultRequestHeaders.Authorization = New AuthenticationHeaderValue("Basic", Convert.ToBase64String(aCredentials))
oClient.DefaultRequestHeaders.Host = HOST
oClient.DefaultRequestHeaders.Add("AppPublisher", String.Empty)
oClient.DefaultRequestHeaders.Add("AppVersion", String.Empty)
oClient.DefaultRequestHeaders.Add("AppName", "None")
Using oAuthenticateResponse As HttpResponseMessage = oClient.GetAsync(oAuthenticateUri).Result
If oAuthenticateResponse.IsSuccessStatusCode Then
sCanary = oAuthenticateResponse.Headers.Single(Function(Pair) Pair.Key = CANARY_HEADER).Value(0)
sCookie = Split(oAuthenticateResponse.Headers.Single(Function(Pair) Pair.Key = COOKIE_HEADER).Value(0), ";")(0)
oClient.DefaultRequestHeaders.Clear()
oClient.DefaultRequestHeaders.Host = HOST
oClient.DefaultRequestHeaders.Add(CANARY_HEADER, sCanary)
oClient.DefaultRequestHeaders.Add(COOKIE_HEADER, sCookie)
Using oDeviceListResponse As HttpResponseMessage = oClient.GetAsync(oDeviceListUri).Result
If oDeviceListResponse.IsSuccessStatusCode Then
sDevices = oDeviceListResponse.Content.ReadAsStringAsync.Result
Else
Console.WriteLine("{0} ({1})", oDeviceListResponse.StatusCode, oDeviceListResponse.ReasonPhrase)
End If
End Using
Else
Console.WriteLine("{0} ({1})", oAuthenticateResponse.StatusCode, oAuthenticateResponse.ReasonPhrase)
End If
End Using
End Using
End Sub
Private Const CANARY_HEADER As String = "Canary"
Private Const COOKIE_HEADER As String = "Set-Cookie"
Private Const USERNAME As String = "domain.admin"
Private Const PASSWORD As String = "admin.password"
Private Const HOST As String = "server"
End Module
I'm just getting started with SocketStream. (v0.1.0) I created the file /app/server/auth.coffee with an exports.actions.login function. I'd like to access #session.setUserId in this file, but I'm have a hard time figuring out where #session lives and how to access it outside of /app/server/app.coffee
Here is my auth.coffee with comments where I'd like to access the session.
users = [
username: 'craig'
password: 'craig',
username: 'joe'
password: 'joe',
]
authenticate = (credentials, cb) ->
user = _.detect users, (user) ->
user.username == credentials.username and user.password == credentials.password
authenticated = true if user?
callback cb, authenticated
exports.actions =
login: (credentials, cb) ->
authenticate credentials, (user) ->
# here is where i'd like to set the userId like so:
# #session.setUserId credentials.username
callback cb user
Interesting you bring a question about sessions up at the moment as I've been re-writing a lot of this code over the last few days as part of SocketStream 0.2.
The good news is the #session variable will be back in 0.2 as I have found an efficient way to pass the session data through to the back end without having to use the ugly #getSession callback.
To answer your question specifically, the #session variable is simply another property which is injected into the export.actions object before the request is processed. Hence you cannot have an action called 'session' (though the name of this 'magic variable' will be configurable in the next release of 0.2).
The exports.authenticate = true setting does not apply in your case.
I'm interested to know how/why you'd like to use the #session object outside of your /app/server code.
I will be committing all the latest session code to the 0.2 preview branch on github in a few days time.
Hope that helps,
Owen
You get the current session only within your server-side code (app/server) using the #getCurrentSession method.
Also you have to add:
exports.authenticate = true
to that file.
I have the following Business Process defined within a Production on an Intersystems Cache Installation
/// Makes a call to Merlin based on the message sent to it from the pre-processor
Class sgh.Process.MerlinProcessor Extends Ens.BusinessProcess [ ClassType = persistent, ProcedureBlock ]
{
Property WorkingDirectory As %String;
Property WebServer As %String;
Property CacheServer As %String;
Property Port As %String;
Property Location As %String;
Parameter SETTINGS = "WorkingDirectory,WebServer,Location,Port,CacheServer";
Method OnRequest(pRequest As sgh.Message.MerlinTransmissionRequest, Output pResponse As Ens.Response) As %Status
{
Set tSC=$$$OK
Do ##class(sgh.Utils.Debug).LogDebugMsg("Packaging an HTTP request for Saved form "_pRequest.DateTimeSaved)
Set dateTimeSaved = pRequest.DateTimeSaved
Set patientId = pRequest.PatientId
Set latestDateTimeSaved = pRequest.LatestDateTimeSaved
Set formName = pRequest.FormName
Set formId = pRequest.FormId
Set episodeNumber = pRequest.EpisodeNumber
Set sentElectronically = pRequest.SentElectronically
Set styleSheet = pRequest.PrintName
Do ##class(sgh.Utils.Debug).LogDebugMsg("Creating HTTP Request Class")
set HTTPReq = ##class(%Net.HttpRequest).%New()
Set HTTPReq.Server = ..WebServer
Set HTTPReq.Port = ..Port
do HTTPReq.InsertParam("DateTimeSaved",dateTimeSaved)
do HTTPReq.InsertParam("HospitalNumber",patientId)
do HTTPReq.InsertParam("Episode",episodeNumber)
do HTTPReq.InsertParam("Stylesheet",styleSheet)
do HTTPReq.InsertParam("Server",..CacheServer)
Set Status = HTTPReq.Post(..Location,0) Quit:$$$ISERR(tSC)
Do ##class(sgh.Utils.Debug).LogDebugMsg("Sent the following request: "_Status)
Quit tSC
}
}
The thing is when I check the debug value (which is defined as a global) all I get is the number '1' - I have no idea therefore if the request has succeeded or even what is wrong (if it has not)
What do I need to do to find out
A) What is the actual web call being made?
B) What the response is?
There is a really slick way to get the answer the two questions you've asked, regardless of where you're using the code. Check the documentation out on the %Net.HttpRequest object here: http://docs.intersystems.com/ens20102/csp/docbook/DocBook.UI.Page.cls?KEY=GNET_http and the class reference here: http://docs.intersystems.com/ens20102/csp/documatic/%25CSP.Documatic.cls?APP=1&LIBRARY=ENSLIB&CLASSNAME=%25Net.HttpRequest
The class reference for the Post method has a parameter called test, that will do what you're looking for. Here's the excerpt:
method Post(location As %String = "", test As %Integer = 0, reset As %Boolean = 1) as %Status
Issue the Http 'post' request, this is used to send data to the web server such as the results of a form, or upload a file. If this completes correctly the response to this request will be in the HttpResponse. The location is the url to request, e.g. '/test.html'. This can contain parameters which are assumed to be already URL escaped, e.g. '/test.html?PARAM=%25VALUE' sets PARAM to %VALUE. If test is 1 then instead of connecting to a remote machine it will just output what it would have send to the web server to the current device, if test is 2 then it will output the response to the current device after the Post. This can be used to check that it will send what you are expecting. This calls Reset automatically after reading the response, except in test=1 mode or if reset=0.
I recommend moving this code to a test routine to view the output properly in terminal. It would look something like this:
// To view the REQUEST you are sending
Set sc = request.Post("/someserver/servlet/webmethod",1)
// To view the RESPONSE you are receiving
Set sc = request.Post("/someserver/servlet/webmethod",2)
// You could also do something like this to parse your RESPONSE stream
Write request.HttpResponse.Data.Read()
I believe the answer you want to A) is in the Server and Location properties of your %Net.HttpRequest object (e.g., HTTPReq.Server and HTTPReq.Location).
For B), the response information should be in the %Net.HttpResponse object stored in the HttpResponse property (e.g. HTTPReq.HttpResponse) after your call is completed.
I hope this helps!
-Derek
(edited for formatting)
From that code sample it looks like you're using Ensemble, not straight-up Cache.
In that case you should be doing this HTTP call in a Business Operation that uses the HTTP Outbound Adapter, not in your Business Process.
See this link for more info on HTTP Adapters:
http://docs.intersystems.com/ens20102/csp/docbook/DocBook.UI.Page.cls?KEY=EHTP
You should also look into how to use the Ensemble Message Browser. That should help with your logging needs.