I am trying to get the "Pipeline Input" to somehow be passed to an external adapter via the $(jobRun.requestBody) pipeline variable and then parsed by a jsonparse task and then sent via a fetch task. I am not sure what format the input should be in when running a webhook job on a Chainlink node. I keep getting this and other errors no matter what I try:
data: key requestBody (segment 1 in keypath jobRun.requestBody): keypath not found
This is what I am seeing on the Chainlink Admin UI:
Here is the closest thing I have found in the documentation:
- https://docs.chain.link/chainlink-nodes/oracle-jobs/job-types/webhook
Here is the job definition if useful:
type = "webhook"
schemaVersion = 1
name = "account-balance-webhook"
forwardingAllowed = false
observationSource = """
parse_request [type="jsonparse" path="data,address" data="$(jobRun.requestBody)"]
fetch [type=bridge name="test" requestData="{\\"id\\": \\"0\\", \\"data\\": { \\"address\\": \\"$(parse_request)\\"}}"]
parse [type=jsonparse path="data,free" data="$(fetch)"]
parse_request -> fetch -> parse
"""
I am running Chainlink in a Docker container with this image: smartcontract/chainlink:1.11.0-root
Some background: I am working on developing an external adapter and want to be able to easily and quickly test.
We use the following webhook job to quickly verify there is no syntax errors, etc. with the bridge to an EA.
type = "webhook"
schemaVersion = 1
name = "[WH-username] cbor0-v0"
observationSource = """
fetch [type="bridge" name="bridge_name" requestData="{ \\"id\\": $(jobSpec.externalJobID), \\"input1\\": \\"value1\\" }"]
parse [type=jsonparse path="data,results" data="$(fetch)"]
fetch -> parse
"""
In general, its best to quickly test an EA directly through curl GET/POST. If the curl works, then a bridge will work as long as you named the bridge correctly in the job-spec.toml
Related
I'm using MIP file sample command line interface to apply labelling.
When trying to apply a label that has protection set, i got "Label requires ad-hoc protection, but protection has not yet been set" error.
Therefore, I tried protecting the file using "--protect" option and got the following error message:
"Something bad happened: The service didn't accept the auth token. Challenge:['Bearer resource="https://aadrm.com", realm="", authorization="https://login.windows.net/common/oauth2/authorize"'], CorrelationId=ce732e4a-249a-47ec-a7c2-04f4d68357da, CorrelationId.Description=ProtectionEngine, CorrelationId=6ff992dc-91b3-4610-a24d-d57e13902114, CorrelationId.Description=FileHandler"
This is my auth.py file:
def main(argv):
client_id = str(argv[0])
tenant_id = str(argv[1])
secret = str(argv[2])
authority = "https://login.microsoftonline.com/{}".format(tenant_id)
app = msal.ConfidentialClientApplication(client_id, authority=authority, client_credential=secret)
result = None
scope = ["https://psor.o365syncservice.com/.default"]
result = app.acquire_token_silent(scope, account=None)
if not result:
logging.info("No suitable token exists in cache. Let's get a new one from AAD.")
result = app.acquire_token_for_client(scopes=scope)
if "access_token" in result:
sys.stdout.write(result['access_token'])
else:
print(result.get("error"))
print(result.get("error_description"))
print(result.get("correlation_id")) # You may need this when reporting a bug
if __name__ == '__main__':
main(sys.argv[1:])
I tried to change the scope to ["https://aadrm.com/.default"] and then I was able to protect the file, but when I try getting the file status or try applying label on it I get the same error message with bad auth token.
These are the permissions as configured in azure portal:
Thank you
I think that scope you have is incorrect: https://psor.o365syncservice.com/.default
It should be https://syncservice.o365syncservice.com/.default.
A good way to handle this is to just append .default to whatever resource the AcquireToken() call gets in the resource param. Something like this.
I'm working on a little proof of concept about Airflow on Google Cloud.
Essentially, I want to create a workflow that download data from an REST API (https), transform this data into JSON format and upload it on a Google Cloud storage unit.
I've already done this with pure Python code and it works. Pretty straightforward! But because I want to schedule this and there is some dependencies, Airflow should be the ideal tool for this.
After careful reading of the Airflow documentation, I've seen the HttpOperator and/or HttpHook can do the trick for the download part.
I've created my Http connection into the WebUI with my email/password for the authorization as the following:
{Conn Id: "atlassian_marketplace", Conn Type: "HTTP", Host: "https://marketplace.atlassian.com/rest/2", Schema: None/Blank, Login: "my username", Password: "my password", Port: None/Blank, Extra: None/Blank}
First question:
-When to use the SimpleHttpOperator versus the HttpHook?
Second question:
-How do we use SimpleHttpOperator or HttpHook with HTTPs calls?
Third question:
-How do we access the data returned by the API call?
In my case, the XCOM feature will not do the trick because these API calls can return a lot of data (100-300mb)!
I've look on Google to find an example code on how to use the operaor/hook for my use case but i didn't find anything useful, yet.
Any ideas?
I put here the skeleton of my code so far.
# Usual Airflow import
# Dag creation
dag = DAG(
'get_reporting_links',
default_args=default_args,
description='Get reporting links',
schedule_interval=timedelta(days=1))
# Task 1: Dummy start
start = DummyOperator(task_id="Start", retries=2, dag=dag)
# Task 2: Connect to Atlassian Marketplace
get_data = SimpleHttpOperator(
http_conn_id="atlassian_marketplace",
endpoint="/vendors/{vendorId}/reporting".format({vendorId: "some number"}),
method="GET")
# Task 3: Save JSON data locally
# TODO: transform_json: transform to JSON get_data.json()?
# Task 4: Upload data to GCP
# TODO: upload_gcs: use Airflow GCS connection
# Task 5: Stop
stop = DummyOperator(task_id="Stop", retries=2, dag=dag)
# Dependencies
start >> get_data >> transform_json >> upload_gcs >> stop
Look at the following example:
# Usual Airflow import
# Dag creation
dag = DAG(
'get_reporting_links',
default_args=default_args,
description='Get reporting links',
schedule_interval=timedelta(days=1))
# Task 1: Dummy start
start = DummyOperator(task_id="Start", retries=2, dag=dag)
# Task 2: Connect to Atlassian Marketplace
get_data = SimpleHttpOperator(
task_id="get_data",
http_conn_id="atlassian_marketplace",
endpoint="/vendors/{vendorId}/reporting".format({vendorId: "some number"}),
method="GET",
xcom_push=True,
)
def transform_json(**kwargs):
ti = kwargs['ti']
pulled_value_1 = ti.xcom_pull(key=None, task_ids='get_data')
...
# transform the json here and save the content to a file
# Task 3: Save JSON data locally
save_and_transform = PythonOperator(
task_id="save_and_transform",
python_callable=transform_json,
provide_context=True,
)
# Task 4: Upload data to GCP
upload_to_gcs = FileToGoogleCloudStorageOperator(...)
# Task 5: Stop
stop = DummyOperator(task_id="Stop", retries=2, dag=dag)
# Dependencies
start >> get_data >> save_and_transform >> upload_to_gcs >> stop
I am trying to integrate QnAmaker knowledge base with Azure Bot Service.
I am unable to find knowledge base id on QnAMaker portal.
How to find the kbid in QnAPortal?
The Knowledge Base Id can be located in Settings under “Deployment details” in your knowledge base. It is the guid that is nestled between “knowledgebases” and “generateAnswer” in the POST (see image below).
Hope of help!
Hey you can also use python to get this by take a look at the following code.
That is if you wanted to write a program to dynamically get the kb ids.
import http.client, os, urllib.parse, json, time, sys
# Represents the various elements used to create HTTP request path for QnA Maker
operations.
# Replace this with a valid subscription key.
# User host = '<your-resource-name>.cognitiveservices.azure.com'
host = '<your-resource-name>.cognitiveservices.azure.com'
subscription_key = '<QnA-Key>'
get_kb_method = '/qnamaker/v4.0/knowledgebases/'
try:
headers = {
'Ocp-Apim-Subscription-Key': subscription_key,
'Content-Type': 'application/json'
}
conn = http.client.HTTPSConnection(host)
conn.request ("GET", get_kb_method, None, headers)
response = conn.getresponse()
data = response.read().decode("UTF-8")
result = None
if len(data) > 0:
result = json.loads(data)
print
#print(json.dumps(result, sort_keys=True, indent=2))
# Note status code 204 means success.
KB_id = result["knowledgebases"][0]["id"]
print(response.status)
print(KB_id)
except :
print ("Unexpected error:", sys.exc_info()[0])
print ("Unexpected error:", sys.exc_info()[1])
I am setting up my first Lambda function on AWS. I use Python 3.6. My code is as follows:
def lambda_handler(event, context):
result = {}
result["Log stream name:"] = context.log_stream_name
result["Log group name:"] = context.log_group_name
result["Request ID:"] = context.aws_request_id
result["Mem. limits(MB)"] = context.memory_limit_in_mb
result["size of event"] = len(event)
result["type of event"] = str(type(event))
return result
I also set up an API Gateway for test Lambda.
However, no matter what query paramters I pass in to the API Gateway, the event is always an empty dict. Below is a sample response. What am I missing?
Request: /test/number?input=5
Status: 200
Latency: 223 ms
Response Body
{
"Log stream name:": "2018/12/05/[$LATEST]9d9fd5dd157046b4a67792aa49f5d71c",
"Log group name:": "/aws/lambda/test",
"Request ID:": "dce7beaf-f8c9-11e8-9cc4-85afb50a0e0c",
"Mem. limits(MB)": "128",
"size of event": 0,
"type of event": "<class 'dict'>"
}
Assuming you don't have request mapping templates, you should turn Lambda Proxy integration on.
I have the following Business Process defined within a Production on an Intersystems Cache Installation
/// Makes a call to Merlin based on the message sent to it from the pre-processor
Class sgh.Process.MerlinProcessor Extends Ens.BusinessProcess [ ClassType = persistent, ProcedureBlock ]
{
Property WorkingDirectory As %String;
Property WebServer As %String;
Property CacheServer As %String;
Property Port As %String;
Property Location As %String;
Parameter SETTINGS = "WorkingDirectory,WebServer,Location,Port,CacheServer";
Method OnRequest(pRequest As sgh.Message.MerlinTransmissionRequest, Output pResponse As Ens.Response) As %Status
{
Set tSC=$$$OK
Do ##class(sgh.Utils.Debug).LogDebugMsg("Packaging an HTTP request for Saved form "_pRequest.DateTimeSaved)
Set dateTimeSaved = pRequest.DateTimeSaved
Set patientId = pRequest.PatientId
Set latestDateTimeSaved = pRequest.LatestDateTimeSaved
Set formName = pRequest.FormName
Set formId = pRequest.FormId
Set episodeNumber = pRequest.EpisodeNumber
Set sentElectronically = pRequest.SentElectronically
Set styleSheet = pRequest.PrintName
Do ##class(sgh.Utils.Debug).LogDebugMsg("Creating HTTP Request Class")
set HTTPReq = ##class(%Net.HttpRequest).%New()
Set HTTPReq.Server = ..WebServer
Set HTTPReq.Port = ..Port
do HTTPReq.InsertParam("DateTimeSaved",dateTimeSaved)
do HTTPReq.InsertParam("HospitalNumber",patientId)
do HTTPReq.InsertParam("Episode",episodeNumber)
do HTTPReq.InsertParam("Stylesheet",styleSheet)
do HTTPReq.InsertParam("Server",..CacheServer)
Set Status = HTTPReq.Post(..Location,0) Quit:$$$ISERR(tSC)
Do ##class(sgh.Utils.Debug).LogDebugMsg("Sent the following request: "_Status)
Quit tSC
}
}
The thing is when I check the debug value (which is defined as a global) all I get is the number '1' - I have no idea therefore if the request has succeeded or even what is wrong (if it has not)
What do I need to do to find out
A) What is the actual web call being made?
B) What the response is?
There is a really slick way to get the answer the two questions you've asked, regardless of where you're using the code. Check the documentation out on the %Net.HttpRequest object here: http://docs.intersystems.com/ens20102/csp/docbook/DocBook.UI.Page.cls?KEY=GNET_http and the class reference here: http://docs.intersystems.com/ens20102/csp/documatic/%25CSP.Documatic.cls?APP=1&LIBRARY=ENSLIB&CLASSNAME=%25Net.HttpRequest
The class reference for the Post method has a parameter called test, that will do what you're looking for. Here's the excerpt:
method Post(location As %String = "", test As %Integer = 0, reset As %Boolean = 1) as %Status
Issue the Http 'post' request, this is used to send data to the web server such as the results of a form, or upload a file. If this completes correctly the response to this request will be in the HttpResponse. The location is the url to request, e.g. '/test.html'. This can contain parameters which are assumed to be already URL escaped, e.g. '/test.html?PARAM=%25VALUE' sets PARAM to %VALUE. If test is 1 then instead of connecting to a remote machine it will just output what it would have send to the web server to the current device, if test is 2 then it will output the response to the current device after the Post. This can be used to check that it will send what you are expecting. This calls Reset automatically after reading the response, except in test=1 mode or if reset=0.
I recommend moving this code to a test routine to view the output properly in terminal. It would look something like this:
// To view the REQUEST you are sending
Set sc = request.Post("/someserver/servlet/webmethod",1)
// To view the RESPONSE you are receiving
Set sc = request.Post("/someserver/servlet/webmethod",2)
// You could also do something like this to parse your RESPONSE stream
Write request.HttpResponse.Data.Read()
I believe the answer you want to A) is in the Server and Location properties of your %Net.HttpRequest object (e.g., HTTPReq.Server and HTTPReq.Location).
For B), the response information should be in the %Net.HttpResponse object stored in the HttpResponse property (e.g. HTTPReq.HttpResponse) after your call is completed.
I hope this helps!
-Derek
(edited for formatting)
From that code sample it looks like you're using Ensemble, not straight-up Cache.
In that case you should be doing this HTTP call in a Business Operation that uses the HTTP Outbound Adapter, not in your Business Process.
See this link for more info on HTTP Adapters:
http://docs.intersystems.com/ens20102/csp/docbook/DocBook.UI.Page.cls?KEY=EHTP
You should also look into how to use the Ensemble Message Browser. That should help with your logging needs.