You need to set up map-remote on multiple phone URLs in autotest, but how do you do it via the command line?
launch example:
case ANDROID -> {
String[] command = {"mitmdump",
"--listen-port", String.valueOf(port),
"--set", "confdir=" + Objects.requireNonNull(Proxy.class.getClassLoader().getResource("mitmproxy")).toURI().getPath() + "/mitmproxy-certs",
"--map-remote", "|" + mapRemotePattern + "|http://localhost:" + String.valueOf(mockPort),
"--modify-headers", "/~hq localhost/Host/" + hostHeader,
"--save-stream-file", fileNameNetworkLog};
startMitmdump(command);
}
I can't find the answer in the documentation
when trying to add 1 more map-remote field with different url, then it doesn't work
If you try to write mapRemote by regular expression OR, to 2 url , then it doesn't work either.
I'm trying to use SimpleHttpOperator for consuming a RESTful API. But, As the name suggests, it only supporting HTTP protocol where I need to consume a HTTPS URI. so, now, I have to use either "requests" object from Python or handle the invocation from within the application code. But, It may not be a standard way. so, I'm looking for any other options available to consume HTTPS URI from within Airflow. Thanks.
I dove into this and am pretty sure that this behavior is a bug in airflow. I have created a ticket for it here:
https://issues.apache.org/jira/browse/AIRFLOW-2910
For now, the best you can do is override SimpleHttpOperator as well as HttpHook in order to change the way that HttpHook.get_conn works (to accept https). I may end up doing this, and if I do I'll post some code.
Update:
Operator override:
from airflow.operators.http_operator import SimpleHttpOperator
from airflow.exceptions import AirflowException
from operators.https_support.https_hook import HttpsHook
class HttpsOperator(SimpleHttpOperator):
def execute(self, context):
http = HttpsHook(self.method, http_conn_id=self.http_conn_id)
self.log.info("Calling HTTP method")
response = http.run(self.endpoint,
self.data,
self.headers,
self.extra_options)
if self.response_check:
if not self.response_check(response):
raise AirflowException("Response check returned False.")
if self.xcom_push_flag:
return response.text
Hook override
from airflow.hooks.http_hook import HttpHook
import requests
class HttpsHook(HttpHook):
def get_conn(self, headers):
"""
Returns http session for use with requests. Supports https.
"""
conn = self.get_connection(self.http_conn_id)
session = requests.Session()
if "://" in conn.host:
self.base_url = conn.host
elif conn.schema:
self.base_url = conn.schema + "://" + conn.host
elif conn.conn_type: # https support
self.base_url = conn.conn_type + "://" + conn.host
else:
# schema defaults to HTTP
self.base_url = "http://" + conn.host
if conn.port:
self.base_url = self.base_url + ":" + str(conn.port) + "/"
if conn.login:
session.auth = (conn.login, conn.password)
if headers:
session.headers.update(headers)
return session
Usage:
Drop-in replacement for SimpleHttpOperator.
This is a couple of months old now, but for what it is worth I did not have any issue with making an HTTPS call on Airflow 1.10.2.
In my initial test I was making a request for templates from sendgrid, so the connection was set up like this:
Conn Id : sendgrid_templates_test
Conn Type : HTTP
Host : https://api.sendgrid.com/
Extra : { "authorization": "Bearer [my token]"}
and then in the dag code:
get_templates = SimpleHttpOperator(
task_id='get_templates',
method='GET',
endpoint='/v3/templates',
http_conn_id = 'sendgrid_templates_test',
trigger_rule="all_done",
xcom_push=True
dag=dag,
)
and that worked. Also notice that my request happens after a Branch Operator, so I needed to set the trigger rule appropriately (to "all_done" to make sure it fires even when one of the branches is skipped), which has nothing to do with the question, but I just wanted to point it out.
Now to be clear, I did get an Insecure Request warning as I did not have certificate verification enabled. But you can see the resulting logs below
[2019-02-21 16:15:01,333] {http_operator.py:89} INFO - Calling HTTP method
[2019-02-21 16:15:01,336] {logging_mixin.py:95} INFO - [2019-02-21 16:15:01,335] {base_hook.py:83} INFO - Using connection to: id: sendgrid_templates_test. Host: https://api.sendgrid.com/, Port: None, Schema: None, Login: None, Password: XXXXXXXX, extra: {'authorization': 'Bearer [my token]'}
[2019-02-21 16:15:01,338] {logging_mixin.py:95} INFO - [2019-02-21 16:15:01,337] {http_hook.py:126} INFO - Sending 'GET' to url: https://api.sendgrid.com//v3/templates
[2019-02-21 16:15:01,956] {logging_mixin.py:95} WARNING - /home/csconnell/.pyenv/versions/airflow/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
[2019-02-21 16:15:05,242] {logging_mixin.py:95} INFO - [2019-02-21 16:15:05,241] {jobs.py:2527} INFO - Task exited with return code 0
I was having the same problem with HTTP/HTTPS when trying to set the connections using environment variables (although it works when i set the connection on the UI).
I've checked the issue #melchoir55 opened (https://issues.apache.org/jira/browse/AIRFLOW-2910) and you don't need to make a custom operator for that, the problem is not that HttpHook or HttpOperator can't use HTTPS, the problem is the way get_hook parse the connection string when dealing with HTTP, it actually understand that the first part (http:// or https://) is the connection type.
In summary, you don't need a custom operator, you can just set the connection in your env as the following:
AIRFLOW_CONN_HTTP_EXAMPLE=http://https%3a%2f%2fexample.com/
Instead of:
AIRFLOW_CONN_HTTP_EXAMPLE=https://example.com/
Or set the connection on the UI.
It is not a intuitive way to set up a connection but I think they are working on a better way to parse connections for Ariflow 2.0.
In Airflow 2.x you can use https URLs by passing https for schema value while setting up your connection and can still use SimpleHttpOperator like shown below.
my_api = SimpleHttpOperator(
task_id="my_api",
http_conn_id="YOUR_CONN_ID",
method="POST",
endpoint="/base-path/end-point",
data=get_data,
headers={"Content-Type": "application/json"},
)
Instead of implementing HttpsHook, we could just put one line of codes into HttpsOperator(SimpleHttpOperator)#above as follows
...
self.extra_options['verify'] = True
response = http.run(self.endpoint,
self.data,
self.headers,
self.extra_options)
...
in Airflow 2, the problem is been resolved.
just check out that :
host name in Connection UI Form, don't end up with /
'endpoint' parameter of SimpleHttpOperator starts with /
I am using Airflow 2.1.0,and the following setting works for https API
In connection UI, setting host name as usual, no need to specify 'https' in schema field, don't forget to set login account and password if your API server request ones.
Connection UI Setting
When constructing your task, add extra_options parameter in SimpleHttpOperator, and put your CA_bundle certification file path as the value for key verify, if you don't have a certification file, then use false to skip verification.
Task definition
Reference: here
I built this script I share with you here that worked perfectly on my Windows 7 computer to dail a tel:-link in a webpage with my Yealink VOIP phone. With the 'reg-file' you see in the comment I activated the tel-link uasage of my script. But now in Windows 10 I can't get it to work any more! Can anyone help me how I can "link" the "tel link" in a browser to my script again in Windows 10?
/*
test usage: cscript Z:\tel_link_open\tel.js [phone number]
create register_me.reg with:
REGEDIT4
[HKEY_CLASSES_ROOT\tel]
#="URL:knoop.frl Custom Telephone Protocol for VoIP phone"
"URL Protocol"=""
[HKEY_CLASSES_ROOT\tel\shell]
[HKEY_CLASSES_ROOT\tel\shell\open]
[HKEY_CLASSES_ROOT\tel\shell\open\command]
#="cscript \"Z:\\tel_link_open\\tel.js\" -c \"\\%1\""
*/
var call_number = WScript.Arguments(0);
call_number = call_number.replace(/\|.+/g,'');
// spatie -.
call_number = call_number.replace(/(\\tel:|%20|\(|\)|[a-z:\\ -]+)/g,'');
// + 31 ( 0 )
call_number = call_number.replace(/(\+|%2b)([0-9]+)(\(|%28)0(\)|%29)/ig,'00$2');
call_number = call_number.replace(/^0031/ig,'0');
WScript.Echo("\n\nGoing to dail: " + call_number + "\n\n");
//WScript.Sleep(50000000);
var outgoing_uri = "31"+"513"+"[number]"+"[internal extension]";
var login_name = "XXX";
var login_pass = "yyy";
var get_url = "http://192.168.xx.yy/servlet?number=" + call_number + "&outgoing_uri=" + outgoing_uri;
// Load the WinHttpRequest object.
var WinHttpReq = new ActiveXObject("WinHttp.WinHttpRequest.5.1");
// HttpRequest SetCredentials flags
HTTPREQUEST_SETCREDENTIALS_FOR_SERVER = 0;
// Specify the target resource.
WinHttpReq.open( "GET",
get_url,
false );
if (login_name.length) { // Set credentials for server.
WinHttpReq.SetCredentials( login_name,
login_pass,
HTTPREQUEST_SETCREDENTIALS_FOR_SERVER);
}
// It might also be necessary to supply credentials
// to the proxy if you connect to the Internet
// through a proxy that requires authentication.
// Send a request to the server and wait for
// a response.
WinHttpReq.send( );
// Display the results of the request.
WScript.Echo( "Result status: " + WinHttpReq.Status + " " + WinHttpReq.StatusText + "\n");
WScript.Echo( WinHttpReq.GetAllResponseHeaders( ) );
/* To save a binary file use this code instead of previous line
BinStream = new ActiveXObject("ADODB.Stream");
BinStream.Type = 1;
BinStream.Open();
BinStream.Write(WinHttpReq.ResponseBody);
BinStream.SaveToFile("out.bin");
*/
I think you can find a solution at MSDN Microsoft:
*Registering the Application Handling the Custom URI Scheme
To register an application to handle a particular URI scheme, add a new key, along with the appropriate subkeys and values, to HKEY_CLASSES_ROOT. The root key must match the URI scheme that is being added. For instance, to add a scheme, add an [Protocol Name] key to HKEY_CLASSES_ROOT, as follows:
HKEY_CLASSES_ROOT
[Protocol Name]
URL Protocol = ""
Under this new key, the URL Protocol string value indicates that this key declares a custom pluggable protocol handler. Without this key, the handler application will not launch. The value should be an empty string.
...
When a user clicks a link containing your custom URI scheme, Windows Internet Explorer launches the pluggable protocol handler registered for that URI scheme. If the specified open command specified in the registry contains a %1 parameter, Internet Explorer passes the URI to the registered pluggable protocol handler application.*
further details see under
Registering an Application to a URI Scheme
This is working for tel-URI, but callto-URI is somewhere registered by Skype. You have to search in registry for it and add your application there.
Got what I needed to work on OSX. On Windows, things went a little south.
I am using 'native' mode to share a local properties file across multiple services (spring cloud client). I have:
-Dspring.cloud.config.server.native.searchLocations=C:/Development/SVN/WSR_20150711/wsr-config/config/
And C:\Development\SVN\WSR_20150711\wsr-config\config\wsr\wsr-dev.properties exists. But looks like spring cloud config server is not picking up this file when I specify the resource via URI /wsr/dev/wsr -- which works in OSX and Linux.
I did a little digging around and found that there might be some issue with the code at NativeEnvironmentRepository, line 135 - 158:
if (normal.startsWith("file:")) {
normal = new File(normal.substring("file:".length()))
.getAbsolutePath();
}
for (String pattern : StringUtils
.commaDelimitedListToStringArray(getLocations(searchLocations,
result.getLabel()))) {
if (!pattern.contains(":")) {
pattern = "file:" + pattern;
}
if (pattern.startsWith("file:")) {
pattern = StringUtils.cleanPath(new File(pattern
.substring("file:".length())).getAbsolutePath()) + "/";
}
if (logger.isTraceEnabled()) {
logger.trace("Testing pattern: " + pattern
+ " with property source: " + name);
}
if (normal.startsWith(pattern)
&& !normal.substring(pattern.length()).contains("/")) {
matches = true;
break;
}
}
Looks like by forcing 'normal' through the new File(..).getAbsolutePath() bit forces the slash to change from / to \ (or \\ some times), and thus is not matching against the patten variable (create at the next for loop).
In debug mode, I can see that pattern is set to C: /Development/SVN/WSR_20150711/wsr-config/config/ while normal is set to C:\\Development\\SVN\\WSR_20150711\\wsr-config\\config\\wsr\\wsr-dev.properties.
Thoughts? Any chance we can fix this soon?
This is a known issue that has been fixed. See https://github.com/spring-cloud/spring-cloud-config/issues/150
I'm using Xamarin.mac. I need to get the fully qualified domain name of the local computer. On Windows this code works:
public string GetFQDN()
{
string domainName = System.Net.NetworkInformation.IPGlobalProperties.GetIPGlobalProperties().DomainName;
string hostName = Dns.GetHostName();
string fqdn = "";
if (!hostName.Contains(domainName))
fqdn = hostName + "." + domainName;
else
fqdn = hostName;
return fqdn;
}
On a mac this code causes this error: System.NotSupportedException: This platform is not supported.
So, what is the equivalent in Xamarin.mac? Or just in Mono?
Just getting the computer name would be a good start.
To do this, you can pretty much do the same you'd do in C on a UNIX system, which is to retrieve the hostname with gethostname() and then use a DNS lookup to find the canonical network name for the host. Luckily, System.Net has ready-made calls for this. The following code should work on both OS X and Linux (in fact, on Linux it is more or less what hostname --fqdn does):
using System;
using System.Net;
class Program {
static void Main() {
// Step 1: Get the host name
var hostname = Dns.GetHostName();
// Step 2: Perform a DNS lookup.
// Note that the lookup is not guaranteed to succeed, especially
// if the system is misconfigured. On the other hand, if that
// happens, you probably can't connect to the host by name, anyway.
var hostinfo = Dns.GetHostEntry(hostname);
// Step 3: Retrieve the canonical name.
var fqdn = hostinfo.HostName;
Console.WriteLine("FQDN: {0}", fqdn);
}
}
Note that with a misconfigured DNS, the DNS lookup may fail, or you may get the rather useless "localhost.localdomain".
If you wish to emulate your original approach, you can use the following code to retrieve the domainname:
var domainname = new StringBuilder(256);
Mono.Unix.Native.Syscall.getdomainname(domainname,
(ulong) domainname.Capacity - 1);
You will need to add the Mono.Posix assembly to your build for this.