How to express the scheme(https) in the model.json file, this is for connection to Elasticsearch?
The following is the model.js file:
{
"version": "1.0",
"defaultSchema": "elasticsearch",
"schemas": [
{
"type": "custom",
"name": "elasticsearch",
"factory": "org.apache.calcite.adapter.elasticsearch.ElasticsearchSchemaFactory",
"operand": {
"coordinates": "{'127.0.0.1': 9200}",
"index": "myIndex",
"useConig": "{}"
}
}
]
}
In the following JAVA code I am trying to connect to Elasticsearch:
Connection conn = DriverManager.getConnection("jdbc:calcite:", properties);
calciteConnection = conn.unwrap(CalciteConnection.class);
String elasticSchema = Resources.toString(somefile.class.getResource("/model.json"), Charset.defaultCharset());
new ModelHandler(calciteConnection, "inline:" + elasticSchema);
String sql = “select field1 from table1”
statement2 = calciteConnection.prepareStatement(sql);
ResultSet set = statement2.executeQuery();
The get a connection closed exception and I can see in the log that it was trying to connect to http not https(desired). Where do I mention https in the model file?
Calcite's Elasticsearch adapter doesn't currently support connections via HTTPS. I'd suggest you open up an issue on Calcite's JIRA. Or better yet, contribute the necessary code changes yourself :)
Related
I successfuly used the "Gateways - Create Datasource" (https://learn.microsoft.com/en-us/rest/api/power-bi/gateways/create-datasource) method from Power BI Rest API to create a SQL datasource, but I´m stuck when I try to create a Databricks datasource.
I saw that there is no Databricks datasource kind, but it could be possible to use the kind "Extension". So a try this code:
OBS: I generated the credential part using the data gateway publick key and a Databricks key.
What am I missing, or doing wrong?
{
"datasourceName": "Databricks AIDA Teste",
"datasourceType":"Extension",
"connectionDetails":{
"path":"{\"host\":\"adb-xxx.azuredatabricks.net\",\"httpPath\":\"\\/sql\\/1.0\\/warehouses\\/xxx\"}",
"kind":"Databricks"
},
"credentialDetails": {
"credentialType": "Key",
"credentials": "xxx",
"privacyLevel": "Organizational"
}
}```
{
"error": {
"code": "BadRequest",
"message": "Bad Request",
"details": [
{
"message": "Unexpected character encountered while parsing value: {. Path 'connectionDetails', line 4, position 30.",
"target": "datasourceToGatewayRequest.connectionDetails"
},
{
"message": "'datasourceToGatewayRequest' is a required parameter",
"target": "datasourceToGatewayRequest"
}
]
}
}
Many thanks!
Im working on cognos dashboard embedded using the reference from -
Cognos Dashboard embedded.
but instead of csv i'm working on JDBC data sources.
i'm trying to connect to JDBC data source as -
"module": {
"xsd": "https://ibm.com/daas/module/1.0/module.xsd",
"source": {
"id": "StringID",
"jdbc": {
"jdbcUrl": "jdbcUrl: `jdbc:db2://DATABASE-HOST:50000/YOURDB`",
"driverClassName": "com.ibm.db2.jcc.DB2Driver",
"schema": "DEFAULTSCHEMA"
},
"user": "user_name",
"password": "password"
},
"table": {
"name": "ROLE",
"description": "description of the table for visual hints ",
"column": [
{
"name": "ID",
"description": "String",
"datatype": "BIGINT",
"nullable": false,
"label": "ID",
"usage": "identifier",
"regularAggregate": "countDistinct",
},
{
"name": "NAME",
"description": "String",
"datatype": "VARCHAR(100)",
"nullable": true,
"label": "Name",
"usage": "identifier",
"regularAggregate": "countDistinct"
}
]
},
"label": "Module Name",
"identifier": "moduleId"
}
Note - here my database is hosted on private network on not hosted on public IP address.
So when i add the above code to add datasources, then the data is not loading from my DB,
even though i mentioned correct user and password for jdbc connection in above code then also when i drag and drop any field from data sources then it opens a pop up and which asks me for userID and Password.
and even after i filled userID and Password details again in popup i'm still unable to load the data.
Errors -
1 . when any module try to fetch data then calls API -
'https://dde-us-south.analytics.ibm.com/daas/v1/data?moduleUrl=%2Fda......'
but in my case this API is failing and giving the error - Status Code: 403 Forbidden
In SignOnDialog.js
At line - 98 call for saveDataSourceCredential method fails and it says saveDataSourceCredential is not a function.
Expectation -
It should not open a pop to asks for userID and password. and data will load directly just as it happens for database hosted on public IP domains.
This does not work in general. If you are using any type of functionality hosted outside your network that needs to access an API or data on your private network, there needs to be some communication channel.
That channel could be established by setting up a VPN, using products like IBM Secure Gateway to create a client / server connection between the IBM Cloud and your Db2 host, or by even setting up a direct link between your company network and the (IBM) cloud.
I have one index in old elasticsearch server in 6.2.0 version (windows server) and now I am trying to move it to new server (Linux) on 7.6.2 version of elasticsearch. I tried below command to migrate my index from old to new server but it is throwing an exception.
POST _reindex
{
"source": {
"remote": {
"host": "http://MyOldDNSName:9200"
},
"index": "test"
},
"dest": {
"index": "test"
}
}
Exception I am getting is -
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "[MyOldDNSName:9200] not whitelisted in reindex.remote.whitelist"
}
],
"type" : "illegal_argument_exception",
"reason" : "[MyOldDNSName:9200] not whitelisted in reindex.remote.whitelist"
},
"status" : 400
}
Note : I did not created any index in new elastic search server. do I have to create it with my old schema and then try to execute the above command ?
The error message is quite clear that your remote host(windows in your case) from which you are trying to build in a index on your new host(Linux) is not whitelisted, Please refer Elasticsearch guide on how to reindex from remote on more info.
As per same doc
Remote hosts have to be explicitly whitelisted in elasticsearch.yml
using the reindex.remote.whitelist property. It can be set to a
comma delimited list of allowed remote host and port combinations
(e.g. otherhost:9200, another:9200, 127.0.10.:9200, localhost:).
Another useful discuss link to troubleshoot the issue.
https://www.elastic.co/guide/en/elasticsearch/reference/8.0/docs-reindex.html#reindex-from-remote
Add this to elasticsearch.yml, modify it according your environment:
reindex.remote.whitelist: "otherhost:9200, another:9200, 127.0.10.*:9200, localhost:*"
I am creating an Amazon RDS Microsoft SQL Server database using an AWS CloudFormation template. It took some time to create and I connected to the database successfully, but it sometimes throws this exception:
Service was unable to open new database connection when requested.
SqlException: A connection was successfully established with the server, but then an error occurred during the pre-login handshake. (provider: SSL Provider, error: 0 - The wait operation timed out.)
Here is the code which I am using to create database using AWS template:
"ProductDatabase":{
"Type":"AWS::RDS::DBInstance",
"Condition":"isNewDatabase",
"Properties":{
"AllocatedStorage":{
"Ref":"AllocatedStorage"
},
"DBInstanceClass":"db.m4.large",
"Port":"1433",
"StorageType":"gp2",
"BackupRetentionPeriod":"7",
"DBName":"",
"DBSubnetGroupName": {
"Ref": "DatabaseSubnetGroup"
},
"MasterUsername":{
"Ref":"DBUserName"
},
"MasterUserPassword":{
"Ref":"DBPassword"
},
"Engine":"sqlserver-se",
"EngineVersion":"13.00.4422.0.v1",
"LicenseModel":"license-included",
"VPCSecurityGroups" : [ { "Ref" : "rdsDBSecurityGroup" } ],
"Tags": [
{
"Key": "Name",
"Value": { "Ref": "AWS::StackName" }
}
]
}
},
I have an ADF pipeline that copies 34 tables from an on premise Oracle database to an Azure data lake store; 32 of these copy just fine on a daily basis, the other 2 consistenly fail with...
Copy activity met an internal service error.
For more information, provide this message to customer support. ErrorCode: 8601 GatewayNodeName=XXXXXXXX,
ErrorCode=SystemErrorOdbcWrapperError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,
Message=Unknown error from wrapper.,
Source=Microsoft.DataTransfer.ClientLibrary.Odbc.OdbcConnector,
''Type=Microsoft.DataTransfer.ClientLibrary.Odbc.Runtime.ValueException,Message=[DataSource.Error] The ODBC driver returned an invalid value.,Source=Microsoft.DataTransfer.ClientLibrary.Odbc.Wrapper,'.
The activity JSON is templated so is identical for all 34 activities. I can run the oracleReaderQuery in Oracle SQL Developer using the same connection details and credentials and get results.
Searches for this have shown 1 unanswered question on here (StackOverflow) and another Microsoft with a response that says "We will get back to you ASAP when we have new updates"....but there are no updates.
It seems I am not the only one having this issue; has anyone found a solution?
I have tried to do a one off copy in ADF but get the same result; I have tried copying the table to blob storage and get the same result.
Can anyone help me try to fathom what is wrong with this please?
The activity JSON is as follows...
{
"type": "Copy",
"typeProperties": {
"source": {
"type": "OracleSource",
"oracleReaderQuery": "SELECT stuff FROM <source table>"
},
"sink": {
"type": "AzureDataLakeStoreSink",
"writeBatchSize": 0,
"writeBatchTimeout": "00:00:00"
}
},
"inputs": [
{
"name": "<source table dataset>"
},
{
"name": "<scheduling dependency dataset>"
}
],
"outputs": [
{
"name": "<destination dataset>"
}
],
"policy": {
"timeout": "02:00:00",
"concurrency": 1,
"retry": 3,
"longRetry": 2,
"longRetryInterval": "03:00:00",
"executionPriorityOrder": "OldestFirst"
},
"scheduler": {
"frequency": "Day",
"interval": 1
},
"name": "Copy Activity 34",
"description": "copy activity"
}
As I said though, this is identical, apart from the table it is accessing, to the 32 activities that work perfectly fine.
What's the data type of stuff in your table?