TApplicationException: Required field 'client_protocol' is unset - hadoop

I am developing a thrift client,
I have build a thrift hive server(apache-hive-0.14.0) on my machine and am also have access to Cloudera Dist Hive 4.6.0
When i connect thrift client to CDH client give following error:
TApplicationException: Required field 'client_protocol' is unset!
Struct:TOpenSessionReq(client_protocol:null, username:
I am passing the right protocol to the server but it seems some thing is over riding it....
Moreover if I point to localhost(where i have my hive server running) every thing seems to working fine....
Please let me know what is wrong here....
Code:
var socket = new TSocket("XXX.XXX.XXX.XXX", 10000);
TStreamTransport sTransport = (TStreamTransport)socket;
var transport = new TBufferedTransport(socket);
underlyingTransport = transport;
var proto = new TBinaryProtocol(transport);
var client = new TCLIService.Client(proto);
transport.Open();
TOpenSessionReq req = new TOpenSessionReq(TProtocolVersion.HIVE_CLI_SERVICE_PROTOCOL_V6);
req.Username = "hive";
req.Password = "hive";
TOpenSessionResp oSResponse = client.OpenSession(req);
TSessionHandle sessionHandle = oSResponse.SessionHandle;
TExecuteStatementReq execReq = new TExecuteStatementReq(sessionHandle, "select * from emp");
TExecuteStatementResp exeRes= client.ExecuteStatement(execReq);
TOperationHandle operationHandle = exeRes.OperationHandle;
TFetchResultsReq fechReq = new TFetchResultsReq(operationHandle,TFetchOrientation.FETCH_FIRST, 1);
TFetchResultsResp fechRes = client.FetchResults(fechReq);
TRowSet results = fechRes.Results;
List<TRow> resultRows = results.Rows;
foreach (var row in resultRows)
{
var val = row.ColVals[0];
System.Console.WriteLine(val.StringVal);
}
TCloseOperationReq closeOprReq = new TCloseOperationReq(operationHandle);
client.CloseOperation(closeOprReq);
TCloseSessionReq creq = new TCloseSessionReq(sessionHandle);
client.CloseSession(creq);

I believe it is the problem of the hive-jdbc version. This solution may solve your problem: Required field 'client_protocol' is unset

Related

In Current Project i am using this syntax to retrieve data from Dynamic CRM

please help with this below code to change to oauth
// Create Server Configuration
ServerConnection.Configuration config = new ServerConnection.Configuration();
config.ServerAddress = serverAddress;
config.DiscoveryUri = new Uri(String.Format("xxx", config.ServerAddress));
config.OrganizationName = organizationName;
config.OrganizationUri = new Uri(String.Format("xxx", config.ServerAddress));
config.Credentials = credentials;
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
crmServiceClient = new CrmServiceClient('xxx', credentials);
//dynamicsServiceProxy = ServerConnection.GetOrganizationProxy(config);
//dynamicsServiceProxy.EnableProxyTypes();
crmServiceClient.ServiceConfiguration.CurrentServiceEndpoint.Behaviors.Add(new
ProxyTypesBehavior(Assembly.GetExecutingAssembly()));
Any one please help me what changes i need to do?
I have Read this article but how to replace IOrganizationService in code?

Elastic Search NEST (.net) SQL

I am trying to use https://www.elastic.co/guide/en/elasticsearch/reference/current/xpack-sql.html
with the .net NEST client. Any ideas how? I do not really see a guide anywhere.
There's a couple of ways that you could write SQL and return results from Elasticsearch
Use the ODBC Driver
Install the ODBC driver and use System.Data.Odbc.OdbcConnection to get records. For example
using var connection = new OdbcConnection("DSN=Local Elasticsearch");
connection.Open();
using var command = connection.CreateCommand();
command.CommandText = "SELECT * FROM my_index";
using var adapter = new OdbcDataAdapter(command);
var table = new DataTable();
adapter.Fill(table);
connection.Close();
// do something with data in table
Note that the ODBC driver is a platinum feature which requires a platinum or enterprise license.
Use the SQL API
Use the Elasticsearch SQL API, which is exposed on NEST, the .NET client. An example
var client = new ElasticClient();
var sqlResponse = client.Sql.Query(q => q
.Query("SELECT * from my_index")
);
foreach (var c in sqlResponse.Columns)
{
// do something with columns
}
foreach (var r in sqlResponse.Rows)
{
// do something with rows
}
The SQL API is part of the features of the default distribution.

Using Network Creds in dotnet core app on a mac using HttpClient

Writing a dotnet core app. I need to log in with network credentials as the service (which happens to be a TFS on-prem server) uses those to authenticate. From my (and another team members') windows machine, the following code works:
Console.WriteLine("Type in your DOMAIN password:");
var pass = GetPassword(); //command line secure string magic from SO
var networkCredential = new NetworkCredential("USERNAME", pass, "DOMAINNAME");
string tfsDefaultCollection = "https://TFSURL/DefaultCollection";
string testUrl = $"{tfsDefaultCollection}/_apis/tfvc/changesets/1234/changes?api-version=2.2";
var httpClientHandler = new HttpClientHandler
{
Credentials = networkCredential
};
var client = new HttpClient(httpClientHandler)
{
BaseAddress = new Uri(testUrl)
};
httpClientHandler.PreAuthenticate = true;
var test = client.GetAsync(testUrl).Result;
Console.WriteLine(test);
But it doesn't work from my mac. I get a 401 unauthorized. Both used the same, hardwired connection. AND this works on my mac:
curl --ntlm --user "DOMAINNAME\USERNAME" "https://TFSURL/DefaultCollection/_apis/tfvc/changesets/1234/changes?api-version=2.2"
So that rules out a connectivity question, I would think. Am I missing something I need to be doing on my mac? Can anybody point me to some documentation or way to troubleshoot what both of these requests are doing at the lowest level to see if there is a difference?
Well finally some google-foo got me there. There's a bug in dotnet core for linux/mac. This issue describes the fix:
https://github.com/dotnet/corefx/issues/25988#issuecomment-412534360
It has to do with the host machine you are connecting to uses both Kerberos and NTLM authentication methods.
Implemented below:
AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", false);
Console.WriteLine("Type in your DOMAIN password:");
var pass = GetPassword(); //command line secure string magic from SO
var networkCredential = new NetworkCredential("USERNAME", pass, "DOMAINNAME");
string tfsDefaultCollection = "https://TFSURL/DefaultCollection";
string testUrl = $"{tfsDefaultCollection}/_apis/tfvc/changesets/1234/changes?api-version=2.2";
var myCache = new CredentialCache
{
{
new Uri(testUrl), "NTLM",
networkCredential
}
};
var httpClientHandler = new HttpClientHandler
{
Credentials = myCache
};
var client = new HttpClient(httpClientHandler)
{
BaseAddress = new Uri(testUrl)
};
httpClientHandler.PreAuthenticate = true;
var test = client.GetAsync(testUrl).Result;
Console.WriteLine(test);
Thanks to #dmcgill50 for getting me on the right googling track.

How to perform a simple select from HBase with criteria (Where clause)

I have the following simple table I've created from the following source: https://hortonworks.com/hadoop-tutorial/introduction-apache-hbase-concepts-apache-phoenix-new-backup-restore-utility-hbase/#start-hbase
using the following:
create 'driver_dangerous_event','events'
put 'driver_dangerous_event','4','events:driverId','78'
put 'driver_dangerous_event','4','events:driverName','Carl'
put 'driver_dangerous_event','4','events:eventTime','2016-09-23 03:25:03.567'
put 'driver_dangerous_event','4','events:eventType','Normal'
put 'driver_dangerous_event','4','events:latitudeColumn','37.484938'
put 'driver_dangerous_event','4','events:longitudeColumn','-119.966284'
put 'driver_dangerous_event','4','events:routeId','845'
put 'driver_dangerous_event','4','events:routeName','Santa Clara to San Diego'
put 'driver_dangerous_event','4','events:truckId','637'
I need to query this row but using a where filter (for future use), I have a rest api or thrift api running on my server.
I tried using rest api but failed to do it, is it possible?
also I tried using this nuget: https://hbasenet.codeplex.com/releases/view/133288 but I can't understand how to filter the data with where clause, I can only select a specific row:
Hbase.Client c = new Hbase.Client(serverHostName, port, 10000);
var res = c.Scan<Driver>("driver_dangerous_event", "events", "1");
Is there any option to do a simple filtered query with REST api/ Thrift API/ some other C# library?
I used Microsoft.HBase.Client for preforming a simple query (https://github.com/hdinsight/hbase-sdk-for-net)
// Connection
RequestOptions scanOptions = RequestOptions.GetDefaultOptions();
scanOptions.Port = int.Parse(hbaseDataConnection.Port);
scanOptions.AlternativeEndpoint = "/";
var nodeIPs = new List<string>();
nodeIPs.Add(hbaseDataConnection.Address);
HBaseClient client = new HBaseClient(null, scanOptions, new LoadBalancerRoundRobin(nodeIPs));
Scanner scanner = new Scanner { batch = 10 };
ScannerInformation scannerInfo = await client.CreateScannerAsync(_tableName, scanner, scanOptions);
var options = RequestOptions.GetDefaultOptions();
options.Port = int.Parse(hbaseDataConnection.Port);
options.AlternativeEndpoint = "/";
options.AlternativeHost = scannerInfo.Location.Host;
var f1 = new SingleColumnValueFilter(
Encoding.UTF8.GetBytes(ColumnFamilyName),
Encoding.UTF8.GetBytes("driverName"),
CompareFilter.CompareOp.Equal,
new SubstringComparator(fld.Values[0].ToString()))
var filter = new FilterList(FilterList.Operator.MustPassAll, f1);
scanner.filter = filter.ToEncodedString();
ScannerInformation scanInfo = client.CreateScannerAsync(_tableName, scanner, scanOptions).Result;
result = RetrieveResults(client, scanInfo, scanOptions).ToList();
Make sure REST API is running on the HBase machine, e.g.
hbase rest start -p 20050 --infoport 20051

Trying to set up Orleans Cluster membership with Consul

I'm trying to use an existing Consul cluster as the membership provider for a test Orleans application.
I get this error when connecting my client app to the Silo
Could not find any gateway in Orleans.Runtime.Host.ConsulBasedMembershipTable. Orleans client cannot initialize.
Digging into the ConsulUtils class, the entries being retrieved have no ProxyPort defined - and are discarded - hence the empty result set.
I initialize the silo like this:
var clusterConfiguration = new ClusterConfiguration();
clusterConfiguration.Globals.DataConnectionString = "http://localhost:8500";
clusterConfiguration.Globals.DeploymentId = "OrleansPlayground";
clusterConfiguration.Globals.LivenessType = GlobalConfiguration.LivenessProviderType.Custom;
clusterConfiguration.Globals.MembershipTableAssembly = "OrleansConsulUtils";
clusterConfiguration.Globals.ReminderServiceType = GlobalConfiguration.ReminderServiceProviderType.Disabled;
var silohost = new SiloHost("Fred", clusterConfiguration);
silohost.InitializeOrleansSilo();
startup = Task.Factory.StartNew(() =>
{
return silohost.StartOrleansSilo();
});
return true;
And I set my client app up like this:
var config = new ClientConfiguration();
config.CustomGatewayProviderAssemblyName = "OrleansConsulUtils";
config.DataConnectionString = "http://localhost:8500";
config.DeploymentId = "OrleansPlayground";
config.GatewayProvider = ClientConfiguration.GatewayProviderType.Custom;
GrainClient.Initialize(config);
Looking at the code in ConsulUtils I can see that the ProxyPort isn't set (i.e. is 0) when the entry is saved. So I'm assuming I have a problem when initializing the silo - but I can't figure out what it is!
Without digging deep in, does sound like a bug. Please repost on GitHub and we will try to help you.

Resources