I am trying to use https://www.elastic.co/guide/en/elasticsearch/reference/current/xpack-sql.html
with the .net NEST client. Any ideas how? I do not really see a guide anywhere.
There's a couple of ways that you could write SQL and return results from Elasticsearch
Use the ODBC Driver
Install the ODBC driver and use System.Data.Odbc.OdbcConnection to get records. For example
using var connection = new OdbcConnection("DSN=Local Elasticsearch");
connection.Open();
using var command = connection.CreateCommand();
command.CommandText = "SELECT * FROM my_index";
using var adapter = new OdbcDataAdapter(command);
var table = new DataTable();
adapter.Fill(table);
connection.Close();
// do something with data in table
Note that the ODBC driver is a platinum feature which requires a platinum or enterprise license.
Use the SQL API
Use the Elasticsearch SQL API, which is exposed on NEST, the .NET client. An example
var client = new ElasticClient();
var sqlResponse = client.Sql.Query(q => q
.Query("SELECT * from my_index")
);
foreach (var c in sqlResponse.Columns)
{
// do something with columns
}
foreach (var r in sqlResponse.Rows)
{
// do something with rows
}
The SQL API is part of the features of the default distribution.
Related
I'm using LLBLGen 5.0 and EntityCollectionBase.GetMulti(filter, relation) to retrieve objects in oracle database.
ObjectCollection objects = new ObjectCollection ();
RelationCollection relationsToUse = new RelationCollection();
relationsToUse.Add(ObjectEntity.Relations.Object2EntityUsingObject2Id);
IPredicateExpression filter = new PredicateExpression(ObjectFields.Code == sectionCode);
objects.GetMulti(filter, relationsToUse);
I would like to add caching system to avoid to do request in database many times.
I saw on LLBLGen documentation that it is possible to use cache on LLBLGen with this code:
var customers = new EntityCollection<CustomerEntity>();
using(var adapter = new DataAccessAdapter())
{
var parameters = new QueryParameters()
{
CollectionToFetch = customers,
FilterToUse = CustomerFields.Country=="USA",
CacheResultset = true,
CacheDuration = new TimeSpan(0, 0, 10) // cache for 10 seconds
};
adapter.FetchEntityCollection(parameters);
}
But I'm not able to found class DataAccessAdapter..
Do you have any idea, suggestion to resolve my issue ?
Thanks in advance,
If you created a llblgen project which uses SelfServicing there is no DataAccessAdapter, so project must be created with the Adapter option.
if this is so then it should be in xxxx.DatabaseSpecific
where xxxx is youre project name.
I have the following simple table I've created from the following source: https://hortonworks.com/hadoop-tutorial/introduction-apache-hbase-concepts-apache-phoenix-new-backup-restore-utility-hbase/#start-hbase
using the following:
create 'driver_dangerous_event','events'
put 'driver_dangerous_event','4','events:driverId','78'
put 'driver_dangerous_event','4','events:driverName','Carl'
put 'driver_dangerous_event','4','events:eventTime','2016-09-23 03:25:03.567'
put 'driver_dangerous_event','4','events:eventType','Normal'
put 'driver_dangerous_event','4','events:latitudeColumn','37.484938'
put 'driver_dangerous_event','4','events:longitudeColumn','-119.966284'
put 'driver_dangerous_event','4','events:routeId','845'
put 'driver_dangerous_event','4','events:routeName','Santa Clara to San Diego'
put 'driver_dangerous_event','4','events:truckId','637'
I need to query this row but using a where filter (for future use), I have a rest api or thrift api running on my server.
I tried using rest api but failed to do it, is it possible?
also I tried using this nuget: https://hbasenet.codeplex.com/releases/view/133288 but I can't understand how to filter the data with where clause, I can only select a specific row:
Hbase.Client c = new Hbase.Client(serverHostName, port, 10000);
var res = c.Scan<Driver>("driver_dangerous_event", "events", "1");
Is there any option to do a simple filtered query with REST api/ Thrift API/ some other C# library?
I used Microsoft.HBase.Client for preforming a simple query (https://github.com/hdinsight/hbase-sdk-for-net)
// Connection
RequestOptions scanOptions = RequestOptions.GetDefaultOptions();
scanOptions.Port = int.Parse(hbaseDataConnection.Port);
scanOptions.AlternativeEndpoint = "/";
var nodeIPs = new List<string>();
nodeIPs.Add(hbaseDataConnection.Address);
HBaseClient client = new HBaseClient(null, scanOptions, new LoadBalancerRoundRobin(nodeIPs));
Scanner scanner = new Scanner { batch = 10 };
ScannerInformation scannerInfo = await client.CreateScannerAsync(_tableName, scanner, scanOptions);
var options = RequestOptions.GetDefaultOptions();
options.Port = int.Parse(hbaseDataConnection.Port);
options.AlternativeEndpoint = "/";
options.AlternativeHost = scannerInfo.Location.Host;
var f1 = new SingleColumnValueFilter(
Encoding.UTF8.GetBytes(ColumnFamilyName),
Encoding.UTF8.GetBytes("driverName"),
CompareFilter.CompareOp.Equal,
new SubstringComparator(fld.Values[0].ToString()))
var filter = new FilterList(FilterList.Operator.MustPassAll, f1);
scanner.filter = filter.ToEncodedString();
ScannerInformation scanInfo = client.CreateScannerAsync(_tableName, scanner, scanOptions).Result;
result = RetrieveResults(client, scanInfo, scanOptions).ToList();
Make sure REST API is running on the HBase machine, e.g.
hbase rest start -p 20050 --infoport 20051
I am developing a thrift client,
I have build a thrift hive server(apache-hive-0.14.0) on my machine and am also have access to Cloudera Dist Hive 4.6.0
When i connect thrift client to CDH client give following error:
TApplicationException: Required field 'client_protocol' is unset!
Struct:TOpenSessionReq(client_protocol:null, username:
I am passing the right protocol to the server but it seems some thing is over riding it....
Moreover if I point to localhost(where i have my hive server running) every thing seems to working fine....
Please let me know what is wrong here....
Code:
var socket = new TSocket("XXX.XXX.XXX.XXX", 10000);
TStreamTransport sTransport = (TStreamTransport)socket;
var transport = new TBufferedTransport(socket);
underlyingTransport = transport;
var proto = new TBinaryProtocol(transport);
var client = new TCLIService.Client(proto);
transport.Open();
TOpenSessionReq req = new TOpenSessionReq(TProtocolVersion.HIVE_CLI_SERVICE_PROTOCOL_V6);
req.Username = "hive";
req.Password = "hive";
TOpenSessionResp oSResponse = client.OpenSession(req);
TSessionHandle sessionHandle = oSResponse.SessionHandle;
TExecuteStatementReq execReq = new TExecuteStatementReq(sessionHandle, "select * from emp");
TExecuteStatementResp exeRes= client.ExecuteStatement(execReq);
TOperationHandle operationHandle = exeRes.OperationHandle;
TFetchResultsReq fechReq = new TFetchResultsReq(operationHandle,TFetchOrientation.FETCH_FIRST, 1);
TFetchResultsResp fechRes = client.FetchResults(fechReq);
TRowSet results = fechRes.Results;
List<TRow> resultRows = results.Rows;
foreach (var row in resultRows)
{
var val = row.ColVals[0];
System.Console.WriteLine(val.StringVal);
}
TCloseOperationReq closeOprReq = new TCloseOperationReq(operationHandle);
client.CloseOperation(closeOprReq);
TCloseSessionReq creq = new TCloseSessionReq(sessionHandle);
client.CloseSession(creq);
I believe it is the problem of the hive-jdbc version. This solution may solve your problem: Required field 'client_protocol' is unset
I'm complete newbie to Oracle DB trying to enable DB change notifications.
private void RegisterNotification()
{
const string connstring = "Data Source=ORA_DB;User Id=USER;Password=pass;";
try
{
var connObj = new OracleConnection(connstring);
connObj.Open();
var cmdObj = connObj.CreateCommand();
cmdObj.CommandText = "SELECT * FROM MYTABLE";
var dep = new OracleDependency(cmdObj);
dep.QueryBasedNotification = false;
dep.OnChange += new OnChangeEventHandler(OnNotificationReceived);
cmdObj.ExecuteNonQuery();
connObj.Close();
connObj.Dispose();
connObj = null;
}
catch (Exception ex)
{
MessageBox.Show(ex.ToString());
}
}
public static void OnNotificationReceived(object src, OracleNotificationEventArgs arg)
{
MessageBox.Show("Table has changed!");
}
I've executed "GRANT CHANGE NOTIFICATION TO USER;" but nothing happens when I change the table data neither manually nor programmatically. Query-based notifications also don't work. I suppose I miss something in Oracle configuration.
I have Oracle 11.2 standard edition.
The CHANGE NOTIFICATION permission is not on the features of the Standard Edition for latest versions :
Licensing information
Oracle TimesTen Application-Tier Database Cache :
Data access using PL/SQL, JDBC, ODBC, ttClasses, OCI, and Pro*C/C++
interfaces Transaction Log API (XLA) for change notification
Multi-node Cache Grid
...
SE2 : N
EE : Y (extra-cost option)
Try to execute your soft under admin permission and as console application. when we used to deal with it we faced with the same stuff. we hadn't managed to use it in webservices.
Using LLBLGen 3.1 (Self Servicing) on SQL Server, how would one execute custom SQL, such as:
delete from UserPreference
select * from UserPreference (into a datatable, for example)
Just noticed this question hadn't been answered. With Self Servicing, you'll probably use the TypedListDAO class.
See: Generated code - Fetching DataReaders and projections, SelfServicing
The TypedListDAO class has what you need to do SQL against your database, and it can automatically do projections onto custom classes for you if you need that (see the article).
But basically, (from memory, so might need some slight adjustments), here's what your code might look like:
// inside the DaoClasses namespace of your generated project
TypedListDAO dao = new TypedListDAO();
// do it yourself, and use your project's connection string
string connectionString = CommonDaoBase.ActualConnectionString;
using (var conn = new SqlConnection(connectionString)) { }
// use a DbConnection directly
DbConnection connection = dao.CreateConnection();
// or
connection = dao.DetermineConnectionToUse(null);
DbCommand cmd = connection.CreateCommand();
cmd.CommandText = "SELECT * FROM UserPreferences";
cmd.CommandType = CommandType.Text;
var reader = cmd.ExecuteReader(CommandBehavior.Default);
while (reader.Read()){}
reader.Close();
// use a datareader
IRetrievalQuery query = new RetrievalQuery(
new SqlCommand("SELECT * FROM UserPreferences"));
// or new RetrievalQuery(cmd);
// where you create the cmd using the dao connection
IDataReader reader = dao.GetAsDataReader(null, query,
CommandBehavior.CloseConnection);
while (reader.Read()){}
reader.Close();
// use a datatable - try something like this
// (BUT honestly, you might want to look at the custom projection
// into custom classes capability, or the data reader, instead of this)
DataTable dt = new DataTable();
dao.GetMultiAsDataTable(new EntityFields(0) /* can't be null, i don't think */,
dt, query, null);
// other methods
dao.ExecuteScalarQuery(query, null);
ActionQuery actionQuery = new ActionQuery(new SqlCommand("INSERT ..."));
dao.ExecuteActionQuery(actionQuery, null);
OR, use a micro-orm to do your sql, and just use the connection from the TypedListDAO class above
Some Light-weight micro-orms, like: Dapper (1 cs file), PetaPoco, Massive, etc...
While it is true that you can access the low level data readers, etc.. I think it kind of defeats the purpose of using the ORM. If you just want to fill a datatable from a collection (with or without filtering), you can use the static method GetMultiAsDataTable (which you can pass a predicate expression to if you want to do filtering). If you want to replace more complex SQL (very useful for reporting), check out the dynamic lists capabilities:
http://www.llblgen.com/documentation/4.0/LLBLGen%20Pro%20RTF/hh_start.htm
The QuerySpec is an even nicer way to specify a dynamic query and project it:
http://www.llblgen.com/documentation/4.0/LLBLGen%20Pro%20RTF/hh_start.htm