Can someone show an example java program of HivePreparedStatement that can be used to connect to hive tables and retrieve data from it.
https://hive.apache.org/javadocs/r1.2.2/api/org/apache/hive/jdbc/HivePreparedStatement.html
I was trying with the following code and I am unable to complete it as there are many parameters for the constructor
TCLIService.Iface client =null;
EmbeddedThriftBinaryCLIService embeddedClient = new EmbeddedThriftBinaryCLIService();
embeddedClient.init(null);
client = embeddedClient;
TOpenSessionReq openReq = new TOpenSessionReq();
JdbcConnectionParams connParams = Utils.parseURL("jdbc:hive2://168.61.32.157:10000/nadb");
Map<String, String> openConf = new HashMap<String, String>();
// for remote JDBC client, try to set the conf var using 'set foo=bar'
for (Map.Entry<String, String> hiveConf : connParams.getHiveConfs().entrySet()) {
openConf.put("set:hiveconf:" + hiveConf.getKey(), hiveConf.getValue());
}
// For remote JDBC client, try to set the hive var using 'set hivevar:key=value'
for (Map.Entry<String, String=> hiveVar : connParams.getHiveVars().entrySet()) {
openConf.put("set:hivevar:" + hiveVar.getKey(), hiveVar.getValue());
}
// switch the database
openConf.put("use:database", connParams.getDbName());
// set the fetchSize
openConf.put("set:hiveconf:hive.server2.thrift.resultset.default.fetch.size",
Integer.toString(fetchSize));
// set the session configuration
Map<String, String> sessVars = connParams.getSessionVars();
if (sessVars.containsKey(HiveAuthFactory.HS2_PROXY_USER)) {
openConf.put(HiveAuthFactory.HS2_PROXY_USER,
sessVars.get(HiveAuthFactory.HS2_PROXY_USER));
}
openReq.setConfiguration(openConf);
// Store the user name in the open request in case no non-sasl authentication
if (JdbcConnectionParams.AUTH_SIMPLE.equals(sessConfMap.get(JdbcConnectionParams.AUTH_TYPE))) {
openReq.setUsername(sessConfMap.get(JdbcConnectionParams.AUTH_USER));
openReq.setPassword(sessConfMap.get(JdbcConnectionParams.AUTH_PASSWD));
}
TOpenSessionResp openResp = client.OpenSession(openReq);
// validate connection
Utils.verifySuccess(openResp.getStatus());
if (!supportedProtocols.contains(openResp.getServerProtocolVersion())) {
throw new TException("Unsupported Hive2 protocol");
}
protocol = openResp.getServerProtocolVersion();
sessHandle = openResp.getSessionHandle();
HivePreparedStatement hivePreparedStatement = new HivePreparedStatement(con, client, sessHandle, sql);
Is there any simple method for doing this?
Related
public PartsServiceCombineModelDTO getServicePartsByModel(String model) {
ServiceInstance instance = loadBalancer.choose("PARTS_APP");
if (instance == null || instance.getUri() == null) {
throw new NoInstanceExistException("Part App Instance Not Available ");
}
String url = instance.getUri().toString();
ParameterizedTypeReference<List<PartsDTO>> responseType = new ParameterizedTypeReference<List<PartsDTO>>() {
};
HttpEntity<String> entity = new HttpEntity<>(new HttpHeaders());
ResponseEntity<List<PartsDTO>> response = restTemplate.exchange(url + properties.getPartModel() + model, HttpMethod.GET, entity, responseType);
List<CarService> service = repository.findAllByModel(model);
List<ServiceDTO> carsDto;
if (service.isEmpty()) {
throw new NoSuchCarExistException("Car Service Not Available");
} else {
carsDto = new ArrayList<>();
for (CarService car : service) {
ServiceDTO carDto = new ServiceDTO();
BeanUtils.copyProperties(car, carDto);
carsDto.add(carDto);
}
}
PartsServiceCombineModelDTO partsServiceCombineModelDTO = new PartsServiceCombineModelDTO(response.getBody(), carsDto);
return partsServiceCombineModelDTO;
}
Mutation Test Report
Here my test method which is not able to to mutate below topics
void getServicePartsByModel() {
when(properties.getPartModel()).thenReturn("/parts/model/");
ServiceInstance instance = mock(ServiceInstance.class);
when(instance.getUri()).thenReturn(URI.create("http://localhost:8080"));
when(loadBalancerClient.choose(eq("PARTS_APP"))).thenReturn(instance);
PartsDTO partsDTO = new PartsDTO();
partsDTO.setId("42");
partsDTO.setManufacturer("Manufacturer");
partsDTO.setMaterial("Material");
partsDTO.setModel("Model");
partsDTO.setPartName("Part Name");
partsDTO.setPartNumber("42");
partsDTO.setPrice(1);
partsDTO.setStock(1);
partsDTO.setWarranty("Warranty");
partsDTO.setYear(1);
PartsDTO partsDTO1 = new PartsDTO();
partsDTO1.setId("42");
partsDTO1.setManufacturer("Manufacturer");
partsDTO1.setMaterial("Material");
partsDTO1.setModel("Model");
partsDTO1.setPartName("Part Name");
partsDTO1.setPartNumber("42");
partsDTO1.setPrice(1);
partsDTO1.setStock(1);
partsDTO1.setWarranty("Warranty");
partsDTO1.setYear(1);
List<PartsDTO> parts = new ArrayList<>();
parts.add(partsDTO1);
parts.add(partsDTO);
ResponseEntity<List<PartsDTO>> partsResponse = ResponseEntity.ok(parts);
when(restTemplate.exchange(any(String.class), eq(HttpMethod.GET), any(HttpEntity.class), eq(new ParameterizedTypeReference<List<PartsDTO>>() {
}))).thenReturn(partsResponse);
List<CarService> carServices = new ArrayList<>();
CarService carService = new CarService();
carService.setCertified(true);
carService.setDealershipTrained(true);
carService.setId("42");
carService.setModel("Model");
carService.setOemParts(true);
carService.setServiceKM(1);
carService.setServiceMileage(1);
carService.setServiceName("Service Name");
carService.setServiceType(1);
carServices.add(carService);
when(serviceRepository.findAllByModel(eq(carService.getModel()))).thenReturn(carServices);
assertNotNull(carService);
verify(beanUtils, atLeastOnce());
BeanUtils.copyProperties(new CarService(),new ServiceDTO());
PartsServiceCombineModelDTO result = carServiceImpl.getServicePartsByModel(partsDTO.getModel());
}
Which throwing error :
Unnecessary stubbings detected.
Clean & maintainable test code requires zero unnecessary code.
Following stubbings are unnecessary (click to navigate to relevant line of code):
-> at com.example.carserviceapp.service.CarServiceImplTest.getServicePartsByModel(CarServiceImplTest.java:728)
-> at com.example.carserviceapp.service.CarServiceImplTest.getServicePartsByModel(CarServiceImplTest.java:731)
-> at com.example.carserviceapp.service.CarServiceImplTest.getServicePartsByModel(CarServiceImplTest.java:763)
-> at com.example.carserviceapp.service.CarServiceImplTest.getServicePartsByModel(CarServiceImplTest.java:778)
Please remove unnecessary stubbings or use 'lenient' strictness. More info: javadoc for UnnecessaryStubbingException class.
org.mockito.exceptions.misusing.UnnecessaryStubbingException:
Can Anyone Please help me in mutating the above method
BeanUtils.copyProperties(car, carDto); and
throw new NoSuchCarExistException("Car Service Not Available");
I followed this example Changing schema name on runtime - Entity Framework where I can create a new EntityConnection from a MetaDataWorkspace that I then use to construct a DbContext with a different schema, but I get compiler warnings saying that RegisterItemCollection method is obsolete and to "Construct MetadataWorkspace using constructor that accepts metadata loading delegates."
How do I do that? Here is the code that is working but gives the 3 warnings for the RegsiterItemCollection calls. I'm surprised it works since warning says obsolete not just deprecated.
public static EntityConnection CreateEntityConnection(string schema, string connString, string model)
{
XmlReader[] conceptualReader = new XmlReader[]
{
XmlReader.Create(
Assembly
.GetExecutingAssembly()
.GetManifestResourceStream(model + ".csdl")
)
};
XmlReader[] mappingReader = new XmlReader[]
{
XmlReader.Create(
Assembly
.GetExecutingAssembly()
.GetManifestResourceStream(model + ".msl")
)
};
var storageReader = XmlReader.Create(
Assembly
.GetExecutingAssembly()
.GetManifestResourceStream(model + ".ssdl")
);
//XNamespace storageNS = "http://schemas.microsoft.com/ado/2009/02/edm/ssdl"; // this would not work!!!
XNamespace storageNS = "http://schemas.microsoft.com/ado/2009/11/edm/ssdl";
var storageXml = XElement.Load(storageReader);
foreach (var entitySet in storageXml.Descendants(storageNS + "EntitySet"))
{
var schemaAttribute = entitySet.Attributes("Schema").FirstOrDefault();
if (schemaAttribute != null)
{
schemaAttribute.SetValue(schema);
}
}
storageXml.CreateReader();
StoreItemCollection storageCollection =
new StoreItemCollection(
new XmlReader[] { storageXml.CreateReader() }
);
EdmItemCollection conceptualCollection = new EdmItemCollection(conceptualReader);
StorageMappingItemCollection mappingCollection =
new StorageMappingItemCollection(
conceptualCollection, storageCollection, mappingReader
);
//var workspace2 = new MetadataWorkspace(conceptualCollection, storageCollection, mappingCollection);
var workspace = new MetadataWorkspace();
workspace.RegisterItemCollection(conceptualCollection);
workspace.RegisterItemCollection(storageCollection);
workspace.RegisterItemCollection(mappingCollection);
var connectionData = new EntityConnectionStringBuilder(connString);
var connection = DbProviderFactories
.GetFactory(connectionData.Provider)
.CreateConnection();
connection.ConnectionString = connectionData.ProviderConnectionString;
return new EntityConnection(workspace, connection);
}
I was able to get rid of the 3 warning messages. Basically it wants you to register the collections in the constructor of the MetadataWorkspace.
There are 3 different overloads for MetadataWorkspace, I chose to use the one which requires to to supply a path (array of strings) to the workspace metadata. To do this I saved readers to temp files and reloaded them.
This is working for me without any warnings.
public static EntityConnection CreateEntityConnection(string schema, string connString, string model) {
var conceptualReader = XmlReader.Create(Assembly.GetExecutingAssembly().GetManifestResourceStream(model + ".csdl"));
var mappingReader = XmlReader.Create(Assembly.GetExecutingAssembly().GetManifestResourceStream(model + ".msl"));
var storageReader = XmlReader.Create(Assembly.GetExecutingAssembly().GetManifestResourceStream(model + ".ssdl"));
XNamespace storageNS = "http://schemas.microsoft.com/ado/2009/11/edm/ssdl";
var storageXml = XElement.Load(storageReader);
var conceptualXml = XElement.Load(conceptualReader);
var mappingXml = XElement.Load(mappingReader);
foreach (var entitySet in storageXml.Descendants(storageNS + "EntitySet")) {
var schemaAttribute = entitySet.Attributes("Schema").FirstOrDefault();
if (schemaAttribute != null) {
schemaAttribute.SetValue(schema);
}
}
storageXml.Save("temp.ssdl");
conceptualXml.Save("temp.csdl");
mappingXml.Save("temp.msl");
MetadataWorkspace workspace = new MetadataWorkspace(new List<String>(){
#"temp.csdl",
#"temp.ssdl",
#"temp.msl"
}
, new List<Assembly>());
var connectionData = new EntityConnectionStringBuilder(connString);
var connection = DbProviderFactories.GetFactory(connectionData.Provider).CreateConnection();
connection.ConnectionString = connectionData.ProviderConnectionString;
return new EntityConnection(workspace, connection);
}
Not wanting to create temp files which slows the process down, I found an alternate answer to this is fairly simple. I replaced these lines of code -
//var workspace2 = new MetadataWorkspace(conceptualCollection, storageCollection, mappingCollection);
var workspace = new MetadataWorkspace();
workspace.RegisterItemCollection(conceptualCollection);
workspace.RegisterItemCollection(storageCollection);
workspace.RegisterItemCollection(mappingCollection);
with this one line of code -
var workspace = new MetadataWorkspace(() => conceptualCollection, () => storageCollection, () => mappingCollection);
and that works fine.
I am using dart powerSNMP for .Net.
I am trying to query a table using GetTable(), it does not work for me.
Below C# code does not return any row,
const string address = "xxx.xxx.xx.x";
using (var mgr = new Manager())
{
var slave = new ManagerSlave(mgr);
slave.Socket.ReceiveTimeout = 13000;
try
{
//Retrieve table using GetNext requests
Variable[,] table = slave.GetTable("1.3.6.1.4.1.14823.2.2.1.1.1.9",
SnmpVersion.Three,
null,
new Security()
{
AuthenticationPassword = "mypassword1",
AuthenticationProtocol = AuthenticationProtocol.Md5,
PrivacyPassword = "mypassword2",
PrivacyProtocol = PrivacyProtocol.Des
},
new IPEndPoint(IPAddress.Parse(address), 161),
0);
}catch(Exception ez)
{
}
}
This is supposed to return a set of records from given OID. but It does not return me anything. When I use MIB Browser, I see GetBulk operation fetches all the records for me.
But what is wrong with GetTable() here?
Am building window form application using VS2010.On my login form i collect useid
and password and then click on the login button, if validation is successful direct
users to the main form.
I want to use a Dictionary to store userid and password read from the DB.
Then close connection. i then compare the inputed values from the textbox againt values:
userd and passward in the dictionary. on sucess direct to main form
here is my code. pls help`
string connectionstring =
"Data Source =localhost;Initial Catalog=HSM;" +
"User Id=sysad;Password=mypassword";
SqlConnection connection = new SqlConnection(connectionstring);
SqlCommand selectcmd = new SqlCommand("Select * from users");
SqlDataReader reader ;
Dictionary<string,string> logintest = new Dictionary<string,string>
try
{
connection.Open();
reader = selectcmd.ExecuteReader();
while (reader.Read())
{
Mainform main1 = new Mainform();
this.Hide();
main1.Show();
}
reader.Close();
`
Simple
foreach (KeyValuePair<string, int> pair in logintest)
{
Console.WriteLine("{0}, {1}", pair.Key, pair.Value);
}
I'm trying to create HFiles programmatically and loading them in a running HBase instance. I found a lot of info in HFileOutputFormat and in LoadIncrementalHFiles
I managed to create the new HFile, send it to the cluster. In the cluster web interface the new store file appears but the new keyrange is unavailable.
InputStream stream = ProgrammaticHFileGeneration.class.getResourceAsStream("ga-hourly.txt");
BufferedReader reader = new BufferedReader(new InputStreamReader(stream));
String line = null;
Map<byte[], String> rowValues = new HashMap<byte[], String>();
while((line = reader.readLine())!=null) {
String[] vals = line.split(",");
String row = new StringBuilder(vals[0]).append(".").append(vals[1]).append(".").append(vals[2]).append(".").append(vals[3]).toString();
rowValues.put(row.getBytes(), line);
}
List<byte[]> keys = new ArrayList<byte[]>(rowValues.keySet());
Collections.sort(keys, byteArrComparator);
HBaseTestingUtility testingUtility = new HBaseTestingUtility();
testingUtility.startMiniCluster();
testingUtility.createTable("table".getBytes(), "data".getBytes());
Writer writer = new HFile.Writer(testingUtility.getTestFileSystem(),
new Path("/tmp/hfiles/data/hfile"),
HFile.DEFAULT_BLOCKSIZE, Compression.Algorithm.NONE, KeyValue.KEY_COMPARATOR);
for(byte[] key:keys) {
writer.append(new KeyValue(key, "data".getBytes(), "d".getBytes(), rowValues.get(key).getBytes()));
}
writer.appendFileInfo(StoreFile.BULKLOAD_TIME_KEY, Bytes.toBytes(System.currentTimeMillis()));
writer.appendFileInfo(StoreFile.MAJOR_COMPACTION_KEY, Bytes.toBytes(true));
writer.close();
Configuration conf = testingUtility.getConfiguration();
LoadIncrementalHFiles loadTool = new LoadIncrementalHFiles(conf);
HTable hTable = new HTable(conf, "table".getBytes());
loadTool.doBulkLoad(new Path("/tmp/hfiles"), hTable);
ResultScanner scanner = hTable.getScanner("data".getBytes());
Result next = null;
System.out.println("Scanning");
while((next = scanner.next()) != null) {
System.out.format("%s %s\n", new String(next.getRow()), new String(next.getValue("data".getBytes(), "d".getBytes())));
}
Did anyone actually make this work ? I have a compilable / testable version up on my github
take a look at the LoadIncrementalHFiles test in the hbase source code : https://github.com/apache/hbase/blob/7c46646994b7a9d6f947cf12796579ef48d0b0bd/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java