The error occured when hbase client batch data. At first it's ok. Some time later it's wrong! The detailed error is:
: 1 time, org.apache.hadoop.hbase.exceptions.FailedSanityCheckException: Requested row out of range for doMiniBatchMutation on HRegion idcard,bfef6945ac273d83\x00\x00\x00\x00\x00\x17\xCC$,1461584032622.dadb8843fe441dac4a3d4d7669597ef5., startKey='bfef6945ac273d83\x00\x00\x00\x00\x00\x17\xCC$', getEndKey()='', row='9a6ec957205e1d74\x00\x00\x00\x00\x01\x90\x1F\xF5'
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:712)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:662)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2046)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32393)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:104)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:745)
The environment is :
hbase hbase-1.1.3
hadoop2.6
hbase-client 1.2.0
The hbase client's code is :
public static void batchPutData(Connection connection, long startNum, long count) throws IOException, ParseException{
//table
Table table = connection.getTable(TableName.valueOf(TABLE_NAME));
//index table
Table index_table = connection.getTable(TableName.valueOf(INDEX_TABLE_NAME));
//random name
RandomChineseName randomChineseName = new RandomChineseName();
//random car
RandomCar randomCar = new RandomCar();
List<Put> puts = new ArrayList<Put>();
List<Put> indexPlateputs = new ArrayList<Put>();
for(long i = 0; i < count; i++){
long index = startNum+i;
Date birthdate = RandomUtils.randomDate();
String birthdateStr = DateUtil.dateToStr(birthdate, "yyyy-MM-dd");
boolean isBoy = i%2==0?true:false;
String name = isBoy?randomChineseName.randomBoyName():randomChineseName.randomGirlName();
String nation = RandomUtils.randomNation();
String plate = randomCar.randomPlate();
byte[] idbuff = Bytes.toBytes(index);
String hashPrefix = MD5Hash.getMD5AsHex(idbuff).substring(0, 16);
//create a put for table
Put put = new Put(Bytes.add(Bytes.toBytes(hashPrefix), idbuff));
put.addColumn(Bytes.toBytes("idcard"), Bytes.toBytes("name"), Bytes.toBytes(name));
put.addColumn(Bytes.toBytes("idcard"), Bytes.toBytes("sex"), Bytes.toBytes(isBoy?1:0));
put.addColumn(Bytes.toBytes("idcard"), Bytes.toBytes("birthdate"), Bytes.toBytes(birthdateStr));
put.addColumn(Bytes.toBytes("idcard"), Bytes.toBytes("nation"), Bytes.toBytes(nation));
put.addColumn(Bytes.toBytes("idcard"), Bytes.toBytes("plate"), Bytes.toBytes(plate));
puts.add(put);
//create a put for index table
String namehashPrefix = MD5Hash.getMD5AsHex(Bytes.toBytes(name)).substring(0, 16);
byte[] bprf = Bytes.add(Bytes.toBytes(namehashPrefix), Bytes.toBytes(name));
bprf = Bytes.add(bprf, Bytes.toBytes(SPLIT), Bytes.toBytes(birthdateStr));
Put namePut = new Put(Bytes.add(bprf, Bytes.toBytes(SPLIT), Bytes.toBytes(index)));
namePut.addColumn(Bytes.toBytes("index"), Bytes.toBytes("idcard"), Bytes.toBytes(0));
indexPlateputs.add(namePut);
//insert for every ten thousands
if(i%10000 == 0){
table.put(puts);
index_table.put(indexPlateputs);
puts.clear();
indexPlateputs.clear();
}
}
}
It seems like conflicts with HBase version . Change HBase version to 1.1.4 or 1.0.0 or other stable version to have a try.
Related
I am trying to update certain rows in my oracle DB using id which is of RAW(255).
Sample ids 0BF3957A016E4EBCB68809E6C2EA8B80, 1199B9F29F0A46F486C052669854C2F8...
#Autowired
private NamedParameterJdbcTemplate jdbcTempalte;
private static final String UPDATE_SUB_STATUS = "update SUBSCRIPTIONS set status = :status, modified_date = systimestamp where id in (:ids)";
public void saveSubscriptionsStatus(List<String> ids, String status) {
MapSqlParameterSource paramSource = new MapSqlParameterSource();
List<String> idsHexToRaw = new ArrayList<>();
String temp = new String();
for (String id : ids) {
temp = "hextoraw('" + id + "')";
idsHexToRaw.add(temp);
}
paramSource.addValue("ids", idsHexToRaw);
paramSource.addValue("status", status);
jdbcTempalte.update(*UPDATE_SUB_STATUS*, paramSource);
}
This above block of code is executing without any error but the updates are not reflected to the db, while if I skip using hextoraw() and just pass the list of ids it works fine and also updates the data in table. see below code
public void saveSubscriptionsStatus(List<String> ids, String status) {
MapSqlParameterSource paramSource = new MapSqlParameterSource();]
paramSource.addValue("ids", ids);
paramSource.addValue("status", status);
jdbcTempalte.update(UPDATE_SUB_STATUS, paramSource);
}
this code works fine and updates the table, but since i am not using hextoraw() it scans the full table for updation which I don't want since i have created indexes. So using hextoraw() will use index for scanning the table but it is not updating the values which is kind of weird.
Got a solution myself by trying all the different combinations :
#Autowired
private NamedParameterJdbcTemplate jdbcTempalte;
public void saveSubscriptionsStatus(List<String> ids, String status) {
String UPDATE_SUB_STATUS = "update SUBSCRIPTIONS set status = :status, modified_date = systimestamp where id in (";
MapSqlParameterSource paramSource = new MapSqlParameterSource();
String subQuery = "";
for (int i = 0; i < ids.size(); i++) {
String temp = "id" + i;
paramSource.addValue(temp, ids.get(i));
subQuery = subQuery + "hextoraw(:" + temp + "), ";
}
subQuery = subQuery.substring(0, subQuery.length() - 2);
UPDATE_SUB_STATUS = UPDATE_SUB_STATUS + subQuery + ")";
paramSource.addValue("status", status);
jdbcTempalte.update(UPDATE_SUB_STATUS, paramSource);
}
What this do is create a query with all the ids to hextoraw as id0, id1, id2...... and also added this values in the MapSqlParameterSource instance and then this worked fine and it also used the index for updating my table.
After running my new function the query look like : update
SUBSCRIPTIONS set status = :status, modified_date = systimestamp
where id in (hextoraw(:id0), hextoraw(:id1), hextoraw(:id2)...)
MapSqlParameterSource instance looks like : {("id0", "randomUUID"),
("id1", "randomUUID"), ("id2", "randomUUID").....}
Instead of doing string manipulation, Convert the list to List of ByteArray
List<byte[]> productGuidByteList = stringList.stream().map(item -> GuidHelper.asBytes(item)).collect(Collectors.toList());
parameters.addValue("productGuidSearch", productGuidByteList);
public static byte[] asBytes(UUID uuid) {
ByteBuffer bb = ByteBuffer.wrap(new byte[16]);
bb.putLong(uuid.getMostSignificantBits());
bb.putLong(uuid.getLeastSignificantBits());
return bb.array();
}
I'm fairly new to Oracle but I have used the Bulk insert on a couple other applications. Most seem to go faster using it but I've had a couple where it slows down the application. This is my second one where it slowed it down significantly so I'm wondering if I have something setup incorrectly or maybe I need to set it up differently. In this case I have a console application that processed ~1,900 records. Inserting them individually it will take ~2.5 hours and when I switched over to the Bulk insert it jumped to 5 hours.
The article I based this off of is http://www.oracle.com/technetwork/issue-archive/2009/09-sep/o59odpnet-085168.html
Here is what I'm doing, I'm retrieving some records from the DB, do calculations, and then write the results out to a text file. After the calculations are done I have to write those results back to a different table in the DB so we can look back at what those calculations later on if needed.
When I make the calculation I add the results to a List. Once I'm done writing out the file I look at that List and if there are any records I do the bulk insert.
With the bulk insert I have a setting in the App.config to set the number of records I want to insert. In this case I'm using 250 records. I assumed it would be better to limit my in memory arrays to say 250 records versus the 1,900. I loop through that list to the count in the App.config and create an array for each column. Those arrays are then passed as parameters to Oracle.
App.config
<add key="UpdateBatchCount" value="250" />
Class
class EligibleHours
{
public string EmployeeID { get; set; }
public decimal Hours { get; set; }
public string HoursSource { get; set; }
}
Data Manager
public static void SaveEligibleHours(List<EligibleHours> listHours)
{
//set the number of records to update batch on from config file Subtract one because of 0 based index
int batchCount = int.Parse(ConfigurationManager.AppSettings["UpdateBatchCount"]);
//create the arrays to add values to
string[] arrEmployeeId = new string[batchCount];
decimal[] arrHours = new decimal[batchCount];
string[] arrHoursSource = new string[batchCount];
int i = 0;
foreach (var item in listHours)
{
//Create an array of employee numbers that will be used for a batch update.
//update after every X amount of records, update. Add 1 to i to compensated for 0 based indexing.
if (i + 1 <= batchCount)
{
arrEmployeeId[i] = item.EmployeeID;
arrHours[i] = item.Hours;
arrHoursSource[i] = item.HoursSource;
i++;
}
else
{
UpdateDbWithEligibleHours(arrEmployeeId, arrHours, arrHoursSource);
//reset counter and array
i = 0;
arrEmployeeId = new string[batchCount];
arrHours = new decimal[batchCount];
arrHoursSource = new string[batchCount];
}
}
//process last array
if (arrEmployeeId.Length > 0)
{
UpdateDbWithEligibleHours(arrEmployeeId, arrHours, arrHoursSource);
}
}
private static void UpdateDbWithEligibleHours(string[] arrEmployeeId, decimal[] arrHours, string[] arrHoursSource)
{
StringBuilder sbQuery = new StringBuilder();
sbQuery.Append("insert into ELIGIBLE_HOURS ");
sbQuery.Append("(EMP_ID, HOURS_SOURCE, TOT_ELIG_HRS, REPORT_DATE) ");
sbQuery.Append("values ");
sbQuery.Append("(:1, :2, :3, SYSDATE) ");
string connectionString = ConfigurationManager.ConnectionStrings["Server_Connection"].ToString();
using (OracleConnection dbConn = new OracleConnection(connectionString))
{
dbConn.Open();
//create Oracle parameters and pass arrays of data
OracleParameter p_employee_id = new OracleParameter();
p_employee_id.OracleDbType = OracleDbType.Char;
p_employee_id.Value = arrEmployeeId;
OracleParameter p_hoursSource = new OracleParameter();
p_hoursSource.OracleDbType = OracleDbType.Char;
p_hoursSource.Value = arrHoursSource;
OracleParameter p_hours = new OracleParameter();
p_hours.OracleDbType = OracleDbType.Decimal;
p_hours.Value = arrHours;
OracleCommand objCmd = dbConn.CreateCommand();
objCmd.CommandText = sbQuery.ToString();
objCmd.ArrayBindCount = arrEmployeeId.Length;
objCmd.Parameters.Add(p_employee_id);
objCmd.Parameters.Add(p_hoursSource);
objCmd.Parameters.Add(p_hours);
objCmd.ExecuteNonQuery();
}
}
JavaFX 8 – FXML – TableView – TableColumn
I have the following objects
CourseResult
SimpleStringProperty CourseID
SimpleStringProperty CourseName
SimpleStringProperty Grade
SimpleIntegerProperty Credits
ATableRow
SimpleStringProperty StudentID
SimpleStringProperty StudentName
CourseResult[] AResult // Occurs 1 to 20.
In my program
javafx.collections.ObservableList<ATableRow> TableData = javafx.collections.FXCollections.observableArrayList();
I populate this Observable list from the database and I am able to see all the values perfectly in debugger.
I create the Tableview and add columns.
public void createTableForThisSemester(int thisSemester, int numberOfCourses, javafx.collections.ObservableList<AResultRow> TableRows) {
TableView<AResultRow> thisTable = new TableView<>();
TableColumn<AResultRow, String> tcolRollNo = new TableColumn<>("Roll Number");
tcolRollNo.setEditable(false);
tcolRollNo.setPrefWidth(120);
TableColumn<AResultRow, String> tcolName = new TableColumn<>("Student Name");
tcolName.setEditable(false);
tcolName.setPrefWidth(350);
tcolRollNo.setCellValueFactory(cellData -> cellData.getValue().StudentIDProperty());
tcolName.setCellValueFactory(cellData -> cellData.getValue().StudentNameProperty());
boolean xyz = thisTable.getColumns().addAll(tcolRollNo, tcolName);
// TableColumn[] courseColumn = new TableColumn[numberOfCourses];
for (int courseNo = 0; courseNo < numberOfCourses; courseNo++) {
String colName = getASemesterCourse(thisSemester, courseNo).getCourseID();
TableColumn<AResultRow, String> thisColumn = new TableColumn<>(colName);
thisColumn.setPrefWidth(80);
thisColumn.setStyle("-fx-alignment: CENTER; font-weight:bold;");
thisColumn.setCellValueFactory(cellData -> cellData.getValue().courseGradeProperty(courseNo));
boolean retVal = thisTable.getColumns().addAll(thisColumn);
}
// System.out.println("# of Rows in Table [" + thisSemester + "] = " + TableRows.size());
thisTable.getSelectionModel().setSelectionMode(SelectionMode.SINGLE);
thisTable.setItems(TableRows);
ScrollPane thisScrollPane = new ScrollPane();
thisScrollPane.setFitToWidth(true);
thisScrollPane.setFitToHeight(true);
thisScrollPane.setMinHeight((theDetails.getHeight() - 25));
thisScrollPane.setMaxHeight((theDetails.getHeight() - 25));
thisScrollPane.setMinWidth((theDetails.getWidth() - 25));
thisScrollPane.setHbarPolicy(ScrollPane.ScrollBarPolicy.ALWAYS);
Tab thisTab = tabs.getTabs().get(thisSemester);
thisTab.setContent(thisScrollPane);
thisScrollPane.setContent(thisTable);
}
Columns StudentID and Name are perfectly populated. But the Results are not being populated in
thisColumn.setCellValueFactory(cellData -> cellData.getValue().courseGradeProperty(courseNo));
I get this error in Netbeans for the line shown above. "Local Variables referenced from a lambda expression must be final or effectively final".
Remember, the GRADE[courseNo] field is a String and is populated.
How to show this value in the TABLE.
I have been trying various methods, like storing this value in a temp String... etc. etc.
For Example,
if i want to insert a double[] like,
Double[] dArr={10.23,25.1,30.5,45.3};
into HBase table.
Could you plzz tell me how to insert it into hbase??
You can store anything you want, you just have to serialize it to a byte[]:
Double[] dArr={10.23,25.1,30.5,45.3};
byte[] value = new byte[0];
byte[] family = "f".getBytes();
byte[] column = "d".getBytes();
for (Double d:dArr) {
value = Bytes.add(value, Bytes.toBytes(d));
}
Put put = new Put( rowKey );
put.add(family, column, Bytes.toBytes(value));
...
You'll have to unserialize the data when you read the value (convert from byte[] to Double[])
Another thing you can do is store each Double (as byte[]) in its own column (d0 to dX)
Double[] dArr={10.23,25.1,30.5,45.3};
Put put = new Put( rowKey );
byte[] family = "f".getBytes();
byte[] column = "d".getBytes();
int i=0;
for (Double d:dArr) {
put.add( family, Bytes.add(column, Bytes.toBytes(i)), Bytes.toBytes(d));
i++;
}
...
I created a list property in my model like so
public virtual List<String> listOfDays { get; set; }
then I converted and stored it in the list like this:
for (int i = 0; i < 30 i++)
{
var enrollment = new Enrollment();
enrollment.StudentID = id;
enrollment.listOfDays = searchString.ToList();
db.Enrollments.Add(enrollment);
db.SaveChanges();
}
I put a breakpoint here... enrollment.listOfDays = searchString.ToList(); ... and all is well. I see that the conversion was performed and I can see the values in listOfDays.
So I thought I would find a column in my database called listOfDays since I'm doing code first but the property is not there.
Then I thought I'd try accessing it anyway like this...
var classdays = from e in db.Enrollments where e.StudentID == id select e.listOfDays;
var days = classdays.ToList();
//here I get an error message about this not being supported in Linq.
Questions:
Why do you think the property was not in the database?
How can I post this array to my model then access it in a controller?
Thanks for any help.
Thanks to Decker: http://forums.asp.net/members/Decker%20Dong%20-%20MSFT.aspx
Here’s how it works:
Using form collection here…
In [HttpPost]…
private void Update (FormCollection formCollection, int id)
for (int sc = 0; sc < theSelectedCourses.Count(); sc++)
{
var enrollment = new Enrollment();
enrollment.CourseID = Convert.ToInt32(theSelectedCourses[sc]);
enrollment.StudentID = id;
enrollment.listOfDays = formCollection["searchString"] ;//bind this as a string instead of a list or array.
Then in [HttpGet]…
private void PopulateAssignedenrolledData(Student student, int id)
{
var dayList = from e in db.Enrollments where e.StudentID == id select e;
var days = dayList.ToList();
if (days.Count > 0)
{
string dl = days.FirstOrDefault().listOfDays;
string[] listofdays = dl.Split(',');
ViewBag.classDay = listofdays.ToList();
}
Thanks to Decker: http://forums.asp.net/members/Decker%20Dong%20-%20MSFT.aspx
Here’s how it works:
Using form collection here…
In [HttpPost]…
private void Update (FormCollection formCollection, int id)
for (int sc = 0; sc < theSelectedCourses.Count(); sc++)
{
var enrollment = new Enrollment();
enrollment.CourseID = Convert.ToInt32(theSelectedCourses[sc]);
enrollment.StudentID = id;
enrollment.listOfDays = formCollection["searchString"] ;//bind this as a string instead of a list or array.
Then in [HttpGet]…
private void PopulateAssignedenrolledData(Student student, int id)
{
var dayList = from e in db.Enrollments where e.StudentID == id select e;
var days = dayList.ToList();
if (days.Count > 0)
{
string dl = days.FirstOrDefault().listOfDays;
string[] listofdays = dl.Split(',');
ViewBag.classDay = listofdays.ToList();
}