I am using BaseX (XML based datastore) for its performance benchmarking. For testing it with ,
TestBeds
I) 10,000 users, 10 friends, 10 resources
II) 100,000 users , 10 friends, 10 resources
I faced below issues:
1) Loading of data is too slow. Gets slowed with eh increase in the number of threads.
2) Plus point - Reading/retriving values from BaseX is faster (17k operation per second)
3) Updating the data in BaseX is very slow. Throughput is ~10 operations per second.
Am I correct to say BaseX is 'TOO' slow for write/update operations (20/sec) compared to read/retrieve (10k/sec)?
Please advice me to make it more efficient for the write and update :
I have a function insertEntity (update or insert function) in to the BaseX datastore as follows -
public int insertEntity(String entitySet, String entityPK,
HashMap<String, ByteIterator> values, boolean insertImage) {
String parentTag ="",childTag ="", key="", entryTag="";
StringBuffer insertData = new StringBuffer();
Set<String> keys = values.keySet();
Iterator<String> iterator = keys.iterator();
while(iterator.hasNext()) {
String entryKey = iterator.next();
if(!(entryKey.equalsIgnoreCase("pic") || entryKey.equalsIgnoreCase("tpic")))
insertData.append("element " + entryKey + " {\"" + StringEscapeUtils.escapeXml(values.get(entryKey).toString()) + "\"},");
}
if(entitySet.equalsIgnoreCase("users")&& insertImage){
byte[] profileImage = ((ObjectByteIterator)values.get("pic")).toArray();
String encodedpImage = DatatypeConverter.printBase64Binary(profileImage);
insertData.append(" element pic {\"" + encodedpImage + "\"},");
profileImage = ((ObjectByteIterator)values.get("tpic")).toArray();
encodedpImage = DatatypeConverter.printBase64Binary(profileImage);
insertData.append(" element tpic {\"" + encodedpImage + "\"},");
}
if(entitySet.equalsIgnoreCase("users"))
{
parentTag = "users";
childTag = "members";
entryTag = "member";
key = "mem_id";
insertData.append("element confirmed_friends {}, element pending_friends {}");
}
if(entitySet.equalsIgnoreCase("resources"))
{
parentTag = "resources";
childTag = "resources";
entryTag = "resource";
key = "rid";
insertData.append("element manipulations {}");
}
try {
session.execute(new XQuery(
"insert node element " + entryTag
+ "{ attribute " + key + "{"
+ entityPK + "}, "
+ insertData.toString()
+ "} "
+ "into doc('" + databaseName + "/" + parentTag +".xml')/" + childTag
));
String q1 = "insert node element " + entryTag
+ "{ attribute " + key + "{"
+ entityPK + "}, "
+ insertData.toString()
+ "} "
+ "into doc('" + databaseName + "/" + parentTag +".xml')/" + childTag;
System.out.println(q1);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return 0;
}
And the below function is acceptFriendship (update function)
public int acceptFriend(int inviterID, int inviteeID) {
// TODO Auto-generated method stub
String acceptFriendQuery1 = "insert node <confirmed_friend id = '"
+ inviterID + "'>"
+ " </confirmed_friend>"
+ "into doc('"+databaseName+"/users.xml')/members/member[#mem_id = '"+inviteeID+"']/confirmed_friends";
String acceptFriendQuery2 = "insert node <confirmed_friend id = '"
+ inviteeID + "'>"
+ " </confirmed_friend>"
+ "into doc('"+databaseName+"/users.xml')/members/member[#mem_id = '"+inviterID+"']/confirmed_friends";
String acceptFriendQuery3 = "delete node doc('"+databaseName+"/users.xml')/members/member[#mem_id = '"
+ inviteeID + "']/pending_friends/pending_friend[#id = '"+ inviterID +"']";
try {
session.execute(new XQuery(acceptFriendQuery1));
session.execute(new XQuery(acceptFriendQuery2));
session.execute(new XQuery(acceptFriendQuery3));
System.out.println("Inviter: "+inviterID +" AND Invitee: "+inviteeID);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return 0;
}
Related
Dears,
I'm new to Streams and want to convert some logic to using them:
this is the logic:
for (String headerKey : map.keySet()) {
int counter = 1;
for (String value : map.get(headerKey)) {
if (map.get(headerKey).size() != 1) {
System.out.println("Response header: " + headerKey + "[" + counter++ + "]: " + value);
} else {
System.out.println("Response header: " + headerKey + ": " + value);
}
}
}
the problem is the "counter"...
I got as far as:
private static long after(Map<String, List<String>> map) {
return map.keySet().stream().map(key -> output(key, map)).count(); // use print() here instead of output()
}
private static String output(String key, Map<String, List<String>> map) {
counter = 1;
long longString = map.get(key).stream().map(value -> print(value, key, map.get(key))).count();
return "" + longString;
}
private static String print(String value, String key, List<String> strings) {
if (strings.size() != 1) {
System.out.println("Response header: " + key + "[" + counter++ + "]: " + value);
} else {
System.out.println("Response header: " + key + ": " + value);
}
return "";
}
And I suppose I can put the print() method at the indicated spot,
but I don't know how to get the counter to behave as in the original code...
All comment/ideas are welcome :)
Thanks beforehand!
Create a helper method like
static Stream<String> values(List<?> list) {
return list.size() == 1? Stream.of(": " + list.get(0)):
IntStream.range(0, list.size()).mapToObj(ix -> "[" + ix + "]: " + list.get(ix));
}
Instead of re-evaluating the list.size() == 1 condition in each iteration, it selects the right shape of the operation upfront. When the size is not one, rather than trying to maintain a counter, stream over the index range in the first place.
This method now can be used when streaming over the map, like
map.entrySet().stream()
.flatMap(e -> values(e.getValue()).map(("Response header: " + e.getKey())::concat))
.forEach(System.out::println);
I am trying to insert the values from this code below:
public int create(Officer officer) {
String sql = "insert into officer values(" + officer.getOfficer_id() + ", " + officer.getCollege_id() + ", " + Officer.getUnit_id() + ", " + officer.getRole_id() + ")";
return template.update(sql);
}
This should be a compilation error. getUnit_id is not a static method , so you need to call it using an instance of Officer.
public int create(Officer officer) {
String sql = "insert into officer values(" + officer.getOfficer_id() + ", " + officer.getCollege_id() + ", " + officer.getUnit_id() + ", " + officer.getRole_id() + ")";
return template.update(sql);
}
I am getting below error in my function.
Error: FROM keyword not found where expected
And here is my Function:
private int BauteilLieferzeit(string Materianummer)
{
try
{
OracleCommand cmd = new OracleCommand(
" Select MATNR, AVG_DAUER" +
" AVG " +
" (DATEDIFF " +
" (mi, Z.APL_ANFDATUM, " +
" Z.STA_LIEFERDATUM)) " +
" as AVG_DAUER " +
" from ZDATA AS Z " +
" where MATNR = '" + Materianummer + "'"
, OraVerbindung._conn);
OracleDataReader r = cmd.ExecuteReader();
if (r.HasRows)
{
int Restminuten = OraVerbindung.Lieferzeit;
while (r.Read())
{
Restminuten = r.GetInt32(1);
}
return Restminuten;
}
else
{
return OraVerbindung.Lieferzeit;
}
}
catch
{
return OraVerbindung.Lieferzeit;
}
}
In Oracle this is not a valid syntax
from ZDATA AS Z
use
from ZDATA Z
instead (remove "AS")
Additionally consider the use of bind variables instead of string concatenation:
" where MATNR = '" + Materianummer + "'"
search for "SQL Injection".
Use this. Included the issue highlighted by Marmite also. But the error FROM keyword not found where expected would be due to missing comma in select statement.
Edit: Removed AVG_DAUER column as it is getting derived later.
private int BauteilLieferzeit(string Materianummer)
{
try
{
OracleCommand cmd = new OracleCommand(
" Select MATNR," +
" AVG " +
" (DATEDIFF " +
" (mi, Z.APL_ANFDATUM, " +
" Z.STA_LIEFERDATUM)) " +
" as AVG_DAUER " +
" from ZDATA Z " +
" where MATNR = '" + Materianummer + "'"
, OraVerbindung._conn);
OracleDataReader r = cmd.ExecuteReader();
if (r.HasRows)
{
int Restminuten = OraVerbindung.Lieferzeit;
while (r.Read())
{
Restminuten = r.GetInt32(1);
}
return Restminuten;
}
else
{
return OraVerbindung.Lieferzeit;
}
}
catch
{
return OraVerbindung.Lieferzeit;
}
}
I have a Spring MVC REST service that accepts two #RequestParams called from and to. These are parsed as java.util.Date and passed to the following method in my DAO class.
#Override
public List<ErrorsDTOEntity> getAllErrors(Date from, Date to) {
try {
Query query = getSession().createQuery(
"SELECT NEW com.mydomain.esb.jpa.dto.ErrorsDTOEntity(ee, ec.message) "
+ "FROM ErrorsEntity ee, EventCodeEntity ec "
+ "WHERE ee.responseTime > " + from.getTime() + " "
+ "AND ee.responseTime < " + to.getTime() + " "
+ "AND ee.serviceResponseCode = ec.code "
+ "GROUP BY ee.domainName, ee.serviceName, ec.message, ee.serviceErrorCount, ee.errorTimestamp, "
+ "ee.deviceName, ee.servErrId, ee.serviceResponseCode, ee.elapsedTime, ee.forwardTime, "
+ "ee.responseCompletionTime, ee.responseSizeAverage, ee.requestSizeAverage, ee.responseTime "
+ "ORDER BY ee.domainName, ee.serviceName, ec.message, ee.errorTimestamp");
#SuppressWarnings("unchecked")
List<ErrorsDTOEntity> services = (List<ErrorsDTOEntity>) query.list();
return services;
} catch (HibernateException hex) {
hex.printStackTrace();
}
return null;
}
This is throwing the following SQL error:
org.hibernate.exception.SQLGrammarException: ORA-00932: inconsistent datatypes: expected TIMESTAMP got NUMBER
What's the proper way to structure this query so I can only fetch results between the from and to dates?
I figured it out, this works:
#Override
public List<ErrorsDTOEntity> getAllErrors(Date from, Date to) {
try {
Query query = getSession().createQuery(
"SELECT NEW com.mydomain.esb.jpa.dto.ErrorsDTOEntity(ee, ec.message) "
+ "FROM ErrorsEntity ee, EventCodeEntity ec "
+ "WHERE ee.responseTime > :from "
+ "AND ee.responseTime < :to "
+ "AND ee.serviceResponseCode = ec.code "
+ "GROUP BY ee.domainName, ee.serviceName, ec.message, ee.serviceErrorCount, ee.errorTimestamp, "
+ "ee.deviceName, ee.servErrId, ee.serviceResponseCode, ee.elapsedTime, ee.forwardTime, "
+ "ee.responseCompletionTime, ee.responseSizeAverage, ee.requestSizeAverage, ee.responseTime "
+ "ORDER BY ee.domainName, ee.serviceName, ec.message, ee.errorTimestamp");
query.setTimestamp("from", from);
query.setTimestamp("to", to);
#SuppressWarnings("unchecked")
List<ErrorsDTOEntity> services = (List<ErrorsDTOEntity>) query.list();
return services;
} catch (HibernateException hex) {
hex.printStackTrace();
}
return null;
}
I would like to select all nodes, that have text in them.
In this example the outer shouldBeIgnored tag, should not be selected:
<shouldBeIgnored>
<span>
the outer Span should be selected
</span>
</shouldBeIgnored>
Some other posts suggest something like this: //*/text().
However, this doesn't work in firefox.
This is a small UnitTest to reproduce the problem:
public class XpathTest {
final WebDriver webDriver = new FirefoxDriver();
#Test
public void shouldNotSelectIgnoredTag() {
this.webDriver.get("http://www.s2server.de/stackoverflow/11773593.html");
System.out.println(this.webDriver.getPageSource());
final List<WebElement> elements = this.webDriver.findElements(By.xpath("//*/text()"));
for (final WebElement webElement : elements) {
assertEquals("span", webElement.getTagName());
}
}
#After
public void tearDown() {
this.webDriver.quit();
}
}
If you want to select all nodes that contain text then you can use
//*[text()]
Above xpath will look for any element which contains text. Notice the text() function which is used to determine if current node has text or not.
In your case it will select <span> tag as it contains text.
You can call a javascript function, which shall return you text nodes:
function GetTextNodes(){
var lastNodes = new Array();
$("*").each(function(){
if($(this).children().length == 0)
lastNodes.push($(this));
});
return lastNodes;
}
Selenium WebDriver code:
IJavaScriptExecutor jscript = driver as IJavaScriptExecutor;
List<IWebElement> listTextNodes = jscript.ExecuteScript("return GetTextNodes();");
FYI: Something like might work for you.
I see no reason why this wouldn't work
(by java)
text = driver.findElement(By.xpath("//span")).getText()
If in the odd case that doesnt work:
text = driver.findElement(By.xpath("//span")).getAttribute("innerHTML")
Finally i found out that there is no way to do it with xpath (because XPaths text() selects also the innerText of a node). As workaround i have to inject Java Script that returns all elements, selected by an XPath, that has some text.
Like this:
public class XpathTest
{
//#formatter:off
final static String JS_SCRIPT_GET_TEXT = "function trim(str) { " +
" return str.replace(/^\s+|\s+$/g,''); " +
"} " +
" " +
"function extractText(element) { " +
" var text = ''; " +
" for ( var i = 0; i < element.childNodes.length; i++) { " +
" if (element.childNodes[i].nodeType === Node.TEXT_NODE) { " +
" nodeText = trim(element.childNodes[i].textContent); " +
" " +
" if (nodeText) { " +
" text += element.childNodes[i].textContent + ' '; " +
" } " +
" } " +
" } " +
" " +
" return trim(text); " +
"} " +
" " +
"function selectElementsHavingTextByXPath(expression) { " +
" " +
" result = document.evaluate(\".\" + expression, document.body, null, " +
" XPathResult.ANY_TYPE, null); " +
" " +
" var nodesWithText = new Array(); " +
" " +
" var node = result.iterateNext(); " +
" while (node) { " +
" if (extractText(node)) { " +
" nodesWithText.push(node) " +
" } " +
" " +
" node = result.iterateNext(); " +
" } " +
" " +
" return nodesWithText; " +
"} " +
"return selectElementsHavingTextByXPath(arguments[0]);";
//#formatter:on
final WebDriver webDriver = new FirefoxDriver();
#Test
public void shouldNotSelectIgnoredTag()
{
this.webDriver.get("http://www.s2server.de/stackoverflow/11773593.html");
final List<WebElement> elements = (List<WebElement>) ((JavascriptExecutor) this.webDriver).executeScript(JS_SCRIPT_GET_TEXT, "//*");
assertFalse(elements.isEmpty());
for (final WebElement webElement : elements)
{
assertEquals("span", webElement.getTagName());
}
}
#After
public void tearDown()
{
this.webDriver.quit();
}
}
I modified the UnitTest that the example testable.
One problem with locating text nodes is that even empty strings are considered as valid text nodes (e.g
<tag1><tag2/></tag1>
has no text nodes but
<tag1> <tag2/> </tag1>
has 2 text nodes, one with 2 spaces and another with 4 spaces )
If you want only the text nodes that have non-empty text, here is one way to do it:
//text()[string-length(normalize-space(.))>0]
or to get their parent elements
//*[text()[string-length(normalize-space(.))>0]]