I have a Datastore query using cursor (Objectify v5) and I want to get the cursor after each item in the result list. Code looks like this:
public List<Puzzle> queryWithCursor(String cursor, String order, int limit) {
Query<Puzzle> query = ObjectifyService.ofy()
.load()
.type(Puzzle.class)
.order(order)
.limit(limit);
query = query.startAt(Cursor.fromWebSafeString(cursor));
List<Puzzle> puzzles = new ArrayList<>();
QueryResultIterator<Puzzle> iterator = query.iterator();
while (iterator.hasNext()) {
Puzzle puzzle = iterator.next();
puzzle.setCursor(iterator.getCursor().toWebSafeString());
puzzles.add(puzzle);
}
return puzzles;
}
While the method works correctly, it triggers so many Datastore queries behind the scene. Basically, every time iterator.getCursor() runs, it triggers an additional query. I learnt from Stackdriver Trace that if limit is 20, the method triggers 19 queries in total (it seems that the last .getCursor() does not trigger additional query). So this method is even slower and more costly than the similar query using offset.
Is this really a bug? Is there a way to avoid the performance hit?
This is actually a fundamental behavior of the datastore, at least in the old sdk (as opposed to the new sdk that Objectify 6 uses, which may be the same maybe not). Calling getCursor() at non-batch boundaries restarts the query. You can try it with the low-level API.
There is a workaround: Make up your own Cursor class. It should consist of the low level Cursor and an offset. Explicitly set a chunk() size, then your cursor should consist of the Cursor at index 0 plus an offset into the chunk.
Then when you want to restart a query at that cursor, use .cursor(batchStartCursor).offset(offsetIntoBatch).
import com.google.appengine.api.datastore.Cursor;
import com.google.appengine.api.datastore.DatastoreService;
import com.google.appengine.api.datastore.DatastoreServiceFactory;
import com.google.appengine.api.datastore.Entity;
import com.google.appengine.api.datastore.FetchOptions;
import com.google.appengine.api.datastore.PreparedQuery;
import com.google.appengine.api.datastore.Query;
import com.google.appengine.api.datastore.Query.SortDirection;
import com.google.appengine.api.datastore.QueryResultList;
import java.io.IOException;
import java.io.PrintWriter;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
public class ListPeopleServlet extends HttpServlet {
static final int PAGE_SIZE = 15;
private final DatastoreService datastore;
public ListPeopleServlet() {
datastore = DatastoreServiceFactory.getDatastoreService();
}
#Override
protected void doGet(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {
FetchOptions fetchOptions = FetchOptions.Builder.withLimit(PAGE_SIZE);
// If this servlet is passed a cursor parameter, let's use it.
String startCursor = req.getParameter("cursor");
if (startCursor != null) {
fetchOptions.startCursor(Cursor.fromWebSafeString(startCursor));
}
Query q = new Query("Person").addSort("name", SortDirection.ASCENDING);
PreparedQuery pq = datastore.prepare(q);
QueryResultList<Entity> results;
try {
results = pq.asQueryResultList(fetchOptions);
} catch (IllegalArgumentException e) {
// IllegalArgumentException happens when an invalid cursor is used.
// A user could have manually entered a bad cursor in the URL or there
// may have been an internal implementation detail change in App Engine.
// Redirect to the page without the cursor parameter to show something
// rather than an error.
resp.sendRedirect("/people");
return;
}
resp.setContentType("text/html");
resp.setCharacterEncoding("UTF-8");
PrintWriter w = resp.getWriter();
w.println("<!DOCTYPE html>");
w.println("<meta charset=\"utf-8\">");
w.println("<title>Cloud Datastore Cursor Sample</title>");
w.println("<ul>");
for (Entity entity : results) {
w.println("<li>" + entity.getProperty("name") + "</li>");
}
w.println("</ul>");
String cursorString = results.getCursor().toWebSafeString();
// This servlet lives at '/people'.
w.println("<a href='/people?cursor=" + cursorString + "'>Next page</a>");
}
}
Related
I am trying to read the cells in an xls file.This is the code I have.Please let me know where I am going wrong.I don't see any error in the logviewer but it isn't printing anything.
import org.apache.commons.csv.CSVFormat;
import org.apache.commons.csv.CSVParser;
import org.apache.commons.csv.CSVRecord;
import java.io.IOException;
import java.io.Reader;
import java.nio.file.Files;
import java.nio.file.Paths;
public class ApacheCommonsCSV {
public void readCSV() throws IOException {
String CSV_File_Path = "C:\\source\\Test.csv";
// read the file
Reader reader = Files.newBufferedReader(Paths.get(CSV_File_Path));
// parse the file into csv values
CSVParser csvParser = new CSVParser(reader, CSVFormat.DEFAULT);
for (CSVRecord csvRecord : csvParser) {
// Accessing Values by Column Index
String name = csvRecord.get(0);
String product = csvRecord.get(1);
// print the value to console
log.info("Record No - " + csvRecord.getRecordNumber());
log.info("---------------");
log.info("Name : " + name);
log.info("Product : " + product);
log.info("---------------");
}
}
}
You're declaring readCSV() function but not calling it anywhere, that's why your code doesn't even get executed.
You need to add this readCSV() function call and if you're lucky enough it will start working and if not - your will see an error in jmeter.log file.
For example like this:
import java.io.IOException;
import java.io.Reader;
import java.nio.file.Files;
import java.nio.file.Paths;
import org.apache.commons.csv.CSVFormat;
import org.apache.commons.csv.CSVParser;
import org.apache.commons.csv.CSVRecord;
public class ApacheCommonsCSV {
public void readCSV() throws IOException {
String CSV_File_Path = "C:\\source\\Test.csv";
// read the file
Reader reader = Files.newBufferedReader(Paths.get(CSV_File_Path));
// parse the file into csv values
CSVParser csvParser = new CSVParser(reader, CSVFormat.DEFAULT);
for (CSVRecord csvRecord : csvParser) {
// Accessing Values by Column Index
String name = csvRecord.get(0);
String product = csvRecord.get(1);
// print the value to console
log.info("Record No - " + csvRecord.getRecordNumber());
log.info("---------------");
log.info("Name : " + name);
log.info("Product : " + product);
log.info("---------------");
}
}
readCSV(); // here is the entry point
}
Just make sure to have commons-csv.jar in JMeter Classpath
Last 2 cents:
Any reason for not using CSV Data Set Config?
Since JMeter 3.1 you should be using JSR223 Test Elements and Groovy language for scripting so consider migrating to Groovy right away. Check out Apache Groovy - Why and How You Should Use It for reasons, tips and tricks.
I'm trying to implement a filter search using spring data solr. I've following filters types and all have a set of filters.
A
aa in (1,2,3)
ab between (2016-08-02 TO 2016-08-10)
B
ba in (2,3,4)
bb between (550 TO 1000)
The Solr query which I want to achieve using Spring data solr is:
q=*:*&fq=(type:A AND aa:(1,2,3) AND ab:[2016-08-02 TO 2016-08-10]) OR (type:B AND ba:(2,3,4) AND bb:[550 TO 1000])
I'm not sure how to group a number of clauses of a type of filter and then have an OR operator.
Thanks in advance.
The trick is to flag the second Criteria via setPartIsOr(true) with an OR-ish nature. This method returns void, so it cannot be chained.
First aCriteria and bCriteria are defined as described. Then bCriteria is flagged as OR-ish. Then both are added to a SimpleFilterQuery. That in turn can be added to the actual Query. That is left that out in the sample.
The DefaultQueryParser in the end is used only to generate a String that can be used in the assertion to check that the query is generated as desired.
import org.junit.jupiter.api.Test;
import org.springframework.data.solr.core.DefaultQueryParser;
import org.springframework.data.solr.core.query.Criteria;
import org.springframework.data.solr.core.query.FilterQuery;
import org.springframework.data.solr.core.query.SimpleFilterQuery;
import static org.junit.jupiter.api.Assertions.assertEquals;
public class CriteriaTest {
#Test
public void generateQuery() {
Criteria aCriteria =
new Criteria("type").is("A")
.connect().and("aa").in(1,2,3)
.connect().and("ab").between("2016-08-02", "2016-08-10");
Criteria bCriteria =
new Criteria("type").is("B")
.connect().and("ba").in(2,3,4)
.connect().and("bb").between("550", "1000");
bCriteria.setPartIsOr(true); // that is the magic
FilterQuery filterQuery = new SimpleFilterQuery();
filterQuery.addCriteria(aCriteria);
filterQuery.addCriteria(bCriteria);
// verify the generated query string
DefaultQueryParser dqp = new DefaultQueryParser(null);
String actualQuery = dqp.getQueryString(filterQuery, null);
String expectedQuery =
"(type:A AND aa:(1 2 3) AND ab:[2016\\-08\\-02 TO 2016\\-08\\-10]) OR "
+ "((type:B AND ba:(2 3 4) AND bb:[550 TO 1000]))";
System.out.println(actualQuery);
assertEquals(expectedQuery, actualQuery);
}
}
I have to run a jdbc update query using preparedstatement/statement without knowing the column type.
I have a query say ' update table set status=? where id=? '
and i am getting a map of values like {("status"="123"), ("id"="546")}
Now I don't know the column type, is there any generic way to run this query using jdbc?
Instead of running - ps.setString(1,map.get(("status"));
beacause i don't know the column type of status field in DB (it may be a int also)
Please help me in solving this without using spring jdbc templates.
ps.setString(1,map.get(("status")) will work for integer also, you just has to take care that value you are putting in integer column is of int type.
Following code explains that:
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.SQLException;
public class SOJDBC
{
public static void main(String rgs[])
{
Connection con = DBConnection.getConnection("TEMP");
try
{
PreparedStatement pstmt = con.prepareStatement("insert into STUDENT(ROLLNO,NAME,AGE) values(?,?,?)");
pstmt.setString(1, "1"); //column type is integer, will work because the value is of int type
pstmt.setString(2, "Bhushan");
pstmt.setString(3, "25"); //column type is integer, will work because the value is of int type
pstmt.executeUpdate();
}
catch (SQLException e)
{
e.printStackTrace();
}
}
}
I currently have a TreeGrid which shows nodes with names. The data is coming from a manually populated DataSource.
When setting the filter on the nodeName field, The filter is not done recursevily and thus I can only filter the Root node.
How can I tell the filter to search for matches in child nodes?
PS: in the code below, I have 3 nodes Root > Run > Child1. If i try the filter and type "R", I get Root and Run. But if i Type "C", I get "no results found"
Code
DataSource:
package murex.otk.gwt.gui.client.ui.datasource;
import java.util.List;
import murex.otk.gwt.gui.client.ui.record.TreeRecord;
import com.smartgwt.client.data.DataSource;
import com.smartgwt.client.data.fields.DataSourceIntegerField;
import com.smartgwt.client.data.fields.DataSourceTextField;
public class ClassesDataSource extends DataSource {
private static ClassesDataSource instance = null;
public static ClassesDataSource getInstance() {
if (instance == null) {
instance = new ClassesDataSource("classesDataSource");
}
return instance;
}
private ClassesDataSource(String id) {
setID(id);
DataSourceTextField nodeNameField = new DataSourceTextField("nodeName");
nodeNameField.setCanFilter(true);
nodeNameField.setRequired(true);
DataSourceIntegerField nodeIdField = new DataSourceIntegerField("nodeId");
nodeIdField.setPrimaryKey(true);
nodeIdField.setRequired(true);
DataSourceIntegerField nodeParentField = new DataSourceIntegerField("nodeParent");
nodeParentField.setRequired(true);
nodeParentField.setForeignKey(id + ".nodeId");
nodeParentField.setRootValue(0);
setFields(nodeIdField, nodeNameField, nodeParentField);
setClientOnly(true);
}
public void populateDataSource(List<String> classNames) {
TreeRecord root = new TreeRecord("Root", 0);
addData(root);
TreeRecord child1 = new TreeRecord("Child1", root.getNodeId());
addData(child1);
TreeRecord child2 = new TreeRecord("Run", child1.getNodeId());
addData(child2);
}
}
Main
public void onModuleLoad() {
ClassesDataSource.getInstance().populateDataSource(new ArrayList<String>());
final TreeGrid employeeTree = new TreeGrid();
employeeTree.setHeight(350);
employeeTree.setDataSource(ClassesDataSource.getInstance());
employeeTree.setAutoFetchData(true);
TreeGridField field = new TreeGridField("nodeName");
field.setCanFilter(true);
employeeTree.setFields(field);
employeeTree.setShowFilterEditor(true);
employeeTree.setAutoFetchAsFilter(true);
employeeTree.setFilterOnKeypress(true);
employeeTree.draw();
}
I solved this.
The problem was that the filter was calling the server to fetch data whereas my datasource was set to Client Only. To fix this, the employeeTree must have employeeTree.setLoadDataOnDemand(false);
You may also use employeeTree.setKeepParentsOnFilter(true)
http://www.smartclient.com/smartgwtee/javadoc/com/smartgwt/client/widgets/tree/TreeGrid.html#setKeepParentsOnFilter(java.lang.Boolean)
It's quite simple with only two operations, you can either push a value, or get all values, clearing all these values from the structure. Can be easily implemented with a queue aggregate.
push(value)
get() -> list of values, clears contents
In my opinion, there is no common name for a data structure which provides this functionality. Depending on the list order provided by get() it is similar to a stack or a queue.
Assuming Java, this could work for you (not tested):
import java.util.ArrayList;
import java.util.Collection;
import java.util.Stack;
public class FlushStack<E> extends Stack<E> {
public Collection<E> get() {
ArrayList<E> elements = new ArrayList<E>(this);
this.clear();
return elements;
}
}
It is basically a stack, only that "get" does a recurring "pop" until the stack is empty.
In pseudocode this would be something like:
def push (v): stack.push (v)
def get ():
retVal = []
while stack.isNotEmpty: retval += [stack.pop () ]
return retVal