I'm using Spring Boot 2.3.1 and JdbcTemplate to query simple table.
I execute following queries:
select * from products
select * from products limit 200 offset 0
select * from products where ... limit 200 offset 0.
All above queries returns 15 rows or less(when where is used). Does anybody know where this limit(15) can be configured?
I have added following properties
spring.jdbc.template.fetch-size=300
spring.jdbc.template.max-rows=300
but this doesn't work for me.
PS. I use PostgreSQL 12
EDIT 1
public List<Product> getList(final ProductFilter filter) {
final String query = filterToWhereClause(filter);
final String where = query.isEmpty() ? "" : " where " + query;
final String sql = String.format("select * from products %s order by id limit %d offset %d", where, filter.getSize(), filter.getPage() * filter.getSize());
return jdbcTemplate.query(sql, (resultSet, rowNumber) -> productRowMapper(resultSet));
}
private static String filterToWhereClause(final ProductFilter filter) {
return Stream.of(
like("name", filter.getName()),
gte("price", filter.getPriceFrom()),
lte("price", filter.getPriceTo()),
gte("total", filter.getTotalFrom()),
lte("total", filter.getTotalTo())
)
.filter(Objects::nonNull)
.collect(Collectors.joining(" AND "));
}
private static Product productRowMapper(final ResultSet resultSet) throws SQLException {
return resultSet.next()
? Product.of(resultSet.getInt("id"), resultSet.getString("name"), resultSet.getBigDecimal("price"), resultSet.getBigDecimal("total"))
: null;
}
Related
Trying to convert java resultSet data to data.frame in R. Up to 5 rows no issue. But when more than 5 records are converting then getting the error "syntax error, unexpected SYMBOL, expecting ','"
public static void main(String[] args) throws ScriptException, FileNotFoundException {
RenjinScriptEngineFactory factory = new RenjinScriptEngineFactory();
ScriptEngine engine = factory.getScriptEngine();
engine.eval("source(\"script.R\")");
engine.eval("workflow.predict("+getData()+")");
}
public static ListVector getData() {
Connection connection = null;
PreparedStatement statement = null;
ResultSet resultSet = null;
StringArrayVector.Builder userid = new StringArrayVector.Builder();
StringArrayVector.Builder defaultCC = new StringArrayVector.Builder();
StringArrayVector.Builder typeId = new StringArrayVector.Builder();
StringArrayVector.Builder amount = new StringArrayVector.Builder();
StringArrayVector.Builder cc = new StringArrayVector.Builder();
StringArrayVector.Builder activity = new StringArrayVector.Builder();
try{
String query = "select top 6 wi.owner_user_id as userId, u.cost_center_id as defaultCC, li.expense_type_id typeId, wi.doc_specific_amount as amount, al.cost_center_id as cc,wi.activity_id as activity from alwf_work_item wi, alco_user u, aler_expense_line_item li,aler_line_allocation al where u.user_id = wi.owner_user_id and wi.business_object_id=li.parent_id and li.exp_line_item_id=al.exp_line_item_id";
connection = getConnection();
statement = connection.prepareStatement(query);
resultSet = statement.executeQuery();
while(resultSet.next()) {
userid.add(resultSet.getLong("userId"));
defaultCC.add(resultSet.getLong("defaultCC"));
typeId.add(resultSet.getLong("typeId"));
amount.add(resultSet.getLong("amount"));
cc.add(resultSet.getLong("cc"));
activity.add(resultSet.getLong("activity"));
}
}catch (Exception e) {
e.printStackTrace();
}
ListVector.NamedBuilder myDf = new ListVector.NamedBuilder();
myDf.setAttribute(Symbols.CLASS, StringVector.valueOf("data.frame"));
myDf.setAttribute(Symbols.ROW_NAMES, new RowNamesVector(userid.length()));
myDf.add("userId", userid.build());
myDf.add("defaultCC", defaultCC.build());
myDf.add("typeId", typeId.build());
myDf.add("amount", amount.build());
myDf.add("cc", cc.build());
myDf.add("activity", activity.build());
return myDf.build();
}
R Script
workflow.predict <- function(abc) {
print(abc)
print(data.frame(lapply(abc, as.character), stringsAsFactors=FALSE))
dataset = data.frame(lapply(abc, as.character), stringsAsFactors=FALSE)
library(randomForest)
classifier = randomForest(x = dataset[-6], y = as.factor(dataset$activity))
new.data=c(62020,3141877,46013,950,3141877)
y_pred = predict(classifier, newdata = new.data)
print(y_pred)
return(y_pred)
}
Below are the errors while running the script with top 6 records. But for top 5 records it is running without any error. Thanks in advance.
Exception in thread "main" org.renjin.parser.ParseException: Syntax error at 1 68 1 70 68 70: syntax error, unexpected SYMBOL, expecting ','
at org.renjin.parser.RParser.parseAll(RParser.java:146)
at org.renjin.parser.RParser.parseSource(RParser.java:69)
at org.renjin.parser.RParser.parseSource(RParser.java:122)
at org.renjin.parser.RParser.parseSource(RParser.java:129)
at org.renjin.script.RenjinScriptEngine.eval(RenjinScriptEngine.java:127)
at RTest.main(RTest.java:27)
Sample Data of the resultset is here
By using Builder userid = new ListVector.Builder() instead of StringArrayVector.Builder userid = new StringArrayVector.Builder() it is accepting more records without any error. I am not sure why StringArrayVector it is restricting to 5 records only.
In the icCube Builder ETL, I want to group the data on more than one field. Also, as aggregation function, I would like to make use of MAX and MIN.
Example data:
(same data in text)
groupId phase startDate endDate
100 start 1-May-2018 5-May-2018
100 start 4-May-2018 7-May-2018
100 start 28-Apr-2018 1-May-2018
100 middle 4-May-2018 11-May-2018
100 middle 1-May-2018 10-May-2018
100 end 12-May-2018 15-May-2018
100 end 11-May-2018 13-May-2018
100 end 13-May-2018 14-May-2018
100 end 9-May-2018 12-May-2018
200 start 4-Apr-2018 2-May-2018
200 middle 18-Apr-2018 3-May-2018
200 middle 1-May-2018 1-May-2018
300 end 21-Apr-2018 24-Apr-2018
I would like to group this data on groupId and phase and get the minimum startDate and the maximum endDate:
How to best do that in the icCube ETL?
We're adding a new version of groupBy View in the ETL layer to support this. However you can create a Java view to perform the groupBy.
Something like :
package iccube.pub;
import java.util.*;
import java.lang.*;
import org.joda.time.*;
import crazydev.iccube.pub.view.*;
public class CustomJavaView implements IOlapBuilderViewLogic
{
private Map<List<Comparable>,List<Agg>> cached;
public CustomJavaView()
{
}
public void onInitMainTable(Map<String, IOlapCachedTable> cachedTables, IOlapDataTableDef mainTable)
{
cached = new HashMap();
}
public boolean onNewRow(IOlapViewContext context, Map<String, IOlapCachedTable> cachedTables, IOlapDataTableDef mainTable, IOlapReadOnlyDataRow mainTableRow)
{
// create the groupby key (list of values)
final List<Comparable> groupBy = Arrays.asList(mainTableRow.get("phase"), mainTableRow.get("groupId"));
// get the aggregators for values for the keys, build them if not already there
final List<Agg> aggs = cached.computeIfAbsent(groupBy, key -> Arrays.asList(new Agg(true), new Agg(false)));
// add values
aggs.get(0).add(mainTableRow.getAsDateTime("startDate"));
aggs.get(1).add(mainTableRow.getAsDateTime("endDate"));
return true; // false to stop
}
public void onProcessingCompleted(IOlapViewContext context, Map<String, IOlapCachedTable> cachedTables)
{
// now we can fire rows
for (Map.Entry<List<Comparable>, List<Agg>> entry : cached.entrySet())
{
final List<Comparable> groupByKey = entry.getKey();
final List<Agg> aggs = entry.getValue();
// create empty row
final IOlapDataTableRow row = context.newRow();
row.set("phase",groupByKey.get(0));
row.set("groupId",groupByKey.get(1));
row.set("startDate",aggs.get(0).date);
row.set("endDate",aggs.get(1).date);
context.fireRow(row);
}
}
// this is the Aggregator, you could implement something more complicated
static class Agg
{
final int isMin;
LocalDateTime date;
Agg(boolean isMin)
{
this.isMin = isMin ? -1 : 1;
}
void add(LocalDateTime ndate)
{
if (ndate != null)
{
date = ( date!= null && ((date.compareTo(ndate) * isMin) > 0)) ? date : ndate;
}
}
}
}
I am trying to update certain rows in my oracle DB using id which is of RAW(255).
Sample ids 0BF3957A016E4EBCB68809E6C2EA8B80, 1199B9F29F0A46F486C052669854C2F8...
#Autowired
private NamedParameterJdbcTemplate jdbcTempalte;
private static final String UPDATE_SUB_STATUS = "update SUBSCRIPTIONS set status = :status, modified_date = systimestamp where id in (:ids)";
public void saveSubscriptionsStatus(List<String> ids, String status) {
MapSqlParameterSource paramSource = new MapSqlParameterSource();
List<String> idsHexToRaw = new ArrayList<>();
String temp = new String();
for (String id : ids) {
temp = "hextoraw('" + id + "')";
idsHexToRaw.add(temp);
}
paramSource.addValue("ids", idsHexToRaw);
paramSource.addValue("status", status);
jdbcTempalte.update(*UPDATE_SUB_STATUS*, paramSource);
}
This above block of code is executing without any error but the updates are not reflected to the db, while if I skip using hextoraw() and just pass the list of ids it works fine and also updates the data in table. see below code
public void saveSubscriptionsStatus(List<String> ids, String status) {
MapSqlParameterSource paramSource = new MapSqlParameterSource();]
paramSource.addValue("ids", ids);
paramSource.addValue("status", status);
jdbcTempalte.update(UPDATE_SUB_STATUS, paramSource);
}
this code works fine and updates the table, but since i am not using hextoraw() it scans the full table for updation which I don't want since i have created indexes. So using hextoraw() will use index for scanning the table but it is not updating the values which is kind of weird.
Got a solution myself by trying all the different combinations :
#Autowired
private NamedParameterJdbcTemplate jdbcTempalte;
public void saveSubscriptionsStatus(List<String> ids, String status) {
String UPDATE_SUB_STATUS = "update SUBSCRIPTIONS set status = :status, modified_date = systimestamp where id in (";
MapSqlParameterSource paramSource = new MapSqlParameterSource();
String subQuery = "";
for (int i = 0; i < ids.size(); i++) {
String temp = "id" + i;
paramSource.addValue(temp, ids.get(i));
subQuery = subQuery + "hextoraw(:" + temp + "), ";
}
subQuery = subQuery.substring(0, subQuery.length() - 2);
UPDATE_SUB_STATUS = UPDATE_SUB_STATUS + subQuery + ")";
paramSource.addValue("status", status);
jdbcTempalte.update(UPDATE_SUB_STATUS, paramSource);
}
What this do is create a query with all the ids to hextoraw as id0, id1, id2...... and also added this values in the MapSqlParameterSource instance and then this worked fine and it also used the index for updating my table.
After running my new function the query look like : update
SUBSCRIPTIONS set status = :status, modified_date = systimestamp
where id in (hextoraw(:id0), hextoraw(:id1), hextoraw(:id2)...)
MapSqlParameterSource instance looks like : {("id0", "randomUUID"),
("id1", "randomUUID"), ("id2", "randomUUID").....}
Instead of doing string manipulation, Convert the list to List of ByteArray
List<byte[]> productGuidByteList = stringList.stream().map(item -> GuidHelper.asBytes(item)).collect(Collectors.toList());
parameters.addValue("productGuidSearch", productGuidByteList);
public static byte[] asBytes(UUID uuid) {
ByteBuffer bb = ByteBuffer.wrap(new byte[16]);
bb.putLong(uuid.getMostSignificantBits());
bb.putLong(uuid.getLeastSignificantBits());
return bb.array();
}
sql3 = "select avg(eff),min(eff),max(eff) from(select ("+baseParam+"/"+denom+"))as eff from ras)as s"; This is query whose output i want.
When i execute the code i get the error stating check your mysql version for syntax. I am using string to store the name of columns. I want to find the efficienccy with respect to 2000 Job_Render i.e. efficiency for each job_render. But what i get is total efficiency of all job_render. when i use the sql syntax with their direct column names. I have commented that query too for the reference. I want to find efficiency of each job render with respect to their 2000 JOBID. Bottom line is i want to find efficiency of 2000 JOBID each whose formula is Job_Render/LC_Final+LC_Preview. I have stored Job_Render in String baseParam and sum of both LC in String Denom. Please help me out.
public class Efficiency {
static final String DB_DRIVER = "com.mysql.jdbc.Driver";
static final String DB_CONNECTION = "jdbc:mysql://localhost:3306/";
static final String DB_USER = "root";
static final String DB_PASSWORD = "root";
static final String dbName = "raas";
public static void main(String[] args) {
try{
effFunc();
}
catch (Exception q){
q.getMessage();
}
}
static void effFunc() throws ClassNotFoundException,SQLException{
Connection conn = null;
// STEP 2: Register JDBC driver
Class.forName(DB_DRIVER);
// STEP 3: Open a connection
System.out.println("Connecting to a selected database...");
conn = DriverManager.getConnection(DB_CONNECTION + dbName, DB_USER,
DB_PASSWORD);
System.out.println("Connected database successfully...");
String baseParam;
//String[] subParam ;
baseParam= "Job_Render";
String sql3="";
String denom="";
final String[] COL={ "LC_Final","LC_Preview"};
denom = "(" + COL[0] + "+" + COL[1] + ")";
Statement stmt = null;
stmt = conn.createStatement();
// sql3 = "select 'Efficiency' Field,avg(eff),min(eff),max(eff) from(select (Job_Render/(LC_Final+LC_Preview))as eff from ras)as s";
sql3 = "select avg(eff),min(eff),max(eff) from(select ("+baseParam+"/"+denom+"))as eff from ras)as s";
System.out.println(sql3);
//
try{
ResultSet res = stmt.executeQuery(sql3);
//System.out.println(res);
while(res.next()){
// String JobID = res.getString("JobID");
// System.out.println("Job ID : " + JobID);
String col1 = res.getString(1);
System.out.println(col1);
double avg = res.getDouble(2);
System.out.println("Average of eff is:"+avg);
double min = res.getDouble(3);
System.out.println("Min of eff is:"+min);
double max = res.getDouble(4);
System.out.println("Max of eff is:"+max);
}}
catch(Exception e){
e.getStackTrace();
System.out.println(e.getMessage());
}
}}
Your code runs the query sql1 which is the empty string,
String sql1="";
// sql1 = "select * from raas.jobs";
ResultSet res = stmt.executeQuery(sql1); // sql1 = ""
should be
ResultSet res = stmt.executeQuery(sql3);
Edit
Then to get the value(s)
String col1 = res.getString(1);
double avg = res.getDouble(2);
double min = res.getDouble(3);
double max = res.getDouble(4);
I use Spring Jdbc Template that way:
public List<User> getUsersForGrid(int rows, int page, String sidx,
String sord) {
int fromRecord = 0;
int toRecord = 0;
toRecord = page * rows;
fromRecord = (page - 1) * rows;
StringBuilder sqlB = new StringBuilder();
sqlB.append("SELECT user_id, username ");
sqlB.append("FROM users ");
sqlB.append("WHERE :fromRecord <= rownum AND rownum <= :toRecord ");
sqlB.append("ORDER BY %s %s ");
String sql = String.format(sqlB.toString(), sidx, sord);
MapSqlParameterSource params = new MapSqlParameterSource();
params.addValue("fromRecord", fromRecord);
params.addValue("toRecord", toRecord);
List<Map<String, Object>> rsRows = this.jdbcTemplate.queryForList(sql
.toString(),params);
List<User> users = new ArrayList<User>();
for (Map<String, Object> row : rsRows) {
BigDecimal id = (BigDecimal) row.get("user_id");
String username = (String) row.get("username");
User user = new User(id.intValue(), username);
users.add(user);
}
return users;
}
and get java.sql.SQLException: Invalid column type
sidx is column nate("user_id" for example) sord is asc/desc
When pass no params(execute only
sql.append("SELECT user_id, username ");
sql.append("FROM users ");
) everything is OK.
Update: Works with:
sqlB.append("WHERE ? <= rownum AND rownum <= ? ");
and
this.jdbcTemplate.queryForList(sql.toString(),new Object[]{fromRecord, toRecord});
Seems like problem with Spring MapSqlParameterSource and named parameters. I use Spring 3.1.3
DB is Oracle 11.2
describe users;
Name Null Type
------------------------------ -------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
USER_ID NOT NULL NUMBER
USERNAME NOT NULL VARCHAR2(40)
PASSWORD NOT NULL VARCHAR2(20)
ENABLED NOT NULL NUMBER
I think the problem is with your order by clause,
you are trying to dynamically change your order by clause.
Just try
StringBuilder sql = new StringBuilder();
sql.append("SELECT user_id, username ");
sql.append("FROM users ");
sql.append("WHERE :fromRecord <= rownum AND rownum <= :toRecord ");
sql.append("ORDER BY user_id asc ");
MapSqlParameterSource params = new MapSqlParameterSource();
params.addValue("fromRecord", fromRecord);
params.addValue("toRecord", toRecord);
If the above works, then instead of using the MapSqlParameterSource for changing the order by clause use something like
StringBuilder sql = new StringBuilder();
sql.append("SELECT user_id, username ");
sql.append("FROM users ");
sql.append("WHERE :fromRecord <= rownum AND rownum <= :toRecord ");
sql.append("ORDER BY %s %s ");
//Format the sql string accordingly
MapSqlParameterSource params = new MapSqlParameterSource();
params.addValue("fromRecord", fromRecord, Types.INTEGER);
params.addValue("toRecord", toRecord, Types.INTEGER);
Hope it helps.
try this
List<Map> rows = getJdbcTemplate().queryForList(sql);
for (Map row : rows) {
BigDecimal id = (BigDecimal) row.get("user_id");
String username = (String) row.get("username");
User user = new User(id.intValue(), username);
users.add(user);
}
ok try
MapSqlParameterSource namedParameters = new MapSqlParameterSource();
namedParameters.addValue("fromRecord", fromRecord);
namedParameters.addValue("toRecord", toRecord);
namedParameters.addValue("sidx", sidx);
namedParameters.addValue("sord", sord);
return this.getNamedParameterJdbcTemplate().query(query,
namedParameters, new UserElementMapper());
public class UserMapper implements RowMapper<User> {
public EmailElement mapRow(ResultSet rs, int rowNum) throws SQLException {
User user = new User();
emailElement.setID(rs.getInt("user_id"));
emailElement.setUsernameo(rs.getString("username"));
return user;
}
}