Spring Batch Dynamic Insert and Update with DB to DB - spring-boot

There is scenario where we want to load data from DB to DB .
But we want to check if data is already present in target system then update it otherwise insert it into DB.
We are using below approach:
#Bean
ItemWriter<Student> onosItemWriter1() {
JdbcBatchItemWriter<Student> databaseItemWriter = new JdbcBatchItemWriter<>();
databaseItemWriter.setDataSource(dataSource);
databaseItemWriter.setJdbcTemplate(namedParameterJdbcTemplate);
databaseItemWriter.setSql(INSERT_QUERY);
ItemPreparedStatementSetter<Student> valueSetter = new ItemPreparedStatementSetter<Student>() {
#Override
public void setValues(Student student, PreparedStatement statement) throws SQLException {
if (Student.getId() < 0) {
log.info("Inserting!");
databaseItemWriter.setSql(INSERT_QUERY);
statement.setString(1, student.getName());
statement.setString(2, student.getEmail());
} else {
log.info("updateing!!!!");
databaseItemWriter.setSql(UPDATE_QUERY);
statement.setString(1, student.getName());
statement.setString(2, student.getEmail());
}
}
};
databaseItemWriter.setItemPreparedStatementSetter(valueSetter);
return databaseItemWriter;
}
above its working only for insert. How I can do it with one JDBC batch Item writer to update it as well dynamically. if record is already present in chunks.

The code you shared won't work, because you are configuring the query on the item writer based on a runtime information from the item itself (Student.getId() < 0).
we want to check if data is already present in target system then update it otherwise insert it into DB
I would keep it simple and write a custom item writer like follows:
public class StudentItemWriter implements ItemWriter<Student> {
private static final String INSERT_QUERY = "insert into student ...";
private static final String UPDATE_QUERY = "update student set ... where id = ?";
private JdbcTemplate jdbcTemplate;
public StudentItemWriter(DataSource dataSource) {
this.jdbcTemplate = new JdbcTemplate(dataSource);
}
public void write(List<? extends Student> students) throws Exception {
for(Student student : students) {
int updated = jdbcTemplate.update(UPDATE_QUERY,...);
if(updated == 0) {
jdbcTemplate.update(INSERT_QUERY,...);
}
}
}
}

Related

Adding custom logic in Odata Create Entity call in java code

In the project am using olingo 2.0.12 jar in the java code.
During the create Entity service call ,
Is there a way to check for which entity data insert requested and,
Alter column values / append new column values before data persisted?
Is there a way to add above?
Code snippet given below,
public class A extends ODataJPADefaultProcessor{
#Override
public ODataResponse createEntity(final PostUriInfo uriParserResultView, final InputStream content,
final String requestContentType, final String contentType) throws ODataJPAModelException,
ODataJPARuntimeException, ODataNotFoundException, EdmException, EntityProviderException {
// Need to check the entity name and need to alter/add column values
}
}
Yes one of the possible ways would be to create your own CustomODataJPAProcessor which extends ODataJPADefaultProcessor.
You will have to register this in JPAServiceFactory by overriding the method
#Override
public ODataSingleProcessor createCustomODataProcessor(ODataJPAContext oDataJPAContext) {
return new CustomODataJPAProcessor(this.oDataJPAContext);
}
Now Olingo will use CustomODataJPAProcessor which can implement the following code to check the entities and transform them if needed
Sample code of CustomODataJPAProcessor
public class CustomODataJPAProcessor extends ODataJPADefaultProcessor {
Logger LOG = LoggerFactory.getLogger(this.getClass());
public CustomODataJPAProcessor(ODataJPAContext oDataJPAContext) {
super(oDataJPAContext);
}
#Override
public ODataResponse createEntity(final PostUriInfo uriParserResultView, final InputStream content,
final String requestContentType, final String contentType) throws ODataException {
ODataResponse oDataResponse = null;
oDataJPAContext.setODataContext(getContext());
InputStream forwardedInputStream = content;
try {
if (uriParserResultView.getTargetEntitySet().getName().equals("Students")) {
LOG.info("Students Entity Set Executed");
if (requestContentType.equalsIgnoreCase(ContentType.APPLICATION_JSON.toContentTypeString())) {
#SuppressWarnings("deprecation")
JsonElement elem = new JsonParser().parse(new InputStreamReader(content));
Gson gson = new GsonBuilder().setFieldNamingPolicy(FieldNamingPolicy.UPPER_CAMEL_CASE).create();
Student s = gson.fromJson(elem, Student.class);
// Change some values
s.setStudentID("Test" + s.getStudentID());
forwardedInputStream = new ByteArrayInputStream(gson.toJson(s).getBytes());
}
}
Object createdJpaEntity = jpaProcessor.process(uriParserResultView, forwardedInputStream,
requestContentType);
oDataResponse = responseBuilder.build(uriParserResultView, createdJpaEntity, contentType);
} catch (JsonIOException | JsonSyntaxException e) {
throw new RuntimeException(e);
} finally {
close();
}
return oDataResponse;
}
}
In Summery
Register your custom org.apache.olingo.odata2.service.factory Code Link
Create your own CustomODataJPAProcessor Code Link
Override createCustomODataProcessor in JPAServiceFactory to use the custom processor Code Link

How to get the type of values from a JDBC query?

I have a spring batch job that takes in a user query, executes that query to find the selected items, and then I want to insert those items into another database. The problem is that I have to convert elements like dates from the resulting query to insert them again. How can I tell the type of the values returned from the query?
This is what I use to read the items which works properly.
#Bean("querySelectiveItems")
#StepScope
public JdbcCursorItemReader querySelectiveItems(#Qualifier("selectiveSourceDatabase") DataSource dataSource,
#Value("#{jobExecutionContext[" + EtlConfiguration.JOB_PARM_MIGRATION_CONFIG + "]}") MigrationDefinition migrationDefinition
) {
JdbcCursorItemReader reader = new JdbcCursorItemReader<>();
reader.setSql(migrationDefinition.getMigrations().getTable().getQuery());
reader.setDataSource(dataSource);
reader.setRowMapper(new ColumnMapRowMapper());
log.info("Queried for items");
return reader;
}
The following is what I wrote to write to the destination database. The problem is that the values I have to insert are unknown because they are the result of a user query. For example if there is a datatype in my insert statement I must have a TO_DATE around the date value. Is there a way to do this?
#Component
#Lazy
class InsertSelectedItems implements ItemWriter<Map<String, Object>> {
private MigrationDefinition migrationDefinition;
private JdbcTemplate destinationTemplate;
public void setDestinationTemplate(JdbcTemplate destinationTemplate) {
this.destinationTemplate = destinationTemplate;
}
public void setMigrationDefinition(MigrationDefinition migrationDefinition) {
this.migrationDefinition = migrationDefinition;
}
#Override
public void write(List<? extends Map<String, Object>> items) throws Exception {
ArrayList<String> columns = new ArrayList<>();
ArrayList<String> values = new ArrayList<>();
System.out.println(items);
for (Map<String, Object> map : items) {
for (Map.Entry<String, Object> entry : map.entrySet()) {
columns.add(entry.getKey());
values.add(String.valueOf(entry.getValue()));
}
}
String sql = String.format("%s ( %s ) VALUES ( %s ) ",
migrationDefinition.getMigrations().getTable().getInsert(),
String.join(",", columns),
String.join(",", values));
log.info(sql);
destinationTemplate.update(sql);
}
}

Read data from an Android Room database in a background Service, no exceptions but no data

I am attempting to read data from an Android Room database in a background Service. There are no exceptions but no data is returned.
I wrote a function to select all rows from a table in the DAO. Calling that function from a background service succeeds, but it returns no data.
My "Contact" class holds contact information (names, phone numbers, emails) and defines the database schema. The database holds rows of contacts, with names, phone numbers, an emails as columns.
The function that returns the LiveData in the DAO is:
#Query("SELECT * FROM contacts_table")
LiveData<List<Contact>> getAll();
where "contacts_table" is the database table holding contact information.
I called getAll as follows:
AppDatabase db = AppDatabase.getDatabase(messageSenderContext.getApplicationContext());
mContactDAO = db.contactDAO();
mAllContacts = mContactDAO.getAll();
where mContactDao is a ContactDAO (The Database Access Object for my Contact class), and mAllContacts is a LiveData>. These are private fields of the class calling getAll().
db.contactDAO() returns an object, as does mContactDAO.getAll(). But attempting to unpack the List from mAllContacts using mAllContacts.getValue() returns null.
This turned out to be a misuse of LiveData. That requires an Observer to actually get the data.
In your ROOM
#Database(entities={Contact.class}, version = 1, exportSchema = false)
public abstract class AppDatabase extends RoomDatabase {
public static final String DATABASE_NAME = "AppDatabase.db";
private static volatile AppDatabase INSTANCE;
private static final int NUMBER_OF_THREADS = 4;
public static final ExecutorService EXECUTOR_SERVICE = Executors.newFixedThreadPool(NUMBER_OF_THREADS);
public abstract ContactDAO contactDAO();
public static AppDatabase getDatabase(final Context context) {
if (INSTANCE == null) {
synchronized (AppDatabase.class) {
if (INSTANCE == null) {
INSTANCE = Room.databaseBuilder(context.getApplicationContext(),
AppDatabase.class, DATABASE_NAME)
.build();
}
}
}
return INSTANCE;
}
}
In your DAO
#Dao
public interface ContactDAO{
#Query("SELECT * FROM contacts_table")
LiveData<List<Contact>> getAll();
}
In your repository:
public class AppRepository {
private ContactDAO mContactDAO;
//constructor
public AppRepository(Application application) {
AppDatabase db = AppDatabase.getDatabase(application);
mContactDAO= db.contactDAO();
}
public LiveData<List<Contact>> getAllContacts(){
LiveData<List<Contact>> contactsList = null;
Future<LiveData<List<Contact>>> futureList = AppDatabase.EXECUTOR_SERVICE.submit(new Callable(){
#Override
public LiveData<List<Contact>> call() {
return contactDAO.getAll();
}
});
try {
contactsList = futureList.get();
} catch (ExecutionException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
return contactsList ;
}
}
In your ViewModel
public class ContactsViewModel extends AndroidViewModel {
private AppRepository appRepository;
private LiveData<List<Contact>> contactsList;
public ContactsViewModel(#NonNull Application application) {
super(application);
appRepository = new AppRepository(application);
}
public LiveData<List<Contacts>> list() {
return appRepository.getAllContacts();
}
}
In your activity (inside of onCreated put)
ContactsViewModel contactsViewModel = new ViewModelProvider(this).get(ContactsViewModel.class);
contactsViewModel.list().observe(getViewLifecycleOwner(), new Observer<List<Contact>>() {
#Override
public void onChanged(List<Contact> contactsList) {
//the contact list will be observed and will return data if there are changes.
//use for example to feed the adapter of a recyclerview
//below an example just to view the contacts data
for(Contact conctact : contactsList){
Log.d("TestApp>>>", "Id: + contact.getId);
Log.d("TestApp>>>", "Name: + contact.getName);
}
});

Spring Integration DSL - JdbcPollingChannelAdapter results not queueing

I swear I had this working, but when I can back to it after a few months (and an upgrade to Boot 1.5.9), I am having issues.
I set up a JdbcPollingChannelAdapter that I can do a receive() on just fine, but when I put the adapter in a flow that does nothing more than queue the result of the adapter, running .receive on the queue always returns a null (I can see in the console log that the adapter's SQL getting executed, though).
Tests below. Why can I get results from the adapter, but not queue the results? Thank you in advance for any assistance.
#RunWith(SpringRunner.class)
#SpringBootTest
#AutoConfigureTestDatabase
#JdbcTest
public class JdbcpollingchanneladapterdemoTests {
#Autowired
#Qualifier("dataSource")
DataSource dataSource;
private static PollableChannel outputQueue;
#BeforeClass
public static void setupClass() {
outputQueue = MessageChannels.queue().get();
return;
}
#Test
public void Should_HaveQueue() {
assertThat(outputQueue, instanceOf(QueueChannel.class));
}
#Test
#Sql(executionPhase = ExecutionPhase.BEFORE_TEST_METHOD,
statements = "Create Table DEMO (CODE VARCHAR(5));")
#Sql(executionPhase = ExecutionPhase.AFTER_TEST_METHOD,
statements = "Drop Table DEMO ;")
public void Should_Not_HaveMessageOnTheQueue_When_No_DemosAreInTheDatabase() {
Message<?> message = outputQueue.receive(5000);
assertThat(message, nullValue()) ;
}
#Test
#Sql(executionPhase = ExecutionPhase.BEFORE_TEST_METHOD,
statements = "Create Table DEMO (CODE VARCHAR(5));")
#Sql(executionPhase = ExecutionPhase.BEFORE_TEST_METHOD,
statements = "Insert into DEMO (CODE) VALUES ('12345');")
#Sql(executionPhase = ExecutionPhase.AFTER_TEST_METHOD,
statements = "Drop Table DEMO ;")
public void Should_HaveMessageOnTheQueue_When_DemosIsInTheDatabase() {
assertThat(outputQueue, instanceOf(QueueChannel.class));
Message<?> message = outputQueue.receive(5000);
assertThat(message, notNullValue());
assertThat(message.getPayload().toString(), equalTo("15317")) ;
}
#Test
#Sql(executionPhase = ExecutionPhase.BEFORE_TEST_METHOD,
statements = "Create Table DEMO (CODE VARCHAR(5));")
#Sql(executionPhase = ExecutionPhase.BEFORE_TEST_METHOD,
statements = "Insert into DEMO (CODE) VALUES ('12345');")
#Sql(executionPhase = ExecutionPhase.AFTER_TEST_METHOD,
statements = "Drop Table DEMO ;")
public void get_message_directly_from_adapter() {
JdbcPollingChannelAdapter adapter =
new JdbcPollingChannelAdapter(dataSource, "SELECT CODE FROM DEMO");
adapter.setRowMapper(new DemoRowMapper());
adapter.setMaxRowsPerPoll(1);
Message<?> message = adapter.receive();
assertThat(message, notNullValue());
}
private static class Demo {
private String demo;
String getDemo() {
return demo;
}
void setDemo(String value) {
this.demo = value;
}
#Override
public String toString() {
return "Demo [value=" + this.demo + "]";
}
}
public static class DemoRowMapper implements RowMapper<Demo> {
#Override
public Demo mapRow(ResultSet rs, int rowNum) throws SQLException {
Demo demo = new Demo();
demo.setDemo(rs.getString("CODE"));
return demo;
}
}
#Component
public static class MyFlowAdapter extends IntegrationFlowAdapter {
#Autowired
#Qualifier("dataSource")
DataSource dataSource;
#Override
protected IntegrationFlowDefinition<?> buildFlow() {
JdbcPollingChannelAdapter adapter =
new JdbcPollingChannelAdapter(dataSource, "SELECT CODE FROM DEMO");
adapter.setRowMapper(new DemoRowMapper());
adapter.setMaxRowsPerPoll(1);
return from(adapter,
c -> c.poller(Pollers.fixedRate(1000L, 2000L)
.maxMessagesPerPoll(1)
.get()))
.channel(outputQueue);
}
}
}
EDIT I've simplified it as much as I can, refactoring to code below. The test passes a flow with a generic message source, and fails on a flow with JdbcPollingChannelAdapter message source. It's just not evident to me how I should configure the second message source so that it will suceed like the first message source.
#Test
#Sql(executionPhase = ExecutionPhase.BEFORE_TEST_METHOD,
statements = "Create Table DEMO (CODE VARCHAR(5));")
#Sql(executionPhase = ExecutionPhase.BEFORE_TEST_METHOD,
statements = "Insert into DEMO (CODE) VALUES ('12345');")
public void Should_HaveMessageOnTheQueue_When_UnsentDemosIsInTheDatabase() {
this.genericFlowContext.registration(new GenericFlowAdapter()).register();
PollableChannel genericChannel = this.beanFactory.getBean("GenericFlowAdapterOutput",
PollableChannel.class);
this.jdbcPollingFlowContext.registration(new JdbcPollingFlowAdapter()).register();
PollableChannel jdbcPollingChannel = this.beanFactory.getBean("JdbcPollingFlowAdapterOutput",
PollableChannel.class);
assertThat(genericChannel.receive(5000).getPayload(), equalTo("15317"));
assertThat(jdbcPollingChannel.receive(5000).getPayload(), equalTo("15317"));
}
private static class GenericFlowAdapter extends IntegrationFlowAdapter {
#Override
protected IntegrationFlowDefinition<?> buildFlow() {
return from(getObjectMessageSource(),
e -> e.poller(Pollers.fixedRate(100)))
.channel(c -> c.queue("GenericFlowAdapterOutput"));
}
private MessageSource<Object> getObjectMessageSource() {
return () -> new GenericMessage<>("15317");
}
}
private static class JdbcPollingFlowAdapter extends IntegrationFlowAdapter {
#Autowired
#Qualifier("dataSource")
DataSource dataSource;
#Override
protected IntegrationFlowDefinition<?> buildFlow() {
return from(getObjectMessageSource(),
e -> e.poller(Pollers.fixedRate(100)))
.channel(c -> c.queue("JdbcPollingFlowAdapterOutput"));
}
private MessageSource<Object> getObjectMessageSource() {
JdbcPollingChannelAdapter adapter =
new JdbcPollingChannelAdapter(dataSource, "SELECT CODE FROM DEMO");
adapter.setRowMapper(new DemoRowMapper());
adapter.setMaxRowsPerPoll(1);
return adapter;
}
}
Looks like you need to add #EnableIntegration to your test configuration.
When you use Spring Boot slices for testing, not all auto-configurations are loaded:
https://docs.spring.io/spring-boot/docs/1.5.9.RELEASE/reference/htmlsingle/#test-auto-configuration
UPDATE
The problem that JdbcPollingChannelAdapter is run in the separate, scheduled thread, already out of the original transaction around test method, where those #Sqls are performed.
The fix for you is like this:
#Sql(executionPhase = ExecutionPhase.BEFORE_TEST_METHOD,
statements = "Insert into DEMO (CODE) VALUES ('12345');",
config = #SqlConfig(transactionMode = SqlConfig.TransactionMode.ISOLATED))
Pay attention to that SqlConfig.TransactionMode.ISOLATED. This way the INSERT transaction is committed and the data is available for that separate polling thread for the JdbcPollingChannelAdapter.
Also pay attention that this JdbcPollingChannelAdapter always returns a List of records. So, your assertThat(jdbcPollingChannel.receive(5000).getPayload(), ...); should be against a List<String> even if there is only one record in the table.

RowMapper returns the list , but execute returned values returns the list size as 1?

please find below my sample code.The Row mapper returns a list. When printed it give me the size in the DB but when i check
(List) employeeDaomap .get("allEmployees") i get the list size as 1 , and entire rows as one item? why what is the wrong in implementation
Also Spring doc says not to use rs.next(), how do we get the list of
values from the DB
public class MyTestDAO extends StoredProcedure {
/** The log. */
static Logger log = Logger.getLogger(MyTestDAO.class);
private static final String SPROC_NAME = "TestSchema.PKG_Test.prc_get_employee_list";
TestRowMapper mapper=new TestRowMapper();
public MyTestDAO(DataSource dataSource){
super(dataSource, SPROC_NAME);
declareParameter(new SqlOutParameter("allEmployees", OracleTypes.CURSOR, mapper));
compile();
}
/**
* Gets the myemplist data from the DB
*
*/
public List<EmployeeDAO> getEmployeeList()
throws Exception {
Map<String,Object> employeeDaomap =new HashMap<String,Object>();
employeeDaomap =execute();
log.info("employeeDaomap after execute ="+employeeDaomap);
log.info("employeeDaomap after execute size ="+employeeDaomap.size()); // expected 1
List<EmployeeDAO> list = (List<EmployeeDAO>) employeeDaomap .get("allEmployees");
log.info("size of the list ="+list.size()); // need to get the size of the list ,
return list;
}
private Map<String, Object> execute() {
return super.execute(new HashMap<String, Object>());
}
}
public class TestRowMapper implements RowMapper<List<EmployeeDAO>> {
static Logger log = Logger.getLogger(TestRowMapper.class);
#Override
public List<EmployeeDAO> mapRow(ResultSet rs, int rowNum)
throws SQLException {
// TODO Auto-generated method stub
rs.setFetchSize(3000);
List<EmployeeDAO> responseItems = new ArrayList<EmployeeDAO>();
EmployeeDAO responseItem = null;
log.info("row num "+rowNum);
while (rs.next()) {
responseItem = new EmployeeDAO();
responseItem.setID(rs.getString("id"));
responseItem.setName(rs.getString("name"));
responseItem.setDesc(rs.getString("desc"));
responseItems.add(responseItem);
}
log.info("TestRowMapper items ="+responseItems);
return responseItems;
}
}
The solution is to use the implements ResultSetExtractor instead of RowMapper and provide implementation for extractData.
public class TestRowMapper implements ResultSetExtractor<List<EmployeeDAO>> {
static Logger log = Logger.getLogger(TestRowMapper.class);
#Override
public List<EMAccountResponse> extractData(ResultSet rs)
throws SQLException, DataAccessException {
rs.setFetchSize(3000);
List<EmployeeDAO> responseItems = new ArrayList<EmployeeDAO>();
EmployeeDAO responseItem = null;
log.info("row num "+rowNum);
while (rs.next()) {
responseItem = new EmployeeDAO();
responseItem.setID(rs.getString("id"));
responseItem.setName(rs.getString("name"));
responseItem.setDesc(rs.getString("desc"));
responseItems.add(responseItem);
}
log.info("TestRowMapper items ="+responseItems);
return responseItems;
}
}

Resources