Spring Boot Gemfire Server Configuration - spring-boot

I am trying to understand how to host a Spring Boot Gemfire server process.
I found this example Spring Gemfire Server
The problem I am having is the the server I am trying to add to the cluster is not showing up in the cluster after I start the process.
Here are the steps I am taking:
Start a new locator locally (default port): gfsh>start locator --name=loc-one
I want to add this SpringBootGemfireServer to the cluster:
note I have commented out the embeded locator start-up - I want to add this to the existing locator already running
#SpringBootApplication
#SuppressWarnings("unused")
public class SpringGemFireServerApplication {
private static final boolean DEFAULT_AUTO_STARTUP = true;
public static void main(String[] args) {
SpringApplication.run(SpringGemFireServerApplication.class, args);
}
#Bean
static PropertyPlaceholderConfigurer propertyPlaceholderConfigurer() {
return new PropertyPlaceholderConfigurer();
}
private String applicationName() {
return SpringGemFireServerApplication.class.getSimpleName();
}
#Bean
Properties gemfireProperties(
#Value("${gemfire.log.level:config}") String logLevel,
#Value("${gemfire.locator.host-port:localhost[10334]}") String locatorHostPort,
#Value("${gemfire.manager.port:1099}") String managerPort) {
Properties gemfireProperties = new Properties();
gemfireProperties.setProperty("name", applicationName());
gemfireProperties.setProperty("log-level", logLevel);
//gemfireProperties.setProperty("start-locator", locatorHostPort);
//gemfireProperties.setProperty("jmx-manager", "true");
//gemfireProperties.setProperty("jmx-manager-port", managerPort);
//gemfireProperties.setProperty("jmx-manager-start", "true");
return gemfireProperties;
}
#Bean
CacheFactoryBean gemfireCache(#Qualifier("gemfireProperties") Properties gemfireProperties) {
CacheFactoryBean gemfireCache = new CacheFactoryBean();
gemfireCache.setClose(true);
gemfireCache.setProperties(gemfireProperties);
return gemfireCache;
}
#Bean
CacheServerFactoryBean gemfireCacheServer(Cache gemfireCache,
#Value("${gemfire.cache.server.bind-address:localhost}") String bindAddress,
#Value("${gemfire.cache.server.hostname-for-clients:localhost}") String hostNameForClients,
#Value("${gemfire.cache.server.port:40404}") int port) {
CacheServerFactoryBean gemfireCacheServer = new CacheServerFactoryBean();
gemfireCacheServer.setCache(gemfireCache);
gemfireCacheServer.setAutoStartup(DEFAULT_AUTO_STARTUP);
gemfireCacheServer.setBindAddress(bindAddress);
gemfireCacheServer.setHostNameForClients(hostNameForClients);
gemfireCacheServer.setPort(port);
return gemfireCacheServer;
}
#Bean
PartitionedRegionFactoryBean<Long, Long> factorialsRegion(Cache gemfireCache,
#Qualifier("factorialsRegionAttributes") RegionAttributes<Long, Long> factorialsRegionAttributes) {
PartitionedRegionFactoryBean<Long, Long> factorialsRegion = new PartitionedRegionFactoryBean<>();
factorialsRegion.setAttributes(factorialsRegionAttributes);
factorialsRegion.setCache(gemfireCache);
factorialsRegion.setClose(false);
factorialsRegion.setName("Factorials");
factorialsRegion.setPersistent(false);
return factorialsRegion;
}
#Bean
#SuppressWarnings("unchecked")
RegionAttributesFactoryBean factorialsRegionAttributes() {
RegionAttributesFactoryBean factorialsRegionAttributes = new RegionAttributesFactoryBean();
factorialsRegionAttributes.setCacheLoader(factorialsCacheLoader());
factorialsRegionAttributes.setKeyConstraint(Long.class);
factorialsRegionAttributes.setValueConstraint(Long.class);
return factorialsRegionAttributes;
}
FactorialsCacheLoader factorialsCacheLoader() {
return new FactorialsCacheLoader();
}
class FactorialsCacheLoader implements CacheLoader<Long, Long> {
// stupid, naive implementation of Factorial!
#Override
public Long load(LoaderHelper<Long, Long> loaderHelper) throws CacheLoaderException {
long number = loaderHelper.getKey();
assert number >= 0 : String.format("Number [%d] must be greater than equal to 0", number);
if (number <= 2L) {
return (number < 2L ? 1L : 2L);
}
long result = number;
while (number-- > 1L) {
result *= number;
}
return result;
}
#Override
public void close() {
}
}
}
When I go to gfsh>connect list members
I only see the locator.

I haven't verified the full configuration to check whether there's something else wrong, but the main issue I see right now is that you seem to be confusing the start-locator property (automatically starts a locator in the current process when the member connects to the distributed system and stops the locator when the member disconnects) with the locators property (the list of locators used by system members, it must be configured consistently for every member of the distributed system). Since you're not correctly setting the locators property when configuring the server, it just can't join the existing distributed system because it doesn't know which locator to connect to.
The default locator port used by GemFire is 10334, so you should change your gemfireProperties method as follows:
#Bean
Properties gemfireProperties(#Value("${gemfire.log.level:config}") String logLevel, #Value("${gemfire.locator.host-port:localhost[10334]}") String locatorHostPort, #Value("${gemfire.manager.port:1099}") String managerPort) {
Properties gemfireProperties = new Properties();
gemfireProperties.setProperty("name", applicationName());
gemfireProperties.setProperty("log-level", logLevel);
// You can directly use the locatorHostPort variable instead.
gemfireProperties.setProperty("locators", "localhost[10334]");
return gemfireProperties;
}
Hope this helps.
Cheers.

Related

How to run Quarkus programatically in Test mode

I am trying to run acceptance tests with concordion fixtures in a quarkus project. Concordion does not work with Junit5 so I am using its original #Run(ConcordionRunner.class).
I am creating a superclass to start my quarkus application before tests like that:
#RunWith(ConcordionRunner.class)
public abstract class AbstractFixture {
public static RunningQuarkusApplication application;
protected static RequestSpecification server;
protected AbstractFixture() {
setUp();
}
public void setUp() {
if(application == null) {
startApplication();
server = new RequestSpecBuilder()
.setPort(8081)
.setContentType(ContentType.JSON)
.build();
}
}
private void startApplication() {
try {
PathsCollection.Builder rootBuilder = PathsCollection.builder();
Path testClassLocation = PathTestHelper.getTestClassesLocation(getClass());
rootBuilder.add(testClassLocation);
final Path appClassLocation = PathTestHelper.getAppClassLocationForTestLocation(
testClassLocation.toString());
rootBuilder.add(appClassLocation);
application = QuarkusBootstrap.builder()
.setIsolateDeployment(false)
.setMode(QuarkusBootstrap.Mode.TEST)
.setProjectRoot(Paths.get("").normalize().toAbsolutePath())
.setApplicationRoot(rootBuilder.build())
.build()
.bootstrap()
.createAugmentor()
.createInitialRuntimeApplication()
.run();
} catch (BindException e) {
e.printStackTrace();
System.out.println("Address already in use - which is fine!");
} catch (Exception e) {
throw new RuntimeException(e);
}
}
}
The code above is working but I can't change the default port 8081 to any other.
If I print the config property in my Test class like below, it prints the port correctly, but quarkus is not running on it:
public class HelloFixture extends AbstractFixture {
public String getGreeting() {
Response response = given(server).when().get("/hello");
System.out.println("Config[port]: " + application.getConfigValue("quarkus.http.port", String.class));
return response.asString();
}
}
How can I specify the configuration file or property programatically before run?
I found the answer. At first, I was referencing the wrong property "quarkus.http.port" instead of "quarkus.http.test-port".
Despite that, I found the way to override properties before run:
...
StartupAction action = QuarkusBootstrap.builder()
.setIsolateDeployment(false)
.setMode(QuarkusBootstrap.Mode.TEST)
.setProjectRoot(Paths.get("").normalize().toAbsolutePath())
.setApplicationRoot(rootBuilder.build())
.build()
.bootstrap()
.createAugmentor()
.createInitialRuntimeApplication();
action.overrideConfig(getConfigOverride());
application = action.run();
...
private Map<String, String> getConfigOverride() {
Map<String, String> config = new HashMap<>();
config.put("quarkus.http.test-port", "18082");
return config;
}

Spring Kafka global transaction ID stays open after program ends

I am creating a Kafka Spring producer under Spring Boot which will send data to Kafka and then write to a database; I want all that work to be in one transaction. I am new to Kafka and no expert on Spring, and am having some difficulty. Any pointers much appreciated.
So far my code writes to Kafka successfully in a loop. I have not yet set up
the DB, but have proceeded to set up global transactioning by adding a transactionIdPrefix to the producerFactory in the configuration:
producerFactory.setTransactionIdPrefix("MY_SERVER");
and added #Transactional to the method that does the Kafka send. Eventually I plan to do my DB work in that same method.
Problem: the code runs great the first time. But if I stop the program, even cleanly, I find that the code hangs the 2nd time I run it as soon as it enters the #Transactional method. If I comment out the #Transactional, it enters the method but hangs on the kafa template send().
The problem seems to be the transaction ID. If I change the prefix and rerun, the program runs fine again the first time but hangs when I run it again, until a new prefix is chosen. Since after a restart the trans ID counter starts at zero, if the trans ID prefix does not change then the same trans ID will be used upon restart.
It seems to me that the original transID is still open on the server, and was never committed. (I can read the data off the topic using the console-consumer, but that will read uncommitted). But if that is the case, how do I get spring to commit the trans? I am thinking my coniguration must be wrong. Or-- is the issue possibly that trans ID's can never be reused? (In which case, how does one solve that?)
Here is my relevant code. Config is:
#SpringBootApplication
public class MYApplication {
#Autowired
private static ChangeSweeper changeSweeper;
#Value("${kafka.bootstrap-servers}")
private String bootstrapServers;
#Bean
public ProducerFactory<String, String> producerFactory() {
Map<String, Object> configProps = new HashMap<>();
configProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
configProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
configProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
DefaultKafkaProducerFactory<String, String> producerFactory=new DefaultKafkaProducerFactory<>(configProps);
producerFactory.setTransactionIdPrefix("MY_SERVER");
return producerFactory;
}
#Bean
public KafkaTransactionManager<String, String> KafkaTransactionManager() {
return new KafkaTransactionManager<String, String>((producerFactory()));
}
#Bean(name="kafkaProducerTemplate")
public KafkaTemplate<String, String> kafkaProducerTemplate() {
return new KafkaTemplate<>(producerFactory());
}
And the method that does the transaction is:
#Transactional
public void send( final List<Record> records) {
logger.debug("sending {} records; batchSize={}; topic={}", records.size(),batchSize, kafkaTopic);
// Divide the record set into batches of size batchSize and send each batch with a kafka transaction:
for (int batchStartIndex = 0; batchStartIndex < records.size(); batchStartIndex += batchSize ) {
int batchEndIndex=Math.min(records.size()-1, batchStartIndex+batchSize-1);
List<Record> nextBatch = records.subList(batchStartIndex, batchEndIndex);
logger.debug("## batch is from " + batchStartIndex + " to " + batchEndIndex);
for (Record record : nextBatch) {
kafkaProducerTemplate.send( kafkaTopic, record.getKey().toString(), record.getData().toString());
logger.debug("Sending> " + record);
}
// I will put the DB writes here
}
This works fine for me no matter how many times I run it (but I have to run 3 broker instances on my local machine because transactions require that by default)...
#SpringBootApplication
#EnableTransactionManagement
public class So47817034Application {
public static void main(String[] args) {
SpringApplication.run(So47817034Application.class, args).close();
}
private final CountDownLatch latch = new CountDownLatch(2);
#Bean
public ApplicationRunner runner(Foo foo) {
return args -> {
foo.send("foo");
foo.send("bar");
this.latch.await(10, TimeUnit.SECONDS);
};
}
#Bean
public KafkaTransactionManager<Object, Object> KafkaTransactionManager(KafkaProperties properties) {
return new KafkaTransactionManager<Object, Object>(kafkaProducerFactory(properties));
}
#Bean
public ProducerFactory<Object, Object> kafkaProducerFactory(KafkaProperties properties) {
DefaultKafkaProducerFactory<Object, Object> factory =
new DefaultKafkaProducerFactory<Object, Object>(properties.buildProducerProperties());
factory.setTransactionIdPrefix("foo-");
return factory;
}
#KafkaListener(id = "foo", topics = "so47817034")
public void listen(String in) {
System.out.println(in);
this.latch.countDown();
}
#Component
public static class Foo {
#Autowired
private KafkaTemplate<Object, Object> template;
#Transactional
public void send(String go) {
this.template.send("so47817034", go);
}
}
}

How to use encrypted store-uri in Spring ImapIdleChannelAdapter

Sample spring configuration is as below.
<int-mail:imap-idle-channel-adapter id="mailAdapter"
store-uri="imaps://${"username"}:${"password"}#imap-server:993/INBOX"
java-mail-properties="javaMailProperties"
channel="emails"
should-delete-messages="false"
should-mark-messages-as-read="true">
</int-mail:imap-idle-channel-adapter>
I wish to keep the password field encrypted in properties file and decrypt it in the code. I am not sure on how to set mailReceiver property of ImapIdleChannelAdapter to my custom version of ImapMailReceiver.
Please let me know if there is any way to do this.
All of my configurations are in XML as described above.
Above solution of adding the defifnation did not work may be I am doing something wrong. Then I tried using XML + Java configuration, as below.
#Configuration
public class EmailConfiguration {
#Bean
public ImapIdleChannelAdapter customAdapter() {
ImapIdleChannelAdapter adapter = new ImapIdleChannelAdapter(mailReceiver());
adapter.setOutputChannel(outputChannel());
adapter.setErrorChannel(errorChannel());
adapter.setAutoStartup(true);
adapter.setShouldReconnectAutomatically(true);
adapter.setTaskScheduler(taskScheduler());
return adapter;
}
#Bean
public TaskImapMailReceiver mailReceiver() {
TaskImapMailReceiver mailReceiver = new TaskImapMailReceiver("imaps://[username]:[password]#imap.googlemail.com:993/inbox");
mailReceiver.setShouldDeleteMessages(false);
mailReceiver.setShouldMarkMessagesAsRead(true);
//mailReceiver.setJavaMailProperties(javaMailProperties());
mailReceiver.setMaxFetchSize(Integer.MAX_VALUE);
return mailReceiver;
}
}
Also created empty errorChannel,outputChannel etc. I observed that Spring creates two instances one with xml config and other with java #Configuration. Where it was expected to use only java configuration. If I remove the xml config tag
then it provides sigle imap instance with my mailReceiver but runs only once does not go periodic. also does not show IMAPS logs.
Just wondering if I need to do so much to encrypt the password. Is somthing wrong with my approach.
Use Java configuration instead of XML...
#Configuration
public class MyConfigClass {
#Bean
public MyMailReceiver receiver() {
...
}
#Bean
public ImapIdleChannelAdapter adapter() {
ImapIdleChannelAdapter adapter = new ImapIdleChannelAdapter(receiver());
...
return adapter;
}
}
If you are using XML for everything else, simply add this class as a <bean/> to your XML.
EDIT
Here's an example that works fine for me...
#SpringBootApplication
public class So42298254Application {
public static void main(String[] args) {
SpringApplication.run(So42298254Application.class, args);
}
#Bean
public TestMailServer.ImapServer imapServer() {
return TestMailServer.imap(0);
}
#Bean
public ImapMailReceiver receiver() {
ImapMailReceiver imapMailReceiver = new ImapMailReceiver(imapUrl("user", "pw"));
imapMailReceiver.setHeaderMapper(new DefaultMailHeaderMapper()); // converts the MimeMessage to a String
imapMailReceiver.setUserFlag("testSIUserFlag"); // needed by the SI test server
Properties javaMailProperties = new Properties();
javaMailProperties.put("mail.debug", "true");
imapMailReceiver.setJavaMailProperties(javaMailProperties);
return imapMailReceiver;
}
private String imapUrl(String user, String pw) {
return "imap://"
+ user + ":" + pw
+ "#localhost:" + imapServer().getPort() + "/INBOX";
}
#Bean
public ImapIdleChannelAdapter adapter() {
ImapIdleChannelAdapter adapter = new ImapIdleChannelAdapter(receiver());
adapter.setOutputChannelName("handleMail");
return adapter;
}
#ServiceActivator(inputChannel = "handleMail")
public void handle(String mail, #Header(MailHeaders.FROM) Object from) {
System.out.println(mail + " from:" + from);
imapServer().resetServer(); // so we'll get the email again
}
}
My intention was to use encrypted passwords in properties files.
So I changed my approach of getting into email receiving classes. I added inherited PropertyPlaceholderConfigurer and implemented method convertPropertyValue() as below.
public class EncryptationAwarePropertyPlaceholderConfigurer extends PropertyPlaceholderConfigurer {
private static final Logger logger = LoggerFactory.getLogger(EncryptationAwarePropertyPlaceholderConfigurer.class);
#Override
protected String convertPropertyValue(String originalValue) {
if (originalValue.contains("{<ENC>}") && originalValue.contains("{</ENC>}")) {
String encryptedTaggedValue = originalValue.substring(originalValue.indexOf("{<ENC>}"), originalValue.indexOf("{</ENC>}") + 8);
String encryptedValue = originalValue.substring(originalValue.indexOf("{<ENC>}") + 7, originalValue.indexOf("{</ENC>}"));
try {
String decryptedValue = EncrypDecriptUtil.decrypt(encryptedValue);//EncrypDecriptUtil is my class for encription and decryption
originalValue = originalValue.replace(encryptedTaggedValue, decryptedValue);
} catch (GeneralSecurityException e) {
logger.error("failed to decrypt property returning original value as in properties file.", e);
}
}
return originalValue;
}
}
And changed properties file to enclose encrypted value in custuom ENC tag
as
mail.imap.task.url=imap://username:{<ENC>}encryptedPassword{</ENC>}#imap.googlemail.com:993/inbox

Read Application Object from GemFire using Spring Data GemFire. Data stored using SpringXD's gemfire-json-server

I'm using the gemfire-json-server module in SpringXD to populate a GemFire grid with json representation of “Order” objects. I understand the gemfire-json-server module saves data in Pdx form in GemFire. I’d like to read the contents of the GemFire grid into an “Order” object in my application. I get a ClassCastException that reads:
java.lang.ClassCastException: com.gemstone.gemfire.pdx.internal.PdxInstanceImpl cannot be cast to org.apache.geode.demo.cc.model.Order
I’m using the Spring Data GemFire libraries to read contents of the cluster. The code snippet to read the contents of the Grid follows:
public interface OrderRepository extends GemfireRepository<Order, String>{
Order findByTransactionId(String transactionId);
}
How can I use Spring Data GemFire to convert data read from the GemFire cluster into an Order object?
Note: The data was initially stored in GemFire using SpringXD's gemfire-json-server-module
Still waiting to hear back from the GemFire PDX engineering team, specifically on Region.get(key), but, interestingly enough if you annotate your application domain object with...
#JsonTypeInfo(use = JsonTypeInfo.Id.CLASS, include = JsonTypeInfo.As.PROPERTY, property = "#type")
public class Order ... {
...
}
This works!
Under-the-hood I knew the GemFire JSONFormatter class (see here) used Jackson's API to un/marshal (de/serialize) JSON data to and from PDX.
However, the orderRepository.findOne(ID) and ordersRegion.get(key) still do not function as I would expect. See updated test class below for more details.
Will report back again when I have more information.
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration(classes = GemFireConfiguration.class)
#SuppressWarnings("unused")
public class JsonToPdxToObjectDataAccessIntegrationTest {
protected static final AtomicLong ID_SEQUENCE = new AtomicLong(0l);
private Order amazon;
private Order bestBuy;
private Order target;
private Order walmart;
#Autowired
private OrderRepository orderRepository;
#Resource(name = "Orders")
private com.gemstone.gemfire.cache.Region<Long, Object> orders;
protected Order createOrder(String name) {
return createOrder(ID_SEQUENCE.incrementAndGet(), name);
}
protected Order createOrder(Long id, String name) {
return new Order(id, name);
}
protected <T> T fromPdx(Object pdxInstance, Class<T> toType) {
try {
if (pdxInstance == null) {
return null;
}
else if (toType.isInstance(pdxInstance)) {
return toType.cast(pdxInstance);
}
else if (pdxInstance instanceof PdxInstance) {
return new ObjectMapper().readValue(JSONFormatter.toJSON(((PdxInstance) pdxInstance)), toType);
}
else {
throw new IllegalArgumentException(String.format("Expected object of type PdxInstance; but was (%1$s)",
pdxInstance.getClass().getName()));
}
}
catch (IOException e) {
throw new RuntimeException(String.format("Failed to convert PDX to object of type (%1$s)", toType), e);
}
}
protected void log(Object value) {
System.out.printf("Object of Type (%1$s) has Value (%2$s)", ObjectUtils.nullSafeClassName(value), value);
}
protected Order put(Order order) {
Object existingOrder = orders.putIfAbsent(order.getTransactionId(), toPdx(order));
return (existingOrder != null ? fromPdx(existingOrder, Order.class) : order);
}
protected PdxInstance toPdx(Object obj) {
try {
return JSONFormatter.fromJSON(new ObjectMapper().writeValueAsString(obj));
}
catch (JsonProcessingException e) {
throw new RuntimeException(String.format("Failed to convert object (%1$s) to JSON", obj), e);
}
}
#Before
public void setup() {
amazon = put(createOrder("Amazon Order"));
bestBuy = put(createOrder("BestBuy Order"));
target = put(createOrder("Target Order"));
walmart = put(createOrder("Wal-Mart Order"));
}
#Test
public void regionGet() {
assertThat((Order) orders.get(amazon.getTransactionId()), is(equalTo(amazon)));
}
#Test
public void repositoryFindOneMethod() {
log(orderRepository.findOne(target.getTransactionId()));
assertThat(orderRepository.findOne(target.getTransactionId()), is(equalTo(target)));
}
#Test
public void repositoryQueryMethod() {
assertThat(orderRepository.findByTransactionId(amazon.getTransactionId()), is(equalTo(amazon)));
assertThat(orderRepository.findByTransactionId(bestBuy.getTransactionId()), is(equalTo(bestBuy)));
assertThat(orderRepository.findByTransactionId(target.getTransactionId()), is(equalTo(target)));
assertThat(orderRepository.findByTransactionId(walmart.getTransactionId()), is(equalTo(walmart)));
}
#Region("Orders")
#JsonTypeInfo(use = JsonTypeInfo.Id.CLASS, include = JsonTypeInfo.As.PROPERTY, property = "#type")
public static class Order implements PdxSerializable {
protected static final OrderPdxSerializer pdxSerializer = new OrderPdxSerializer();
#Id
private Long transactionId;
private String name;
public Order() {
}
public Order(Long transactionId) {
this.transactionId = transactionId;
}
public Order(Long transactionId, String name) {
this.transactionId = transactionId;
this.name = name;
}
public String getName() {
return name;
}
public void setName(final String name) {
this.name = name;
}
public Long getTransactionId() {
return transactionId;
}
public void setTransactionId(final Long transactionId) {
this.transactionId = transactionId;
}
#Override
public void fromData(PdxReader reader) {
Order order = (Order) pdxSerializer.fromData(Order.class, reader);
if (order != null) {
this.transactionId = order.getTransactionId();
this.name = order.getName();
}
}
#Override
public void toData(PdxWriter writer) {
pdxSerializer.toData(this, writer);
}
#Override
public boolean equals(Object obj) {
if (obj == this) {
return true;
}
if (!(obj instanceof Order)) {
return false;
}
Order that = (Order) obj;
return ObjectUtils.nullSafeEquals(this.getTransactionId(), that.getTransactionId());
}
#Override
public int hashCode() {
int hashValue = 17;
hashValue = 37 * hashValue + ObjectUtils.nullSafeHashCode(getTransactionId());
return hashValue;
}
#Override
public String toString() {
return String.format("{ #type = %1$s, id = %2$d, name = %3$s }",
getClass().getName(), getTransactionId(), getName());
}
}
public static class OrderPdxSerializer implements PdxSerializer {
#Override
public Object fromData(Class<?> type, PdxReader in) {
if (Order.class.equals(type)) {
return new Order(in.readLong("transactionId"), in.readString("name"));
}
return null;
}
#Override
public boolean toData(Object obj, PdxWriter out) {
if (obj instanceof Order) {
Order order = (Order) obj;
out.writeLong("transactionId", order.getTransactionId());
out.writeString("name", order.getName());
return true;
}
return false;
}
}
public interface OrderRepository extends GemfireRepository<Order, Long> {
Order findByTransactionId(Long transactionId);
}
#Configuration
protected static class GemFireConfiguration {
#Bean
public Properties gemfireProperties() {
Properties gemfireProperties = new Properties();
gemfireProperties.setProperty("name", JsonToPdxToObjectDataAccessIntegrationTest.class.getSimpleName());
gemfireProperties.setProperty("mcast-port", "0");
gemfireProperties.setProperty("log-level", "warning");
return gemfireProperties;
}
#Bean
public CacheFactoryBean gemfireCache(Properties gemfireProperties) {
CacheFactoryBean cacheFactoryBean = new CacheFactoryBean();
cacheFactoryBean.setProperties(gemfireProperties);
//cacheFactoryBean.setPdxSerializer(new MappingPdxSerializer());
cacheFactoryBean.setPdxSerializer(new OrderPdxSerializer());
cacheFactoryBean.setPdxReadSerialized(false);
return cacheFactoryBean;
}
#Bean(name = "Orders")
public PartitionedRegionFactoryBean ordersRegion(Cache gemfireCache) {
PartitionedRegionFactoryBean regionFactoryBean = new PartitionedRegionFactoryBean();
regionFactoryBean.setCache(gemfireCache);
regionFactoryBean.setName("Orders");
regionFactoryBean.setPersistent(false);
return regionFactoryBean;
}
#Bean
public GemfireRepositoryFactoryBean orderRepository() {
GemfireRepositoryFactoryBean<OrderRepository, Order, Long> repositoryFactoryBean =
new GemfireRepositoryFactoryBean<>();
repositoryFactoryBean.setRepositoryInterface(OrderRepository.class);
return repositoryFactoryBean;
}
}
}
So, as you are aware, GemFire (and by extension, Apache Geode) stores JSON in PDX format (as a PdxInstance). This is so GemFire can interoperate with many different language-based clients (native C++/C#, web-oriented (JavaScript, Pyhton, Ruby, etc) using the Developer REST API, in addition to Java) and also to be able to use OQL to query the JSON data.
After a bit of experimentation, I am surprised GemFire is not behaving as I would expect. I created an example, self-contained test class (i.e. no Spring XD, of course) that simulates your use case... essentially storing JSON data in GemFire as PDX and then attempting to read the data back out as the Order application domain object type using the Repository abstraction, logical enough.
Given the use of the Repository abstraction and implementation from Spring Data GemFire, the infrastructure will attempt to access the application domain object based on the Repository generic type parameter (in this case "Order" from the "OrderRepository" definition).
However, the data is stored in PDX, so now what?
No matter, Spring Data GemFire provides the MappingPdxSerializer class to convert PDX instances back to application domain objects using the same "mapping meta-data" that the Repository infrastructure uses. Cool, so I plug that in...
#Bean
public CacheFactoryBean gemfireCache(Properties gemfireProperties) {
CacheFactoryBean cacheFactoryBean = new CacheFactoryBean();
cacheFactoryBean.setProperties(gemfireProperties);
cacheFactoryBean.setPdxSerializer(new MappingPdxSerializer());
cacheFactoryBean.setPdxReadSerialized(false);
return cacheFactoryBean;
}
You will also notice, I set the PDX 'read-serialized' property (cacheFactoryBean.setPdxReadSerialized(false);) to false in order to ensure data access operations return the domain object and not the PDX instance.
However, this had no affect on the query method. In fact, it had no affect on the following operations either...
orderRepository.findOne(amazonOrder.getTransactionId());
ordersRegion.get(amazonOrder.getTransactionId());
Both calls returned a PdxInstance. Note, the implementation of OrderRepository.findOne(..) is based on SimpleGemfireRepository.findOne(key), which uses GemfireTemplate.get(key), which just performs Region.get(key), and so is effectively the same as (ordersRegion.get(amazonOrder.getTransactionId();). The outcome should not be, especially with Region.get() and read-serialized set to false.
With the OQL query (SELECT * FROM /Orders WHERE transactionId = $1) generated from the findByTransactionId(String id), the Repository infrastructure has a bit less control over what the GemFire query engine will return based on what the caller (OrderRepository) expects (based on the generic type parameter), so running OQL statements could potentially behave differently than direct Region access using get.
Next, I went onto try modifying the Order type to implement PdxSerializable, to handle the conversion during data access operations (direct Region access with get, OQL, or otherwise). This had no affect.
So, I tried to implement a custom PdxSerializer for Order objects. This had no affect either.
The only thing I can conclude at this point is something is getting lost in translation between Order -> JSON -> PDX and then from PDX -> Order. Seemingly, GemFire needs additional type meta-data required by PDX (something like #JsonTypeInfo(use = JsonTypeInfo.Id.CLASS, include = JsonTypeInfo.As.PROPERTY, property = "#type") in the JSON data that PDXFormatter recognizes, though I am not certain it does.
Note, in my test class, I used Jackson's ObjectMapper to serialize the Order to JSON and then GemFire's JSONFormatter to serialize the JSON to PDX, which I suspect Spring XD is doing similarly under-the-hood. In fact, Spring XD uses Spring Data GemFire and is most likely using the JSON Region Auto Proxy support. That is exactly what SDG's JSONRegionAdvice object does (see here).
Anyway, I have an inquiry out to the rest of the GemFire engineering team. There are also things that could be done in Spring Data GemFire to ensure the PDX data is converted, such as making use of the MappingPdxSerializer directly to convert the data automatically on behalf of the caller if the data is indeed of type PdxInstance. Similar to how JSON Region Auto Proxying works, you could write AOP interceptor for the Orders Region to automagicaly convert PDX to an Order.
Though, I don't think any of this should be necessary as GemFire should be doing the right thing in this case. Sorry I don't have a better answer right now. Let's see what I find out.
Cheers and stay tuned!
See subsequent post for test code.

Spring, autowire #Value from a database

I am using a properties File to store some configuration properties, that are accessed this way:
#Value("#{configuration.path_file}")
private String pathFile;
Is it possible (with Spring 3) to use the same #Value annotation, but loading the properties from a database instead of a file ?
Assuming you have a table in your database stored key/value pairs:
Define a new bean "applicationProperties" - psuedo-code follows...
public class ApplicationProperties {
#AutoWired
private DataSource datasource;
public getPropertyValue(String key) {
// transact on your datasource here to fetch value for key
// SNIPPED
}
}
Inject this bean where required in your application. If you already have a dao/service layer then you would just make use of that.
Yes, you can keep your #Value annotation, and use the database source with the help of EnvironmentPostProcessor.
As of Spring Boot 1.3, we're able to use the EnvironmentPostProcessor to customize the application's Environment before application context is refreshed.
For example, create a class which implements EnvironmentPostProcessor:
public class ReadDbPropertiesPostProcessor implements EnvironmentPostProcessor {
private static final String PROPERTY_SOURCE_NAME = "databaseProperties";
private String[] CONFIGS = {
"app.version"
// list your properties here
};
#Override
public void postProcessEnvironment(ConfigurableEnvironment environment, SpringApplication application) {
Map<String, Object> propertySource = new HashMap<>();
try {
// the following db connections properties must be defined in application.properties
DataSource ds = DataSourceBuilder
.create()
.username(environment.getProperty("spring.datasource.username"))
.password(environment.getProperty("spring.datasource.password"))
.url(environment.getProperty("spring.datasource.url"))
.driverClassName("com.mysql.jdbc.Driver")
.build();
try (Connection connection = ds.getConnection();
// suppose you have a config table storing the properties name/value pair
PreparedStatement preparedStatement = connection.prepareStatement("SELECT value FROM config WHERE name = ?")) {
for (int i = 0; i < CONFIGS.length; i++) {
String configName = CONFIGS[i];
preparedStatement.setString(1, configName);
ResultSet rs = preparedStatement.executeQuery();
while (rs.next()) {
propertySource.put(configName, rs.getString("value"));
}
// rs.close();
preparedStatement.clearParameters();
}
}
environment.getPropertySources().addFirst(new MapPropertySource(PROPERTY_SOURCE_NAME, propertySource));
} catch (Throwable e) {
throw new RuntimeException(e);
}
}
}
Finally, don't forget to put your spring.factories in META-INF. An example:
org.springframework.boot.autoconfigure.EnableAutoConfiguration=
com.baeldung.environmentpostprocessor.autoconfig.PriceCalculationAutoConfig
Although not having used spring 3, I'd assume you can, if you make a bean that reads the properties from the database and exposes them with getters.

Resources