I have a problem with the Apache Flink Streaming API.
I could manage to set up the whole CEP-Environment with a custom DataSource and when using a standard sink on that source like "print()", everything works fine.
This is what my sink looks like now:
#RequiredArgsConstructor
public class EventDataConsumer extends RichSinkFunction<EventData>{
private final transient Consumer<EventData> consumer;
#Override
public void invoke(EventData eventData) throws Exception {
consumer.accept(eventData);
}
}
What I try to achieve ist, to pass a method reference to this SinkFunction, which shall be executed for each element in my DataStream.
This is how I initialize the SinkFunction:
EventDataConsumer consumer = new EventDataConsumer(someService::handleEventData);
outStream.addSink(consumer);
My Problem is, that when I set a breakpoint in the "invoke" method of my custom sink, the consumer appears to be null even though I call the constructor explicitly, which assigns the consumer.
As the Sink is distributed to as many instances as the parallelism of the sink it should be serializable. When executing on cluster the Sink is serialised sent to TaskManagers where it is deserialised.
In your example the consumer field is transient, that is why after serialisation it becomes null.
Related
I'm able to make Spring+Rabbit work with the non-functional way (prior to 2.0?), but I'm trying to use with the functional pattern as the previous one is deprecated.
I've been following this doc: https://docs.spring.io/spring-cloud-stream/docs/3.1.0/reference/html/spring-cloud-stream.html#_binding_and_binding_names
The queue (consumer) is not being created in Rabbit with the new method. I can see the connection being created but without any consumer.
I have the following in my application.properties:
spring.cloud.stream.function.bindings.approved-in-0=approved
spring.cloud.stream.bindings.approved.destination=myTopic.exchange
spring.cloud.stream.bindings.approved.group=myGroup.approved
spring.cloud.stream.bindings.approved.consumer.back-off-initial-interval=2000
spring.cloud.stream.rabbit.bindings.approved.consumer.queueNameGroupOnly=true
spring.cloud.stream.rabbit.bindings.approved.consumer.bindingRoutingKey=myRoutingKey
which is replacing:
spring.cloud.stream.bindings.approved.destination=myTopic.exchange
spring.cloud.stream.bindings.approved.group=myGroup.approved
spring.cloud.stream.bindings.approved.consumer.back-off-initial-interval=2000
spring.cloud.stream.rabbit.bindings.approved.consumer.queueNameGroupOnly=true
spring.cloud.stream.rabbit.bindings.approved.consumer.bindingRoutingKey=myRoutingKey
And the new class
#Slf4j
#Service
public class ApprovedReceiver {
#Bean
public Consumer<String> approved() {
// I also saw that it's recommended to not use Consumer, but use Function instead
// https://docs.spring.io/spring-cloud-stream/docs/3.1.0/reference/html/spring-cloud-stream.html#_consumer_reactive
return value -> log.info("value: {}", value);
}
}
which is replacing
// BindableApprovedChannel.class
#Configuration
public interface BindableApprovedChannel {
#Input("approved")
SubscribableChannel getApproved();
}
// ApprovedReceiver.class
#Service
#EnableBinding(BindableApprovedChannel.class)
public class ApprovedReceiver {
#StreamListener("approved")
public void handleMessage(String payload) {
log.info("value: {}", payload);
}
}
Thanks!
If you have multiple beans of type Function, Supplier or Consumer (which could be declared by third party libraries), the framework does not know which one to bind to.
Try setting the spring.cloud.function.definition property to approved.
https://docs.spring.io/spring-cloud-stream/docs/3.1.3/reference/html/spring-cloud-stream.html#spring_cloud_function
In the event you only have single bean of type java.util.function.[Supplier/Function/Consumer], you can skip the spring.cloud.function.definition property, since such functional bean will be auto-discovered. However, it is considered best practice to use such property to avoid any confusion. Some time this auto-discovery can get in the way, since single bean of type java.util.function.[Supplier/Function/Consumer] could be there for purposes other then handling messages, yet being single it is auto-discovered and auto-bound. For these rare scenarios you can disable auto-discovery by providing spring.cloud.stream.function.autodetect property with value set to false.
Gary's answer is correct. If adding the definition property alone doesn't resolve the issue I would recommend sharing what you're doing for your supplier.
This is also a very helpful general discussion for transitioning from imperative to functional with links to repos with more in depth examples: EnableBinding is deprecated in Spring Cloud Stream 3.x
I would like to propagate JTA state (= the transaction) between a transactional REST endpoint that emits a message to a reactive-messaging connector.
#Inject
#Channel("test")
Emitter<String> emitter;
#POST
#Transactional
public Response test() {
emitter.send("test");
}
and
#ApplicationScoped
#Connector("test")
public class TestConnector implements OutgoingConnectorFactory {
#Inject
TransactionManager tm;
#Override
public SubscriberBuilder<? extends Message<?>, Void> getSubscriberBuilder(Config config) {
return ReactiveStreams.<Message<?>>builder()
.flatMapCompletionStage(message -> {
tm.getTransaction(); // = null
return message.ack();
})
.ignore();
}
}
As I understand, context-propagation is responsible for making the transaction available (see io.smallrye.context.jta.context.propagation.JtaContextProvider#currentContext). The problem seems to be, that currentContext gets created on subscription, which happens when the injection point (Emitter<String> emitter) get its instance. Which is too early to properly capture the transaction.
What am I missing?
By the way, I am having the same problem when using #Incoming / #Outgoing instead of the emitter. I have decided to give you this example because it is easy to understand and reproduce.
At the moment, you need to pass the current Transaction in the message metadata. Thus, it will be propagated to your different downstream components (as well as the connector).
Note that, Transaction tends to be attached to the request scope, which means that in your connector, it may already be too late to use it. So, make sure your endpoint is asynchronous and only returns when the emitted message is acknowledged.
Context Propagation is not going to help in this case as the underlying streams are built at startup time (at build time in Quarkus) so, there are no capture contexts.
I have to create a batch job where I need to fetch data from 1 DB and after processing dump that data to another DB where auto generated ID would be assigned to persisted data. I need to send that data along with generated ID to solace queue.
Reader(DB1) --data1--> Processor --data2--> Writer (DB2) --data3--> Writer (Solace Publisher)
I am using spring boot-2.2.5.RELEASE and spring-boot-starter-batch.
I have created a job having 1 step that read data from DB1 and write data to DB2 via RepositoryItemReader and RepositoryItemWriter respectively. This is working fine.
Now next task is to send persisted data having generated ID to solace stream (using spring-cloud-starter-stream-solace).
I have below questions. Please assist as I am totally new to spring batch
How can I get the complete record after it's saved to DB2 based on some parameter? Do I have to write my own RepositoryItemWriter having StepExecution Context or can I somehow use the existing RepositoryItemWriter.
Once I got the record I need to use solace stream and there I have publish method which expects argument(record) to be published. I think again I need to write my own Item Writer and either I could use the record passed from above repositoryItemWriter by StepExecutionContext or should I query into DB2 directly from here based on some parameter ?
Either of the above case I need to use stepexecution context but can I use available RepositoryItemWriter or do I have to write my own?
Is there any other concept which is handy in this handy instead of using above approaches?
Passing data to future steps is a common pattern in Spring Batch. According to the documentation https://docs.spring.io/spring-batch/docs/current/reference/html/common-patterns.html#passingDataToFutureSteps you can use stepExecution to store and retrieve your generated IDs. In your case the writers are also listeners which has before step methods annotated with #BeforeStep. For example:
public class DB2ItemWriter implements ItemWriter<Object> {
private StepExecution stepExecution;
public void write(List<? extends Object> items) throws Exception {
// ...
ExecutionContext stepContext = this.stepExecution.getExecutionContext();
stepContext.put("generatedIds", ids);
}
#BeforeStep
public void saveStepExecution(StepExecution stepExecution) {
this.stepExecution = stepExecution;
}
}
and then you retrieve the ids in the next writer
public class SolacePublisherItemWriter implements ItemWriter<Object> {
public void write(List<? extends Object> items) throws Exception {
// ...
}
#BeforeStep
public void retrieveGeneratedIds(StepExecution stepExecution) {
ExecutionContext stepExecutionContext = stepExecution.getExecutionContext();
this.generatedIds = stepExecutionContext.get("generatedIds");
}
}
I have created a job having 1 step that read data from DB1 and write data to DB2 via RepositoryItemReader and RepositoryItemWriter respectively. This is working fine.
I would add a second step that reads data from the table (in which records have been persisted by step 1 and have their IDs generated) and push it to solace using a custom writer.
Spring Boot 1.5
Quartz 2.2
I dynamically create and schedule Quartz jobs during runtime with a Quartz-configured as a jdbc-job-store. These jobs need to be persistent between app executions.
During the job execution, I need access to the full Spring context (Spring-managed beans and JPA transactions).
However, if I try to Autowire anything into my job, then I get an error like..
"Unsatisfied dependency expressed through field myAutowiredField"
I can't figure this out. I have found tons of examples of people showing how to get autowiring to work in a Quartz job, but almost all of these just have a static, hard-coded job. Not a job dynamically created at runtime.
The example at the following URL comes the closest. It dynamically creates jobs and autowiring works great in them. However, it's a ram job store. As soon as I switch to jdbc, I'm back to square one.
https://icecarev.com/2016/11/05/spring-boot-1-4-and-quartz-scheduling-runtime-created-job-instances-from-a-configuration-file/
I have also looked at these..
Spring + Hibernate + Quartz: Dynamic Job
inject bean reference into a Quartz job in Spring?
.... etc.
But again, their solutions all seem to be missing something. For example, they rely on static jobs, triggers, or just plain don't work, etc.
Anybody have any tips or links to up-to-date resources?
Thanks!
Edit 1
Something happens to the spring context when the job is fired. Here's some code to illustrate.
In the first autowireBean() call (this is done during the Spring Boot configuration), it doesn't throw an error. NOTE: At this point, there's no use for this, I'm just showing that it does 'work' here.
In the second autowireBean() call (this is when the job is fired), it fails. This is the 'real' call.
public class AutowiringSpringBeanJobFactory extends SpringBeanJobFactory
{
private transient AutowireCapableBeanFactory beanFactory;
public AutowiringSpringBeanJobFactory(ApplicationContext context)
{
super();
this.beanFactory = context.getAutowireCapableBeanFactory();
MyJobClass job = new MyJobClass();
beanFactory.autowireBean(new MyJobClass()); /** no problem **/
beanFactory.initializeBean(job, "job");
}
#Override
protected Object createJobInstance(final TriggerFiredBundle bundle) throws Exception
{
final Object job = super.createJobInstance(bundle); /* job is an instance MyJobClass.. same as above */
beanFactory.autowireBean(job); /** "Unsatisfied dependency" exception **/
beanFactory.initializeBean(job, "job");
return job;
}
}
Edit 2
Well, I seem to have got it working, however, I don't know if there will be consequences.
Here's the culprit in org.springframework.scheduling.quartz.AdaptableJobFactory
protected Object createJobInstance(TriggerFiredBundle bundle) throws Exception {
return bundle.getJobDetail().getJobClass().newInstance();
}
It seems that bundle.getJobDetail().getJobClass() returns some 'proxy' type of my job class. But basically it 'says' it's returning my correct Job class, but it's different. It has the same name, but I can't cast it. For example...
Object job = bundle.getJobDetail().getJobClass().newInstance();
MyJobClass myJob = (MyJobClass) job;
Throws an error saying that I can't cast com.company.MyJobClass to com.company.MyJobClass.
So here's my 'fix'...
#Override
protected Object createJobInstance(final TriggerFiredBundle bundle) throws Exception
{
String className = bundle.getJobDetail().getJobClass().getName();
Object job = Class.forName(className).newInstance();
beanFactory.autowireBean(job);
job = beanFactory.applyBeanPostProcessorsAfterInitialization(job, "job"); // Without this, #Transactional in my job bean doesn't work
beanFactory.initializeBean(job, "job");
return job;
}
Since I'm not calling super() anymore, I lose a handy feature of SpringBeanJobFactory, but I'm OK with that for the moment.
I guess bottom line is that autowireBean() needs to be called before this sucker gets wrapped in a transactional proxy.
When you mention "Not a job dynamically created at runtime.", I'm going to assume that at some point you do something similar to: MyJob job = new MyJob();
If you want to inject dependencies in MyJob using Spring's #Autowire, one approach that comes to my mind is:
Annotate MyJob class with: #Configurable(dependencyCheck = true)
Run the Java process using a Java agent: java -javaagent:<path to spring-agent-${spring.version}.jar> ...
We need to pass an object when creating a topology so that the bolt can access that and do some further processing based on that object. Is it possible to pass the object via TopplogyContext and if yes, how? Or are there any other ways to pass an object when submitting a topology, before submitting so that bolt can have a handle/control on it?
We need to pass the object via a context so that all bolts can access it and there is no need to force an implementation of constructor in all the bolts for that topology. So, Wanted to know if any API exists to do the same?
You can pass the object in the storm configuration map, provided that it is serializable.
In the prepare() method of any spout or bolt in the topology you can retrieve this object.
This is how you put your object in the configuration map on topology submission:
Config conf = new Config();
MyObject myPreciousObject = new MyObject("precious");
conf.put("my.object",myPreciousObject);
StormSubmitter.submitTopology(args[0], conf, builder.createTopology());
This is how you retrieve it in the prepare() method of a bolt or spout:
prepare(Map stormConf,TopologyContext context) {
MyObject myPreciousObject = (MyObject) stormConf.get("my.object");
}
I am not very sure what you mean, but your bolt class can always take a parameter on initialization, and you can initialize that with the object you want to pass when creating the topology.
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout( "spout", new mySpout() );
builder.setBolt( "bolt", new myBolt1(myObj) ).shuffleGrouping("spout");
And your bolt constructor could take this object as the argument.
Edit:
If you want the data to be accessible without passing it explicitly to a constructor, you can again always make a static class to store this data and access it from the bolt objects