I have set up a single node Spring XD and running the XD Shell. The hosts running these are not exposed to internet due to which I am not able to use the twittersearch stream which is out of box from Spring XD. It throws java.net.UnknownHostException: api.twitter.com
Now my question is, how do I set my proxy details to Spring XD so it can use them to reach out to Twitter?
Running single node in verbose mode you'll see an number of properties that have been set from environment variables such as HTTP_PROXY. There are also a number that will require Java arguments to be past in such -Dhttp.proxyHost=localhost. Try setting these in DEFAULT_JVM_OPTS= in the xd-singlenode script.
Related
I am creating a spring boot microservice project with intelij IDEA.
Currently I have developed three seperate spring boot rest services as customer service, vehicle service and spring cloud config server. Spring cloud config server is pointing to a github repository.
The issue is sometimes above projects take more than 10 minutes to run and sometimes does't run and give an error message as "failed to check application readystate intellij attached provider for the vm is not found". I have no idea why this happens ?
There are two possible causes:
1. IntelliJ IDEA and the Spring application are running in different JVMs.
There is a bug for IntelliJ IDEA regarding that:
https://youtrack.jetbrains.com/issue/IDEA-210665
Here is short summary:
IntelliJ IDEA uses local JMX connector for retrieving Spring Boot actuator endpoint's data by default. However, it could be impossible to get local JMX connector address via attach api if Spring Boot application and IntelliJ IDEA are run by different JVMs. In this case, add the following lines to VM options of your Spring Boot run configuration:
-Dcom.sun.management.jmxremote.port={some_port}
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
As mentioned in the official Oracle documentation, this configuration is insecure. Any remote user who knows (or guesses) your port number and host name will be able to monitor and control your Java applications and platform.
2. Prolonged time to retrieve local hostname
You can check that time using inetTester. Normally it should take only several milliseconds to complete. If it takes a long time then you can add the hostname returned by inetTester to /etc/hosts file like this:
127.0.0.1 localhost winsky
::1 localhost winsky
I am new to SCDF and am trying to get started with a RabbitMQ transport layer and SCDF version 1.2.2. I have setup RabbitMQ in a separate VM and have the SCDF local server and SCDF shell jar in one VM. Can someone suggest how I can specify the server details of my RabbitMQ (which is in a different host in the same network) for SCDF to use as a transport.
For reasons outside my control I need to use the MQ setup in a different machine. Please advise.
SCDF doesn't require RabbitMQ and I think you are trying to use RabbitMQ as the binder for your Spring Cloud Stream applications that are orchestrated via SCDF.
You would need to configure the properties mentioned here
You can find more information here on how to specify these properties at SCDF.
I can monitor individual SPRING INTEGRATION applications via visualvm changing the command line parameters when starting the JVM (-Dcom.sun.....)
My application has components in multiple jvm's, each of which i can name.
I would like my operational console to connect per server to one JMX service via one port. Then as I add JVM's(services) they are discoverable by the operational console(lets assume its visualvm) by name.
Any help is greatly appreciated
Take a look at Jolokia I believe a single client can connect to multiple agents (you install a jolokia agent on each JVM).
I am working on Spring XD and GemFire XD. I want to understand how Spring XD's distributed environment works. I know spring xd uses either redis or rabittmq as the transport.
I am clear about this, I have install spring xd and rabittmq on one machine. I changed the redis.properties file and added hostnames.
Do I need to install spring xd on all the machines? If so, after installing, how to bring those up.
On the master machine, I will do ./xd-admin and ./xd-container
How do you start up the nodes (spring xd instances/workers) so that they can listen for instructions from xd-admin?
Please help me on this.
Thanks,
-Suyodhan
Redis is used for analytics as only supported platform. For transport, you need either Redis or Rabbit.
Basically you just need to install Redis and RabbitMQ per their respective documentation. They can be in same or different servers, Ideally you would use their high availability option. For example Redis Sentinal. YOu don't need RabbitMQ unless you want to change the default transport from Redis to Rabbit. Once you install Redis and Rabbit, bring them up and provide their host:port info (and any additional as applicable) to the servers.yml in XD install (in all nodes) and bring up admin and containers. Evrything should work automatically by using zookeeper as the means to manage the distributed runtime.
If you use Spring XD in distributed mode, I assume you have set up zookeeper as well. (If not check this http://docs.spring.io/spring-xd/docs/1.0.0.M7/reference/html/#_setting_up_zookeeper )
Admin and Container instances register themselves with Zookeeper as they come up. Admin queries zookeeper for available containers and assign tasks like deploying modules. Zookeeper is the trick behind Distributed mode.
Hope this helps.
You will install Spring xd one time on one machine, Spring XD will be connected to your hdfs distributed scaled out environment.
You need to start the followings:
1. redis or rappitMQ in your case
2. hsqldb server
3. container
4. admin
when you start spring xd, you need to register the name node firstly using the command:
hadoop config fs --name hdfs://serverip:8020
then you can use any module defined in spring xd (using stream or batch) by specifying its parameters directly without specifying those in the server.yml file.
Moha.
I am new to spring batch. I want to run spring batch jobs on server a and want to launch those jobs from server b using spring batch admin.is it possible? I have searched the following two ways:
1.JMX way: i could convert spring batch beans into mbeans but i cant read them from spring batch admin.can you tell how to read mbeans from spring batch admin and launch them?
2.common repository: i think if i use the same db repository for both spring batch and spring batch admin then i can launch remote jobs from spring batch admin (from server b).but in the job xml file in spring batch admin what should be the classpath for tasklet?
can you help in the above or tell me if any new way exists?
we ended up implementing a framework using mq communication to handle this. each 'batch node' registers itself and any 'batch class' parameters such as 'nodeType=A' or 'jobSizeiCanHandle=BIG' (these are fictitious but you get the point). The client console reads this information and queries the nodes via MQ for the job list. It then submits job requests with parameters via a rudimentary text based protocol (property file format).
command=START_JOB
job=JobABC
param1=x
param2=y
One of the batch nodes will pick up the message and start the job, it will return success/fail status in the same manner with a message with the same correlation id. so the client can show response to the user.
this allows us to do what you're talking about AND spark the jobs via an external scheduler (Control-M) . The 'nodeType=A' mentioned above allows us to query individual nodes (the nodes listen where 'nodeType=A or nodeType=*'. This allows commands to be 'targeted' to specific nodes if that is necessary.
Keep in mind, this is our own console, not the spring batch admin console. So perhaps that doesn't help you, but building up a simple console doesn't take that long using the spring batch APIs (4 or 5 asps).
The batch nodes could also have started up simple services like HTTP REST services or 'whatever' but we use MQ heavily and i liked the idea of not having to preregister nodes (the framework code doesn't know/care that it's in an HTTP container, so it couldn't register the endpoint easily). With MQ, the channel is preconfigured and all apps just 'use it' so it seemed easier.
Good luck.
I am trying to do the same thing. But it seems that in order to launch job directly from Spring batch admin, all the job resource has to be added to the spring batch web app. May be try restful job submission with spring MVC
#chau
One way to use Spring batch admin as is, but "discover" and "invoke" remote jobs is to provide your own implementations for org.springframework.batch.admin.service.JobService and org.springframework.batch.core.launch.JobOperator that can query and invoke jobs from remote job registry/repository.
You can find custom implementation for JobService and JMX enabled Job administrator in : https://github.com/regunathb/Trooper/tree/master/batch-core as: org.trpr.platform.batch.impl.spring.admin.SimpleJobService and org.trpr.platform.batch.impl.spring.jmx.JobAdministrator
Spring beans XML that uses these beans are here : https://github.com/regunathb/Trooper/blob/master/batch-core/src/main/resources/packaged/common-batch-config.xml