I'm deploying Spring Cloud Dataflow 1.4.0.RELEASE on my org's Cloud Foundry. I took the CF deployable server and used a manifest to deploy it. We use it mostly for tasks developed using Spring Cloud Task. In the documentation (that uses PCF) suggests this configuration:
SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_TASK_SERVICES my_mysql
Where my_mysql is the DB to keep track of task related information. Documentation suggests using this as CF's cups but that won't work since my org uses Vault to store keys and secrets so DB connections are not allowed in such way.
Since I did not provided any value for SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_TASK_SERVICES in my manifest when I start SPDF it throws an error indicating that bean "taskService" could not be created.
Haven't found a workaround for this. Is this the only way to specify DB connection for tasks in version 1.4?
You can also pass a deployment property when launching the task:
task launch --name mytask --properties "deployer.mytask.cloudfoundry.services=-Duser.timezone=my_mysql"
This way, you don't have to use the server level environment property SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_TASK_SERVICES
Related
I am creating a spring boot microservice project with intelij IDEA.
Currently I have developed three seperate spring boot rest services as customer service, vehicle service and spring cloud config server. Spring cloud config server is pointing to a github repository.
The issue is sometimes above projects take more than 10 minutes to run and sometimes does't run and give an error message as "failed to check application readystate intellij attached provider for the vm is not found". I have no idea why this happens ?
There are two possible causes:
1. IntelliJ IDEA and the Spring application are running in different JVMs.
There is a bug for IntelliJ IDEA regarding that:
https://youtrack.jetbrains.com/issue/IDEA-210665
Here is short summary:
IntelliJ IDEA uses local JMX connector for retrieving Spring Boot actuator endpoint's data by default. However, it could be impossible to get local JMX connector address via attach api if Spring Boot application and IntelliJ IDEA are run by different JVMs. In this case, add the following lines to VM options of your Spring Boot run configuration:
-Dcom.sun.management.jmxremote.port={some_port}
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
As mentioned in the official Oracle documentation, this configuration is insecure. Any remote user who knows (or guesses) your port number and host name will be able to monitor and control your Java applications and platform.
2. Prolonged time to retrieve local hostname
You can check that time using inetTester. Normally it should take only several milliseconds to complete. If it takes a long time then you can add the hostname returned by inetTester to /etc/hosts file like this:
127.0.0.1 localhost winsky
::1 localhost winsky
In terraform we get a state file, and CloudFormation also has a notion of a working state. How does go cloud handle the state, do we have to create it ourselves?
For more info on Go Cloud
https://github.com/google/go-cloud
https://godoc.org/github.com/google/go-cloud
Terraform wants to solve the problem of managing and provisioning Cloud services.
Go Cloud wants to solve the problem of using Cloud services in application code.
So, they work well together. For example, the Go Cloud sample guestbook app (https://github.com/google/go-cloud/tree/master/samples/guestbook) uses Terraform to provision the resources needed to run the app on various Cloud providers; the application code in the sample has a small amount of provider-specific setup code, but the application logic itself is provider-agnostic.
go-cloud:
The Go Cloud Project is an initiative that will allow application developers to seamlessly deploy cloud applications on any combination of cloud providers. It does this by providing stable, idiomatic interfaces for common uses like storage and databases. Think database/sql for cloud products.
Terraform:
Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions.
So with go-cloud you could create a tool like terraform that, for now can provide generic APIs for:
Unstructured binary (blob) storage
Variables that change at runtime (configuration)
Connecting to MySQL databases
Server startup and diagnostics: request logging, tracing, and health checking
I am trying to register a spring cloud task jar which is residing in a ftp server , from a spring cloud data flow container which is running on PCF.
Can somebody help me in terms of how to do a "app register" from a ftp location from SCDF if SCDF supports that ? How do i register a ftp location as remote repository with user name & passwd ?
Thanks!
Nilanjan
As discussed in the email, currently, in SCDF, we support file://, maven://, http://, docker://, and hdfs:// as the file resolution strategies. Besides, for cloud platforms, it'd be recommended that you resolve and register artifacts from maven or docker to be more resilient.
If you're running SCDF on PCF, you could bind the SCDF-server to volume-services. By having all the custom apps in volume-services, SCDF-server could then resolve artifacts from the designated directory via file:// resolution - this is another good alternative for PCF deployments.
I am a newbie with CF microservices and I am trying to deploy a service multiple times. As far as I understood, each time I deploy into a space the application is getting a different database server and schema. Is there a way to tell the Cloud Foundry to use only a fixed DB server all the times across multiple deployments in one environment?
The keyword for your case is 'Service Instance'
You can create a service instance of database server within the environment specific for your application and bind it via application manifest.
e.g.
cf create-service rabbitmq small-plan myapplication-rabbitmq-instance
As long as you have a binding to myapplication-rabbitmq-instance in your application manifest it would be preserved/be the same between application deployments within this space.
e.g. in your application manifest:
---
...
services:
- myapplication-rabbitmq-instance
More on https://docs.cloudfoundry.org/devguide/services/
We are planning to retire the existing legacy java batch applications and recreate it with the latest available batch framework.
Given that we have a large number of batch jobs to be modernised, we are looking for a framework or architecture that would allow us to
Develop a batch solution that would allow us to dynamically deploy a new batch as and when they are created, without disturbing the existing deployed applications. - Does Spring cloud Task provide any of this feature. Note: We are looking only to deploy the apps to our local server, and has nothing to do with cloud.
If Spring Batch/Boot can provide us the feature we typically expect from a batch application, what is the special value add to go for Spring Cloud Task? - I wasn't able to completely understand this from the Spring documentation available online.
From the documentation of the Spring Cloud Task, I was able to understand that it allows an application to have many tasks within it. What should I do if each of the tasks have their own library dependencies, which might contradict with the dependencies of other Tasks? So in that case, should each of these tasks moved to a new application or this there a work around for that?
To answer your questions:
Does Spring Cloud Task handle orchestration - No. Spring Cloud Task does not handle orchestration of tasks or jobs. The component in this ecosystem that handles the deployment/orchestration of tasks or jobs is really Spring Cloud Data Flow (which is why I asked if you use any type of cloud platform including YARN, Cloud Foundry, Kubernetes, or Mesos...the environments supported by Spring Cloud Data Flow).
What added value does Spring Cloud Task provide over Spring Boot/Spring Batch - Spring Cloud Task is designed to provide a few things:
Similar abilities to Spring Batch with regards to state management without needing to create a batch job. When running a Boot application on a cloud environment, there is no standard way of getting the results from environment to environment (YARN handles job results differently from tasks on Cloud Foundry which is different from jobs on Kubernetes, etc). Spring Batch provides this but now all short lived processes need the overhead of the Batch API so Spring Cloud Task provides a lighter touch to those use cases.
Automatically adds informational listeners. With Spring XD, when you ran a job in an XD container, the XD container automatically added a number of informational listeners that broadcast events that you could listen for. Spring Cloud Task brings the same functionality without the need for the XD container.
Integration with Spring Cloud Stream. Spring Cloud Task provides the ability to launch tasks from messages received from Spring Cloud Stream. Also, the informational messages previously mentioned (both Batch events as well as Task events) are sent via Spring Cloud Stream channels.
The DeployerPartitionHandler. When working in a cloud environment, this PartitionHandler implementation allows you to launch workers for a partitioned batch job as tasks. This allows for the dynamic scaling of partitioned batch jobs instead of the traditional option of pre-deploying workers that listen for work which wastes resources in a modern cloud environment.
How does the packaging of multiple tasks work with dependencies - In short, this is not recommended. The idea of a Spring Cloud Task is that the execution of the Spring Boot application is the Task. While you could package up multiple tasks and using different methods, have them execute based on different stimulus, that goes against the 12 factor application concepts which are essential for correct use of Spring Cloud Task.
My two cents
For the best option for a modern batch platform, you really need to look into some from of platform first and that begins at the Cloud Foundry/Kubernetes/Mesos/YARN layer. Without that, you end up building a large part of the infrastructure yourself. That is why Spring XD evolved into Spring Cloud Data Flow. The added complexity that lived in the containers of Spring XD is removed by requiring a modern platform to run on (since they all handle those guarantees themselves). Without that piece, you're going to spend a lot of time managing the deployment and orchestration of applications that most modern platforms handle for you.
From there, the choice becomes pretty easy IMHO with Spring Cloud Task for simple tasks, Spring Batch for batch jobs, and Spring Cloud Data Flow for orchestration.