I was wondering if it is possible to distribute an elasticsearch instance packed within some sort of installer to run locally on a client's machine.
I know that elastic is ok to distribute, but I'm looking for some way to pack it up as a dependency in a larger project.
Thanks!
I guess docker container is the best solution for your use case.
You can see this project :
https://github.com/deviantony/docker-elk
A requirement is the host/workstation targeted must have docker (+ optionnaly docker-composer) installed.
docker-compose allow to package a lot of container for example we could run a multi node cluster + kibana
Related
I have a single Java application. We developed the application in Eclipse. It is a Maven project. We already have a system for launching our application to AWS EC2. It works but is rudimentary and we would like to learn about the more common and modern approaches other teams use to launch their Java Maven apps to EC2. We have heard of Docker and I researched the tool yesterday. I understand the basics of building an image, tagging it and pushing to either Docker Hub or Amazon's ECS service. I have also read through a few tutorials describing how to pull a Docker image into an EC2 instance. However, I don't know if this is what we are trying to do, given that I am a bit confused about the role Docker can play in our situation to help make our dev ops more robust and efficient.
Currently, we are building our Maven app in Eclipse. When the build completes, we run a second Java file that uses the AWS JDK for Java to
launch an EC2 instance
copy the.jar artifact from the build into this instance
add the instance to a load balancer and
test the app
My understanding of how we can use Docker is as follows. We would Dockerize our application and push it to an online repository according to the steps in this video.
Then we would create an EC2 instance and pull the Docker image into this new instance according to the steps in this tutorial.
If this is the typical flow, then what is the purpose of using Docker here? What is the added benefit, when we are currently ...
creating the instance,
deploying the app directly to the instance and also
testing the running app
all using a simple single Java file and functions from the AWS SDK for Java?
#GNG what are your objectives for containerization?
Amazon ECS is the best method if you want to operate in only AWS environment.
Docker is effective in hybrid environments i.e., on physical servers and VMs.
the Docker image is portable and complete executable of your application: it delivers your jar, but it can also include property files, static resources, etc... You package everything you need and deploy to AWS, but you could decide also to deploy the same image on other platforms (or locally).
Another benefit is the image contains the whole runtime (OS, jdk) so you dont rely on what AWS provides ensuring also isolation from the underlying infrastructure.
How to configure Netflix conductor to run standalone elastic search rather than embedded elastic search ?
if you have a conductor-config.properties just make sure you have these pointing to the valid elasticsearch you have up and running:
workflow.elasticsearch.instanceType=EXTERNAL
workflow.elasticsearch.url=http://elasticsearch:9200
Then should be able to run conductor up with that config:
java conductor-server-2.15.0-SNAPSHOT-all.jar conductor-config.properties
https://github.com/Netflix/conductor/blob/master/es5-persistence/src/main/java/com/netflix/conductor/dao/es5/index/ElasticSearchRestDAOV5.java
You can inspect this as an example, swapping the elastic container by your own, modifying the conductor-config.properties. It will be copied in when you run:
check out https://github.com/s50600822/conductor-cheat
inside the repo just do
docker-compose up
Check out https://github.com/Netflix/conductor/blob/master/es5-persistence/src/main/java/com/netflix/conductor/dao/es5/index/ElasticSearchRestDAOV5.java for other options.
To add external elastic search we need to follow code changes as mentioned in
below link.
https://github.com/Netflix/conductor/tree/master/es5-persistence.
And rebuild jar and run conductor server again with properties.
if you still get errors , I suggest to follow below link
https://github.com/Netflix/conductor/issues/489.
You can use the standalone installation of elasticsearch2 or elasticsearch5 because the associated support classes are already provided with Netflix Conductor binary.
To configure externally you have to do the following
Install and configure standalone elasticsearch. By default the
installation would expose 2 ports 9200/http or 9300/tcp.
Update server.properties file with the host and port so that the
communication will start happening with the standalone instance of
elasticsearch.
Hope this helps.
I am new to Docker. I have developed a webapp that needs elasticsearch server up and running, and the server itself listens for any user request. IN our workflow, we would like to see elasticsearch up and running, then logstash to be able to publish data (by reading the data we have with the help of logstash conf files), and finally launch the webapp we have. I am advised to use docker compose as it helps to compose multiple servers.
SO, we have three sub-directories, one for each es, logstash and webapp.
In my first step, I have in my elasticsearch dockerfile the below lines
FROM elasticsearch:latest
RUN plugin -i elasticsearch/marvel/latest
Similarly, I have Dockerfile in other sub-directories as well.
I use the command 'docker-compose up' to start the process. Once, ES is built ad logstash will be built. When logstash is being built, it tries to publish data to ES. But it finds here that ES is not up and running.I see connection refused exception. Can someone tell why this error comes? The contents in the Dockerfile is ok?
I think, I am not using docker / docker-compose the way I should use. May be, couple of pointers to learning materials will be helpful. I find plenty but could not relate to the use case I have.
There are two phases when running up. The "build" phase, which runs docker build, and the "run" phase, which runs the equivalent of docker run. During the build phase there is no support for linking to other containers, so you won't be able to connect to them.
An easy solution is to run part of the setup during the "run" phase, but the downside is that it isn't cached and has to be performed each time.
To include it as part of the build phase, you can export the json that needs to be sent to ES, include it as part of the ES build, and then have a step that does something like this:
start es
wait for es to be available
run curl to PUT/POST the data to ES
stop es
I have the 2 containers mentioned running with Vagrant how do I connect my Django app from one container to Postgresql on the other?
The idea:
TL;DR: Read this to know how to link containers
I should be able to easily connect containers when they are running with a project already setup/coded, either in the host machine or a VM.
I have read a bit here and there. Found out how to install everything I need in a container, but that would not be isolating db setup from app. These are the conclusions I've drawn so far:
Docker's use cases are to run/test projects in an isolated manner, the process of programming and setting up the project (in my case, a Django app) is done in a Vagrant VM or on the host machine. Then after a certain point in development, I'd run the project in a container to test the behavior. So the premise that I'd be developing in a Docker, while possible, is not part of their philosophy.
Docker's purpose is, after testing/development, to deploy in containers using the same or better setup used to test the project.
My organization's website is a Django app running on front end webservers + a few background processing servers in AWS.
We're currently using Ansible for both :
system configuration (from a bare OS image)
frequent manually-triggered code deployments.
The same Ansible playbook is able to provision either a local Vagrant dev VM, or a production EC2 instance from scratch.
We now want to implement autoscaling in EC2, and that requires some changes towards a "treat servers as cattle, not pets" philosophy.
The first prerequisite was to move from a statically managed Ansible inventory to a dynamic, EC2 API-based one, done.
The next big question is how to deploy in this new world where throwaway instances come up & down in the middle of the night. The options I can think of are :
Bake a new fully-deployed AMI for each deploy, create a new AS Launch config and update the AS group with that. Sounds very, very cumbersome, but also very reliable because of the clean slate approach, and will ensure that any system changes the code requires will be here. Also, no additional steps needed on instance bootup, so up & running more quickly.
Use a base AMI that doesn't change very often, automatically get the latest app code from git upon bootup, start webserver. Once it's up just do manual deploys as needed, like before. But what if the new code depends on a change in the system config (new package, permissions, etc) ? Looks like you have to start taking care of dependencies between code versions and system/AMI versions, whereas the "just do a full ansible run" approach was more integrated and more reliable. Is it more than just a potential headache in practice ?
Use Docker ? I have a strong hunch it can be useful, but I'm not sure yet how it would fit our picture. We're a relatively self-contained Django front-end app with just RabbitMQ + memcache as services, which we're never going to run on the same host anyway. So what benefits are there in building a Docker image using Ansible that contains system packages + latest code, rather than having Ansible just do it directly on an EC2 instance ?
How do you do it ? Any insights / best practices ?
Thanks !
This question is very opinion based. But just to give you my take, I would just go with prebaking the AMIs with Ansible and then use CloudFormation to deploy your stacks with Autoscaling, Monitoring and your pre-baked AMIs. The advantage of this is that if you have most of the application stack pre-baked into the AMI autoscaling UP will happen faster.
Docker is another approach but in my opinion it adds an extra layer in your application that you may not need if you are already using EC2. Docker can be really useful if you say want to containerize in a single server. Maybe you have some extra capacity in a server and Docker will allow you to run that extra application on the same server without interfering with existing ones.
Having said that some people find Docker useful not in the sort of way to optimize the resources in a single server but rather in a sort of way that it allows you to pre-bake your applications in containers. So when you do deploy a new version or new code all you have to do is copy/replicate these docker containers across your servers, then stop the old container versions and start the new container versions.
My two cents.
A hybrid solution may give you the desired result. Store the head docker image in S3, prebake the AMI with a simple fetch and run script on start (or pass it into a stock AMI with user-data). Version control by moving the head image to your latest stable version, you could probably also implement test stacks of new versions by making the fetch script smart enough to identify which docker version to fetch based on instance tags which are configurable at instance launch.
You can also use AWS CodeDeploy with AutoScaling and your build server. We use CodeDeploy plugin for Jenkins.
This setup allows you to:
perform your build in Jenkins
upload to S3 bucket
deploy to all the EC2s one by one which are part of the assigned AWS Auto-Scaling group.
All that with a push of a button!
Here is the AWS tutorial: Deploy an Application to an Auto Scaling Group Using AWS CodeDeploy