I have been creating some Lambda functions on AWS using the serverless framework, Flask and SLS WSGI. Some dynamodb tables but that should not matter in this case.
The problem that I am facing is that I can not debug the whole thing end to end, I am able to run sls wsgi serve and run a local instance of my lambda functions, happy days. However, I am a little bit spoiled by other dev tools, languages and IDEs (even just Flask itself) that allow me to set breakpoints and see the scope, step through etc. So I would really like to be able to get that done here as well.
I tried launching the sls command mentioned above in a launch configuration inside vs code, no luck. Next thing I tried was to run the default flask launch config but that obviously didn't include all the configuration stored in the sls.yml file which is essential for accessing the local dynamodb instance.
The last thing I tried was to attach to ptvsd at the end of my app.py file. So I would hit a wait action from ptvsd, attach the debugger in vs code to the specified port, which seems to be successful and returning the code execution. However, it seems like sls wsgi runs the file twice, so that the attaching happens for the first instance and not the second, which then does not trigger a breakpoint when I try to execute an API call through Postman.
I guess I could include the wait step everywhere manually, then attach for each method that I am trying to debug inside the code instead of in the IDE, but that seems like overkill and not very convenient.
I have been looking for answers online and reading through docs and could find no further.
I figured out that I can use Attach using Process Id It is however a little bit tricky because there are always 2 processes running in the list (different pid's). Its not great, but it does the trick
One technique I have found useful, albeit in a node environment but should apply here, is to make use of unit testing as a way to execute code locally with the ability to tie in a debuggerm as well as make use of mocking to stub away the external dependencies such as AWS services (S3, DynamoDB, etc). I wrote a blog post about setting this up for node but you may find it useful as a way to consider setting things up with Pythoin as well: https://serverless.com/blog/serverless-local-development/
However, in the world of serverless development, it ultimately doesn't matter how sophisticated you get with your local development environment, you will have to to test in the cloud environment as well. The unit testing technique I described is good for catching those basic syntactical and logical errors but you still will need to perform a deployment into the cloud and test in that environment. Its one of the reasons at Serverless we are working very hard on ways to improve the ability and time it takes to deploy to the cloud so that testing in AWS is replaces local testing.
Related
i'm working on a GitLab CI implementation of ZAP.
What i'm trying to archive, is to perform tests directly in the project, and check the results in the pipeline. I need your help to understand how i can write a yml file to test all the urls present in the application to test.
Is there anyone who already did this?
Thx
GitLab Team member here. I definitely don't pretend to be an expert on ZAP and how to integrate it, however, this StackOverflow thread seems like a promising example of executing successfully using something like Docker.
Let me know if this reply misses the mark, but I hope it at least gets you one step closer.
I have a (slightly complex) spring webservice which communicates with multiple frontends via a RESTful API (JSON) and additionally with other devices via SOAP or REST. I'd like setup an automated test environment which is capable of the following things:
create preconditions via fixtures (POSTGRES DB)
send REST or SOAP messages against the API
is able to run certain task (requests against the API) at a specific
time/date
assert and validate the produced results (return of the API call or
check the DB)
run all tests independet from any frontend/UI
integrate the testing environment in my infrastructure (i.e. create a
docker container which runs all tests deployed by Jenkins)
preferably I'd like to build reusable components (i.e. for creating a user that is needed in multiple different tests and so on...). I know there are a lot of tools and frameworks (SoapUI, JMETER,...). But before trying them all and getting lost, I'd like to get an experience report from someone who has a simular setup.
we are using JMeter for API testing. We tried SOAPui but it had some memory issues. So we are pushing forward with JMeter and so far so good.
For your questions:
We are using MySQL, but this post seems to show how to set up a postgres connection in JMeter: https://hiromia.blogspot.com/2015/03/how-to-perform-load-testing-on.html
JMeter can send REST API requests
I'm not sure if this is possible but you could probably have your Jenkins job scheduled to run when you need the API to run the specific task at the specific time.
There are quite a few assertions in JMeter. I use the Response and the BeanShell Assertions a lot.
JMeter is independent from any front end UI which helps pinpoint bugs.
I have not run docker but I am running via Jenkins. This jenkins plugin has been helpful: https://wiki.jenkins.io/display/JENKINS/Log+Parser+Plugin
A few more Tips:
Use the HTTP Request Defaults element. It will save you from having to update all your HTTP requests.
Use the User Defined Variables to define variables you need.
You can combine user defined variables like: ${namePrefix}${myTime} but it will have to be in a 2nd User Defined Variable element(you cant combine them in the same element)
If you have multiple test environments, set up a user defined variable with a value like this: ${__P(testenv,staging)} This way, you can change it from a CLI like this: -Jtestenv=HOTFIX
We are using ExtentReports for pretty html results reports with a custom JSR223 Listener(find my old post on this site).
If your site uses cookies, use the HTTP Cookie Manager.
If you need things to happen in order, on the Test Plan element, check this option: Run Threat Groups consecutively. If you dont, JMeter runs them in a random order.
Hope this is helpful. Happy Testing!
I'm setting up my front-end application to use continuous integration in CircleCI. Unit tests work fine, but end-to-end tests are not.
The problem is that it requires the backend (API) server to be running, and ours is in another completely different application. So, what is the best way to setup this backend server (thinking about CI)?
I thought about uploading it on heroku, but then I'd have to keep manually updating the code via git. Another option was to download the code to the CI VM and run the server directly there, but it is just too much work (install ruby, postgres, gems...), and it doesn't seem in no way the best option.
Have anyone passed through the same situation? How do you guys usually deal with this kind of situations?
I ended up doing everything inside the CI. I made some custom scripts that configure the backend project every time the test suite is ran. Also, I cached the folder with the backend code and the gems (which was taking ~2min to install).
The configuring part now adds ~20 seconds to the total time, so it wasn't a big deal. Although I still think that this is probably not the best way to do this, it has some advantages, such as not worrying about updating the backend code (it pulls from master automatically) or its database (it runs rake db:reset after updating the code).
Assuming the API server is running somewhere, configure the front-end application to point there while in the test/CI environment, at least to start out. If there are multiple API environments, choose the one the most closely matches the front-end environment (e.g. dev, staging, etc).
It gets more complicated if/when you need to run the e2e tests each time the API is built or match up specific build versions of the front-end and the API. In that case you will have to run the API server as part of the test.
I really like the approach of a 12factor app, which you are kinda forced into, when you deploy an application to Heroku. For this question I'm particularly interested in setting environment variables for configuration, like one would do on Heroku.
As far as I can tell, there's no way to change the ENV for one or multiple instances within the EC2 console (though it's seems to be possible to set 5 ENV vars when using elastic beanstalk). Therefore my next bet on an Ubuntu based system would be to use /etc/environment, /etc/profile, ~/.profile or just the export command to set ENV variables.
Is this the correct approach or am I missing something?
And if so, is there a best practice on how to do it? I guess I could use something like Capistrano or Fabric, get a list of servers from the AWS api, connect to all of them and change the mentioned files/call export. Though 12factor is pretty well known, I couldn't find any blog post describing how to handle the ENV for a non-trivial amount of instances on EC2. And I don't want to implement such a thing, if somebody already did it very well and I just missed something.
Note: I want a solution without using elastic beanstalk and I don't care about git push deployment or any other Heroku-like feature, this is solely related to app configuration.
Any hints appreciated, thanks!
Good question. There are many ways you can approach your deployment/environment setup.
One thing to keep in mind is that with Heroku (or Elastic Beanstalk for that matter) you only push the code. Their service takes care of the scalability factor and replication of your services across their infrastructure (once you push the code).
If you are using fabric (or capistrano) you are using a push model too, but you have to take care of all the scalability/replication/fault tolerance of your application.
Having said that, if you are using EC2, in my opinion it's better if you leverage AMIs, Autoscale and Cloudformation for your deployments. This is the beauty of elasticity and Virtualization in that you can think of resources as ephemeral. You can still use fabric/capistrano to automate the AMI builds (I use Ansible) and configure environment variables, packages, etc. Then you can define a Cloudformation stack (with a JSON file) and in it you can add an autoscaling group with your prebaked AMI.
Another way of deploying your app is to simply use the AWS Opsworks service. It's pretty comprehensive and it has a lot of options but it may not be for everybody since some people may want a bit more flexibility.
If you want to go 'pull' model you can use Puppet, Chef or CFEngine. In this case you have a master policy server somewhere in the cloud (Puppetmaster, Chef Server or Policy Server). When a server gets spun up, an agent (Puppet agent, Chef Client, Cfengine agent) connects to its master to pick up its policy and then executes it. The policy may contains all the packages and environment variables that you need for your application to function. Again, it's a different model. This model scales pretty well but it depends on how many agents the master can handle and how you stagger the connections from the agents to the master. You can load balance multiple masters too if you want to scale to thousands of servers or you can just simply use multiple masters. From experience, if you want something really "fast" Cfengine works pretty good, there's a good blog comparing the speed of Puppet and CFengine here: http://www.blogcompiler.com/2012/09/30/scalability-of-cfengine-and-puppet-2/
You can also go "push" completely with tools like fabric, Ansible, Capistrano. However, you are constrained by how much a single server (or laptop) can handle multiple connections to thousands of servers that its trying to push to. This is also constrained by network bandwidth, but hey you can get creative and stagger your push updates and perhaps use multiple servers to push. Again it works and it's a different model so it depends which direction you want to go.
Hope this helps.
If you dont need beanstalk, you can look at AWS Opsworks (http://aws.amazon.com/opsworks/). Ideal for Web worker kind of deployment scenerios. You can pass any variable from outside the code here (even Chef recipies)
It's might be late but they what we are doing.
We have python script that take env var in Json and send that to as post data to another python script that convert those vars to ymal file.
After that we use Jenkins pipline groovy using multibranch. Jenkins do all the build and then code deploy copies those env vars to ec2 instanced running in autoscaling.
Off course we are doing some manapulation from yaml to simple text file so code deploy can paste it on /etc/envoirments
I'm looking for a way to create an app which has a realtime web interface as well as an API which can be called by a node.js client while sharing most of its code.
I'd like to be able to manage data, monitor and execute tasks inside of my app via browser, but also have an automation/scheduling program which connects to my web app and tells it to run various tasks and get results of each task.
Unfortunately it doesn't look like I can connect to Meteor from the server, so I'm wondering if there's another approach? Is what I described even possible using Meteor?
I have done some testing using socket.io and I think I may be able to do it this way, but Meteor seems like it'd be really great for the realtime user interface.
Yes, you can use npm packages to do what you want. Just like standard Node.js programming.
There might be one error you run into when calling Meteor between external code, but it is easy to solve.
I guess in your case you could set up a TCP server that way and make it update a collection, then you could get the clients to update through the reactive collection publishing mechanism.