Since a few days, our aws elastic beanstalk fails to deploy our code through npm Install.
We are using our own fork from parse-server repo, and it always worked fine.
Unfortunately it now fails without obvious reason. When looking at the instance logs it clearly shows it tries to use the original parse-server repo (on a very old branch) instead of our very own fork, but I can't figure why.
Our package.json file indicates:
"parse-server": "git+https://github.com/hulab/parse-server.git#patched/5.3.0-hulab-2"
and our npm-shrinkwrap.json file reflects it with a
"parse-server": {
"version": "git+https://github.com/hulab/parse-server.git#54bfd65181f19d4296f0ebea79cf3a4ab542f2fc",
"from": "git+https://github.com/hulab/parse-server.git#patched/5.3.0-hulab-2",
...
}
Whereas the EC2 instance logs indicate failures when installing:
parse-server#github:parse-community/parse-server#892c6f94d50b6dced8a5e1948e058dc7b095c914
I can't figure out why this branch is used while not being referred to in any of our files!
Any help would be very appreciated :)
It is likely another dependency that points to that specific version of Parse Server.
For example, the Parse JS SDK has a devDependency for integration tests:
"parse-server": "github:parse-community/parse-server#alpha"
Since you are using a customized version of Parse Server, check the Parse JS SDK (which is a component of Parse Server), and any other Parse dependency that you added, whether they have a parse-server dependency and to where it points. In package.json you may only see the branch name but in package-lock.json of that dependency you may see the actual commit hash you are referring to.
Related
I've been experimenting with using the new AWS CDK pipelines construct (https://docs.aws.amazon.com/cdk/api/latest/docs/pipelines-readme.html), and have successfully setup a couple of projects, including following the blog post announcement here: https://aws.amazon.com/blogs/developer/cdk-pipelines-continuous-delivery-for-aws-cdk-applications/.
However, as soon as I try to add a dependency to my lambdas the build fails with a
lib/lambda/handler.ts(2,24): error TS2307: Cannot find module 'stripe' or its corresponding type declarations.
I have installed a package.json file and the node_modules in the directory with the lambdas, tried zipping the lambdas and node_modules, tried uploading the zip file with the console, and tried getting the 'buildCommand' during the 'synthAction' step to install the dependencies. Nothing works.
The asset appears to get created in the cdk.out directory, and code changes are being uploaded, but node_modules themselves never get packaged along with the lambda functions.
I am using the 'SimpleSynthAction.standardNpmSynth' action, along with an 'npm run build' command in the 'buildCommand' step.
I've looked at Lambda can't find modules from outer folders when deployed with CDK, How to install dependencies of lambda functions upon cdk build with AWS CDK and https://github.com/aws-samples/aws-cdk-examples/issues/110#issuecomment-550506116 without luck.
What am I missing?
Looks like the issue has to do with bundling. I abandoned the 'all-in on CDK approach' and went back to using a SAM/CDK hybrid, but this new blog post suggests bundling is the answer.
https://aws.amazon.com/blogs/devops/building-apps-with-aws-cdk/
Specifically, it references this construct, which likely would have helped in my situation: https://docs.aws.amazon.com/cdk/api/latest/docs/aws-lambda-nodejs-readme.html
Something to try in future.
I'm currently working something on AWS Cloudformation which using this repo https://github.com/awslabs/goformation. Because I did some customise so I made a fork https://github.com/vrealzhou/goformation.
Now in my other project (using go module) I'm trying to using go get github.com/vrealzhou/goformation#v2.3.1 and I've got this error:
go: github.com/vrealzhou/goformation#v0.0.0-20190513073615-ff3b65adb278: parsing go.mod: unexpected module path "github.com/awslabs/goformation"
go: error loading module requirements
Does anyone know the reason and how to solve this problem? Thanks
You can use replace in your go.mod to use a fork instead of the upstream version. That way, you can make whatever modifications you need to the code without having to update the module path or import paths.
To be specific, in this case, you can do the following in your go.mod (I tested this by forking the repo, making a small change, and confirming it showed up):
require github.com/awslabs/goformation v1.4.1
replace github.com/awslabs/goformation => github.com/vrealzhou/goformation master
The first time you build or test, master will be replaced by the latest pseudo-version for your fork to make sure you get repeatable builds. The replace requires a specific version for the replacement.
My Gitlab CI Auto DevOps job failed with
Status: Downloaded newer image for gliderlabs/herokuish:latest
-----> Unable to select a buildpack
ERROR: Job failed: exit code 1
I've gone through
Auto DevOps and
Getting started with Auto DevOps
and am still not sure where I should put the buildpack.
Mine should be heroku-buildpack-go, which I've lost track where I get that from.
My repo consist only a single-character README.md, and the "Hello, playground" main.go.
Conclusion:
Thanks to David's comprehensive explanation, I was able to get DevOps started with correct buildpack:
From this I would conclude that your single .go file at the root of the directory tree does not meet the activation criteria for auto-building Go projects. I'd suggest picking one of the dependency managers in the requirements above and modifying your project to support it.
FTA, I just did touch go.mod then git add & git push and the AutoDevops started building my GO project indeed.
However it seems to me that Gitlab AutoDevops is not able to build any GO projects very easily, as I get the following error (with project variable TRACE=true):
...
!! The go.mod file for this project does not specify a Go version
!!
!! Defaulting to go1.11.1
!!
!! For more details see: https://devcenter.heroku.com/articles/go-apps-with-modules#build-configuration
!!
-----> Installing go1.11.1
-----> Fetching go1.11.1.linux-amd64.tar.gz... done
!! Installing package '.' (default)
!!
!! To install a different package spec add a comment in the following form to your `go.mod` file:
!! // +heroku install ./cmd/...
!!
!! For more details see: https://devcenter.heroku.com/articles/go-apps-with-modules#build-configuration
!!
-----> Running: go install -v -tags heroku .
go: cannot determine module path for source directory /tmp/build (outside GOPATH, no import comments)
ERROR: Job failed: exit code 1
The easier solution is to use .gitlab-ci.yml file instead, documented in
https://blog.boatswain.io/post/build-go-project-with-gitlab-ci/
(and followed up at Gitlab CI - Start Shared Runner for normal repos).
From the AutoDevops documentation:
Auto Build creates a build of the application in one of two ways:
If there is a Dockerfile, it will use docker build to create a Docker image.
Otherwise, it will use Herokuish and Heroku buildpacks to automatically detect and build the application into a Docker image.
Then looking at the build activation criteria, as per the Heroku Go buildpack documentation:
This buildpack will detect your repository as Go if you are using either:
go modules
dep
govendor
glide
GB
Godep
Or more specifically for godep, govendor or GB:
The Heroku Go buildpack is used when an application meets one of the following requirements:
has a Godeps/Godeps.json file, identifying the application as being managed by godep;
has a vendor/vendor.json file, identifying the application as being managed by govendor;
has a src directory, which has sub directories, contains one or more .go files, identifying the application as being managed by gb.
From this I would conclude that your single .go file at the root of the directory tree does not meet the activation criteria for auto-building Go projects. I'd suggest picking one of the dependency managers in the requirements above and modifying your project to support it. After that AutoDevops should start building your project.
If you are still having issues after that, this debugging note might help:
After making sure that the project meets the buildpack requirements;
if it still fails, setting a project variable TRACE=true will enable verbose logging which​ may help to continue troubleshooting.
My Problem
Elastic Beats is an open source project for log shippers written in Go. It features several log outputs, including console, Elasticsearch and Redis. I would like to add an output of my own - to AWS Kinesis.
I have cloned the repo to ~/github/beats, and tried building it:
$ cd filebeat; go build main.go
However, it failed due to a missing library which is a part of the project:
main.go:6:2: cannot find package "github.com/elastic/beats/filebeat/cmd" in any of:
/usr/local/go/src/github.com/elastic/beats/filebeat/cmd (from $GOROOT)
/Users/adam/go/src/github.com/elastic/beats/filebeat/cmd (from $GOPATH)
A directory of the project is dependent on a package from the same repo, but instead of looking one directory up the hierarchy it looks in the GOPATH.
So, go get github.com/elastic/beats/filebeat/cmd fetched the code, and now go build main.go works. Changing the code in my GOPATH is reflected in these builds.
This leaves me with an structural inconvenience. Some of my code is at a working directory, and some of it is at my GOPATH and included by the code in my working directory.
I would like to have all my code in a single directory for various reasons, not the least being keeping everything under version control.
What Have I Tried
Mostly searching for the problem. I am quite new to Go, so I might have missed the correct terminology.
My Question
What is the right way to edit the code of an imported library in Go?
One of the recommended ways to work with other's packages is:
Get the sources of the original package:
go get github.com/elastic/beats
As a result you will clone project's git repository to the folder
$GOPATH/src/github.com/elastic/beats
Make some fixes, compile code, fix, compile... When you make go install package will be compiled and installed to your system. When you need merge updates from original repository you can git pull them.
Everything is OK. What's next? How to share your work with others?
Fork project on github, suppose it will be github.com/username/beats
Add this fork as another remote mycopy (or any other name you like) to your local repository
git remote add mycopy git://github.com/username/beats.git
When all is done you can push updated sources to your repo on github
git push mycopy
and then open a pull-request to original sources. This way you can share your work with others. And keep your changes in sync with mainstream.
Previous answers to this question are obsolete when developing projects that using Go Modules.
For projects that using Go Modules, one may use the following command to replace an imported library(eg. example.com/imported/module) with a local module(eg. ./local/module):
go mod edit -replace=example.com/imported/module=./local/module
Or by adding the following line into the go.mod file:
replace example.com/imported/module => ./local/module
Reference Docs: https://golang.org/doc/modules/managing-dependencies#unpublished
A project working copy should be checked out into $GOPATH/src/package/import/path - for example, this project should be checked out into /Users/adam/go/src/github.com/elastic/beats. With the project in the correct location, the go tooling will be able to operate on it normally; otherwise, it will not be able to resolve imports correctly. See go help gopath for more info.
I've been trying to convert and deploy one of our node.js apps into a lambda function and have been having some problems with the node_modules dependencies - saying that it can't find certain modules. I started by creating a package.json, npm install the dependencies locally then copy the node modules folder up to lambda.
For instance I have a project that requires sequelize and convict and have been getting errors saying that it cannot find the moment module as a sub-dependency. I see that moment is included in the root of my node_modules but it was not included in the sub folder under the sequelize module.
However, this project runs fine locally. What is the difference in lambda and what's the best practice for deploying a somewhat long list of node modules with it - just a copy of the node_modules folder? On some of the other simpler projects I have, the small amount of node_modules can be copied up with no problems.
{
"errorMessage": "Cannot find module 'moment'",
"errorType": "Error",
"stackTrace": [
"Function.Module._resolveFilename (module.js:338:15)",
"Function.Module._load (module.js:280:25)",
"Module.require (module.js:364:17)",
"require (module.js:380:17)",
"VERSION (/var/task/node_modules/sequelize/node_modules/moment-timezone/moment-timezone.js:14:28)",
"Object. (/var/task/node_modules/sequelize/node_modules/moment-timezone/moment-timezone.js:18:2)",
"Module._compile (module.js:456:26)",
"Object.Module._extensions..js (module.js:474:10)",
"Module.load (module.js:356:32)",
"Function.Module._load (module.js:312:12)"
]
}
I resolved this by uploading all from a zip file which contains all the data I need for my lambda function.
you can just create your project in your local machine and make all the changes that you need then the file you are going to zip should have this same structure and also see that there is an option to load your code from a zip file.
This sounds to me like an issue caused by different versions of npm. Are you running the same version of nodejs locally as is used by Lambda (ie. v0.10.36)?
Depending on the version of npm you're using to install the modules locally, the node_modules directory's contents are laid out slightly differently (mainly in order to de-duplicate things), and that may be why your dependencies can't find their dependencies in Lambda.
After a bit of digging, it sounds like a clean install (ie. rm your node_modules directory and run npm install) might clean things up for you. The reason why is that it seems that npm doesn't install sub-dependencies if they're already present at the top level (ie. you installed moment before sequelize, etc).