Speed up many installations of node modules - performance

I wrote a package that analyzes the structure of many yeoman generators:
https://github.com/tobiasoberrauch/yeoman-analyzer
Therefore I have to install all generators. This solution tooks very long (12 hours). I'm not quite sure is the current solution is the best way to do this:
Get the list of generator names: https://github.com/tobiasoberrauch/yeoman-analyzer/blob/master/lib/analyzer/data/generators.json
Install generators to node_modules regarding a list of generator names (VERY SLOW) : https://github.com/tobiasoberrauch/yeoman-analyzer/blob/master/lib/analyzer/generators.js#L24
Analyze the index.js from each generator: https://github.com/tobiasoberrauch/yeoman-analyzer/blob/master/lib/analyzer/generators.js#L47
Write report to a json file: https://github.com/tobiasoberrauch/yeoman-analyzer/blob/master/lib/analyzer/reporter/json.js#L21
Is there a way to speed up the process? 2. tooks very long.
I read about caching, local registry (npmd) but I didn't found a proper, fesiable solution.
I would be very happy about any hint :+1:
Cheers Tobias

If you need index.js only, you don't have to install all generators. Just download a tarball from npm registry using something like wget, and unpack it with tar xz.

Thanks for the hints.
Here's my (temporarily) solution:
Get package data from registry: https://github.com/tobiasoberrauch/yeoman-analyzer/blob/master/lib/analyzer/generators.js#L50
Get latest version: https://github.com/tobiasoberrauch/yeoman-analyzer/blob/master/lib/analyzer/generators.js#L64
Download and / or extract tarball file: https://github.com/tobiasoberrauch/yeoman-analyzer/blob/master/lib/analyzer/generators.js#L129
Read package.json from extracted file and get main file: https://github.com/tobiasoberrauch/yeoman-analyzer/blob/master/lib/analyzer/generators.js#L79
Find expressions in main file: https://github.com/tobiasoberrauch/yeoman-analyzer/blob/master/lib/analyzer/generators.js#L90
Sometimes is the zip file empty, the download stops suddenly or the esprima parser shows unexpected error. But I have a rough plan to optimize it.
This is just a awful callback hell. Now it's time to optimize and restructure.

Related

go-swagger restapi/configure_todo_list.go - api.TodoGetHandler undefined error

I am a newbie in go and go-swagger. I am following steps in Simple Server tutorial in goswagger.io.
I am using Ubuntu 18.04, swagger v0.25.0 and go 1.15.6.
Following the same steps, there are a few differences of the files generated. For instance, goswagger.io's has find_todos_okbody.go and get_okbody.go in models but mine does not. Why is that so?
Link to screenshot of my generated files vs
Link to screenshot of generated files by swagger.io
Starting the server as written in the tutorial go install ./cmd/todo-list-server/ gives me the following error. Can anyone please help with this?
# my_folder/swagger-todo-list/restapi
restapi/configure_todo_list.go:41:8: api.TodosGetHandler undefined (type *operations.TodoListAPI has no field or method TodosGetHandler)
restapi/configure_todo_list.go:42:6: api.TodosGetHandler undefined (type *operations.TodoListAPI has no field or method TodosGetHandler)
The first step in goswagger.io todo-list is swagger init spec .... Which directory should I run this command in? I ran it in a newly created folder in my home directory. However, from the page, it shows the path to be ~/go/src/github.com/go-swagger/go-swagger/examples/tutorials/todo-list. I am not sure whether I should use go get ..., git clone ... or create those folders. Can someone advise me?
Thanks.
This is likely the documentation lagging behind the version of the code that you are running. As long as it compiles, the specific files the tool generates isn't so crucial.
This is a compilation error. When you do go install foo it will try to build the foo package as an executable and then move that to your GOPATH/bin directory. It seems that the generated code in restapi/configure_todo_list.go isn't correct for the operations code generated.
All you need to run this tutorial yourself is an empty directory and the swagger tool (not its source code). You run the commands from the root of this empty project. In order not to run into GOPATH problems I would initialise a module with go mod init todo-list-example before doing anything else.
Note that while the todo-list example code exists inside the go-swagger source, it's there just for documenting example usage and output.
What I would advice for #2 is to make sure you're using a properly released version of go-swagger, rather than installing from the latest commit (which happens when you just do a go get), as I have found that to be occasionally unstable.
Next, re-generate the entire server, but make sure you also regenerate restapi/configure_todo_list.go by passing --regenerate-configureapi to your swagger generate call. This file isn't always refreshed because you're meant to modify it to configure your app, and if you changed versions of the tool it may be different and incompatible.
If after that you still get the compilation error, it may be worth submitting a bug report at https://github.com/go-swagger/go-swagger/issues.
Thanks #EzequielMuns. The errors in #2 went away after I ran go get - u -f ./... as stated in
...
For this generation to compile you need to have some packages in your GOPATH:
* github.com/go-openapi/runtime
* github.com/jessevdk/go-flags
You can get these now with: go get -u -f ./...
I think it's an error of swagger code generation. You can do as folloing to fix this:
delete file configure_todo_list.go;
regenerate code.
# swagger generate server -A todo-list -f ./swagger.yml
Then, you can run command go install ./cmd/todo-list-server/, it will succeed.

Go install exclude file

I have created a go script that compiles, starts, checks the status, and ends a web service I created (that is also in go). However, I have come to a road block.
With the compile feature I run the following command:
go install .
Which gives the following error:
./script.go:55: main redeclared in this block
previous declaration at ./hello.go:8
Which makes sense as I have two different files, both with the main func and main package. I also tried moving the script to another folder and then changing the command ran to:
go install {path}
Where {path} is equal to the path I want installed/compiled. Which I then got the following error:
exit status 1: can't load package: package /var/www/test.com/go: import "/var/www/test.com/go": cannot import absolute path
So in conclusion I have thought of only one solution (and I am up to hear others if mine isn't the best approach). My idea is to exclude the script file from compiling with the rest of the files, but I am unsure how to.
I did some research and couldn't find an easy way to do it (such as an --exclude flag with the go install command). Does anybody know how to accomplish what I am trying to achieve?
Thank you.
you could give the hello.go a different package name, that should work. Or i am missing something?
Regards
Tim

J language's "load" command

I'm working through the J primer, and getting stuck when it comes to the load command.
In particular, there are times when the next step in a tutorial is load 'foo' and I'll get an error like the following:
load 'plot'
not found: /users/username/j64-801/addons/graphics/plot/plot.ijs
|file name error: script
| 0!:0 y[4!:55<'y'
When I do ls /users/username/j64/addons/ I only have config and ide in there, so it's sensible that graphics is not found.
My question:
if given an example that says load 'foo', how do I go about finding and installing foo?
I'd recommend simply installing all the JAL packages ("Addons"). There aren't too many, so the download won't take long, and you'll have access to everything you need to run the Labs, Wiki examples, and any code posted by the community (e.g. on the J Forums).
To install all available Addons, type the following into Jconsole (you could theoretically type it into JHS or JQT instead, but since those are distributed as Addons, you might not be able to upgrade them while they're running):
load'pacman' NB. J PACkage MANager
install'all'
The package manager will start running, and you'll see output like:
Updating server catalog...
Installing 52 packages
Downloading base library...
Installing base library...
Downloading api/gl3...
Installing api/gl3...
Downloading api/ncurses...
Installing api/ncurses...
Then stop and restart Jconsole, and run:
load 'pacman'
'update' jpkg 'all'
To make sure all recursive dependencies were satisfied and all packages are up to date (in particular, the base library). Ultimately, you want to see something like:
Updating server catalog...
Local JAL information was last updated: <datetime>
All available packages are installed and up to date.
Then stop & restart J one last time. When that's done, you should have everything you need to run the Labs.
To answer your final question, if you see a line like:
load'foo'
The first thing you should do is run getscripts_j_ 'foo'. In your example:
getscripts_j_ 'plot'
+--------------------------------------------------------------+
|c:/users/user/j64-801/addons/graphics/plot/plot.ijs|
+--------------------------------------------------------------+
Here, you can see the fully-qualified path of where J expects the package to live.
In particular, you can see it where it is relative to the addons directory, which will always be in the form addons/category/module/foo.ijs. The category and module name indicate which addon you need to install, so all you have to do pick the desired entry from the catalog visible in the package manager.

appledoc Exception: at least one directory

After wasting some time to figure out what goes wrong, I finally have to ask for help. I want to use appledocs from Gentle Bytes. I followed every step of the quick install guide, but I´m not able to compile the project.
Here is what I´ve done:
1. cloned it from git://github.com/tomaz/appledoc.git
2. installed the templates to ~/Library/Application Support/appledoc
3. tried to compile the project
Everytime I try to compile, I get following error:
ERROR: AppledocException: At least one directory or file name path is required, use 'appledoc --help'
What do I have to do now?
Sounds like you've compiled it just fine and are now running the program. If it's a command-line program try command-option-R in Xcode to provide some arguments (i.e. names of files that you want to process).
The error means you didn't give it source paths: after all switches, you must give it at least one path to your source files. Can be either file or directory. In later case it will recursively scan the dir. Here's example
appledoc <options> ~/MyProject
Above example will use ~/MyProject directory as a source. You can also add multiple source paths. Note that you need to give the tool few options, see this page for minimum command line and other usage examples.
You either have to copy appledoc executable to one of directories in your path, as suggested by Caleb, or use full path to it when invoking (for example: /path/to/appledoc)

How to install a masked package in Gentoo 2008?

I searched the net and handbook, but I only managed to learn what is the masked package, and not how to install it. I did find some commands, but they don't seem to work on 2008 (looking at it, it seems those are for earlier versions). I have something like this:
localhost ~ # emerge flamerobin
Calculating dependencies
!!! All ebuilds that could satisfy "dev-db/flamerobin" have been masked.
!!! One of the following masked packages is required to complete your request:
- dev-db/flamerobin-0.8.6 (masked by: ~x86 keyword)
- dev-db/flamerobin-0.8.3 (masked by: ~x86 keyword)
I would like to install version 0.8.6, but don't know how? I found some instructions, but they tell me to edit or write to some files under /etc/portage. However, I don't have /etc/portage on my system:
localhost ~ # ls /etc/portage
ls: cannot access /etc/portage: No such file or directory
There are two different kinds of masks in gentoo. Keyword masks and package masks. A keyword mask means that the package is either not supported (or untested) by your architecture, or still in testing. A package mask means that the package is masked for another reason (and for most users it is not smart to unmask). The solutions are:
Add a line to /etc/portage/package.keywords (Check man portage in the package.keywords section). This is for the keyword problems.
Add a line to /etc/portage/package.unmask for "package.mask" problems (you can also use package.mask for the converse). This is in the same man file, under the section package.unmask. I advise to use versioned atoms here to avoid shooting in your own foot with really broken future versions a couple of months down the line.
These days there's also a more 'automated' solution, called "autounmask". No more file editing needed to unmask!
The great benefit of the package is, it also unmasks / handles keywords of dependencies if needed. It's provided in the package app-portage/autounmask.
/etc/portage/package.keywords and
/etc/portage/package.unmask
can be directories as well nowadays (but autounmask handles single files as well). In those directories, multiple can place multiple "autounmask" files, one file in each dir per "unmask"-package. If you use single files instead of dirs, 'autounmask' will place some kind of header / footer, and this way it becomes easy to remove "unmasks" if wanted.
Simply mkdir /etc/portage and edit as mentioned here: http://gentoo-wiki.com/TIP_Dealing_with_masked_packages#But_you_want_to_install_the_package_anyway...

Resources