TypeScript checks the entire codebase on transpiling, even if only one file has actually changed. For small projects, that is fine, yet since our codebase grew, it takes quite a long time.
During development, I want quick response time of my unit tests. The unit test should run as soon as possible.
Unfortunately, I have to wait on each run about 10-15 seconds for the unit test to even start as the the tsc takes a long time to transpile, and of that time 60%-80% is spent on checking.
These example runs are just from removing and adding a newline in one file:
yarn tsc v0.27.5
$ "/home/philipp/fancyProject/node_modules/.bin/tsc" "--watch" "--diagnostics"
Files: 511
Lines: 260611
Nodes: 898141
Identifiers: 323004
Symbols: 863060
Types: 302553
Memory used: 704680K
I/O read: 0.17s
I/O write: 0.09s
Parse time: 2.61s
Bind time: 0.95s
Check time: 7.65s
Emit time: 1.45s
Total time: 12.65s
00:35:34 - Compilation complete. Watching for file changes.
00:41:58 - File change detected. Starting incremental compilation...
Files: 511
Lines: 260612
Nodes: 898141
Identifiers: 323004
Symbols: 863060
Types: 302553
Memory used: 1085950K
I/O read: 0.00s
I/O write: 0.04s
Parse time: 0.68s
Bind time: 0.00s
Check time: 12.65s
Emit time: 1.36s
Total time: 14.69s
00:42:13 - Compilation complete. Watching for file changes.
00:42:17 - File change detected. Starting incremental compilation...
Files: 511
Lines: 260611
Nodes: 898141
Identifiers: 323004
Symbols: 863060
Types: 302553
Memory used: 1106446K
I/O read: 0.00s
I/O write: 0.12s
Parse time: 0.32s
Bind time: 0.01s
Check time: 9.28s
Emit time: 0.89s
Total time: 10.50s
00:42:27 - Compilation complete. Watching for file changes.
I wonder if there is a way to tell typescript:
Just treat everything as OK and just dump the JavaScript as quickly as possible to the disk.
I want to ensure first that my unit test pass in order to have a quick feedback loop.
And since my IDE takes care of the type checks already within the file I am currently working on, I rarely have mistake in the check of the transpiling anyway. And if there was a big issue, my unit tests should catch them.
When building the project, I would just use the classic tsc with the checks. As I have said, this is only for development and having a quick feedback loop.
Start using WebPack
Add awesome-typescript-loader
Set transpileOnly in setting to true
Also you can change other paramers, which can boost speed: ignoreDiagnostics, forceIsolatedModules, etc.
Related
When starting bitbake I get the following info that the dependency cache is empty and parsing all recipes takes several minutes. This is very frustrating as with every bitbake command I have to wait 6-7 minutes only for parsing the recipes.
Loading cache: 100% | ETA: --:--:--
Loaded 0 entries from dependency cache.
Parsing recipes: 1% |# | Time: 0:06:20
From time to time the cache gets loaded correctly. So caching seems to work in principal, I just could not find out why it is not loaded most of the time.
I already tried to get rid of .lock files in the build directory, but this also did not solve the problem.
I found out that the bb_cache.dat file that is loaded does not exist. E.g it tries to load bb_cache.dat -> bb_cache.dat.8766c4ab6f5e02381cb595498695990e54b0e58d7e7aed06cfdf517975 which is not existing. Then it parses the recipies and at the end it generates this file that was missing before. Next time when starting bitbake it either loads the previously generated one or (most of the time) looks for a new file and does not find it again. And so on...
Are you changing any of the configuration between bitbake invocations? Are you updating any of the repositories or layers?
It will reparse only when some configuration file has been changed, be that a local.conf file, a bblayers.conf file, or a configuration file in one of the checkouts. "bitbake -D" which enables debug output might give some hints as to why it is reparsing too.
Sometimes when I run this command npx hardhat compile on my windows cli I get the error below:
Compiling 72 files with 0.7.0
contracts/libraries/ERC20.sol: Warning: SPDX license identifier not provided in source file. Before publishing, consider adding a comment containing "SPDX-License-Identifier: <SPDX-License>" to each source file. Use "SPDX-License-Identifier: UNLICENSED" for non-open-source code.
Please see https://spdx.org for more information.
contracts/libraries/ERC1155/EnumerableSet.sol:158:5: Warning: Variable is shadowed in inline assembly by an instruction of the same name
function add(Bytes32Set storage set, bytes32 value) internal returns (bool) {
^ (Relevant source part starts here and spans across multiple lines).
contracts/libraries/ERC1155/EnumerableSet.sol:224:5: Warning: Variable is shadowed in inline assembly by an instruction of the same name
function add(AddressSet storage set, address value) internal returns (bool) {
^ (Relevant source part starts here and spans across multiple lines).
Compiling 1 file with 0.8.0
<--- Last few GCs --->
[8432:042B0460] 263058 ms: Mark-sweep (reduce) 349.8 (356.3) -> 248.2 (262.4) MB, 434.4 / 0.2 ms (+ 70.9 ms in 3 steps since start of marking, biggest step 69.6 ms, walltime since start of marking 800 ms) (average mu = 0.989, current mu = 0.990) memory[8432:042B0460] 263627
ms: Mark-sweep (reduce) 248.2 (259.4) -> 248.2 (252.1) MB, 555.5 / 0.0 ms (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 556 ms) (average mu = 0.969, current
mu = 0.023) memory p
<--- JS stacktrace --->
FATAL ERROR: NewNativeModule Allocation failed - process out of memory`
After some time the error just kind of goes away.
It goes away probably after I've restarted my system or created a new Hardhat project and imported the code there.
But this is happening too often, what could be the cause?
I've done quite some research and some answers suggested it might be a problem with Node and the application's memory allocation, but I don't know how I would apply the solutions to a Hardhat project.
Here is a link to one possible solution: https://medium.com/#vuongtran/how-to-solve-process-out-of-memory-in-node-js-5f0de8f8464c
OS: WINDOWS 10
CLI: WINDOWS CMD
Issue
I have a go package, with a test suite.
When I run the test suite for this package, the total runtime is ~ 7 seconds :
$ go test ./mydbpackage/ -count 1
ok mymodule/mydbpackage 7.253s
However, when I add a -cpuprofile=cpu.out option, the sampling does not cover the whole run :
$ go test ./mydbpackage/ -count 1 -cpuprofile=cpu.out
ok mymodule/mydbpackage 7.029s
$ go tool pprof -text -cum cpu.out
File: mydbpackage.test
Type: cpu
Time: Aug 6, 2020 at 9:42am (CEST)
Duration: 5.22s, Total samples = 780ms (14.95%) # <--- depending on the runs, I get 400ms to 1s
Showing nodes accounting for 780ms, 100% of 780ms total
flat flat% sum% cum cum%
0 0% 0% 440ms 56.41% testing.tRunner
10ms 1.28% 1.28% 220ms 28.21% database/sql.withLock
10ms 1.28% 2.56% 180ms 23.08% runtime.findrunnable
0 0% 2.56% 180ms 23.08% runtime.mcall
...
Looking at the collected samples :
# sample from another run :
$ go tool pprof -traces cpu.out | grep "ms " # get the first line of each sample
10ms runtime.nanotime
10ms fmt.(*readRune).ReadRune
30ms syscall.Syscall
10ms runtime.scanobject
10ms runtime.gentraceback
...
# 98 samples collected, for a total sum of 1.12s
The issue I see is : for some reason, the sampling profiler stops collecting samples, or is blocked/slowed down at some point.
Context
go version is 1.14.6, platform is linux/amd64
$ go version
go version go1.14.6 linux/amd64
This package contains code that interact with a database, and the tests are run against a live postgresql server.
One thing I tried : t.Skip() internally calls runtime.Goexit(), so I replaced calls to t.Skip and variants with a simple return ; but it didn't change the outcome.
Question
Why aren't more samples collected ? I there some known pattern that blocks/slows down the sampler, or terminates the sampler earlier than it should ?
#Volker guided me to the answer in his comments :
-cpuprofile creates a profile in which only goroutines actively using the CPU are sampled.
In my use case : my go code spends a lot of time waiting for the answers of the postgresql server.
Generating a trace using go test -trace=trace.out, and then extracting a network blocking profile using go tool trace -pprof=net trace.out > network.out yielded much more relevant information.
For reference, on top of opening the complete trace using go tool trace trace.out, here are the values you can pass to -pprof= :
from go tool trace docs :
net: network blocking profile
sync: synchronization blocking profile
syscall: syscall blocking profile
sched: scheduler latency profile
I'm trying to run those Flink Benchmarks:
https://github.com/dataArtisans/flink-benchmarks
I've generated the jar file using maven with that command:
mvn clean package -Pbuild-jar
Then I'm trying to run the benchmark on a Flink Cluster with that command:
./bin/flink run -c org.apache.flink.benchmark.WindowBenchmarks ~/flinkBenchmarks/target/flink-hackathon-benchmarks-0.1.jar
I've used the -c option to add to the classpath the Main of the benchmark (WindowBenchmarks) I want to run.
Finally, I get that error:
# JMH version: 1.19
# VM version: JDK 1.8.0_151, VM 25.151-b12
# VM invoker: /usr/lib/jvm/java-8-oracle/jre/bin/java
# VM options: -Dlog.file=/home/user/flink-1.3.2/flink-dist/target/flink-1.3.2-bin/flink-1.3.2/log/flink-user-client-mypc.log -Dlog4j.configuration=file:/home/user/flink-1.3.2/flink-dist/target/flink-1.3.2-bin/flink-1.3.2/conf/log4j-cli.properties -Dlogback.configurationFile=file:/home/user/flink-1.3.2/flink-dist/target/flink-1.3.2-bin/flink-1.3.2/conf/logback.xml -Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
# Warmup: 10 iterations, 1 s each
# Measurement: 10 iterations, 1 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.flink.benchmark.WindowBenchmarks.sessionWindow
# Run progress: 0.00% complete, ETA 00:04:00
# Fork: 1 of 3
Error: Could not find or load main class org.openjdk.jmh.runner.ForkedMain
<forked VM failed with exit code 1>
<stdout last='20 lines'>
</stdout>
<stderr last='20 lines'>
Error: Could not find or load main class org.openjdk.jmh.runner.ForkedMain
</stderr>
# Run complete. Total time: 00:00:00
Benchmark Mode Cnt Score Error Units
The program didn't contain a Flink job. Perhaps you forgot to call execute() on the execution environment.
I don't have any previous experience with Flink and Maven so I find out what is missing. My first thought was that it's a missing dependencies error, but they look fine. Any suggestions?
Thank you in advance!
flink-benchmarks is a repository that contains sets of micro benchmarks designed to run on single machine, not on the cluster. The main functions defined in the various classes (test cases) are 'JMH' runners, not Flink programs. As such you can either execute whole benchmark suite (which takes ~1hour):
mvn -Dflink.version=1.5.0 clean install exec:exec
or if you want to execute just one benchmark, the best approach is to execute selected main function manually. For example from your IDE (don't forget about selecting flink.version, default value for the property is defined in pom.xml).
There is also a possibility to execute single benchmark from console, but I haven't tried it for very long time.
I am using travis-ci and biicode to build my project who is depending on boost log. But boost log times are longer than 10 min so I get this message:
No output has been received in the last 10 minutes, this potentially indicates a
stalled build or something wrong with the build itself.
The build has been terminated
The build is working correctly, it's just that boost log is really long to compile with limited resources (I tried to compile it on a VM with 1 CPU and 2GB of RAM and it took almost more than 15 min)
I know this is happening because there is not enough verbose going on so I tried the following flags:
>bii cpp:build -- VERBOSE=1
In the CMakeList.txt, set BII_BOOST_VERBOSE ON as mentionnened here
Set BOOST_LOG_COMPILE_FAST_ON as explained here
Using travis_wait
Actually travis_wait seems to be the correct solution but when I put it in my .travis.yml like this
script: travis_wait bii cpp:build
It does actually doesn't output logs like usually and just time out after 20 min. I don't think the actual building is taking place
What is the correct way to handle this problem?
This is a known issue, Boost.Log takes a long time to compile.
You can use travis_wait to call bii cpp:configure, but I'm with you, I need log feedback (No pun intended). However, I have tried that too and leaded to >50min build, which means travis aborts build on free accounts :( Of course my repo does not build Boost.Log only.
Just as a note, here's part of the settings.py file from the boost-biicode repo:
#Boost.Log takes so much time to compile, leads to timeouts on Travis CI
#It was tested on Windows and linux, works 'ok' (Be careful with linking settings)
if args.ci: del packages['examples/boost-log']
I'm currently working on a solution, launching asynchronous builds while printing progress. Check this issue. It will be ready for this week :)
To speed-up your build, try to play with BII_BOOST_BUILD_J variable to set the number of threads you want for building Boost components. Here's an example:
script:
- bii cpp:configure -DBII_BOOST_BUILD_J=4
Be careful, more threads means more RAM needed to compile at a time. Be sure you don't make the travis job VM go out of memory.