I have a project consisting of several packages in a directory tree. If I change one of the packages then stack does usually automatically recompiles packages which depend on the changed one (sometimes the automatism fails and i have no hint when - but this is not the focus of the question).
during the recompilation, i see often a recompilation of all the packages listed as extra-deps. these have not changed and recompilation is unnecessary (but time consuming). I understand that there is some mechanism to compile such dependencies only once (as long as they are not changing) and to be reused. i saw once a message from stack saying that some previous compilation of a package was reused pointing to this caching.
i was asked in a comment to provide the stack.yaml in the project directory (I tried also with entries like
- location: /home/frank/Workspace8/repo8/litMain
extra-dep: true
but did not see a difference):
flags: {}
extra-package-dbs: []
packages:
- litMain
- rdf4store
- rdfhextension
- uniform/uniform-strings
- uniform/uniform-error
- uniform/uniform-algebras
- uniform/uniform-convenience
- uniform/uniform-fileio
- uniform/uniform-time
extra-deps:
- chatter-0.8.0.1
- data-easy-0.7.0
- fullstop-0.1.4
- hgal-2.0.0.2
- non-negative-0.1.1
- numeric-prelude-0.4.2
- rdf4h-3.0.1
- test-invariant-0.4.5.0
- tokenize-0.3.0
resolver: lts-8.2
How to achieve the reuse of previously compiled and not changed extra-deps in general?
I tried build, build --dependencies-only and build --snapshot but did not succeed. All the dependencies are rebuilt... i could not find more information in the documentation.
Helpful comments made me rethink and edit the question. I hope the problem is now better exposed. Thank you for help!
You are right that a change in a local package shouldn't cause recompilation of any extra-deps as long as the extra-deps don't depend on any local packages. (Local packages are those that are listed in the extra-deps section.)
Usually my extra-deps are only recompiled when I upgrade stack or change the resolver.
This indicates that there might be a bug in stack's behavior, so it would be great if you could make a bug report for stack the next time you see this redundant recompilation again!
Related
I have been using Clojure, ClojureScript, lein, shadow-cljs, re-frame, reagent, Emacs, and CIDER to work on a Clojure/ClojureScript dynamic web app project.
Usually, I build the project by executing shadow-cljs watch app. It works fine. I can use the application and watch changes.
Currently, I am working on a Continuous Integration project via GitHub Actions. In this new environment, I wanna use a different command: shadow-cljs compile app.
But, I am having problems. Both on GitHub Actions env and in my local env (which is reproducing the steps on GitHub Actions).
The result returned by the command is quite weird. First, it says the compilation went well, Build completed, and the terminal displays some non-harmful warnings:
shadow-cljs compile app
shadow-cljs - config: /Users/pedro/projects/my_project/shadow-cljs.edn
WARNING: random-uuid already refers to: #'clojure.core/random-uuid in namespace: portal.runtime.browser, being replaced by: #'portal.runtime.browser/random-uuid
[:app] Compiling ...
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
WARNING: abs already refers to: #'clojure.core/abs in namespace: day8.re-frame-10x.inlined-deps.garden.v1v3v10.garden.color, being replaced by: #'day8.re-frame-10x.inlined-deps.garden.v1v3v10.garden.color/abs
WARNING: abs already refers to: #'clojure.core/abs in namespace: garden.color, being replaced by: #'garden.color/abs
[:app] Build completed. (1121 files, 27 compiled, 8 warnings, 45.85s)
------ WARNING #1 - :redef -----------------------------------------------------
(... omitted ...)
Unfortunately, the terminal stays like this forever. It behaves like the watch command... But, compile is not supposed to be like this.
The terminal does not finish the process. There is no exit. Even though the build is complete, as stated in the last line before WARNING.
Feels like a deadlock situation. The terminal is still responsive because I can cancel the process with Control-c. But, besides canceling it, it is frozen. This can be catastrophic on GitHub actions since you are going to be paying for an ongoing process.
On shadow-clj.edn, I tried commenting out compilation-options:
;:compiler-options {:optimizations :none}
But it did not make any difference. The same problem happens with shadow-cljs release app.
After finding a GitHub issue with a similar problem, I also tried:
:js-options {:resolve {"highlight.js" {:target :npm :require "highlight.js/lib/core"}}}
One of my hypotheses is that highlight.js is causing some trouble while parsing strings and/or files...
I am also afraid the problem could be some REPL being fired up by some dependency and blocking the process for the exit... But I am not sure where to look for it.
Why is this happening? What could be causing this? How can I solve it?
;;;;
UPDATE:
After #thomasheller kind answer, I tried some things. I believe all of the hints can be discarded for the present situation, except for the macros:
1 - The project does not use user.clj
2 - Build hooks is the default setting which is basically a comment:
:build-hooks [;; this will create a build report for every release build
;; which includes a detailed breakdown of the included sources
;; and how much they each contributed to the overall size:
#_(shadow.cljs.build-report/hook
{:output-to "build-reports/report.html"})]
3- There is no shadow-cljs start, server, or watch process going before trying to compile. The problem of an endless compile app happens just after re-starting my MacBook, on the CI, in others' people's PC... So, it is unrelated.
Ok. Now, let's talk about the macros... The project's main repository has 6 defmacros - according to a git grep (I have not checked the dependencies).
I am suspicious about one of the macros which is involved in reading files:
(defmacro slurp [file & [default]]
(if (.exists (io/file file))
(clojure.core/slurp file)
default))
Why does this exist? Well, it is being used inside another macro that reads config files related to environment variables:
(defmacro read-open-config [env-var]
(clojure.edn/read-string (slurp (str "config/" (System/getenv env-var) "-open.edn"))))
We have multiple config files for different purposes (I know it is not the standard practice...).
It feels like too much dynamicity...
First of all a few clarifications.
compile produces a development build, no optimizations are applied. Setting anything related to that does nothing.
CI systems should likely use release. This avoid including all the development related code (eg. highlight.js, re-frame-10x).
A dependency cannot "fire up a REPL". The shadow-cljs "server mode" provides it, which compile or release don't normally enter.
highlight.js errors would fail earlier with a visible error, you would not get to "Build completed."
This could be happening for a variety of reasons, very hard to debug without seeing the build config/setup. A few guesses:
You have a user.clj on the classpath, which starts additional "stuff" when the process is launched and this namespace is loaded? Clojure will load this unconditionally and shadow-cljs cannot prevent whatever that may do.
You have configured :build-hooks that launches additional stuff. It could launch an addition css watch process that doesn't exit?
You use a macro that launches additional stuff without shutting it down?
You previously started a shadow-cljs start, server or watch process? Coupled with the above that may then wait for the server to stop. On your local machine is the watch maybe still running? You can prevent the attempted server connect via npx shadow-cljs release build --force-spawn, but that is only useful if it is actually running.
Normally shadow-cljs will shut down after compile or release. I'm not aware of anyone ever having any issues with this after a successful build. The issue you linked is not related to yours, since that never got to a completed build.
After a long research process, it was discovered that the problem came from tests inside a private dependency. Apparently, some tests with asynchronous processes profile were holding the compilation process from finishing.
After commenting-out some olds tests, the compilation happened as expected.
This seems to be a tricky question, also because it is difficult to present here all settings involved. So let me state the total picture:
I use Visual Studio 2022 to build a solution (all in x64 mode) comprising of two projects
CapopConsole
and
CapopWrap
CapopConsole is a C# Console-Project generating an exe and CapopWrap is a C++/CLI project wrapping a C++ native library called Capop. It generates a dll called CapopWrap.dll. CapopConsole has CapopWrap as dependency.
CapopWrap has three extra .lib dependencies, namely
capoplib.lib, libpredicates.lib, jsoncpp.lib
in Release mode and
capoplib.lib, libpredicates_d.lib, jsoncppd.lib
in Debug mode.
The first two of each row are from a certain project, special for me, the last one is from the well known jsoncpp project. All three libs are built with Visual Studio 2022 with the new CMake mode (no 'solutions').
Now the point is: Compiling and linking CapopWrap produces both for release and debug mode respectively a result dll, in both cases called CapopWrap.dll.
But: CapopWrap.dll from debug mode has a dependency on jsoncppd.dll whereas CapopWrap.dll from release mode has no explicit jsoncpp related dependency whatsoever (as shown with dumpbin, for example).
By symmetry one might expect that CapopWrap.dll (Release) would incur a dependency on jsoncpp.dll but this is not the case.
I would really be happy, if someone could help me find an explanation for this - my own attempts so far where fruitless.
Addendum: I do not give concrete excerpts from my .csproj and cmake files at first, because they are a bit lengthy and maybe an answer is possible without them. If not, please tell me, what you want to see concretely from my configuration files (or filesystem contents), I will edit it in then.
I am trying to build GridDB from source code on MacOS as recommended How to setup GridDB on macOS with instructions https://docs.griddb.net/gettingstarted/using-source-code/#build-a-server-and-client-java.
During the automake step, I get the following warning(s) for many source files:
3rd_party/MessagePack/Makefile.am:8: warning: source file '$(srcdir)/src/objectc.c' is in a subdirectory,
3rd_party/MessagePack/Makefile.am:8: but option 'subdir-objects' is disabled
Has anyone else had this? Any advice of how to make solve this warning? Is it critical in the make process?
There is a subtle difference between a warning and an error.
Both indicate problems, but while an error indicates a fatal problem that requires a solution right now, a warning is just that: a hint that there's a potential problem.
So: there's no need to solve anything right now.
As a user (of GridDB), you (normally) can ignore any such warning.
However, the maintainers (of GridDB) should look into that, as the warning might turn into an error as the world outside evolves.
Having said that, afaict not using subdir-objects with sources in subdirectories is not a problem per se, and some might prefer it that way. automake tries to push towards subdir-objects, as it fixes problems if you have multiple files with the same name in different directories. However, this (multiple same-named files in different directories) is not the case here.
I am working on a few libraries for coding Arduinos in Ada. Each library is its own project, and I have an aggregate project that aggregates the libraries. I need to specify the runtime for each project since they are running on different chips. So for example I have something like this:
aggregate project Agg is
for Project_Files use ("due/arduino_due.gpr",
"uno/arduino_uno.gpr",
"nano/arduino_nano.gpr");
-- ...
end Agg;
library project Arduino_Due is
-- Library_Dir, _Name, and _Kind attributes ...
-- Target attribute ...
for Runtime ("Ada") use "../runtimes/arduino_due_runtime";
package Compiler is
-- Driver and Switches attributes ...
end Compiler;
And similar projects for the Uno and Nano. Building arduino_due.gpr directly works fine. It finds my runtime in the specified folder as it should. However, when I build agg.gpr, I get
fatal error, run-time library not installed correctly
cannot locate file system.ads
This occurs whether I use an absolute path or a relative path, and also occurs when the relative path is concatenated with Project'Project_Dir. However, if rather than using the Runtime attribute I use the compiler switch --RTS=..., then it works, but only if I use a relative path that is prefixed with Project'Project_Dir. An absolute path or a plain relative path will result in the error gprbuild: invalid runtime directory runtimes/arduino_due_runtime.
So what's going on here? This behavior seems inconsistent and I couldn't find anything in the docs about it so I suspect a bug. But I thought I'd ask here first in case I'm doing something wrong. Maybe I should just be using child projects, or project extension?
This isn’t a bug, it’s a feature :-).
See this rejected issue.
There are two things:
Several options are only recognised in the main project, and if you use an aggregate project that is the main project.
Package Builder is ignored in aggregated projects.
My conclusion: aggregate projects don’t suit your use case, or mine. As I said in the issue noted above, back to Makefiles (or scripts).
Part of the design intent is that aggregate projects should share code and compilations: as 2.8.4 of the manual says,
The loading of aggregate projects is optimized in GPRbuild, so that all files are searched for only once on the disk (thus reducing the number of system calls and yielding faster compilation times, especially on systems with sources on remote servers). As part of the loading, GPRbuild computes how and where a source file should be compiled, and even if it is located several times in the aggregated projects it will be compiled only once.
Since there is no ambiguity as to which switches should be used, files can be compiled in parallel (through the usual -j switch) and this can be done while maximizing the use of CPUs (compared to launching multiple GPRbuild commands in parallel).
As of May 2016, does ghc-mod support projects compiled with ghcjs?
I realise that in a lot of cases, the source is the same, but this is not the case when working with ghcjs-dom library.
I currently have 2 project folders, configured with stack. From outside these folders:
ghc-mod check folder1/executable/Main.hs
works perfectly. The stack.yaml file here specifies ghc-7.10.2
ghc-mod check folder2/executable/Main.hs
fails. The stack.yaml file here specifies ghcjs-0.2.0.0_ghc-7.10.2
Is this expected behaviour? I can't tell frrm the feature requests and wikis what the current state is.
Not currently - https://github.com/ghcjs/ghcjs/issues/271
Note that this is actually pretty tricky as ghc-mod would need to be linked with ghcjs. As far as I know ghcjsi is the only such tool to work with ghcjs