Scons - How Can I Detect When a File is Pulled from Cache? - caching

I have an issue in which I need some code to run as a result of a command builder:
node = env.Command (target, dependencies, function)
In this case, function will run if the target is out of date, which is what I want, but if the target is in the cache, function doesn't run. What I'd like is to run a different function if the target is pulled from cache.
I tried:
env.AddPostAction(node, function2)
but that function doesn't get called either.
Any ideas? Thanks.

Afaik, scons will not know how it will satisfy the demand while executing your code. It makes that decision after completing the 1st pass. So, even if you could tell, I don't believe you could act on it within your code.
An easy, obvious way is to parse the scons output for 'Retrieved ... File name'.
And of course, the problem suggests the dependencies are set up badly, and it looks like you fixed that.

Related

Sourcing data into rstudio [duplicate]

This is meant to be a FAQ question, so please be as complete as possible. The answer is a community answer, so feel free to edit if you think something is missing.
This question was discussed and approved on meta.
I am using R and tried some.function but I got following error message:
Error: could not find function "some.function"
This question comes up very regularly. When you get this type of error in R, how can you solve it?
There are a few things you should check :
Did you write the name of your function correctly? Names are case sensitive.
Did you install the package that contains the function? install.packages("thePackage") (this only needs to be done once)
Did you attach that package to the workspace ?
require(thePackage) (and check its return value) or library(thePackage) (this should be done every time you start a new R session)
Are you using an older R version where this function didn't exist yet?
Are you using a different version of the specific package? This could be in either direction: functions are added and removed over time, and it's possible the code you're referencing is expecting a newer or older version of the package than what you have installed.
If you're not sure in which package that function is situated, you can do a few things.
If you're sure you installed and attached/loaded the right package, type help.search("some.function") or ??some.function to get an information box that can tell you in which package it is contained.
find and getAnywhere can also be used to locate functions.
If you have no clue about the package, you can use findFn in the sos package as explained in this answer.
RSiteSearch("some.function") or searching with rdocumentation or rseek are alternative ways to find the function.
Sometimes you need to use an older version of R, but run code created for a newer version. Newly added functions (eg hasName in R 3.4.0) won't be found then. If you use an older R version and want to use a newer function, you can use the package backports to make such functions available. You also find a list of functions that need to be backported on the git repo of backports. Keep in mind that R versions older than R3.0.0 are incompatible with packages built for R3.0.0 and later versions.
Another problem, in the presence of a NAMESPACE, is that you are trying to run an unexported function from package foo.
For example (contrived, I know, but):
> mod <- prcomp(USArrests, scale = TRUE)
> plot.prcomp(mod)
Error: could not find function "plot.prcomp"
Firstly, you shouldn't be calling S3 methods directly, but lets assume plot.prcomp was actually some useful internal function in package foo. To call such function if you know what you are doing requires the use of :::. You also need to know the namespace in which the function is found. Using getAnywhere() we find that the function is in package stats:
> getAnywhere(plot.prcomp)
A single object matching ‘plot.prcomp’ was found
It was found in the following places
registered S3 method for plot from namespace stats
namespace:stats
with value
function (x, main = deparse(substitute(x)), ...)
screeplot.default(x, main = main, ...)
<environment: namespace:stats>
So we can now call it directly using:
> stats:::plot.prcomp(mod)
I've used plot.prcomp just as an example to illustrate the purpose. In normal use you shouldn't be calling S3 methods like this. But as I said, if the function you want to call exists (it might be a hidden utility function for example), but is in a namespace, R will report that it can't find the function unless you tell it which namespace to look in.
Compare this to the following:
stats::plot.prcomp
The above fails because while stats uses plot.prcomp, it is not exported from stats as the error rightly tells us:
Error: 'plot.prcomp' is not an exported object from 'namespace:stats'
This is documented as follows:
pkg::name returns the value of the exported variable name in namespace pkg, whereas pkg:::name returns the value of the internal variable name.
I can usually resolve this problem when a computer is under my control, but it's more of a nuisance when working with a grid. When a grid is not homogenous, not all libraries may be installed, and my experience has often been that a package wasn't installed because a dependency wasn't installed. To address this, I check the following:
Is Fortran installed? (Look for 'gfortran'.) This affects several major packages in R.
Is Java installed? Are the Java class paths correct?
Check that the package was installed by the admin and available for use by the appropriate user. Sometimes users will install packages in the wrong places or run without appropriate access to the right libraries. .libPaths() is a good check.
Check ldd results for R, to be sure about shared libraries
It's good to periodically run a script that just loads every package needed and does some little test. This catches the package issue as early as possible in the workflow. This is akin to build testing or unit testing, except it's more like a smoke test to make sure that the very basic stuff works.
If packages can be stored in a network-accessible location, are they? If they cannot, is there a way to ensure consistent versions across the machines? (This may seem OT, but correct package installation includes availability of the right version.)
Is the package available for the given OS? Unfortunately, not all packages are available across platforms. This goes back to step 5. If possible, try to find a way to handle a different OS by switching to an appropriate flavor of a package or switch off the dependency in certain cases.
Having encountered this quite a bit, some of these steps become fairly routine. Although #7 might seem like a good starting point, these are listed in approximate order of the frequency that I use them.
If this occurs while you check your package (R CMD check), take a look at your NAMESPACE.
You can solve this by adding the following statement to the NAMESPACE:
exportPattern("^[^\\\\.]")
This exports everything that doesn't start with a dot ("."). This allows you to have your hidden functions, starting with a dot:
.myHiddenFunction <- function(x) cat("my hidden function")
I had the error
Error: could not find function some.function
happen when doing R CMD check of a package I was making with RStudio. I found adding
exportPattern(".")
to the NAMESPACE file did the trick. As a sidenote, I had initially configured RStudio to use ROxygen to make the documentation -- and selected the configuration where ROxygen would write my NAMESPACE file for me, which kept erasing my edits. So, in my instance I unchecked NAMESPACE from the Roxygen configuration and added exportPattern(".") to NAMESPACE to solve this error.
This error can occur even if the name of the function is valid if some mandatory arguments are missing (i.e you did not provide enough arguments).
I got this in an Rcpp context, where I wrote a C++ function with optionnal arguments, and did not provided those arguments in R. It appeared that optionnal arguments from the C++ were seen as mandatory by R. As a result, R could not find a matching function for the correct name but an incorrect number of arguments.
Rcpp Function : SEXP RcppFunction(arg1, arg2=0) {}
R Calls :
RcppFunction(0) raises the error
RcppFunction(0, 0) does not
Rdocumentation.org has a very handy search function that - among other things - lets you find functions - from all the packages on CRAN, as well as from packages from Bioconductor and GitHub.
If you are using parallelMap you'll need to export custom functions to the slave jobs, otherwise you get an error "could not find function ".
If you set a non-missing level on parallelStart the same argument should be passed to parallelExport, else you get the same error. So this should be strictly followed:
parallelStart(mode = "<your mode here>", N, level = "<task.level>")
parallelExport("<myfun>", level = "<task.level>")
You may be able to fix this error by name spacing :: the function call
comparison.cloud(colors = c("red", "green"), max.words = 100)
to
wordcloud::comparison.cloud(colors = c("red", "green"), max.words = 100)
I got the same, error, I was running version .99xxx, I checked for updates from help menu and updated My RStudio to 1.0x, then the error did not come
So simple solution, just update your R Studio

Trying to make SCons Ada Builder work with VariantDir

I'm struggling with the last pieces of logic to make our Ada builder work as expectedly with variantdir. The problem is caused by the fact that the inflexible tools gnatbind and gnatlink doesn't allow the binder files to be placed in a directory other than the current one. This leaves me with two options:
Let gnatbind write the the binder files to topdir and then let gnatlink pick it from there. This may however cause race conditions if we want to allow simulatenous builds for different architectures and compiler versions which we want.
Modify the calls to gnatbind and gnatlink to temporarily go down to the build directory, in our case build/$ARCH/src-path. I successfully fixed the gnatbind step as this is explicitly called using a env.Execute from within the Ada builder. To try to fix the linking step I've modified the Program env using
env["LINKCOM"] = SCons.Action.Action(ada_linkcom)
where ada_linkcom is defined as
def ada_linkcom(source, target,env ):
....
return ret
where ret is a string describing what should be done in the shell. I need this to be a function it contains a bit complicated logic to convert paths from being relative to top-level to just containing their basenames.
This however fails with an error in scons-2.3.1/SCons/Executor.py on line 347 in function do_execute. Isn't env["LINKCOM"] allowed to be a function with ada_linkcom's signature?
No, it's not. You seem to think that 'env["LINKCOM"]' is what actually calls/executes the final build command, and that's not quite correct. Instead, environment variables like LINKCOM get expanded by the Executor/Builder for each specified Action, and are then executed.
You can have Python functions as Actions, and also use a so-called "generator" to create your Action strings on-the-fly. But you have to assign this Action to a Builder, and can't set it as an environment variable directly.
Please also have a look at the UserGuide ( http://www.scons.org/doc/production/HTML/scons-user.html ), especially section 18.4 "Builders That Execute Python Functions". Our basic guide for writing Builders and Tools might also prove to be helpful: http://www.scons.org/wiki/ToolsForFools

llvm: is it possible to merge validation and compilation in a single stage?

Generally speaking, when writing a llvm frontend, one will take an AST and first check that its semantics is well-defined. After this, one will take the AST and perform the IR build phase.
I was wondering, how realistic is to perform directly the IR build phase onto the AST, and if errors are found during the build process, revert any partial changes to the module object?
I assume something like this would be required:
remove defined Types
remove defined Globals
anything else i'm missing?
Any ideas about this? what are the general guidelines of what needs to done for a clean revert of module changes after a failed build phase?
Now, this is thinking in terms of optimistically compiling, and failing gracefully it somethings goes wrong. It might very well be that this is completely impossible or discouraged under the current LLVM model. A clear and well-documented answer in this regard is also completely acceptable
Edit In the end, I just want a reasonable way to add functions incrementally but revert gracefully to previous state of module and/or LLVMContext if a function build fails. Whatever is the preferred approach for that will be entirely satisfactory.
thanks!
Many compilers (not necessarily LLVM-related) mix semantic analysis with code generation, so it can definitely be done. However, I'm puzzled by your reference to "revert any partial changes to the module object". When you start building an IR module and encounter a semantic error in the AST, what is your plan? Do you want to spit an incomplete module? Why? Thinking about the way any regular compiler works, if there are semantic errors in the code (i.e. reference to an undefined variable), no output is created. Would you like something different?

Inconsistency running a clojure jar from command line

I have a clojure program that at some point executes a function called db-rebuild-files-table.
This function takes a directory filename as a single string argument and calls a recursive function that descends into the directory file tree, extracts certain data from the files there and logs each file in a mysql database. The end result of this command is a "files" table populated by all files in the tree under the given directory.
What I need is to be able to run this command periodically from the shell.
So, I added the :gen-class directive in the file containing my -main function that actually calls (db-rebuild-files-table *dirname*). I run lein uberjar and generate a jar which I can then execute with:
java -jar my-project-SNAPSHOT-1.0.0-standalone.jar namespace.containing.main
Sure enough, the function runs, but in the database there only exists a single entry, for the directory *dirname*. When I execute the exact same sexp in the clojure REPL I get the right behaviour: all the file tree under *dirname* get processed.
What am I doing wrong? Why does the call (db-rebuild-files-table *dirname*) behave inconsistently when called from the REPL and when executed from the command line?
[EDIT] Whats even weirder is that I get no error anywhere. All function calls seem to work as they should. I can even run the -main function in the REPL and it updates the table correctly.
If this works in the REPL, but not when executed stand-alone, then I would guess that you may be bitten by the laziness of Clojure.
Does your code perhaps need a doseq in order to get the benefits of a side-effect (e.g. writing to your database)?
Nailed it. It was a very insidious bug in my program. I got bitten by clojure's laziness.
My file-tree function used map internally, and so produced just the first value, the root directory. For some reason I still can't figure out, when executed at the REPL, evaluation was actually forced and the whole tree seq was produced. I just added a doall in my function and it solved it.
Still trying to figure why executing something at the REPL forces evaluation though. Any thoughts?

How can I make an external toolbox available to a MATLAB Parallel Computing Toolbox job?

As a continuation of this question and the subsequent answer, does anyone know how to have a job created using the Parallel Computing Toolbox (using createJob and createTask) access external toolboxes? Is there a configuration parameter I can specify when creating the function to specify toolboxes that should be loaded?
According to this section of the documentation, one way you can do this is to specify either the 'PathDependencies' property or the 'FileDependencies' property of the job object so that it points to the functions you need the job's workers to be able to use.
You should be able to point the way to the KbCheck function in PsychToolbox, along with any other functions or directories needed for KbCheck to work properly. It would look something like this:
obj = createJob('PathDependencies',{'path_to_KbCheck',...
'path_to_other_PTB_functions'});
A few comments, based on my work troubleshooting this:
It appears that there are inconsistencies with how well nested functions and anonymous functions work with the Parallel Computation toolkit. I was unable to get them to work, while others have been able to. (Also see here.) As such, I would recommend having each function stored in it's own file, and including those files using the PathDependencies or FileDependencies properties, as described by gnovice above.
It is very hard to troubleshoot the Parallel Computation toolkit, as everything happens outside your view. Use breakpoints liberally in your code, and the inspect command is your friend. Also note that if there is an error, task objects will contain an error parameter, which in turn will contain ErrorMessage string, and possibly the Error.causes MException object. Both of these were immensely useful in debugging.
When including Psychtoolbox, you need to do it as follows. First, create a jobStartup.m file with the following lines:
PTB_path = '/Users/eliezerk/Documents/MATLAB/Psychtoolbox3/';
addpath( PTB_path );
cd( PTB_path );
SetupPsychtoolbox;
However, since the Parallel Computation toolkit can't handle any graphics functionality, running SetupPsychtoolbox as-is will actually cause your thread to crash. To avoid this, you need to edit the PsychtoolboxPostInstallRoutine function, which is called at the very end of SetupPsychtoolbox. Specifically, you want to comment out the line AssertOpenGL (line 496, as of the time of this answer; this may change in future releases).

Resources