I have a script which collected together a number of files to be installed. This includes a number of empty directories.
Previously I would use the D flag in the manifest file which would copy empty directories. However due to the way I generate the manifest files (as part of our build process) I can sometimes end up with two D entries with the same destination folder. e.g:
D;${A_LIB}/all/pysys/${PYSYS_VERSION}/lib/python2.7/site-packages;./third_party/python/lib/python2.7/site-packages;COMMON;${UNIX}
D;${A_LIB_BT}/python/${PYTHON_VERSION};./third_party/python;COMMON;${ALL}
This causes InstallAnywhere to fail to build the installer.
To get around this I rewrote the manifest generation code to parse the directories previously pointed to by a D and replace the D entry with F entries for each file in the directory.
Unfortunately this will not include empty directories (which we may / may not need in the installer but in general it's just safer to create them than have some piece of code fail because they're not there).
I've tried the following in the manifest. Reference, Reference3 and Reference4 are empty, Reference2 contains a single directory (which is itself empty). Only Reference2 is present in the install - the other three which are empty directories seem to get excluded.
D,$IA_PROJECT_DIR$/samples/pysys/cor_002/Reference,./samples/pysys/cor_002/Reference
D,$IA_PROJECT_DIR$/samples/pysys/cor_002/Reference2,./samples/pysys/cor_002/Reference2
D,$IA_PROJECT_DIR$/samples/pysys/cor_002/Reference3/.,./samples/pysys/cor_002/Reference3/.
D,$IA_PROJECT_DIR$/samples/pysys/cor_002/Reference4/../Reference4,./samples/pysys/cor_002/Reference4/../Reference4
I've also tried increasing the log level but this has not revealed anything. Is there a way to increase this log level?
export LAX_DEBUG=true
Any suggestions?
DISCLAIMER: I've cross posted this to InstallAnywhere's forums but I will do my best to keep the answers in sync and spread the knowledge.
I can't speak to your manifest challenges. However, my first thought is to change the manifest generator to be sensitive to duplicate output locations -- maybe by storing them in a Map or Set -- and then handling collisions when they occur by failing the build or adjusting the output location(s).
On the other hand, I can tell you how to increase the verbosity of your installer.
Make the installer more verbose by adding:
-Dlax.debug.all=true -Dlax.debug.level=3
to Project > JVM Settings > Installer Settings (tab) > Optional Installer Arguments > Additional Arguments. You'll want to remove these before you ship. You can also add these to the command line when you start the installer. Level values of 4 and 5 work, too, and are even more verbose.
You can also make your installer print its progress to the console by going to Project > JVM Settings > Log Settings. Here, uncheck Include debug output (stderr and stdout). Then enter the word console in Send stderr to: and Send stdout to:. Rather than console, you can also set a specific file name. You'll also want to undo these settings before you ship.
The solution turns out to be so blindingly simple that I never tried it.
To get EMPTY directories installed by Install Anywhere you have to specify the directories as files in the manifest. So with the following directory structure:
Reference <empty>
Reference2
testdir <empty>
Reference3 <empty>
Reference4 <empty>
You need to specify the entries in the manifest as F. Specifying then as D will only result in the "Reference2" directory being included.
F,$IA_PROJECT_DIR$/samples/pysys/cor_002/Reference,./samples/pysys/cor_002/Reference
F,$IA_PROJECT_DIR$/samples/pysys/cor_002/Reference2,./samples/pysys/cor_002/Reference2
F,$IA_PROJECT_DIR$/samples/pysys/cor_002/Reference3/.,./samples/pysys/cor_002/Reference3/.
F,$IA_PROJECT_DIR$/samples/pysys/cor_002/Reference4/../Reference4,./samples/pysys/cor_002/Reference4/../Reference4
Sorry to answer my own question, really wasn't the plan!
Related
I am recovering Stata following a Windows upgrade. I have a list of my packages generated from ado dir in the following format:
[1] package mdesc from http://fmwww.bc.edu/RePEc/bocode/m
'MDESC': module to tabulate prevalence of missing values
[2] package univar from http://fmwww.bc.edu/RePEc/bocode/u
'UNIVAR': module to generate univariate summary with box-and-whiskers plot
[3] package tabmiss from http://www.ats.ucla.edu/stat/stata/ado/analysis
tabmiss. Shows tabulation of number of missing and non-missing values
I have many packages and would like to reinstall them without having to designate each directory/url via net cd. While using net cd along with net install or ssc install along with package names in a loop is trivial (as below), it would seem that an automated method for this task might be available.
net cd http://www.ats.ucla.edu/stat/stata/ado/analysis
local ucla tabmiss csgof powerlog ldfbeta
foreach x of local ucla {
net install `x'
}
To my knowledge, there is no built-in or automated method of tracking and managing your installed packages outside of what is available through ado or net.
I would also tend to agree with #Nick Cox that this task seems strange and I can't imagine how a new Stata install or reinstall could know what was installed previously, but I find the question interesting for other reasons.
The main reason being for users who have Stata installed on multiple machines who need the same packages on both machines. I faced a similar issue when I purchased a new computer and installed Stata but wanted all of the packages I use to be available as well. Outside of moving the ado directory or selected contents I'm not aware of any quick solution.
Here it would be possible to use the output of ado dir on one machine to determine what you need to install on a second machine with a new Stata install.
The method you propose using a foreach loop could save you time from having to type in or copy/paste a lot of packages and URLs. At the same time however, this is only beneficial if you have many packages from only a few repositories because you will need to net cd to the URL each time as you show in your example.
An alternative solution is the programmatic solution. As you know, ado dir will list each installed package, the URL and a short description of the package. Using this, a log file, and the built in I/O functionality, a short program could be written to automate the process and dynamically build a do file that contains the commands to install the already installed packages.
The code below generates a do file containing commands (in this case, net describe package, from(url)) for each package I have installed on my computer.
clear *
tempfile log1
log using "`log1'", text name(mylog)
ado dir
log close mylog
tempname logfile
file open `logfile' using "`log1'", read
file read `logfile' line
file open dfh using "path/to/your/dofile.do", write replace
local pckage "package"
while r(eof) == 0 {
if `: list pckage in line' {
local packageName : word 3 of `line'
local dirName : word 5 of `line'
di "`packageName' `dirName'"
file write dfh "net describe `packageName', from(`dirName')"
file write dfh _newline
}
file read `logfile' line
}
file close `logfile'
file close dfh
In the above code, I create a temp file to write a .txt log file to and store the contents of ado dir in that file.
Then, I open the log file using file open and read it line by line in the while loop.
Above the loop, I'm creating a do file at /path/to/your/dofile.do to hold the output of the loop - the dynamically created commands relating to the installed packages on my machine.
The loop will iterate so long as r(eof) = 0, where r(eof) is an end of file marker. I use an if statement to sort out lines of the log file which contain the word package, as I'm only interested in those lines with the package name and URL in them.
Inside of the if block, I parse the local macro line to pull the package name and the URL/directory name.
this is important: this section of code assumes that the 3rd and 5th words in the macro will always be the package name and URL respectively - Confirm this from the output of ado dir before executing.
You will also need to change the command that is being written to the file handle dfh inside of the loop to what you want (net install, etc) when you are ready to execute.
For more help on using file, locals, and tempfiles execute any of the following in Stata:
help file
help extended_fcn
help macrolists
There may be nicer ways to parse the contents of ado dir but this has worked for me. And of course I'd always advise that you take the time to understand what the code is doing so that you can make any necessary tweaks to fit your particular situation.
Is it possible to tell ASDF that it should produce only one fas(l) file for entire system? This file should be concatenation (in right order) of all compiled files of the system, including all files of systems on which target system depends.
Yes, with compile-bundle-op (ASDF 3.1): http://common-lisp.net/project/asdf/asdf/Predefined-operations-of-ASDF.html
edit: Actually, monolithic-compile-bundle-op seemes to be asked for (as shown in other answers).
If you have to predict the extension, use uiop:compile-file-type.
And/or you can just call (asdf:output-files 'asdf:monolithic-compile-bundle-op :my-system) to figure out what is actually used.
Option monolithic-compile-bundle-op will create single compiled file which includes all dependencies, while compile-bundle-op creates a file for every system.
Example of use:
(asdf:operate 'asdf:monolithic-compile-bundle-op :my-system)
This command will create file my-system--all-systems.fas(l) in output directory of target project, as well as "bundle" files for every system, named like my-system--system.fas(l).
I have a project that contains a folder to manage file templates, but it doesn't look like Go provides any support for non-Go-code project files. The project itself compiles to an executable, but it needs to know where this template folder is in order to operate correctly. Right now I do a search for $GOPATH/src/<templates>/templates, but this feels like kind of a hack to me because it would break if I decided to rename the package or host it somewhere else.
I've done some searching and it looks like a number of people are interested in being able to "compile" the asset files by embedding them in the final binary, but I'm not sure how I feel about this approach.
Any ideas?
Either pick a path (or a list of paths) that users are expected to put the supporting data in (/usr/local/share/myapp, ...) or just compile it into the binary.
It depends on how you are planning to distribute the program. As a package? With an installer?
Most of my programs I enjoy just having a single file to deploy and I just have a few templates to include, so I do that.
I have an example using go-bindata where I build the html template with a Makefile, but if I build with the 'devel' flag it will read the file at runtime instead to make development easier.
I can think of two options, use a cwd flag, or infer from cwd and arg 0:
-cwd path/to/assets
path/to/exe -cwd=$(path/to/exe/assets)
Internally, the exectable would chdir to wherever cwd points to, and then it can use relative paths throughout the application. This has the added benefit that the user can change the assets without having to recompile the program.
I do this for config files. Basically the order goes:
process cmd arguments, looking for a -cwd variable (it defaults to empty)
chdir to -cwd
parse config file
reparse cmd arguments, overwriting the settings in the config file
I'm not sure how many arguments your app has, but I've found this to be very useful, especially since Go doesn't have a standard packaging tool that will compile these assets in.
infer from arg 0
Another option is to use the first argument and get the path to the executable. Something like this:
here := path.Dir(os.Args[0])
if !path.IsAbs(os.Args[0]) {
here = path.Join(os.Getwd(), here)
}
This will get you the path to where the executable is. If you're guaranteed the user won't move this without moving the rest of your assets, you can use this, but I find it much more flexible to use the above -cwd idea, because then the user can place the executable anywhere on their system and just point it to the assets.
The best option would probably be a mixture of the two. If the user doesn't supply a -cwd flag, they probably haven't moved anything, so infer from arg 0 and the cwd. The cwd flag overrides this.
I'm using Sphinx on Windows.
Most of my documentation is for regular users, but there are some sub-pages with content for administrators only.
So I want to build two versions of my documentation: a complete version, and a second version with the "admin" pages excluded.
I used the exclude_patterns in the build configuration for that.
So far, it works. Every file in every subfolder whose name contains "admin" is ignored when I put this into the conf.py file:
exclude_patterns = ['**/*admin*']
The problem is that I'd like to run the build once to get both versions.
What I'm trying to do right now is running make.bat twice and supply different parameters on each run.
According to the documentation, I can achieve this by setting the BUILDDIR and SPHINXOPTS variables.
So now I have a build.bat that looks like this:
path=%path%;c:\python27\scripts
rem BUILD ADMIN DOCS
set SPHINXOPTS=
set BUILDDIR=c:\build\admin
call make clean
call make html
rem BUILD USER DOCS
set SPHINXOPTS=-D exclude_patterns=['**/*admin*']
set BUILDDIR=c:\build\user
call make clean
call make html
pause
The build in the two different directories works when I delete the line set BUILDDIR=build from the sphinx-generated make.bat file.
However, the exclude pattern does not work.
The batch file listed above outputs this for the second build (the one with the exclude pattern):
Making output directory...
Running Sphinx v1.1.3
loading translations [de]... done
loading pickled environment... not yet created
Exception occurred:
File "C:\Python27\lib\site-packages\sphinx-1.1.3-py2.7.egg\sphinx\environment.
py", line 495, in find_files
['**/' + d for d in config.exclude_dirnames] +
TypeError: coercing to Unicode: need string or buffer, list found
The full traceback has been saved in c:\users\myusername\appdata\local\temp\sphinx-err-kmihxk.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
Either send bugs to the mailing list at <http://groups.google.com/group/sphinx-dev/>,
or report them in the tracker at <http://bitbucket.org/birkenfeld/sphinx/issues/>.
What am I doing wrong?
Is the syntax for exclude_patterns in the sphinx-build command line different than in the conf.py file?
Or is there a better way to build two different versions in one step?
My first thought was that this was a quoting issue, quoting being notoriously difficult to get right on the Windows command line. However, I wasn't able to come up with any combination of quoting that changed the behavior at all. (The problem is easy to replicate)
Of course it could still just be some quoting issue I'm not smart enough to figure out, but I suspect this is a Sphinx bug of some kind, and hope you will report it to the Sphinx developers.
In the meantime, here's an alternate solution:
quoting from here:
There is a special object named tags available in the config file. It can be used to query and change the tags (see Including content based on tags). Use tags.has('tag') to query, tags.add('tag') and tags.remove('tag') to change
This allows you to essentially pass flags into the conf.py file from the command line, and since the conf.py file is just Python, you can use if statements to set the value of exclude_patterns conditionally based on the tags you pass in.
For example, you could pass Sphinx options like:
set SPHINXOPTS=-t foradmins
to pass the "foradmins" tag, and then check for it in your conf.py like so:
exclude_patterns = blah
if tags.has('foradmins'):
exclude_patterns = []
That should allow you to do what you want. Good Luck!
A Day with Winrar
All I wanted to do was exclude folders and their contents using wildcards, and even after reading the docs, it turned into a guessing game...
So my test bed looks like:
C:\!tmp1\f1
C:\!tmp1\f1\f1.txt
C:\!tmp1\f1\a
C:\!tmp1\f1\a\a.txt
C:\!tmp1\f2
C:\!tmp1\f2\f2.txt
C:\!tmp1\f2\a
C:\!tmp1\f2\a\a.txt
And I am executing:
C:\>"c:\program files\winrar\winrar.exe" a -r !tmp1.rar !tmp1
which gives me a rar with !tmp1 as the root (sole top level folder).
The exclude switch is -x<filepathpattern> and may be included multiple times.
So, given that we want to exclude f2, and all its subcontents...
-x*\f2\*
removes the contents, but leaves f2
-xf2
does nothing - includes all
-x\f2
does nothing - includes all
-x*\f2
does nothing - includes all (now I'm mad), so surely it must be..
-x\f2\
nope, does nothing - includes all. So it has GOT to be...
-x*\f2\
hell no, does nothing - includes all. and I already know that
-x*\f2\*
removes the contents, but leaves f2. Onward we go...
-x*f2\
does nothing - includes all. Grrrr. Aha! how about...
-x!tmp1\f2\
nope, does nothing - includes all. WTF. Alright, So it has GOT to be...
-x!tmp1\f2
Holy moly, it worked! Hmmm, then how come....
-x*\f2
does not work? This was the little demon that sent me down this crazed path to begin with and should have worked!
Given all that, do I dare try to go after */a/* directories, removing contents and the dirs?
-x*\a
does not work, of course, does nothing.
-x*\*\a
does not work, of course, does nothing.
-x!tmp1\*\a
nope. But...
-x*\a\*
removes contents of both dirs, but leaves the folders. So, in desperation I can use the -ed switch which will not store empty folders, but this is a broad hack, I want to eliminate the folders specified not all empty folders.
With my animosity growing toward winrar, I am passing the baton of information forward with an eye to that glorious day when we will know how to specifically exclude a folder and its contents using wildcards and not using the -ed switch.
(Quite old question but still may be relevant)
Maybe what you simply needed was this :
-x*\f2 -x*\f2\*
two exclude switches, should remove directory f2 and all its contents.
An even older question by now, but came across this question so I reproduced your folder structure and, at least nowadays (Winrar 5.11, not the latest but quite new), this works:
-x*\f2
So the whole command line is:
"C:\Program Files\WinRAR\Rar.exe" a -m5 -s !tmp1.rar !tmp1 -x*\f2
And this is what is stored in the .rar file:
!tmp1\f1\a\a.txt
!tmp1\f1\f1.txt
!tmp1\f1\a
!tmp1\f1
!tmp1
Similarly, if you use -x*\a, all a folders are excluded, storing this:
!tmp1\f1\f1.txt
!tmp1\f2\f2.txt
!tmp1\f1
!tmp1\f2
!tmp1
Finally, combining both parameters (-x*\f2 -x*\a), you get this:
!tmp1\f1\f1.txt
!tmp1\f1
!tmp1
To manage large list of files to be excluded, you can create text fie and write all excluded files/folders relative to the source folder:
1) create file list.txt, write the name of excluded files/folders
note: * refer to the source, all files/folders are relative to the source folder
*\f2
*\f3
2) Run the command
rar a -r -x#list.txt target.rar source-folder