Some context
I need to emulate the behaviour of the shell command source but in Python. Specifically sourcing tcsh scripts, but it would be nice for it to be as generic as possible.
I've seen this answer where it makes the shell run env to print all environment variables.
My question
env or printenv only prints environment variables but not local variables. However the tcsh scripts I want to source use commands like
set FOO=bar
So is there a way in tcsh (and other shells, just for completeness) to print all local variables?
Edit: Yes I know I'd have to run TCSH as a subprocess for it to parse the script, and yes python can't read into TCSH's memory which is why I want a way to print all local variables within TCSH so Python can read them.
Related
When I need to run a bash script that runs cd somedir to affect the current shell I run it with . scriptname. However, if scriptname is a python script even with #!/usr/bin/env python3 in the first line, it doesn't work, it seems it expects the script to be a bash script. How can I make it work with python scripts (or any other language with the appropriate shebang)?
It is not possible.
The only thing that can affect current process environment is the process itself. Because the current shell is bash, the only thing that can be executed that could affect bash environment is something that can be run by bash itself. That "something" are statements interpreted by bash. Because bash doesn't support interpreting and running python statements, it is not possible.
The usual way around this, is to output from your python script properly escaped assignment statements that would assign environment variables. Then the output from your script is evalulated by bash. This is for example how eval "$(docker-machine ...)" works.
I have a set of environment variables that need to be set on the basis of the arguments specified in the shell script.
But the problem is that those variables are already defined in the bash profile
FOR EXAMPLE:
bash_profile has a variable called "KARAN":
export KARAN=/config/1
Now on running the shell script, this is what it should do:
export KARAN=/config/2 (Changed the bash profile's KARAN value to 2)
Your question is not clear. If your script needs to set the env var to a specific value just do so using export VAR=val. What I think you're asking is how to have a script modify the environment of the current shell. And that is impossible without the cooperation of both shells. That is because environment vars are inherited by child processes. But a child process cannot directly modify the environment of its parent process (or some other random process for that matter). To do so the two processes must coordinate the exchange of data. This is typically done by using the source command if the child process is a shell script. Or by having the child process write a series of export statements to stdout and having the parent shell capture and execute those statements. For example, let's say I have a script named set_env that looks like this
#!/bin/sh
echo export KARAN=/config_2
echo export VAR2=val2
The current shell would then do
eval $(set_env)
Note, however, eval is dangerous. I prefer to do this which is slightly safer:
set_env | source /dev/stdin
That, however, only works in shells like ksh and zsh. Due to how bash handles pipelines the source is actually executed in a child shell and therefore the vars won't be set in the current shell.
You can create a new Profile with all the new definitions. and then call the line below on top of your shell script. Similarly, you can create as many profiles as you want and use it.
source bash_profile_new
my project, which uses mapreduce without hadoop, is composed of two files:
bash.sh and mapreduce.py.
I would like to use environment variables to communicate the information between bash.sh and mapreduce.py.
Within bash.sh I use export myvariable and on mapreduce.py, I use os.environ ['myvariable'].
I would like to edit myvariable within mapreduce.py and print the result via bash.sh. I tried to execute this istruction: os.environ ['myvariable'] = 'hello', but on bash.sh 'myvariable' is empty. Do you have any suggestions?
You can't do it from python, but some clever bash tricks can do
something similar. The basic reasoning is this: environment variables
exist in a per-process memory space. When a new process is created
with fork() it inherits its parent's environment variables. When you
set an environment variable in your shell (e.g. bash) like this:
/why-cant-environmental-variables-set-in-python-persist
So you can only make available to Bash script if the bash script is called inside python process space. A simple example can be
bash script
#!/bin/bash
echo "var from python is $myvariable"
python script
import os
os.environ ['myvariable'] = 'hello'
print(os.environ['myvariable'])
# all environment varaibles will be availbe to bash script in this case
os.system('sh ./ab.sh')
This is the way that you can try. Otherwise, no way to set it and make it available to bash script.
Setting an environment variable sets it only for the current process
and any child processes it launches. So using os.system will set it
only for the shell that is running to execute the command you
provided. When that command finishes, the shell goes away, and so does
the environment variable. Setting it using os.putenv or os.environ has
a similar effect; the environment variables are set for the Python
process and any children of it.
ENV via python
You can also try vice versa as you mention in question
Here is sequence
First set in bash script
call python script from bash ( based on argument to avoid loop)
update ENV in python
call bash again from python, if you call outside it will vanish.
bash script
#!/bin/bash
export myvariable="hellobash"
echo "myvariable form bash $myvariable"
if [ ! -z $1 ]; then
./py.py
else
echo "myvariable after updated from python $myvariable"
fi
Call bash script outside from python with the argument like myscript.sh bash, without argument in python myscript.sh
#!/usr/bin/python
import os
print("myvar form python",os.environ['myvariable'])
os.environ ['myvariable'] = 'hello'
print("myvar form python after update",os.environ['myvariable'])
os.system('sh ./ab.sh')
In vim I can access my bash environment variables such as $PWD and $PATH. I would like to know how to access my temporary shell variables in vim too.
For example, suppose I was in my terminal and define a variable foo="bar". Then I enter vim and try to access this variable with the following command :!echo $foo, but it does not recognize this variable. From my understanding, vim starts a new shell each time a bash command is invoked and then closes it immediately after. Is there a way to use the same shell in vim that my local variable foo was defined in?
No, you can't interact with the parent shell from a subprocess it spawned (without that shell's active participation, which isn't reasonably/practically available in the scenario at hand) -- but you can export your variables to make them accessible to new shells started in child processes.
Running
set -a
...will make any variable defined going forward be automatically exported to the environment, even without an explicit export command.
Since (unlike the C system() function) vim's system() honors the SHELL environment variable, if SHELL=/bin/bash (or :set shell=/bin/bash has been run in vim), you can also invoke exported functions from vim. That is, if you define the function and export it as follows:
foo() { echo "bar"; }
export -f foo
...then you can invoke it with !foo from inside vim.
Even then, however, this is running in a new, transient shell instance, not the original parent process.
Explanation
Environment variables and shell variables are two entirely different concepts, but as we manipulate them in a similar way in bash, it's easy to get confused.
Whenever a process is created (by fork), it may include an environment, given by its parent at fork-time. The child process may then access and modify its content. How this is done as a user depends on the program :
In vim, you can access an environment variable like this : :echo $foo
In bash, you can access it like this : $ echo "$foo"
In most programming languages, you can access it with a syntax coherent with the rest of the language, such as ENV['foo'] in ruby
On the other hand, a program may allocate memory for any internal use, but notably, it will quite often define and use variables. Once again, this depends on the program :
In vim, you would use the :let command to assign an internal variable
In bash, you would assign a variable with $ foo='bar', and then read it with $ echo "$foo"
In most programming languages, you have a variation of the foo='bar' syntax, sometimes with type declarations, etc
As you can see, bash uses the same syntax to read an environment variable and one of its own private variables, which can lead to some confusion.
When you execute vim from your bash shell, the environment is copied over from the parent process (bash) to the child (vim), but the private memory of bash (including the variables you may have defined) are not.
Thus, accessing them from the child process would require some inter-process communication mechanism, between parent and child. While technically doable, this option is not implemented in bash nor vim.
Solution
In order for your variable to be accessible from vim (or any forked process, for that matter), you need it to be present in the environment of your vim process.
Several options to do that :
$ export foo='bar' : This will mark your variable for export to the environment of subsequently executed commands. That's what you want in most cases.
$ foo='bar' vim : This adds your variable to the environment of this vim command. Very useful for troubleshooting, or for one-liners.
$ set -a : As you can see in bash manpage, this marks every subsequent definitions for export to the environment of subsequent commands. It's essentially equivalent to prepending every subsequent definition by export.
To go further
The question uses the :!echo $foo syntax to display the value of foo, which is yet another usecase. The ! here is actually an escape sequence that allows you to execute a shell command from vim.
However, vim cannot execute anything in the parent shell (the one you executed the vim command in), so it creates a new bash shell in a child process, executes echo in it, and displays the result.
In the current case, the result is mostly the same, but it could easily be misleading in other situations, so it's important to understand what is happening here.
There is another vim syntax, using expand, that allows one to lookup variables : :echo expand("$foo")
It however works entirely differently.
If no internal variable named foo exists, vim will invoke a shell to look it up (similarly to what ! would do).
This options is way slower than an environment lookup, and not recommended for most usecases.
If you want to use a value from your shell on the :substitute command, there's actually a way to do it.
I don't know if it solves your need but here we go.
Let's say we want to substitute Mydir by your PWD:
:s/Mydir/\=expand($PWD)/g
I'm trying to access a variable declared by previous command (inside a Makefile).
Here's the Makefile:
all:
./script1.sh
./script2.sh
Here's the script declaring the variable I want to access,script1.sh:
#!/usr/bin/env bash
myVar=1234
Here's the script trying to access the variable previously defined, script2.sh:
#!/usr/bin/env bash
echo $myVar
Unfortunately when I run make, myVar isn't accessible. Is there an other way around? Thanks.
Make will run each shell command in its own shell. And when the shell exits, its environment is lost.
If you want variables from one script to be available in the next, there are constructs which will do this. For example:
all:
( . ./script1.sh; ./script2.sh )
This causes Make to launch a single shell to handle both scripts.
Note also that you will need to export the variable in order for it to be visible in the second script; unexported variables are available only to the local script, and not to subshells that it launches.
UPDATE (per Kusalananda's comment):
If you want your shell commands to populate MAKE variables instead of merely environment variables, you may have options that depend on the version of Make that you are running. For example, in BSD make and GNU make, you can use "variable assignment modifiers" including (from the BSD make man page):
!= Expand the value and pass it to the shell for execution and
assign the result to the variable. Any newlines in the result
are replaced with spaces.
Thus, with BSD make and GNU make, you could do this:
$ cat Makefile
foo!= . ./script1.sh; ./script2.sh
all:
#echo "foo=${foo}"
$
$ cat script1.sh
export test=bar
$
$ cat script2.sh
#!/usr/bin/env bash
echo "$test"
$
$ make
foo=bar
$
Note that script1.sh does not include any shebang because it's being sourced, and is therefore running in the calling shell, whatever that is. That makes the shebang line merely a comment. If you're on a system where the default shell is POSIX but not bash (like Ubuntu, Solaris, FreeBSD, etc), this should still work because POSIX shells should all understand the concept of exporting variables.
The two separate invocations of the scripts create two separate environments. The first script sets a variable in its environment and exits (the environment is lost). The second script does not have that variable in its environment, so it outputs an empty string.
You can not have environment variables pass between environments other than between the environments of a parent shell to its child shell (not the other way around). The variables passed over into the child shell are only those that the parent shell has export-ed. So, if the first script invoked the second script, the value would be outputted (if it was export-ed in the first script).
In a shell, you would source the first file to set the variables therein in the current environment (and then export them!). However, in Makefiles it's a bit trickier since there's no convenient source command.
Instead you may want to read this StackOverflow question.
EDIT in light of #ghoti's answer: #ghoti has a good solution, but I'll leave my answer in here as it explains a bit more verbosely about environment variables and what we can do and not do with them with regards to passing them between environments.