Unix export command path - bash

I want to run a single line command via NiFi ExecuteStreamCommand processor. I want to run a gsutil command and before doing that I want to export environment variable GOOGLE_APPLICATION_CREDENTIALS.
So the command would be
export GOOGLE_APPLICATION_CREDENTIALS='/temp/abc.json'
However, NiFi needs the path of the command. On the server when I checked which export I do not get its path:
[user#server1 ~]$ which export
/usr/bin/which: no export in (/opt/teradata/client/14.10/tbuild/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin:/home/user/.local/bin:/home/user/bin:/usr/local/google-cloud-sdk/bin/)
If its a builtin command, how do I get NiFi to run it?

how do I get NiFi to run it?
Use env to run a command with modified environment.
/usr/bin/env GOOGLE_APPLICATION_CREDENTIALS='/temp/abc.json' gsutil

Related

Why variables that I export in bash script does not work on EC2?

When I create EC2 inst. I use bash script into user data where I export variable AWS credentials and then run the command to copy files from S3 bucket. But this command is not executed.
#! /bin/bash
export AWS_ACCESS_KEY_ID=MYACCESSKEY
export AWS_SECRET_ACCESS_KEY=MYSECRETKEY
aws s3 cp s3://mys3bucket/ ./
How to fix it?
It is run on its own bash process which dies at the end of your script.
Exported variables are preserved only for the lifetime of your script and they are also visible from child processes of your script.

Why exporting env var in bash script does not affect env?

I want to set env variable in shell script. Shell script content is:
#!/bin/bash
export XDEBUG_CONFIG="idekey=PHPSTORM"
I tried both bash bin/enable_debug and bin/enable_debug. After both command I get:
$ echo $XDEBUG_CONFIG
$
However if I run export XDEBUG_CONFIG="idekey=PHPSTORM" directly in cli it works. What's wrong with my method?
You can try running your script as below:
. bin/enable_debug
OR
source bin/enable_debug
as indicated by #Aserre

Pass environment variable from command line to yarn

I have a code that reads port number from environment variable or from config. Code looks like this
const port = process.env.PORT || serverConfig.port;
await app.listen(port);
To run app without defining environment variable, I run following yarn command.
yarn start:dev
This command works successfully in Linux shell and Windows command line.
Now, I want to pass environment variable. I tried following,
PORT=2344 yarn start:dev
This commands works successfully in Linux shell but failing in Windows command line. I tried following ways but couldn't get it to work.
Tried: PORT=2344 yarn start:dev
I got error: 'PORT' is not recognized as an internal or external command,
operable program or batch file.
Tried: yarn PORT=2344 start:dev
I got error: yarn run v1.17.3
error Command "PORT=2344" not found.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Any idea please? I know, I can define environment variables from System Properties in Windows. But any way if I can do it from command line?
i'd suggest you use the NPM module called cross-env. it allows adding particular env variables on the command line regardless of platform. with that said, you may try:
$ cross-env PORT=2344 yarn start:dev
You can chain commands on the Windows command prompt with &(or &&). To set an environment variable you need to use the set command.
The result should look like this: set PORT=1234 && yarn start:dev.
Found a solution for this problem in Windows command prompt.
Create a .env file in project root folder (outside src folder).
Define PORT in it. In my case, contents of .env file will be,
PORT=2344
Run yarn start:dev
Application will use port number that you have specified in .env file.
Put .env file at root. Then following command will expose content of .env file and then run yarn start command
$ source .env && yarn start
or this command
$ export $(cat .env) && yarn start
If update any variable in .env then close the terminal and open new terminal window and can again run above command. Or else can also run unset command to remove existing var.
unset VAR_NAME
You can use popular package dotenv:
create a file .env in root directory
put all your env vars
e.g.:
ENV=DEVELOPMENT
run your code like this
$ node -r dotenv/config your_script.js
here the explanation:
[https://github.com/motdotla/dotenv#preload]
To define environment variables in the Windows command prompt we can use the set command, you can then split your call into two lines.
set PORT=2344
yarn start:dev
The set command persists within the current command prompt, so you only need to run it once.
The equivalent command in bash is 'export'.
FYI (not a direct answer). I was attempting this in VS Code - passing .env variables through yarn to a JavaScript app. Google had very few examples so I'm sharing this for posterity as it's somewhat related.
The following simply substitutes text normally placed directly into the package.json or script file. Use this to quickly obfuscate or externalize your delivery configurations.
In Environment Variable File (.env)
PORT=2344
In Yarn File (package.json)
source .env; yarn ./start.sh --port $PORT
In Yarn Script (start.sh)
#!/bin/bash
while [ $? != 0 ]; do
node dist/src/index.js $1; #replace with your app call#
done
The app then accepts port as a variable. Great for multi-tenant deployments.

How to Set docker container ip to environment variable dynamically on startup?

I want to export docker container hostname as an environment variable which I can later use in my app. In my docker file I call my script "run" as last command
CMD run
The run file is executable and works fine with rest of commands I perform but before them I want to export container hostname to an env. variable as follows
"run" File Try 1
#!/bin/bash
export DOCKER_MACHINE_IP=`hostname -i`
my_other_commands
exec tail -f /dev/null
But when I enter docker container and check, the variable is not set. If I use
echo $DOCKER_MACHINE_IP
in run file after exporting, it shows ip on console when I try
docker logs
I also tried sourcing another script from "run" file as follows
"run" File Try 2
#!/bin/bash
source ./bin/script
my_other_commands
exec tail -f /dev/null
and the script again contains the export command. But this also does not set the environment variable. What I am doing wrong?
When you execute a script, any environment variable set by that script will be lost when the script exits.
But for both the cases you've posted above the environment variable should be accessible for the commands in your scripts, but when you enter the docker container via docker run you will get a new shell, which does not contain your variable.
tl;dr Your exported environment variable will only be available to sub shells of the shell which set the variable. And if you need it when logging in you should source the ./bin/script file.

Run bash alias from Apache Hive

I am trying to create an alias on Hadoop machine and run it from Hive JVM.
When I explicitly run the command from Hive with ! prefix it works, however when I add the alias, source the .bashrc file and call the alias from Hive, I get an error. Example:
.bashrc content:
# Environment variables required by hadoop
export JAVA_HOME=/usr/lib/jvm/java-7-oracle
export HADOOP_HOME_WARN_SUPPRESS=true
export HADOOP_HOME=/home/hadoop
export PATH=$PATH:/home/hadoop/bin
alias load-table='java -cp /home/hadoop/userlib/MyJar.jar com.MyClass.TableLoader';
Call on Hive:
!load-table;
Output:
Exception raised from Shell command Cannot run program "load-table": error=2, No such file or directory
Aliases have several limitations compared to shell functions (e.g. by default you cannot call an alias from a non-interactive shell).
Define in your ~/.bashrc:
function load-table() {
# Make sure the java executable is accessible
if which java > /dev/null 2>&1; then
java -cp /home/hadoop/userlib/MyJar.jar com.MyClass.TableLoader
else
echo "java not found! Check your PATH!"
fi
}
export -f load-table # to export the function (BASH specific)
Source you .bashrc to apply the changes. Then, call load-table.

Resources