I am trying to install confluent kafka on my databrick drivers and using init scripts there.
I am using below command to write an script to DBFS like below:
%python
dbutils.fs.put("dbfs:/databricks/tmp/sample_n8.sh",
"""
#!/bin/bash
wget -P /dbfs/databricks/tmp/tmp1 http://packages.confluent.io/archive/1.0/confluent-1.0.1-2.10.4.zip
cd /dbfs/databricks/tmp/tmp1
unzip confluent-1.0.1-2.10.4.zip
cd confluent-1.0.1
./bin/zookeeper-server-start ./etc/kafka/zookeeper.properties &
exit 0
""")
The I edit my intiscripts and add en entry there to denote to above location
[![init scripts entry adding][1]][1]
However, when I try to run my cluster it nevers starts and it always halts. If I go to event log, it shows that it is stuck at 'Starting init scripts execution.'
I know there should be tweak in my script to run it on the background but even I am using & at the end of the start command for zookeper.
Can someone give me any hint how to resolve above?
[1]: https://i.stack.imgur.com/CncIL.png
EDIT: I guess this question could be the same if I ask how I can run my script in a %sh databricks cell while the cell can finish the running of above bash script, but at the moment it always telling me that the command is running
Related
As written in the title, I am having some problem with my cron job script not executing. I am using CentOS 7.
My crontab -e looks like this:
30 0 * * * /opt/abc/efg/cron_jobs.sh >> /opt/abc/logs/cron_jobs.log
My cron_jobs.sh looks like this:
#!/bin/bash
#keep this script in efg folder
#run this daily through crontab -e
#45 0 * * * /opt/abc/efg/cron_job.sh
cd "$(dirname "$0")"
export PYTHONPATH=$PYTHONPATH:`pwd`
#some daily jobs script for abc
date
#send email to users whose keys will expire 7 days later
/usr/local/bin/python2.7 scripts/send_expiration_reminder.py -d 7
#send email to key owners whos keys will expire
/usr/local/bin/python2.7 scripts/send_expiration_reminder.py -d -1
# review user follow status daily task
# Need to use venv due to some library dependencies
/opt/abc/virtualenv/bin/python2.7 scripts/review_user_status.py
So, what I've found is that the log for the cron jobs in /var/logs/cron states that the cron ran at 0:30 am accordingly.
Strangely, I find that /opt/abc/logs/cron_jobs.log empty, and the scripts does not seem to run at all. It used to output some log before I re-inputted the crontab (to re-install the cron jobs), and replaced cron_jobs.sh, so I think the problem might have arose from those actions.
And also, I would like to know if there are any ways to log the error from executing a python script. I have been trying to run /opt/abc/virtualenv/bin/python2.7 scripts/review_user_status.py but it never seem to work as intended (does not run the main function at all), and there is no log output whatsoever.
I tried to run this on a different machine and it works properly, so I am not sure what is wrong with the cron job.
Here is a snippet of the log I got from /var/log/cron to show that the cron called the job:
Mar 22 18:32:01 web41 CROND[20252]: (root) CMD (/opt/abc/efg/cron_jobs.sh >> /opt/abc/logs/cron_jobs.log)
There are a few areas to check if you haven't performed these already,
if your executable permissions set on the script,
chmod +x <python file>
in addition permissions for the user to access the directories.
Run the script manually to test the script works from beginning to end, as the user who will be running the script, will be more realistic.
You can test your crontab schedule by temporarily setting every minute for testing, unlike Windows where you can right, click and click Run.
First, thank you all for the suggestions and heads up. I found out that what was ruining my script is the existence of /r in the line break. Apparently, Linux in general does not accept /r and only accepts /n.
It is because I ftp my files to the machine where the script breaks. On the other hand, it works fine on another machine because I used git pull instead of ftp.
Hope that this info will also be a helpful to others!
I have this bash script that opens a terminal repeatedly with no way of closing them and keeps one after another. The goal is annoy anyone who happens to execute this program. Here's the source code of it:
#!/bin/bash
while true
do
gnome-terminal -x sh -c "./<name-of-script.sh>; bash"
clear
sleep 1
trap '' 2
exit
done
I have tested this and it does work the way I want it to. What I'm trying to do is execute this script on a remote host. On a ruby on rails website I have set an image to attempt to run this script by clicking on it. When someone does they will run a controller function called call_script2 and here's what I have so far:
def call_script2
remote_ip = request.ip
#system("scp /home/ncs/<script.sh> root##{remote_ip}:/root/")
system("ssh root##{remote_ip} sudo home/ncs/./<script.sh>")
render 'script/index'
end
What I'm trying to accomplish is to run the contents of the bash script on the remote host when they click on a certain image. As you can see I've tried uploading the script of the host under the root directory and then trying to run the bash program with no success. I'm perfectly fine using something other than SSH if this not do able.
Please feel free to share your thoughts on this and thank you for taking the time to read my post. Have a great day!
Update: I was able to accomplish this goal by using the following:
system("ssh -t root##{remote_ip} DISPLAY=:0 ./Thorgrim.sh")
Thank you to everyone who have commented on this post to help me out!
I am trying to run a bash script from Groovy in Jenkins but I am not able to find the lucky commands.
I can run this and it creates my "RJ" directory:
process = "mkdir /app/jenkins/workspace/TEST/RJ"
println process.execute()
But when I try to run my bash it is not creating my output file. I am able to run this bash script on the server directly and it is creating my expected output file.
process = "/app/jenkins/workspace/TEST/info_file.sh"
println process.execute()
why run it through groovy and not directly via a ssh command.
If you do want to do via groovy, you'll need to add a ssh libray, and do the whole connection, auth, execute. process.execute won't run a linux box.
first, you don't check the stderr and not waiting for process to end.
similar problem was here:
Curl request from command line and via Groovy script
with error text it's easier to solve the error.
I'm creating Dockerfile script and it has a command line that executes a program and requires user input 1 from keyboard as selected option to go to further steps.
Xdotool, man yes or expect cannot help in this situation.
Update source-code:
First off, download and extract RevoMath library, navigate to RevoMath folder then execute the install script.
...
RUN wget -q https://mran.microsoft.com/install/mro/3.2.4/RevoMath-3.2.4.tar.gz
RUN tar -xzf RevoMath-3.2.4.tar.gz
RUN cd RevoMath/
RUN ./RevoMath.sh
...
Install script has some select options as follow:
echo "1. Install MKL"
echo "2. Uninstall MKL"
echo "3. Exit utility"
We need to enter 1 from keyboard to install. How can we do it via Docker command?
Any help would be appreciated!
If I correctly understand you, you would like to add echo 1 | before ./RevoMath.sh in your Dockerfile:
...
RUN cd RevoMath/ && echo 1 | ./RevoMath.sh
...
BTW: In your example this lines will not work as you expected:
RUN cd RevoMath/
RUN ./RevoMath.sh
Because each RUN is an independent execution.
You should use && if you want to execute RevoMath.sh script from specific folder (see my example in the beginning)
I suggest to use redirect from standard input.
For example install.sh required some input(s) from user at execution time.
Suppose you need to enter 1 as a response to first interaction(questions) and
then you have another response as y for further interaction then it's good to use redirect from stdin.
$#>install.sh <EOF
$#>1
$#>y
$#>EOF
This way whenever script is waiting for inputs it will answer as 1 for the first question and y for the second question.
I'm trying to create a job in Jenkins that will execute a simple shell script. However, it seems that Jenkins isn't actually doing anything with my script. No matter what value I put into the Execute Shell Command section, Jenkins always says that it passes. Even if I put in a bogus filename like "RandomBogusFilename.sh" it'll say the job was a success.
Can anyone point out what I'm doing wrong and how I can get Jenkins to actually use my shell script?
The shell script, the job config, and the console output are all shown below. I'm currently trying to do this on a Windows Server 2008 R2 Standard machine.
Thanks.
My .sh file
File Name: surveyToolRequest.sh
File Location: /jobs/Jeff Shell Script Test/workspace
Description:
Hit a web address and retrieve the HTTP Response. Then print out the HTTP Response.
#!/bin/bash
response_code=$(curl -s -o /dev/null -w "%{http_code}" http://SOME-WEBSITE.COM)
echo "The response code is " $response_code
My Jenkins Job Config
Jenkins Console Output
I played with this and found that it worked if I specified the path to the script. If the script is in your job's workspace directory,
./surveyToolRequest.sh
should work as Jenkins looks for files relative to the root of the workspace.
It's more common to just put the contents of the script file directory into the job configuration; that way you can see what the job is doing and you'll avoid problems like this one.
You should use run "Execute windows batch command" and not "Execute Shell"