I am planning to use Jenkins with Hive. I have created some Hive scripts: test1.hql, test2.hql, test3.hql which lie on the sample server "ssh rt#test1.test2.net" location "/home/rt"
test1.hql:
select id, name from tblA;
test2.hql:
select id, name from tblB;
test3.hql:
select id, name from tblC;
How can I execute it from Jenkins?
On Jenkins up-to below is what I did:
New Job > Job name as test > Build a free-style software project
How can I point to my .hql scripts from Jenkins and what settings do I need to have? I am new to Jenkins.
Any help?
Thanks.
Found something:
Create a Jenkins job
In the configuration of that job there is a "Build" section with "Add build step"
Add a "Execute Shell" step, and put your code.
that code would be basically:
hive -f /pathtohql/test1.hql
The only issue i see with this is that the m/c with jenkins will have to have hive installed and pointing to the correct metastore & cluster.
Related
I am trying to setup a Jenkins pipeline using scripted pipeline with a windows 2019 server, however I ran into this error while trying to build
> webpack --config ./config/webpack-cli-prod.config.js
C:\myProject\node_modules\webpack\lib\javascript\JavascriptModulesPlugin.js:143
throw new TypeError(
^
TypeError: The 'compilation' argument must be an instance of Compilation
at Function.getCompilationHooks (C:\myProject\node_modules\webpack\lib\javascript\JavascriptModulesPlugin.js:143:10)
at SourceMapDevToolModuleOptionsPlugin.apply (C:\myProject\node_modules\webpack\lib\SourceMapDevToolModuleOptionsPlugin.js:50:27)
at C:\myProject\node_modules\webpack\lib\SourceMapDevToolPlugin.js:163:53
at Hook.eval [as call] (eval at create (C:\myProject\node_modules\tapable\lib\HookCodeFactory.js:19:10), <anonymous>:100:1)
at Hook.CALL_DELEGATE [as _call] (C:\myProject\node_modules\tapable\lib\Hook.js:14:14)
at Compiler.newCompilation (C:\myProject\node_modules\webpack\lib\Compiler.js:1122:26)
at C:\myProject\node_modules\webpack\lib\Compiler.js:1166:29
at Hook.eval [as callAsync] (eval at create (C:\myProject\node_modules\tapable\lib\HookCodeFactory.js:33:10), <anonymous>:6:1)
at Hook.CALL_ASYNC_DELEGATE [as _callAsync] (C:\myProject\node_modules\tapable\lib\Hook.js:18:14)
at Compiler.compile (C:\myProject\node_modules\webpack\lib\Compiler.js:1161:28)
I tried to run same command/step using freestyle Jenkins job, and it works without this error.
I tried to run the same command on the Jenkins agent locally, and it works without this error.
I looked up on google, and came across this link here, I tried to use newer version of html-webpack-plugin, and we also tried to build without the plugin. All come to the same result, that the error would occur only when running from Jenkins scripted pipeline.
I also tried with a different server, while keeping the same agent and job configuration, and I also get the same error.
The version of npm is 8.11.0, the node version is 16.16.0. The agent is connected through running the agent.jar file from the agent.
The only difference I see between the freestyle job and the scripted pipeline job is the freestyle job appears to be run as SYSTEM by the Jenkins server, whereas the pipeline job is probably run with lower privilege (I am not entirely sure though). I saw also this post, where it says
in the Freestyle job everything is executed in the agent, but for the Scripted Pipeline Job, the pipeline code is translated in the controller to atomic commands that are sent to the agents.
But I have no idea how to make the scripted pipeline job run just like the freestyle job.
On one hand, it appears to have to do with webpack, and on the other it appears to be related to Jenkins since running freestyle and locally on the server is without errors.
This is how my Jenkins scripted pipeline looks like (with sensitive information removed)
node("My-Server"){
dir("C:\\MyProject"){
stage('Pre-Test Build Client (Web)') {
dir("aFolder"){
bat 'npm run build-all-prod' // This is the script that invoke the webpack build command
}
}
}
}
I have run out of options, and do not where to go from here, and I couldn't find any more information on google that would be helpful. Any help here would be really appreciated. Thank you.
I'm not sure if jenkins creates the same environment variables, and command line tools (you can configure some on the node's configuration page). I would check if the node's environment variables and tools are the same in freestyle vs pipeline job by running something like this in each job, and comparing the output:
bat 'echo %PATH%'
bat 'which webpack'
bat 'npm list webpack'
Another thing worth checking is whether you're using batch script in both jobs, and not eg shell in freestyle.
Lastly I found a gh issue whith the same error as you, caused by having 2 different installations of webpack, one on v5. May be worth looking into, if everything else fails.
I have a jenkins job in freestyle mode. In the build step, I add a section "execute shell script". And in this shell section script, I have a command to connect to bitbucket with a user et token. My issue is I want to obfuscate the token to not allow everyone to see it.
I know that this possible with pipeline but I not find any solution for freestyle job ...
Anyone have solution for this please?
Finally I manage to do it with option "Use secret text(s) or file(s)" available in the job settings
A Bash script is CRON-ed to run on server A every day and connects to a remote MySQL database to run a query, dump the results to a file, and send an email.
In my freestyle project, I have set up my build to pull that Bash script from my Github repo and execute it. Jenkins starts it but the script fails. Based on my discovery,
Jenkins runs my script in a clone of my repo on its own server, not on server A where it is CRON-ed
The above explains why my shell script fails to find the mysql client executable
Based on my understanding, this means that my freestyle project needs to connect to server A where my script is CRON-ed to run like so
ssh serverA_username#serverA #using ServerA user key
SCRIPT="$WORKSPACE/shell/bash_script_name.sh"
sh -x $SCRIPT
Am I correct? Any pointers welcome. Thank you!
My jenkins is installed in C:\Program Files (x86)\Jenkins
bat file is located in C:\Users\Admin\workspace\demoWork\run.bat
When i run this bat file from cmd everything works fine. But when i try from jenkins executing batch command as mentioned in Image, Jenkins displays error as
Build step 'Execute Windows batch command' marked build as failure
Also inside jenkins folder automatically workspace folder gets created with Job title name. Can you guys please explain me in detail
Tatkal, you can't execute a command like in your image,
why don't you simply try
C:\users\admin\workspace\demowork\run.bat
or
call "C:\users\admin\workspace\demowork\run.bat"
"Also inside jenkins folder automatically workspace folder gets created with Job title name. Can you guys please explain me in detail" -
Jenkins creates folder with job title name automatically, saves jobs data and other build info... this is how it works. By default in jenkins job you can access your workspace using $WORKSPACE variable
You have put very little detail into this so I'm going by pure guess..
The Execute Windows batch command is to literally execute code, not execute a file.. to execute the file you could use this command :
start cmd.exe /c C:\myprj\mybat.bat
or you could take the contents of the .bat file and rewrite in in that command line..
The way Jenkins works is it creates its own workspace for each job, essentially to sandbox the environment, its a testing framework so it should be used to stage changes to code, which will then be pushed to your live(working) environment. People use it to automate some tasks, but this isnt the primary use of Jenkins.. if the above doesn't help you let me know more details of the error and I can try help you with it.
node {
bat 'D:\\gatling-charts-highcharts-bundle-3.0.2\\bin\\gatling.bat'
}
I do have a jenkins job where I execute command from within Maven plugin which executes ant build script. The job also does 2 ant calls as there are 2 mirror servers. Something like this:
usr/bin/ant -v -d -f /utils_repo/build.xml ${target} -propertyfile /tmp/myjob/install.properties
Where Maven connects to each server and executes something similar.
My question is how can I share timestamp of when jenkins job starts within 2 instances of ant calls. In my ant job I have a backup build step before rolling in a new code, but I need to put the logic if dump/backup was done on the first host, do not do it on the second one as they do share mysql instance and core files on nfs mount, What happens right now is there is no logic and when second ant call runs dump on the second server it overwrites the previous dump from the first instance with the new data and updated mysql.
So I was thinking on creating a touch task to touch some file since I have shared directory between 2 servers, but I have the same build.xml for both server instances, so the touch will executed on the second ant call and overwrite the modification time of the first ant call.
I thought of if I could share jenkins timestamp property of when job starts within 2 ant jobs. Do not know if this is possible.
Thanks in advance for advise.
I suggest you should use the environment variable BUILD_NUMBER set by Jenkins and make it stored on you nfs, as a property file for instance.
So if that property file doesn't exists, or if the value of the property loaded from there doesn't match the environment variable set by Jenkins, it means the current node is the first to run for that Jenkins job. So it can do the backup. And that node would overwrite the shared property file with the current build number.
If the loaded property match the current build number, then it means the first run has already been done, no backup to do.
Implementation hints:
add the build number to the command line
usr/bin/ant -v -d -f /utils_repo/build.xml ${target} -propertyfile /tmp/myjob/install.properties -Dexpected.jenkins.build.number=$BUILD_NUMBER
use ant contrib if/then/else tasks: http://ant-contrib.sourceforge.net/tasks/tasks/index.html
write the property file with:
<echo file="/mnt/nfs/shared/jenkins.properties">jenkins.build.number=$expected.jenkins.build.number</echo>