I want to make some changes to hadoop hdfs according to a published paper. After that I just need to build HDFS and get it running. How can I do that?
Refer the following Hadoop documentation
http://wiki.apache.org/hadoop/HowToContribute
This assumes you build on Linux. If you use a different OS, you may need to do some extra steps; for details see this - I've never done this on non-Linux myself.
Install Git, Java (JDK), Maven and ProtocolBuffer (2.5+ version required)
Clone https://github.com/apache/hadoop-common.git by typing something like this in your command line:
git clone https://github.com/apache/hadoop-common.git
Note: you may want to use a particular branch corresponding to the version of HDFS you're looking to build. To list all branches, type git branch -a. Then to switch to branch 2.3, for example, type:
git checkout --track origin/branch-2.3
If you did everything correctly, you should see a message about tracking the remote branch you've selected.
Make whatever changes you need to make in HDFS; the code lives under hadoop-hdfs-project.
Compile the project by running the following from the root of your tree:
mvn install -DskipTests
This will take some time the first time you do it, but will be a lot quicker during re-runs.
Your final jars will be placed into directories like hadoop-hdfs-project/hadoop-hdfs/target (this is accurate for at least 2.3, but it might have been different in older version, or it may change in the future).
Related
I am trying to build my first image with Yocto for a Xilinx Eval Board ZCU104. As I am just a beginner I installed the required packages to work with Poky and stuck to the following simple steps:
I cloned Poky: git clone git://git.yoctoproject.org/poky
source oe-init-build-env: a build directory was created.
In the build directory I cloned the meta-xilinx layers: git clone git#github.com:Xilinx/meta-xilinx.git
I read the README.md and README.building.md files from meta-xilinx/meta-xilinx-bsp to find what to do next.
I used the command bitbake-layers add-layer to add the following layers: meta-xilinx-bsp, meta-xilinx-standalone and meta-xilinx-contrib. I can see those layers in bblayers.conf.
I edited MACHINE in local.conf: MACHINE ?= zcu104-zynqmp
I think I am good to go so I call bitbake -k core-image-minimal and expect a succesful build.
Bitbake started to parse until it throwed the following error:
ERROR: ParseError at
//poky/build/meta-xilinx/meta-xilinx-bsp/recipes-bsp/uboot-device-tree/uboot-device-tree.bb:11:
Could not inherit file classes/xsctdt.bbclass
I looked for that class under /meta-xilinx-bsp/classes and it is not there, but I found it in the meta-xilinx repository so I started wondering why I didn't get that class (among others).
Note that I did not switch to any branch yet (either yocto or meta-xilinx) and I stayed on the master branches. I gave it a second try after switching to the latest tags from both projects but I got the following error:
ERROR: Layer xilinx is not compatible with the core layer which only
supports these series: gatesgarth (layer is compatible with zeus)
What should I try next? Should I checkout other branches? If so, is there any index to link the right yocto and meta-xilinx branches?
Anyways, how come I am missing necessary classes from the repository when I clone it?
Thanks a lot!
Checkout the branches named the same way in each layer you're using (poky included). The branches are usually named after Yocto releases. It's usually wise to avoid the master branch.
If there is no branch of a given Yocto release for a layer, look into what seems to be a recent branch and check in conf/layer.conf for the LAYERSERIES_COMPAT variable. Find a branch which has a LAYERSERIES_COMPAT with the Yocto release you want to use. If there are none, you'll need to add the Yocto release to said variable and fix things along the way until it compiles and everything is working fine.
c.f.: https://docs.yoctoproject.org/ref-manual/variables.html#term-LAYERSERIES_COMPAT
I have a source code using which I want to test some processes through Jenkins.For that I've installed Jenkins and plugins as well. But I do not have internet access to my machine so I have my source code in a local folder instead of Git.
Please help me to configure Jenkins without git.
Thanks a ton in advance.
in the build steps, you can do something like
cd C:\users\user\Documents\Projects\
// build here
if you're in a different directory, you need to do:
cd D:\Projects
D:
// build here
But deleting the workspace option won't work now. since your artifacts will be stored in the current directory and the workspace is a different directory. you can add an additional step, in this case, to clean up things or you can change the workspace from the advanced option in the general tab.
I was looking at https://yarnpkg.com/lang/en/docs/cli/policies/ and it seems like a good idea to use it to allow my team to easily be on the same yarn version. However, yarn policies set 1.18 downloads the full yarn release into .yarn/releases (a 4.5mb js file) and sets a config entry in the repo's .yarnrc file.
It feels weird to check in this 4.5mb yarn executable, but if I don't, my colleagues are not going to be able to run yarn, because the entry in the .yarnrc won't exist on their system and it's not magically downloaded...
So, is it best practice to check the .yarn/releases folder into version control?
Yes your assumption is correct, you check yarnrc and the actual yarn JS file into source control.
I have a Jenkins multiphase job that
gets an update from version control (Subversion)
does a Maven build. Couldn't be more cut and dry.
The 1. above, svn update, does return a latest revision number. I would like to fetch that into 2. so that the build and its associated artifacts do have that number pinned to them, if any way possible in the artifact name itself but, if not, in the build history. Is there a way to do it and how (e.g. using the subversion or another plugin)?
I am using the buildnumber plugin to fetch the build number and generate a small text file that is contained inside my WAR artifacts, which makes it subsequently available via HTTP. But to see it, one must either first deploy the artifact or at least extract its contents. I would like it more readily visible in Jenkins.
You should take a look at How to get SVN revision number in Jenkins Workflow Plugin? - He solved the problem with:
def revision = 'svn info'.execute().in.text.split('\n').find { it.startsWith('Revision') }.split(':')[1].trim()
He's obviously using the latest Jenkins version, but if you're using one of the old Jenkins versions you can simply run the following command and parse the result to get the revision number you want:
svn info -r HEAD
BTW you can use a Jenkins constant too. In the browser you can open http://your-jenkins-host/env-vars.html/ and you will find a constant named SVN_REVISION. Each job build keep the SVN revision into that variable.
I would to run buildr on my svn repository - to build the files from \src\main\java in the subversion repository with username and password and save it to anther location on my disc.
How can I do it?
if i understand your question correctly, you are trying to setup a contiuous build system. checking out source code is the responsibility of your CI system. it would poll your svn, git or whatever cvs you use for changes and then build it on the configuration you provide.
a free build-server is jenkins (aka hudson). if you want a hosted solution you can go for travis-ci (it's free). or something expensive like atlassian bamboo.
You will have to write an Ruby build script - that when you execute will:
Checkout the source
do the 'buildr'
Commit the built file(s)
http://www.markdeepwell.com/2010/06/ruby-subversion-bindings/
http://blog.carlossanchez.eu/2009/08/18/using-subversion-ruby-bindings/
the easiest way is run external svn commands
task :checkout do
system 'svn checkout ..'
end