How do I compile OpenJ9 on Fedora/RHEL/CentOS? - j9

The default OpenJ9 build instructions use Docker, but I'd like to compile without Docker; specifically, on RHEL 7.3.

The build instructions on RHEL 7 (Or CentOS) should be fairly similar to OpenJDK with HotSpot.
I have a DockerFile that creates an OpenJ9 development environment based on CentOS 7. But if you want to create the environment manually, do this:
Grab the build dependency for compiling OpenJDK yum builddep java-1.8.0-openjdk
Grab the rest of the build dependencies which are not installed in the minimal centos image.
java-1.8.0-openjdk-devel for a "Boot" JDK
hg, file, which, unzip, make for compiling
numactl-devel is needed for J9 numa support
yum install java-1.8.0-openjdk-devel make file which unzip hg numactl-devel
grab a git 2 package from WANDisco: rpm -U http://opensource.wandisco.com/centos/7/git/x86_64/wandisco-git-release-7-2.noarch.rpm && yum install -y git
Of course you can always modify the build file to make it work with git 1.7, as suggested
download freemarker and put the location into ./configure

I got this working by looking at the Dockerfile and making a few changes for Fedora-based systems:
sudo yum install autoconf ca-certificates ccache cpio file gcc-c++ gcc git \
git-core alsa-lib-devel cups-devel elfutils-devel \
freetype-devel numactl-devel libX11-devel libXext-devel \
libXrender-devel libXt-devel libXtst-devel make \
java-1.8.0-openjdk-devel pkgconfig coreutils \
openssh-clients unzip wget zip
# Put java-1.8.0-openjdk-devel at the front of the PATH; for example:
export PATH=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-1.b16.el7_3.x86_64/bin/:${PATH}
cd ~/
wget https://sourceforge.net/projects/freemarker/files/freemarker/2.3.8/freemarker-2.3.8.tar.gz/download -O freemarker.tgz
tar -xzf freemarker.tgz freemarker-2.3.8/lib/freemarker.jar --strip=2
rm -f freemarker.tgz
git clone https://github.com/ibmruntimes/openj9-openjdk-jdk9
cd openj9-openjdk-jdk9
bash ./get_source.sh
# I'm on RHEL 7.3 with git 1.8 which doesn't support -C ${DIR}, so that
# needs to change to --work-tree=${DIR} --git-dir=${DIR}/.git
# Apply the patch at the bottom of this answer now before continuing.
bash ./configure --with-freemarker-jar=~/freemarker.jar
make all
[...]
Finished building target 'all' in configuration 'linux-x86_64-normal-server-release'
build/linux-x86_64-normal-server-release/images/jre/bin/java -version
openjdk version "9-internal"
OpenJDK Runtime Environment (build 9-internal+0-adhoc.kevin.openj9-openjdk-jdk9)
Eclipse OpenJ9 VM (build 2.9, JRE 9 Linux amd64-64 Compressed References 20170915_000000 (JIT enabled, AOT enabled)
J9VM - ead54ef
JIT - ead54ef
OMR - 617de12
OpenJDK - f38f4d6 based on jdk-9+181)
Patch for older Git 1.8:
diff --git a/closed/OpenJ9.gmk b/closed/OpenJ9.gmk
index 977fb40..2d19477 100755
--- a/closed/OpenJ9.gmk
+++ b/closed/OpenJ9.gmk
## -27,8 +27,8 ## ifeq (,$(BUILD_ID))
BUILD_ID := 000000
endif
-OPENJ9_SHA := $(shell git -C $(OPENJ9_TOPDIR) rev-parse --short HEAD)
-OPENJ9OMR_SHA := $(shell git -C $(OPENJ9OMR_TOPDIR) rev-parse --short HEAD)
+OPENJ9_SHA := $(shell git --work-tree=$(OPENJ9_TOPDIR) --git-dir=$(OPENJ9_TOPDIR)/.git rev-parse --short HEAD)
+OPENJ9OMR_SHA := $(shell git --work-tree=$(OPENJ9OMR_TOPDIR) --git-dir=$(OPENJ9OMR_TOPDIR)/.git rev-parse --short HEAD)
ifeq (,$(OPENJ9_SHA))
$(error Could not determine OpenJ9 SHA)
endif
diff --git a/closed/autoconf/custom-hook.m4 b/closed/autoconf/custom-hook.m4
index 30d8a98..ed8e6af 100644
--- a/closed/autoconf/custom-hook.m4
+++ b/closed/autoconf/custom-hook.m4
## -127,8 +127,8 ## AC_DEFUN_ONCE([OPENJ9_PLATFORM_SETUP],
AC_DEFUN_ONCE([OPENJDK_VERSION_DETAILS],
[
- OPENJDK_SHA=`git -C $SRC_ROOT rev-parse --short HEAD`
- OPENJDK_TAG=`git -C $SRC_ROOT describe --abbrev=0 --tags`
+ OPENJDK_SHA=`git --work-tree=$SRC_ROOT --git-dir=$SRC_ROOT/.git rev-parse --short HEAD`
+ OPENJDK_TAG=`git --work-tree=$SRC_ROOT --git-dir=$SRC_ROOT/.git describe --abbrev=0 --tags`
AC_SUBST(OPENJDK_SHA)
AC_SUBST(OPENJDK_TAG)
])
diff --git a/closed/make/ReleaseFile.gmk b/closed/make/ReleaseFile.gmk
index d6be94c..2feca5b 100644
--- a/closed/make/ReleaseFile.gmk
+++ b/closed/make/ReleaseFile.gmk
## -16,8 +16,8 ##
# 2 along with this work; if not, see <http://www.gnu.org/licenses/>.
#
# ===========================================================================
-
-COMPANY_NAME := Eclipse OpenJ9
-OPENJ9OMR_SHA := $(shell git -C $(OPENJ9OMR_TOPDIR) rev-parse --short HEAD)
-OPENJ9_SHA := $(shell git -C $(OPENJ9_TOPDIR) rev-parse --short HEAD)
-SOURCE_REVISION = OpenJDK:$(OPENJDK_SHA) OpenJ9:$(OPENJ9_SHA) OMR:$(OPENJ9OMR_SHA)
+
+COMPANY_NAME := Eclipse OpenJ9
+OPENJ9OMR_SHA := $(shell git --work-tree=$(OPENJ9OMR_TOPDIR) --git-dir=$(OPENJ9OMR_TOPDIR)/.git rev-parse --short HEAD)
+OPENJ9_SHA := $(shell git --work-tree=$(OPENJ9_TOPDIR) --git-dir=$(OPENJ9_TOPDIR)/.git rev-parse --short HEAD)
+SOURCE_REVISION = OpenJDK:$(OPENJDK_SHA) OpenJ9:$(OPENJ9_SHA) OMR:$(OPENJ9OMR_SHA)

Related

Singularity v3.9.4 Installation - mconfig not inside a git repository and no VERSION file found

I'm trying to install Singularity after installing Go. I've confirmed that Go has been installed successfully:
$ go version
go version go1.17.6 linux/amd64
After that, I run the following commands:
$ export VERSION=3.9.4
$ wget https://github.com/sylabs/singularity/archive/refs/tags/v${VERSION}.tar.gz
$ tar -xzf v${VERSION}.tar.gz
$ cd singularity-${VERSION}
$ ./mconfig # This is where it fails
E: Not inside a git repository and no VERSION file found. Abort.
The tarball you are downloading is the one generated automatically for a tag, not the release tarball. From the release page:
Source Code
Please use the singularity-ce-3.9.4.tar.gz download below to obtain and install SingularityCE 3.9.4. The GitHub auto-generated 'Source Code' downloads do not include required dependencies etc.
Working snippet using the correct url:
export VERSION=3.9.4
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-ce-${VERSION}.tar.gz && \
tar xf singularity-ce-${VERSION}.tar.gz && \
cd singularity-ce-${VERSION} && \
./mconfig

git auto-complete doesn't prompt branch names

I tried all that mentioned in this thread, still when I double tap tab key for command git checkout I get only following suggestions
FETCH_HEAD HEAD ORIG_HEAD
whereas I am expecting branch names.
Any other suggestion to get this working?
I am on ubuutu 14.04
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Codename: trusty
UPDATE
global gitconfig file
[user]
email = abc#zxc.com
name = Abc Zxc
[credential]
helper = cache --timeout=3600
[filter "lfs"]
process = git-lfs filter-process
required = true
clean = git-lfs clean -- %f
smudge = git-lfs smudge -- %f
[diff]
tool = meld
[merge]
tool = meld
[difftool]
prompt = false
git version
kishor#kishor-ThinkCentre-E73:~$ git version
git version 1.9.1
bash-completion details
# bash/zsh completion support for core Git.
#
# Copyright (C) 2006,2007 Shawn O. Pearce <spearce#spearce.org>
# Conceptually based on gitcompletion (http://gitweb.hawaga.org.uk/).
# Distributed under the GNU General Public License, version 2.0.
Command output
git --git-dir=".git" for-each-ref --shell --format="%(refname:short)" refs/tags refs/heads refs/remotes
'160117_whole_flow_demo'
'170109_aws_mgmt_plane'
'170123_django_angular_lib'
'170124_authentication'
'170125_merging_html'
'170206_code_integration'
'170208_ng_code_snippets'
'170214_user_mapping_and_dropdown'
... and so on
P.S. I am facing the same issue in all other repos also.
I may be wrong, but I think you are trying to use a too old version of git compared to your version of git-completion.
You can try running:
git for-each-ref --format="(refname:strip=2)" refs/heads/*
If there is no output, of if there is an error, I am probably right (I expect something along the line of fatal: unknown refname: format strip=2).
In this case, either upgrade your version of git, or replace your git-completion so that it matches your version of git. If you choose the latter, you can use this version (from git's github repository).

How to freeze micro version with dependencies?

I want to build a docker image with a fixed version of micro and go dependencies. I plan to do it with dep:
git checkout git#github.com:micro/micro.git
dep ensure
git add Gopkg.toml
git add Gopkg.lock
# Build micro
CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '-w' -i -o micro ./main.go
# Build docker image
...
So, my question is does it the best solution to build consistent micro docker image?
An example of a Dockerfile can be:
FROM golang:1.9-alpine3.6 as builder
# Install package manager
RUN apk add --no-cache --virtual .go-dependencies git curl \
&& curl https://glide.sh/get | sh
# Copy files from context
WORKDIR /go/src/github.com/foo/bar
COPY . .
# Install project dependencies, test and build
RUN glide install \
&& go test ./... \
&& CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '-w' -i -o ./entry ./main.go ./plugins.go
# Build final image with binary
FROM alpine:3.6
RUN apk add --update ca-certificates && \
rm -rf /var/cache/apk/* /tmp/*
WORKDIR /
COPY --from=builder /go/src/github.com/foo/bar/entry .
ENTRYPOINT [ "/entry" ]
And the glide.yaml would look like this:
package: .
import:
- package: github.com/micro/go-micro
version: ^0.3.0
subpackages:
- client
- server
- package: github.com/micro/go-plugins
version: ^0.6.1
subpackages:
- wrapper/trace/opentracing
- broker/nats
- transport/nats
- package: github.com/opentracing/opentracing-go
version: ^1
- package: github.com/openzipkin/zipkin-go-opentracing
version: ^0.3
testImport:
- package: github.com/golang/mock
subpackages:
- gomock
- package: github.com/smartystreets/goconvey
subpackages:
- convey
In my case, dep looks great and fast enough, moreover, it's official dependency manager in go so I think it's a right choice.

The compilation of ocamlfind failed at "install -m 0755 ocaml-stub

I'm new to OCaml and I'm trying to install cohttp. After a failure because I hadn't accepted the Xcode license agreement (OS X Sierra 10.12.6) I get an error while the package manager (opam) is trying to install a sub-dependency, ocamlfind.
=-=- Processing actions -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= 🐫
[ERROR] The compilation of ocamlfind failed at "install -m 0755 ocaml-stub /Users/greg/.opam/system/bin/ocaml".
Processing 1/27: [ocamlfind: rm]
#=== ERROR while installing ocamlfind.1.7.3-1 =================================#
# opam-version 1.2.2
# os darwin
# command install -m 0755 ocaml-stub /Users/greg/.opam/system/bin/ocaml
# path /Users/greg/.opam/system/build/ocamlfind.1.7.3-1
# compiler system (4.05.0)
# exit-code 127
# env-file /Users/greg/.opam/system/build/ocamlfind.1.7.3-1/ocamlfind-93888-487c34.env
# stdout-file /Users/greg/.opam/system/build/ocamlfind.1.7.3-1/ocamlfind-93888-487c34.out
# stderr-file /Users/greg/.opam/system/build/ocamlfind.1.7.3-1/ocamlfind-93888-487c34.err
The log files are empty. The env file has this (lightly edited):
PATH=/Users/greg/.opam/system/bin:/usr/local/google-cloud-sdk/bin:/Users/greg/.nvm/versions/node/v6.5.0/bin::/opt/chefdk/bin:/usr/local/mysql/bin:/usr/local/share/npm/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/greg/.chefdk/gem/ruby/2.1.0/bin
OCAML_TOPLEVEL_PATH=/Users/greg/.opam/system/lib/toplevel
PERL5LIB=/Users/greg/.opam/system/lib/perl5:
MANPATH=/Users/greg/.opam/system/man:/Users/greg/.nvm/versions/node/v6.5.0/share/man:/usr/local/mysql/man:/usr/local/share/man:/usr/share/man:/Applications/Xcode.app/Contents/Developer/usr/share/man:/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/share/man
OPAMSWITCH=system
OPAMUTF8MSGS=1
CAML_LD_LIBRARY_PATH=/Users/greg/.opam/system/lib/stublibs:/usr/local/lib/ocaml/stublibs
TERM_PROGRAM=Apple_Terminal
TERM=xterm-256color
SHELL=/bin/bash
TMPDIR=/var/folders/fq/9ggcftcj1lsb150mldyjn5sh0000gn/T/
NVM_PATH=/Users/greg/.nvm/versions/node/v6.5.0/lib/node
Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.jNLKBcDJTX/Render
TERM_PROGRAM_VERSION=388.1.1
TERM_SESSION_ID=E845D49D-9C60-4CEA-8777-19F521ED83A1
NVM_DIR=/Users/greg/.nvm
USER=greg
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.0CA6ZcRBJQ/Listeners
__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x52
NVM_NODEJS_ORG_MIRROR=https://nodejs.org/dist
PWD=/Users/greg/dev/ocaml
EDITOR=emacs
LANG=en_CA.UTF-8
NODE_PATH=./src
XPC_FLAGS=0x0
NODE_ENV=development
XPC_SERVICE_NAME=0
HOME=/Users/greg
SHLVL=1
LOGNAME=greg
NVM_BIN=/Users/greg/.nvm/versions/node/v6.5.0/bin
NVM_IOJS_ORG_MIRROR=https://iojs.org/dist
PROMPT_COMMAND=set_tab_title
_=/usr/local/bin/opam
OPAM_PACKAGE_VERSION=1.7.3-1
OPAM_PACKAGE_NAME=ocamlfind
MAKELEVEL=
MAKEFLAGS=
I don't have a lot to go on to debug this. Any suggestions on where to go from here?
This github issue for opam led me to a solution that worked for me: https://github.com/ocaml/opam-repository/issues/10064
The solution to the github issue was that the user had the local directory, ., in their $PATH which led to the wrong install command being run. This wasn't my problem, but I had an extra : in my path.
/Users/greg/.nvm/versions/node/v6.5.0/bin::/opt/chefdk/bin:/usr/local/mysql/bin:/usr/local/share/npm/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
According to the comment below having two colons in a row in the path string will add the empty string as part of the path which is interpreted as the current directory.
Removing the extra colon allows the install of ocamlfind to succeed.

How do I build Boost from github?

I git clone boost latest source code on master branch from Github. I tried to build it but failed,
$ ./bootstrap.sh
./bootstrap.sh: line 188: ./tools/build/src/engine/build.sh: No such file or directory
-n Building Boost.Build engine with toolset ...
Failed to build Boost.Build build engine
Consult 'bootstrap.log' for more details
The content of bootstrap.log
1 ./bootstrap.sh: line 218: cd: ./tools/build/src/engine: No such file or directory
Question:
I understand there is no ./tools/build/src/engine, how do I solve this? I also noted that
-n Building Boost.Build engine with toolset ...
however, bootstrap.sh doesn't have -n option.
My develop environment:
MacOS X10.9 Xcode5.1
The current documentation for building directly from the Git repo is at Getting Started. Basically there are some additional steps to create the include directory tree and to run the build itself. NOTE, please also make sure you use the b2 command from the cloned repo. Not whatever you might have prebuilt in your system.
To checkout latest version of Boost libraries from GitHub do the following:
Checkout Boost super-project (only minimum required): git clone --single-branch --branch master --depth=1 https://github.com/boostorg/boost.git.
cd boost/
Checkout all the submodules (only minimum required): git submodule update --init --recursive --remote --no-fetch --depth=1.
If an error like following occured:
Cloning into 'libs/predef'...
remote: Counting objects: 243, done.
remote: Compressing objects: 100% (163/163), done.
remote: Total 243 (delta 128), reused 126 (delta 70), pack-reused 0
Receiving objects: 100% (243/243), 142.82 KiB | 209.00 KiB/s, done.
Resolving deltas: 100% (128/128), done.
Checking connectivity... done.
fatal: Needed a single revision
Unable to find current origin/master revision in submodule path 'libs/predef'
then use script (reget.bash):
#! /usr/bin/env bash -vex
rm -rf $3/$1 .git/modules/$1
git clone --depth=1 --branch=$2 --single-branch --separate-git-dir .git/modules/$1 https://github.com/boostorg/$1 $3/$1
where $1 is predef, $2 is master, $3 is libs, i.e. run bash reget.bash predef master libs.
An error can occur many times for different submodules, just use above script to clean up unrecoverable git error and to checkout latest commit for failed submodule. Then reuse git submodule update --init --recursive --remote --no-fetch --depth=1.
After complete checking out of all the submodules, build b2 executable. For clang it looks like:
export CC=clang
export CFLAGS="-march=native -Ofast"
export CXX=clang++
export CXXFLAGS="-march=native -Ofast"
bash bootstrap.sh --with-toolset=clang
You have obtained b2 executable. Use it to build whole Boost:
sudo ./b2 -j`nproc` toolset=clang --build-dir=/tmp/build-boost --without-mpi install
ADDITIONAL:
If you want to clone only HEAD of boost itself and HEADS of all the its submodules, then you may use following Lua script (tested on https://github.com/boostorg/boost.git repository):
-- mkdir boost ; cd boost ; lua ../git-submodules-clone-HEAD.lua https://github.com/boostorg/boost.git .
local module_url = arg[1] or 'https://github.com/boostorg/boost.git'
local module = arg[2] or module_url:match('.+/([_%d%a]+)%.git')
local branch = arg[3] or 'master'
function execute(command)
print('# ' .. command)
return os.execute(command)
end
-- execute('rm -rf ' .. module)
if not execute('git clone --single-branch --branch master --depth=1 ' .. module_url .. ' ' .. module) then
io.stderr:write('can\'t clone repository from ' .. module_url .. ' to ' .. module .. '\n')
return 1
end
-- cd $module ; git submodule update --init --recursive --remote --no-fetch --depth=1
execute('mkdir -p ' .. module .. '/.git/modules')
assert(io.input(module .. '/.gitmodules'))
local lines = {}
for line in io.lines() do
table.insert(lines, line)
end
local submodule
local path
local submodule_url
for _, line in ipairs(lines) do
local submodule_ = line:match('^%[submodule %"([_%d%a]-)%"%]$')
if submodule_ then
submodule = submodule_
path = nil
submodule_url = nil
else
local path_ = line:match('^%s*path = (.+)$')
if path_ then
path = path_
else
submodule_url = line:match('^%s*url = (.+)$')
end
if submodule and path and submodule_url then
-- execute('rm -rf ' .. path)
local git_dir = module .. '/.git/modules/' .. path:match('^.-/(.+)$')
-- execute('rm -rf ' .. git_dir)
execute('mkdir -p $(dirname "' .. git_dir .. '")')
if not execute('git clone --depth=1 --single-branch --branch=' .. branch .. ' --separate-git-dir ' .. git_dir .. ' ' .. module_url .. '/' .. submodule_url .. ' ' .. module .. '/' .. path) then
io.stderr:write('can\'t clone submodule ' .. submodule)
return 1
end
path = nil
submodule_url = nil
end
end
end
I really missed some important part of the documentation: .\b2 headers. So here follows all I've done to build thread, chrono and date-time, both release and debug, both static and shared, from version 1.60.0, cloning from the GitHub repo (in a Windows machine):
git clone --recursive https://github.com/boostorg/boost.git
cd boost
git checkout boost-1.60.0
git submodule update
bootstrap.bat
.\b2.exe headers
.\b2.exe variant=release,debug link=static,shared address-model=32 architecture=x86 --with-thread --with-chrono --with-date_time --stagedir=stage\win32 stage
.\b2.exe variant=release,debug link=static,shared address-model=64 architecture=x86 --with-thread --with-chrono --with-date_time --stagedir=stage\x64 stage
Don't forget the:
.\b2.exe headers
I had exactly the same problem. The shortest answer is that, tools/build.git is missing. https://github.com/boostorg/build.git. Simply do a git submodule update --recursive --remote on the directory. This will download the missing tools/build and other depending directories.
Note, many more submodules would be correspondingly downloaded apart from the build submodule.

Resources