I know there are two yarns:
Yarn, the node.js package manager
Apache Hadoop's Yarn
I noticed that both commands work fine:
yarn --version
yarnpkg --version
Is yarnpkg a CLI alias that yarn, the node.js package manager, implemented to avoid conflicts?
You can check if they point to the same binary by typing which yarn and which yarnpkg.
If they point to the same binary file then it's just an alias or a symlink, but if not, then they are not the same, although it is most likely that they both do the same job, and it is just that apache named it differently.
(Debian's yarn also uses yarnpkg)
Related
I am trying to add vtex using yarn but getting below error,
Usage Error: The 'yarn global' commands have been removed in 2.x - consider using 'yarn dlx' or a third-party plugin instead
yarn version: 3.2.0
Node: 16.14.0
NPM: 8.5.2
OS Details: MacOS Monterey, chip: apple m1 pro
Thanks in advance.
quick answer
Did you try to use yarn dlx instead of yarn global (removing the word add)? The resulting command:
yarn dlx vtex
explanation & official doc reference
The above command is what the instruction in the error message means to recommend. Granted, "consider using 'yarn dlx' or a third-party plugin instead", could be clearer. Nonetheless it has some useful info.
The yarn dlx documentation is probably more to the point. Their example:
yarn dlx create-react-app ./my-app
The documentation explains that the dlx command uses a temporary environment in place of using global packages.
Most package manager maintainers have come to the conclusion that global package versions are problematic. You can find various discussions of why package manager maintainers are giving (and encouraging) alternatives, usually citing the issues developers encounter with different projects requiring different versions.
In any case, it's hard to argue that the best option is downgrading. Likely you'll not regret trying the up-to-date approach.
On mac OS check yarn version by running command in terminal,
yarn --version
If it is greater than or equal to 2. Please downgrade yarn version to 1.xx.x using,
yarn set version 1.22.1
I was using lerna yarn workspaces to manage the my monorepos but due to dependencies incompatible version issues i downgraded my yarn version and after that yarn cmds start failing. Then i again upgraded my yarn version to latest 1.22.10 but after that no yarn cmd is running. running any cmd gives below error attached in SS:
Can someone help me in this, i have tried multiple times reinstalling node, nvm and yarn but nothing is working for me.
I have tried downloading yarn using brew, npm globally but nothing works
Finally, after slogging for past 2 days and searching insanely online. Able to resolve this by hit and trial.
If you have yet to install or see yarn at /.yarn/ you can install it directly following https://classic.yarnpkg.com/en/docs/install/ or
curl -o- -L https://yarnpkg.com/install.sh | bash
Then follow below steps:-
mkdir ~/.yarn/releases/
cp ~/.yarn/lib/cli.js ~/.yarn/releases/yarn-1.19.0.cjs
yarn -v
voila, it works, Never been so happy before
I read that on OS X you can install Yarn either by
curl -o- -L https://yarnpkg.com/install.sh | bash
brew install yarn
npm i -g yarn
What functional difference is there between these three methods? Why would someone choose one over the others?
when using brew to install packages, you install them system wide. that is, you cannot have more than one version for the same package, which is usually problematic. for this reason, many other technologies have spawn, such as docker, snap.
moreover, each package manager has its own lifecycle and packs original package in a different manner for ease of use, distribution and maintenance. for instance, npm container is based on the release of npm package itself.
usually, you should stick to the package manager of the same ecosystem that you are using. specifically to your case, it will be recommended to use npm to install and update your package (using package.json). which will give each of your project to pin and lock the desired yarn version that you like, without any affecting your system wide.
speaking of npm, you might wish to look at this answer
curl downloads the installation script from yarnpkg.com, and installs yarn using that script
brew is a package manager for MacOS. It's meant to make things easier for people when installing commands for the terminal. When you install with brew, the package get's put into /usr/local/bin instead of /usr/bin so I believe that this is kind of like a virtual environment, and yarn wouldn't be installed into the core of your machine. You'll have to install homebrew before you can use it, and you install it using curl. I believe that there is less risk when using homebrew because of the reasons that it is kind of like a virtual environment
npm is a package manager for javascript, it's the same as yarn. It's meant for easy installation of javascript packages.
I use brew for all installs to the terminal, and npm for all installs of javascript packages.
I have nvm:
nvm ls
v8.11.3
v8.11.4
-> v11.1.0
default -> 8.11.4 (-> v8.11.4)
node -> stable (-> v11.1.0) (default)
stable -> 11.1 (-> v11.1.0) (default)
I installed yarn with:
sudo apt-get install --no-install-recommends yarn
I also added in .bashrc alias node=nodejs. But when I try yarn install I see:
Yarn requires Node.js 4.0 or higher to be installed.
How can I fix it?
This gist helped on this problem.
Run the following commands
echo "==> Installing Yarn package manager"
rm -rf ~/.yarn
curl -o- -L https://yarnpkg.com/install.sh | bash
# Yarn configurations
export PATH="$HOME/.yarn/bin:$PATH"
yarn config set prefix ~/.yarn -g
And add the following in ~/.bashrc
export PATH="$HOME/.yarn/bin:$HOME/.config/yarn/global/node_modules/.bin:$PATH"
It should then work properly without the warning after restarting the shell.
First uninstall the nodejs package:
sudo apt remove nodejs
Ubuntu 16.04 contains a version of Node.js in its default repositories that can be used to easily provide a consistent experience across multiple systems. At the time of writing, the version in the repositories is v4.2.6. This will not be the latest version, but it should be quite stable and sufficient for quick experimentation with the language.
In order to get this version, we just have to use the apt package manager. We should refresh our local package index first, and then install from the repositories:
sudo apt-get update
sudo apt-get install nodejs
If the package in the repositories suits your needs, this is all you need to do to get set up with Node.js. In most cases, you’ll also want to also install npm, which is the Node.js package manager. You can do this by typing:
sudo apt-get install npm
This will allow you to easily install modules and packages to use with Node.js.
Because of a conflict with another package, the executable from the Ubuntu repositories is called nodejs instead of node. Keep this in mind as you are running software.
To check which version of Node.js you have installed after these initial steps, type:
nodejs -v
Screenshot for nodejs version
I just want to mention that my configuration file looked something like that
export PATH=$PATH:`yarn global bin`
#NVM INITIALIZATION STUFF
(yarn docs recommended $PATH)
the export was before my nvm initialization. Which meant node was not available during the runtime of that line. So I just switched my configuration file to be
#NVM INITIALIZATION STUFF
export PATH=$PATH:`yarn global bin`
I had the same issue. by putting nvm path above yarn path didn't solve the issue then I looked up for a solution in man page and solve the issue by setting default node version on a shell.
Current lts version is v14.17.6 so i install it use it and set default node version on a shell.
nvm install --lts
nvm use --lts
nvm alias default <version>
Additional you can set always default to the latest available node version on a shell by running below command.
nvm alias node <version>
I am new to Spark, Hadoop and Cloudera. We need to use a specific version (1.5.2) of Spark and also have the requirement to use Cloudera for the cluster management, also for Spark.
However, CDH 5.5 comes with Spark 1.5.0 and can not be changed very easily.
People are mentioning to "just download" a custom version of spark manually. But how to manage this "custom" spark version by Cloudera, so I can distribute it across the cluster? Or, does it need to be operated and provisioned completely separate from Cloudera?
Thanks for any help and explanation.
Yes, It is possible to run any Apache Spark version .!!
Steps we need to make sure before doing it:
You have YARN configured in the CM. After which you can run your application as a YARN application with spark-submit. please refer to this link. It will be used to work like any other YARN application.
It is not mandatory to install spark, you can run your application.
Under YARN, you can run any application, with any version of Spark. After all, Spark it's a bunch of libraries, so you can pack your jar with your dependencies and send it to YARN. However there are some additional, small tasks to be done.
In the following link, dlb8 provides a list of tasks to be done to run Spark 2.0 in an installation with a previous version. Just change version/paths accordingly.
Find the version of CDH and Hadoop running on your cluster using
$ hadoop version
Hadoop 2.6.0-cdh5.4.8
Download Spark and extract the sources. Pre built Spark binaries should work out of the box with most CDH versions, unless there are custom fixes in your CDH build in which case you can use the spark-2.0.0-bin-without-hadoop.tgz.
(Optional) You can also build Spark by opening the distribution directory in the shell and running the following command using the CDH and Hadoop version from step 1
$ ./dev/make-distribution.sh --name custom-spark --tgz -Psparkr -Phadoop-2.6 -Phive -Phive-thriftserver -Pyarn
Note: With Spark 2.0 the default build uses Scala version 2.11. If you need to stick to Scala 2.10, use the -Dscala-2.10 property or
$ ./dev/change-scala-version.sh 2.10
Note that -Phadoop-provided enables the profile to build the assembly without including Hadoop-ecosystem dependencies provided by Cloudera.
Extract the tgz file.
$tar -xvzf /path/to/spark-2.0.0-bin-hadoop2.6.tgz
cd into the custom Spark distribution and configure the custom Spark distribution by copying the configuration from your current Spark version
$ cp -R /etc/spark/conf/* conf/
$ cp /etc/hive/conf/hive-site.xml conf/
Change SPARK_HOME to point to folder with the Spark 2.0 distribution
$ sed -i "s#\(.*SPARK_HOME\)=.*#\1=$(pwd)#" conf/spark-env.sh
Change spark.master to yarn from yarn-client in spark-defaults.conf
$ sed -i 's/spark.master=yarn-client/spark.master=yarn/' conf/spark-
defaults.conf
Delete spark.yarn.jar from spark-defaults.conf
$ sed '-i /spark.yarn.jar/d' conf/spark-defaults.conf
Finally test your new Spark installation:
$ ./bin/run-example SparkPi 10 --master yarn
$ ./bin/spark-shell --master yarn
$ ./bin/pyspark
Update log4j.properties to suppress annoying warnings. Add the following to conf/log4j.properties
echo "log4j.logger.org.spark_project.jetty=ERROR" >> conf/log4j.properties
However, it can be adapted to the opposite, since the bottom line is "to use a Spark version on an installation with a different version".
It's even simpler if you don't have to deal with 1.x - 2.x version changes, because you don't need to pay attention to the change of scala version and of the assembly approach.
I tested it in a CDH5.4 installation to set 1.6.3 and it worked fine. I did it with the "spark.yarn.jars" option:
#### set "spark.yarn.jars"
$ cd $SPARK_HOME
$ hadoop fs mkdir spark-2.0.0-bin-hadoop
$ hadoop fs -copyFromLocal jars/* spark-2.0.0-bin-hadoop
$ echo "spark.yarn.jars=hdfs:///nameservice1/user/<yourusername>/spark-2.0.0-bin-hadoop/*" >> conf/spark-defaults.conf