How do I validate my YAML file from command line? - yaml

I am having issues pulling from a YAML config file:
Fatal error: while parsing a block mapping; expected <block end>, but found block entry
While there are plenty of online YAML validators, which I have tried and have helped, I'd like to validate my YAML files from the command line and integrate this into my continuous integration pipeline.

With basic Ruby installation this should work:
ruby -ryaml -e "p YAML.load(STDIN.read)" < data.yaml
Python version (thx #Murphy):
pip install pyyaml
python -c 'import yaml, sys; print(yaml.safe_load(sys.stdin))' < data.yaml

You could use yamllint. It's available in Homebrew, etc. It can be used for syntax validation as well as for linting.

Given that you have a perl install on the server you are working on, and it has some of the basic YAML tools, you can use...
perl -MYAML -e 'use YAML;YAML::LoadFile("./file.yaml")'
It should be noted that this will be strict in it's interpretation of the file, but useful.

To correct your .yaml files I recommend the tool yamllint. It can be launched easily from the local console.
The package yamllint is available for all major operating systems.
It's installable from the system's package sources. (e.g. sudo apt-get install yamllint).
See documentation for quick start and installation.

My preferd way is
yamllint -d "{extends: default, rules: {quoted-strings: enable}}" .
Since I really want to catch quote errors,
e.g. validate: bash -c ' ' \""
This is valid yaml, since yaml will just quote the string and turn it into:
validate: "bash -c ' ' \\\"\""
Whilst there was just clearly a quote missing at the beginning of the validate comand.
So a normal yaml checker will not detect this, yamllint wil not even detect this in it's default configuration, so turn on quoted-strings checker.

If you got no interpreter installed in your environment but still got a curl, then you could use an online linter project like Lint-Trilogy:
curl -X POST --data "data=$(cat myfile.yml)" https://www.lint-trilogy.com/lint/yaml/json
It delivers the validation result incl. error descriptions (if any) as json or csv or, where sufficient, as plain text true or false.
It's available as docker file, too. So if you often need a REST based linter, perhaps in a CI/CD pipeline, it may be handy to host an own instance on your site.

Or alternately installed (free) Eclipse IDE and then YEdit yaml editor and see your yaml with syntax highlighting, error flags, and outline views. One time setup cost works pretty well for me.

Related

which command doesn't work on my computer

In the past, we 'which' command to get the info of the relevant software on our computer, Like:
which python
which git
But now it seems don't work on my MacOS Mojave. Is there anything wrong with my setting?
New edition:
The result will turn out to be like this
AA:~ AA$ which python
/usr/bin/which: illegal option -- -
usage: which [-as] program ...
New edition2:
AA:~ AA$ type --all which
which is aliased to `alias | /usr/bin/which --tty-only --read-alias --show-dot --show-tilde'
which is /usr/bin/which
AA:~ AA$ type -all python
python is /Users/AA/anaconda3/bin/python
python is /Users/AA/anaconda/bin/python
python is /usr/bin/python
AA:~ AA$
The alias is what causes the error message. Apparently the alias definition is simply erroneous for your system. It's not clear what defined this alias or why; it seems wrong on several levels, so I doubt it's part of the standard install.
You can remove the alias with unalias which; but I'm guessing it's defined in one of your startup files, and should be removed from there - after this point, it should be gone for good the next time you log in.
But anyway, you should not be using which - it's better to accustom yourself to the POSIX standard command type. It was introduced specifically to replace which but it's apparently still hard to eradicate the old command from people's minds.
For me, this was when something from a yarn global upgrade replaced my system's /usr/bin/which.
/usr/bin/which -> ../local/share/.config/yarn/global/node_modules/.bin/which
Since your error message is exactly the same as the text strings inside that program, it is possible that your issue is from a similar source.
console.error('which: illegal option -- ' + flag)
console.error('usage: which [-as] program ...')
"Fixed" with reinstalling which through the system's package manager (e.g. sudo dnf reinstall which). That may now interfere with whatever JavaScript package depended on that, but I plan on removing those globals, so I will not find out.
Better fix: I feel like my system is dirty now and needs all of its bin files checked... Comparing everything in /usr/local/share/.config/yarn/global/node_modules/.bin to see if it has an equivalent in /usr/bin seems like an easy enough way to look for other points of interference.
for x in $(ls /usr/local/share/.config/yarn/global/node_modules/.bin/*); do
ls -l /usr/bin/"$(basename "${x}")"
done
Then those can be checked against the system's package manager with commands like the following. Then reinstalled if needed.
sudo dnf info ...
sudo dnf provides ...
More:
Seems weird to me that it would allow clobbering existing system programs. At the very least, I would expect it to have used /usr/local/bin instead. That extra system and system-like package management seems to be why they have done away with yarn global in yarn 2 (berry).
I have different packages for sudo yarn global list and yarn global list. Maybe using sudo at some point was my mistake?
P.S. Thanks goes to #tripleee, I was unaware of type. I have seen various other ways of attempting to handle different which programs and different versions of which, along with alternatives like test (and others I have forgotten), but type looks worth trying as a replacement.
P.P.S. Annoying to find out that this OS is configured to run tab completion through which.
P.P.P.S. Yes, I know scripting with the output ls is a bad habit.

using ctags or cscope(without interactive mode) from commandline

I am using a custom editor for an embedded systems project. For source code I would like to get ctags working from command line and give me search results on commandline. Other option is to work with cscope in non interactive mode so I can include it in my editor at a later date. I did some initial web search but couldn't find anything relevant to accomplish this.
Does anyone know how to use either of these tools from command line?? Any tutorial?
Thanks.
Have a great day.
Using readtags.c shipped as part of ctags implementation, you can search a tag from given tags file.
Let me show an example:
$ ctags -R main
$ readtags -t tags kindDefinition
kindDefinition main/types.h /^typedef struct sKindDefinition kindDefinition;$/
$ readtags -t tags -e kindDefinition
kindDefinition main/types.h /^typedef struct sKindDefinition kindDefinition;$/;" kind:t typeref:struct:sKindDefinition

pandoc version 1.12.3 or higher is required and was not found (R shiny)

I have a problem generating a pdf report from my app shiny which is hosted on a server.
the app works fine but when I press the button to download the report, I get this error :
pandoc version 1.12.3 or higher is required and was not found.
The proble is that if I type pandoc -v I get:
pandoc 1.12.3.3
Compiled with texmath 0.6.6, highlighting-kate 0.5.6.1.
Syntax highlighting is supported for the following languages:
actionscript, ada, apache, asn1, asp, awk, bash, bibtex, boo, c, changelog,
clojure, cmake, coffee, coldfusion, commonlisp, cpp, cs, css, curry, d,
diff, djangotemplate, doxygen, doxygenlua, dtd, eiffel, email, erlang,
fortran, fsharp, gnuassembler, go, haskell, haxe, html, ini, java, javadoc,
javascript, json, jsp, julia, latex, lex, literatecurry, literatehaskell,
lua, makefile, mandoc, markdown, matlab, maxima, metafont, mips, modelines,
modula2, modula3, monobasic, nasm, noweb, objectivec, objectivecpp, ocaml,
octave, pascal, perl, php, pike, postscript, prolog, python, r,
relaxngcompact, restructuredtext, rhtml, roff, ruby, rust, scala, scheme,
sci, sed, sgml, sql, sqlmysql, sqlpostgresql, tcl, texinfo, verilog, vhdl,
xml, xorg, xslt, xul, yacc, yaml
Default user data directory: /home/daniele/.pandoc
Copyright (C) 2006-2013 John MacFarlane
Web: http://johnmacfarlane.net/pandoc
This is free software; see the source for copying conditions. There is no
warranty, not even for merchantability or fitness for a particular purpose.
So I suppose I have the right version for that. TexLive is also installed and the path is in $PATH.
Server.R
library(shiny)
library(drsmooth)
library(shinyBS)
library(knitr)
library(xtable)
library(rmarkdown)
shinyServer(function(input, output,session) {
output$downloadReport <- downloadHandler(
filename = function() {
paste('report', sep = '.','pdf')
},
content = function(file) {
src <- normalizePath('report.Rmd')
# temporarily switch to the temp dir, in case you do not have write
# permission to the current working directory
owd <- setwd(tempdir())
on.exit(setwd(owd))
file.copy(src, 'report.Rmd')
library(rmarkdown)
out <- render('report.Rmd')
file.rename(out, file)
})
output$tb <- renderUI({
p(h4("Report")),
"Dowload a the report of your analysis in a pdf format",
tags$br(),downloadButton('downloadReport',label="Download report"),
tags$em("This option will be available soon")
})
})
* report.Rmd* does not contain any sort of calculation, it's only text.
The pdf generation works fine on my local version (MacOS) but not on the server.
I'm here to give other information if needed.
Go into RStudio and find the system environment variable for RSTUDIO_PANDOC
Sys.getenv("RSTUDIO_PANDOC")
Then put that in your R script prior to calling the render command.
Sys.setenv(RSTUDIO_PANDOC="--- insert directory here ---")
This worked for me after I'd been struggling to find how rmarkdown finds pandoc. I had to check github to look at the source.
Another option so that this works for all your R scripts is to define this variable globally.
On Debian/Ubuntu, add the following line to your .bashrc file:
export RSTUDIO_PANDOC=/usr/lib/rstudio/bin/pandoc
On macOS, add the following to your .bash_profile file:
export RSTUDIO_PANDOC=/Applications/RStudio.app/Contents/MacOS/pandoc
On Windows (using Git Bash), add the following to your .bashrc file:
export RSTUDIO_PANDOC="/c/Program Files/RStudio/bin/pandoc/"
The easiest way I solved this issue is to pass the Sys.setenv(..) command inside the crontab command prior to calling the RMarkdown::render. You need to separate the two commands with a semicolon:
R -e "Sys.setenv(RSTUDIO_PANDOC='/usr/lib/rstudio-server/bin/pandoc'); rmarkdown::render('File.Rmd', output_file='output.html')"
(Remember that the rstudio-server path differs from the non-server version)
For those not using RStudio, you may just need to install pandoc on your system. For me it was
sudo pacman -S pandoc
and it worked (Arch Linux).
I'm using Arch Linux, and RStudio as well..
the only thing that worked for me was:
sudo pacman -S pandoc
:)
If anyone is having this issue and also use anaconda, its possible they were having my issue. The rstudio shell does not load the .bashrc file when it starts up meaning if your version of pandoc is installed within anaconda Rstudio will not find it. Installing pandoc separately with a command like sudo pacman -S pandoc worked for me!
I had a similar problem with pandoc on Debian 10 while building a bookdown document. In the Makefile what I did was:
# use rstudio pandoc
# this rule sets the PANDOC environment variable from the shell
build_book1:
export RSTUDIO_PANDOC="/usr/lib/rstudio/bin/pandoc";\
Rscript -e 'bookdown::render_book("index.Rmd", "bookdown::gitbook")'
# use rstudio pandoc
# this rule sets the environment variable from R using multilines
build_book2:
Rscript -e "\
Sys.setenv(RSTUDIO_PANDOC='/usr/lib/rstudio/bin/pandoc');\
bookdown::render_book('index.Rmd', 'bookdown::gitbook')"
These two rules are equivalent and knit the book successfully.
I just didn't like the long Rscript command:
Rscript -e "Sys.setenv(RSTUDIO_PANDOC='/usr/lib/rstudio/bin/pandoc'); bookdown::render_book('index.Rmd', 'bookdown::gitbook')"
Hey I just beat this error. I solved this by deleting the 2 pandoc files, "pandoc" and "pandoc-citeproc" from the shiny-server folder. I then created a link for each of these files from the rstudio-server folder. It worked like a charm. This was an issue for me when I was trying to embed leaflet in the rmarkdown documents from running a shiny-server on a linux machine. I found it odd that when I ran it in rstudio on the same linux machine it worked fine, but not when I ran it using shiny-server. So the shiny-server install of pandoc is old/outdated.
Cheers
For Windows 10, RStudio 2022.12.0
Pandoc is installed with RStudio, so I prefer to use the already-installed pandoc.exe. As far as I can tell where it is installed changes from time to time. In the last couple of years, I've seen it in the below locations (the top one is where it is with my current verison of RStudio).
January 2023-
"C:/Program Files/RStudio/resources/app/bin/quarto/bin/tools"
August 2022-
"C:/Program Files/RStudio/bin/quarto/bin/tools"
"C:/Program Files/RStudio/bin/quarto/bin"
"C:/Program Files/RStudio/bin/pandoc"
Once you know where the pre-installed pandoc is you can include this line in your .R file as the top answer from Chris/Yihui indicate and it works for me.
Sys.setenv(RSTUDIO_PANDOC = "C:/Program Files/RStudio/resources/app/bin/quarto/bin/tools")
If you are trying to run a script from the command line on Windows you just need to have the directory path in the PATH variable*. You can also create a separate User variable named RSTUDIO_PANDOC and give this variable the directory*. Then close and reopen any terminals to refresh the system paths.**
*Experiment with a trailing / if you are having issues.
**I was unable to point to a UNC path. The // at the beginning of the path hosed the rmarkdown package pandoc functions. If you are using a UNC path, you must map it to a drive and reference the drive letter. There are ways to do this dynamically. I use a DOS/batch script which I found via Google.
I was facing a similar issue in IntelliJ R plugin. I solved it by copying the pandoc file in ~/.IntelliJIdea2019.3/config/plugins/rplugin/pandoc
On Windows, and without RStudio, you can install pandoc with choco install pandoc or via the pandoc website, https://pandoc.org/.
Make sure to restart your IDE to ensure it picks up the new install.

I want to make a shell command program in ruby for windows but everything is in linux

I have a ruby program named options that I want to run from the command line with a few options like
options -add 400
options -sub 600
options -h
I am already using optparse to interpret the commands but I want to run the program as its own script, but have to run it as
ruby options -add 400
I've looked it up and found some answers like How to create a shell command supporting Ruby? which seems to be linux and I'm not sure of part of the explanation like which bin to put in, or answers like Shell execute from Ruby whose answer still requires ruby in the command. Can anyone explain how to do this in more depth, or direct me to a source that explains it without relying on a linux platform?
Assuming that you have ruby in your PATH on Windows couln't you just create wrapper script and place it in folder present at your PATH?
ruby options %*
Do you have thought about to make it as gem ? There is a simple guide http://guides.rubygems.org/, then you can make the gem which will be working on linux and windows too. I don't know if you know but when I did software avaible under shell or bash I used slop (https://github.com/injekt/slop) it is very convenient the gem. Please see the source of slop, because it is an example of gem, all structure and necessary files to build the gem.

curl runs from command line, but not in ruby script

The following command (and variations of it) run okay in terminal in mac, run okay in a ruby program executed on a mac, run okay directly in the windows command prompt, but fail with a parsing error when I try to run it inside of a ruby file on windows.
curl -u"user:pwd" -d"{\"name\":\"new_repo_beepo\"}" https://api.github.com/user/repos --insecure
I've tried executing it with backticks, %x() and system. I've also tried substitution of strings and json'ing pieces of it, without any luck. From what I can determine, the failure point is in the -d"{\"name\":\"repo_name\"}" section, but that's only from trying the command without it. Regardless, in each variation of the command on windows in ruby, I get a JSON parsing error.
Have you tried using rest-client?
It is a gem and works quite nice. It is probably better than using system()or %x() and it is definitely more secure (You can inject malicious bash commands on system() so it needs to be used carefully).
It is quite simple to use. Just install the gem and require it one your ruby file.
NOTE: If using Rails just add it to your Gemfile(No need to require it on each file).
require 'rest_client'
RestClient.get 'https://api.github.com/user/repos', {params: {id: 50, foo: 'bar'}}
You can also use some params for --insecure ssl.
It seems likely this is a parsing/quoting problem with the shell mechanism used by ruby to run the command on windows. Have you tried the tokenized form of system, eg:
system('curl', '-u"user:pwd"', '-d"{"name":"new_repo_beepo"}"', 'https://api.github.com/user/repos', '--insecure')
On posix that will send the arg vector as specified directly to the exec'ed program without letting the shell get in the way; probably the same semantics hold for windows.
You shouldn't need to exec curl to do this as ruby has stdlib Net::HTTP. In comments you mentioned that you've had problems with this module under jruby, but we have jruby services here exercising http[s] requests just fine, so you might try posting a question addressing the specific problems you have with jruby and native http client libs.

Resources