How can I convert Markdown documents to HTML en masse? - bash

I'm writing some documentation in Markdown, and creating a separate file for each section of the doc. I would like to be able to convert all the files to HTML in one go, but I can't find anyone else who has tried the same thing. I'm on a Mac, so I would think a simple bash script should be able to handle it, but I've never done anything in bash and haven't had any luck. It seems like it should be simple to write something so I could just run:
markdown-batch ./*.markdown
Any ideas?

Use pandoc — it's a commandline tool that lets you convert from one format to another. This tool supports Markdown to HTML and back.
E.g. to generate HTML from Markdown, run:
pandoc -f markdown index.md > index.html

This is how you would do it in Bash.
for i in ./*.markdown; do perl markdown.pl --html4tags $i > $i.html; done;
Of course, you need the Markdown script.

If you have Node.js installed, then you can use the [MdPugToHtml] converter (https://www.npmjs.com/package/md-pug-to-html). It massively converts Markdown to Html. Moreover, it is possible to use Pug templates, but you can use them without templates.
The conversion is performed in the terminal with just one command:
npx md-pug-to-html /home/content
where:
npx is an npm command that installs md-pug-to-html at the first launch, and then launches the md-pug-to-html converter.
/home/content is a directory with your Markdown files. You may have another one.
The converter has various settings and can be used both in the CLI command line and has an API for use in applications.
There is detailed documentation on the MdPugToHtml converter in English and Russian.

You can do this really easily with VS Code. (Well, this is not a command line tool, but proved itself to be super helpful.)
Install the Markdown All In One extension by Yu Zhang
Open the VS Code Command Palette (Ctrl-Shift-P), and select Markdown All In One: Print documents to HTML (select a source folder)
Tip: If you want to make your export portable, you want to change absolute image paths to relative paths by using the following setting in your settings.json (Ctrl-Shift-P -> Preferences: Open Settings (JSON))
"markdown.extension.print.absoluteImgPath": false
In this way, after conversion, just copy all non-markdown files (images) to the destination folder and the HTML pages are portable.

I use this in a .bat file:
#echo off
for %i in (*.txt) python markdown.py "%i"

// using Bash in mac
for i in *.md; do asciidoc $i; done;

Related

how to generate html output in sphinx-needs instead of needs.json

i am trying to use/understand sphinx-needs.
As soon as i run the following command it is generating needs.json
sphinx-build -b needs source build
But how to generate the html output from needs.json?
i don't want to have any custom layouts a default or predefined layouts (such as'complete') will do
Thanks for your answers
I have generated my sphinx folders using the quickstart command
$ sphinx-quickstart
you will have something that looks like this
After writing your needs in .rst format
the following command will generate the html files.
$ make html
PS: You should not have any dependency problems.
Sphinx-needs need to be installed separately.

How to convert images from TIFF to JPG preserving the comments and tags

I am using preview (that comes with OS X El Capitan) feature to convert a file form TIFF format into JPG for example. I expected the export process will include the original comments, but it doesn't happen (it applies also for the tag fields).
The generated JPG file has no comment
The compression and change image format work, but the META INFO such as comment or tags are not exported.
Any suggestion or workaround about how to include that information. I need to convert about 500 images so manually copy/paste doesn't work for me.
Updated Answer
In the light of your comments, I think the best way forward is to try and identify how/where the comments are stored for each platform (Windows vs macOS) and then to decide which method you want to use going forward.
macOS Finder/Spotlight comments will not be legible on Windows, so if you want Windows compatibility, you need to standardise on JPEG or EXIF comments.
I recommend using exiftool which you can install with homebrew, using:
brew install exiftool
Then I suggest you try extracting the comments from your files to see how/where they are stored:
exiftool -a image.jpg
will show you all tags in image.jpg. Your comments may be under:
comment - which is the JPEG comment, or
EXIF:UserComment - which is the EXIF comment
If you find your comments in the JPEG or the EXIF section, you can extract just the comments with:
exiftool -comment image.jpg # extract JPEG comment
exiftool -EXIF:UserComment image.jpg # extract EXIF UserComment
Add the option -s3 to suppress the field-names in the above to save having to parse them out.
Likewise, you can set the comments with:
exiftool -comment="FUNKY JPEG COMMENT" image.jpg # set JPEG comment
exiftool -EXIF:UserComment="FUNKY EXIF USER COMMENT" image.jpg # set EXIF UserComment
You can also extract the EXIF user comments to a CSV with:
exiftool -EXIF:UserComment -csv *.jpg
SourceFile,UserComment
a.jpg,FUNKY EXIF:UserComment
b.jpg,b FUNKY EXIF:UserComment
You can also apply comments from a CSV.
You should also be able to extract macOS/Spotlight/Finder comments using the script in my main answer:
$HOME/macOSGetFinderComment "/Users/someone/soneFile.tif"
Original Answer
I would suggest you try the following using ImageMagick.
First, use the Finder, or any other tool you are familiar with, to make a copy of your photos including the entire directory structure to some new place where we cannot damage your existing photos. So, let's say you copy (NOT move) the entire tree of TIFs to a subdirectory called "NEW" inside your HOME directory.
Then start the Terminal and change directory to "NEW":
cd NEW
Easy Method
If all the TIFs are in a single directory or two, just use mogrify:
mogrify -format jpg *.tif
Harder Method
If the TIF files are in multiple directories, you will need to work a bit harder. Inside Terminal copy and paste this:
find NEW -name \*.tif -exec sh -c 'new="${1%.tif}.jpg"; convert "{}" "$new"' _ {} \;
That starts looking in the "NEW" directory for files named "*.tif". When it finds one, it starts a new shell (sh) passing it the filename of the TIF. It then works out the new filename by replacing a trailing "tif" with "jpg" and invokes ImageMagick convert to do the conversion.
As regards the Finder/Spotlight comments, here is a little script to get the Finder comment of a file:
#!/bin/bash
# macOSGetFinderComment
# Pass an absolute path to the file!
file=$1
osascript<<EOF
tell application "Finder" to get comment of item POSIX file "$file"
EOF
And here is one to set the Finder/Spotlight comment:
#!/bin/bash
# macOSSetFinderComment
# Pass an absolute path to the file!
file=$1
comm=$2
osascript<<EOF
tell application "Finder" to set comment of item POSIX file "$file" to "$comm"
EOF
So, I would save those 2 scripts in your HOME directory and then make them executable with:
cd
chmod +x macOS*FinderComment
Then save this file in your HOME directory under $HOME/CopyComments:
#!/bin/bash
shopt -s nullglob
for f in $(pwd)/*.tif; do
comment=$($HOME/macOSGetFinderComment "$f")
new="${f%.tif}.jpg"
echo Setting comment of $new to $comment
$HOME/macOSSetFinderComment "$new" "$comment"
done
and make it executable with:
chmod +x $HOME/CopyComments
and run it with:
cd NEW
$HOME/CopyComments
I have posted this problem also in Apple Community, here is the solution proposed by VikingOSX. It is a big piece of code, so better download it from here or directly from the Apple Community Link mentioned. Here is a description about the solution as described in the original post:
Prompts for a source folder, and a destination folder.
Duplicates folder hierarchy from source to destination folder.
Selects all TIFF images in the folder hierarchy and converts them to JPEG.
For sub-folders and their files, transfers the original Finder comments, color tags and tag name(s) to the destination hierarchy.
The compression level for the JPG file is high, it can be modified for: medium or low in the line: save this_img as JPEG in outfile_name with compression level medium with icon
Limitation: Source folder can only contain one-level of sub-folders. Ignoring this will result in unplanned results.
Additional Comments
Uses a with timeout clause to allow for large number of files. AppleScript does not yet support Finder tag names, so this script uses AppleScript/Objective-C to get and set those tag name(s). Due to this extension, the script now requires AppleScript 2.4 and must be run on OS 10.10 or later.
Due to the AppleScript/Objective-C code, the script cannot be run interactively as a script/script bundle without using the control+command+R keyboard shortcut. A test is made when the script starts, and will warn appropriately. It is best to save the script as an application to avoid this keyboard shortcut altogether.
Usage
Save the script and then copy and paste the file contains into the Script Editor (you can find the application in the folder: Utilities under the name: Script Editor), compile and save the file with the format: Application, then double click on it to run the script application.
I have tested the script under with Mac Air 2010, with OS El Capitan, for a folder with 884 TIFF files with 2.25GB size and it takes about 18 minutes to convert them into JPG files with medium compression level. The generated files will contain the tags and comments from the original equivalent TIFF file.
Disclaimer
Comment and tags generated in one platform for example Windows or mac OS are not visualized in the other platform. Tags created in Windows are treated in mac OS as keywords (Comand+i for visualizing them), but comments generated in Windows are not visualized in mac OS. This is general incompatibility problem that apply for photos in any format (for example TIFF or JPG).
EDIT (updated solution for solving cross-platform problem with comments)
Taking the idea from #MarkSetchell, I adapted the original script to at least solve the cross-platform problem from macOS to Windows, i.e. a comment from macOS can be seen in Windows platform. The idea is to use EXIF metadata. Then the Applescript will invoke the shell script for invoking the exiftool:
set uxFilepath to POSIX path of NewIMG
do shell script "/usr/local/bin/exiftool -overwrite_original -EXIF:UserComment=\"" & cmtstr & "\" " & uxFilepath
Windows processes the UserComment metadata from EXIF as a regular file comment. Now same comment on the TIF file will be on the JPG and also because such comments were copied (copy-paste) into an EXIF metadata the same information will be visualized under Windows. The same idea can be used for other file properties, in case Windows/Mac read it.
The EXIF metadata in macOS can be visualized from command line as suggest #MarkSetchell, but also from Finder: Command+o (to launch preview app), then Command+i (to launch the inspector). Then click on tap: "More Info", then the tab EXIF.
For the opposite process will require an script that does the opposite, i.e., copy EXIF comment using exiftool, into macOS comment. I have verified that in such case the Windows comment will appear under the label: XPComment. The script uses: UserComment, but it works using XPComment as label in both directions.

Install .ttf font using command line

On OS X you can install .ttf font files by double clicking them. This is a hassle when dealing with multiple files. Is there a command to install font files using the Terminal app ?
You could copy the fonts using
cp myfont.ttf /Library/Fonts/
or multiple files
cp fontsFolder/*.ttf /Library/Fonts/
Install fonts with the following command line. Replace BRLNSR with your font and add in more lines if you need more fonts.
cd ~/Library/Fonts && { curl -O 'https://github.com/bloomberg/scatteract/raw/master/fonts/BRLNSR.TTF' ; cd -; }
This code does the following:
cd into the fonts directory
curl downloads the font
pops back to the original directory
This relies on the very nice bloomberg fonts github repo with a bunch of fonts stored - but you could change the curl url to wherever the font you want is located online.
The clever way the cd into a directory, download and pop out again came from user Atle's answer here.
To make the newly copied fonts available to applications requires activating them (for the process, for the user, or for the whole system). This you can do programmatically through various CoreText commands, depending on what you want to do with the font(s). I’m not sure if there’s a way to do this from the command line without turning on auto-activation for everything. See atsutil man pages for (scant) details.

Atom and Pandoc

Using Atom, I'm trying to generate a PDF file from markdown.
I installed markdown-preview-plus plugin, which supports pandoc, and then installed pandoc and configured the plugin to use it.
Now, markdown-preview-plus does recognize pandoc, but I cannot see any command to generate a PDF. Plugin's web page seems to say nothing about that.. Can you help me?
markdown-preview-plus
Atom's markdown-preview-plus package generates an HTML preview, as #mb21 alluded to. This is clear by the fact that you can right click on the preview and select Copy as HTML. With the Enable Pandoc Parser option enabled, MPP is indeed using pandoc to generate this HTML preview.
In light of your question, I tried adding the following to the Pandoc Options: Commandline Arguments setting in MPP:
--to=latex, --output=temp.pdf
Note 1: you can't specify --to=pdf because pandoc can only generate PDF by first generating a LaTeX file.
Note 2: the above doesn't work because MPP essentially passes the contents of the editor window through pandoc 'on-the-fly', so you can't really hijack the --output setting.
Workarounds
AFAIK, there is no way to get a "live" PDF preview (the way you can get a "live" HTML preview using MPP). This means you will have to build the document whenever you'd like to view what's changed.
Assuming you want to view the PDF in the Atom window, you can install pdf-view and open the PDF in a pane, side-by-side with your source. Otherwise, you could simply open the PDF in your favorite PDF reader.
Build from the command line
As #mb21 suggested, you can build from the command line. I sometimes use BAT/CMD files to store lengthy, or complicated build commands (since I'm on Windows). For example, in my document source directory, I might have make.cmd, which contains:
pandoc --filter=pandoc-crossref --filter=pandoc-citeproc --smart --listings --number-sections --standalone file.md -o file.pdf
Then I run make.cmd from the command prompt using `./make.cmd'.
Alternatively, you could use Makefile.
Build from Atom
Install atom-script. Then, configure your run options (Ctrl-Alt-Shift-O, on Windows) with something along the lines of the following:
Command: pandoc
Command Arguments: --filter=pandoc-crossref --filter=pandoc-citeproc --standalone file.md -o file.pdf
As you edit your source and would like to update the PDF, you can execute the command via Ctrl-Shift-B
Build from Atom using panzer
You'll still need Atom's script package, but you'll also need to get panzer which is a utility that helps manage build configurations for pandoc.
Edit:
Automatically Build-on-Save via Grunt
Rather than having to press a key combination (e.g. building from command line or using atom-script), I thought of automatically building the output PDF upon saving using Grunt. I've captured the basic idea in this gist
I'm working on the same thing right now, only I am using embedded latex mathematical formulae etc.
The markdown-preview-plus gives a fair representation of what I'm likely to see, but I've been running the following command from the command line in order to compile my pdf
pandoc -f markdown myfile.md -o pdffile.pdf
This works in most simple cases, for trickier ones, or where I want to stitch a few things together, I'm taking my markdown files first to their latex equivalents using a command like
pandoc -f markdown+tex_math_dollars+pipe_tables myfile.md -o myfile.tex
Which creates a latex version of my original file that I can compile/combine with other latex files, or convert to pdf using
pandoc myfile.tex -o myfile.pdf
It's supposed to be possible to embed these various scripts into Atom, using the 'scripts' package, but I'm yet to try that - would be great if someone were to post up their methods too in this direction.
I got pandoc-pdf toolbar working on Atom after installing Perl on Windows, though PDF compilation is much slower with "Latexmk".
I recommend using markdown-preview enhanced and then use several other options available including pandoc, ebook etc.
markdown-preview-enhanced in Atom has support for PDF-on-save, similar to the Grunt workflow suggested above. The output command has several options in addition to PDF, such as MS Word.
Example:
---
layout: post
title: tentative tentacles
date: 2020-09-15 15:01
bibliography: bibliography.bib
output: pdf_document
export_on_save:
pandoc: true
---

How to download multiple numbered images from a website in an easy manner?

I'd like to download multiple numbered images from a website.
The images are structured like this:
http://website.com/images/foo1bar.jpg
http://website.com/images/foo2bar.jpg
http://website.com/images/foo3bar.jpg
... And I'd like to download all of the images within a specific interval.
Are there simple browser addons that could do this, or should I use "wget" or the like?
Thank you for your time.
Crudely, on Unix-like systems:
#!/bin/sh
for i in {1..3}
do
wget http://website.com/images/foo"$i"bar.jpg
done
Try googling "bash for loop".
Edit LOL! Indeed, in a haste I omitted the name of the very program that downloads the image files. Also, this goes into a text editor, then you save it with an arbitrary file name, make it executable with the command
chmod u+x the_file_name
and finally you run it with
./the_file_name

Resources