Why do comments invalidate my Makefile commands? - makefile

I'm trying to add some comments to my Makefile but am seeing some odd behavior that I don't understand. The following succeeds with a make plan-all:
plan-all: \
plan-master \
plan-log-archive \
plan-shared \
plan-audit \
However, if I try to add comments:
plan-all: \
# --------------------
# Global Accounts |
# --------------------
plan-master \
plan-log-archive \
plan-shared \
plan-audit \
I get a failure:
make: plan-master: No such file or directory

There is no way to add comments into a logical line that uses backslashes. It can't be done.
You can either add them before the rule, or you can use multiple rules instead of backslashes:
# --------------------
# Global Accounts |
# --------------------
plan-all: \
plan-master \
plan-log-archive \
plan-shared \
plan-audit \
or
# --------------------
# Global Accounts |
# --------------------
plan-all: \
plan-master
# -------------------
# Other Accounts |
# -------------------
plan-all: \
plan-log-archive \
plan-shared \
plan-audit \

Related

Output training losses over iterations/epochs to file from trainer.py in HuggingFace Transfrormers

In the Transformer's library framework, by HuggingFace only the evaluation step metrics are outputted to a file named eval_resuls_{dataset}.txt in the "output_dir" when running run_glue.py. In the eval_resuls file, there are the metrics associated with the dataset. e.g., accuracy for MNLI and the evaluation loss.
Can a parameter be passed to run_glue.py to generate a training_results_{dataset}.txt file that tracks the training loss? Or would I have to build the functionality myself?
My file named run_python_script_glue.bash:
GLUE_DIR=../../huggingface/GLUE_SMALL/
TASK_NAME=MNLI
ID=OT
python3 run_glue.py \
--local_rank -1 \
--seed 42 \
--model_type albert \
--model_name_or_path albert-base-v2 \
--task_name $TASK_NAME \
--do_train \
--do_eval \
--data_dir $GLUE_DIR/$TASK_NAME \
--max_seq_length 128 \
--per_gpu_train_batch_size 8 \
--per_gpu_eval_batch_size 8 \
--gradient_accumulation_steps 2\
--learning_rate 3e-5 \
--max_steps -1 \
--warmup_steps 1000\
--doc_stride 128 \
--num_train_epochs 3.0 \
--save_steps 9999\
--output_dir ./results/GLUE_SMALL/$TASK_NAME/ALBERT/$ID/ \
--do_lower_case \
--overwrite_output_dir \
--label_noise 0.2\
--att_kl 0.01\
--att_se_hid_size 16\
--att_se_nonlinear relu\
--att_type soft_attention \
--adver_type ot \
--rho 0.5 \
--model_type whai \
--prior_gamma 2.70 \
--three_initial 0.0
In the trainer.py file in the transformer library, the training loss variable during the training step is called tr_loss.
tr_loss = self._training_step(model, inputs, optimizer, global_step)
loss_scalar = (tr_loss - logging_loss) / self.args.logging_steps
logs["loss"] = loss_scalar
logging_loss = tr_loss
In the code, the training loss is first scaled by the logging steps and later passed to a logs dictionary. The logs['loss'] is later printed to the terminal but not to a file. Is there a way to upgrade this to include an update to a txt file?

For loop values in a Makefile not being used as arguments to a shell command

I'm trying to use a Makefile to iterate over several date values and execute a python script for each one, here's the Makefile I'm using (Makefile.study.s1):
include Makefile
# Dates to test
SNAP_TST := 2019-10-12 2020-02-08 2020-10-10 2021-01-02 2021-07-24 2021-12-31 2022-05-27
buildDataset:
for date in $(SNAP_TST) ; do \
python src/_buildDataset.py --table $(TABLE) \
--nan-values $(NAN_CONFIG) \
--patterns-rmv $(PATTERNS_RMV) \
--target-bin $(TARGET_CLASS) \
--target-surv $(TARGET_SURV) \
--target-init $(TARGET_INIT) \
--train-file $(DATA_TRN) \
--test-file $(DIR_DATA)/main/churnvol_test_$$date.csv \
--train-date $(SNAP_TRN) \
--test-date $$date \
--config-input $(CONFIG_INPUT) \
--feats $(FEATS) ; \
done
.PHONY: buildDataset
When I run make -f Makefile.study.s1 buildDataset it replaces the value date with the string "$date" instead of one of the dates in SNAP_TST. Can you guys help me figure out what I did wrong here, and how I can fix this makefile so that $$date is replaced with one of the dates in SNAP_TST? Thank you in advance.

Check If Vim Syntax Region Exists and Remove It

Background:
Syntax highlighting for perl files is extremely slow at times for large files (1k+ lines).
I profiled using:
:syntime on
"*** Do some slow actions ***
:syntime report
There were many slowly performaning regions, like: perlStatementProc
I significantly improved performance by removing some of the slowly performing syntax regions (there are more):
:syntax clear perlStatementProc
Now I want to use this vimrc with these improvements on a different machine which may not have a specific region defined.
I am seeing this ERROR when opening Vim:
E28: No such highlight group name: perlStatementProc
How can I check if the syntax region name perlStatementProc exists?
I found out about hlexists and implemented this solution in my vimrc:
" Remove some syntax highlighting from large perl files.
function! RemovePerlSyntax()
if line('$') > 1000
let perl_syntaxes = [
\ "perlStatementProc",
\ "perlMatch",
\ "perlStatementPword",
\ "perlQR",
\ "perlQW",
\ "perlQQ",
\ "perlQ",
\ "perlStatementIndirObjWrap",
\ "perlVarPlain",
\ "perlVarPlain",
\ "perlOperator",
\ "perlStatementFiledesc",
\ "perlStatementScalar",
\ "perlStatementInclude",
\ "perlStatementNumeric",
\ "perlStatementSocket",
\ "perlFloat",
\ "perlFormat",
\ "perlStatementMisc",
\ "perlStatementFiles",
\ "perlStatementList",
\ "perlStatementIPC",
\ "perlStatementNetwork",
\ "perlStatementTime",
\ "perlStatementIOfunc",
\ "perlStatementFlow",
\ "perlStatementControl",
\ "perlHereDoc",
\ "perlHereDocStart",
\ "perlVarPlain2",
\ "perlVarBlock",
\ "perlVarBlock2",
\ "perlDATA",
\ "perlControl",
\ "perlStatementHash",
\ "perlStatementVector",
\ "perlIndentedHereDoc",
\ "perlLabel",
\ "perlConditional",
\ "perlRepeat",
\ "perlNumber",
\ "perlStatementRegexp",
\ ]
for perl_syntax in perl_syntaxes
" NEW - Was missing this check before.
if hlexists( perl_syntax )
exec "syntax clear " . perl_syntax
endif
endfor
let b:remove_perl_syntax = 1
else
let b:remove_perl_syntax = 0
endif
endfunction
augroup remove_perl_syntax
autocmd!
autocmd BufNewFile,BufRead,BufReadPost,FileType perl call RemovePerlSyntax()
augroup END

Learning rate not set in run_mlm.py?

I want to run (or resume) the run_mlm.py script with a specific learning rate, but it doesn't seem like setting it in the script arguments does anything.
os.system(
f"python {script} \
--model_type {model} \
--config_name './models/{model}/config.json' \
--train_file './content/{data}/train.txt' \
--validation_file './content/{data}/test.txt' \
--learning_rate 6e-4 \
--weight_decay 0.01 \
--warmup_steps 6000 \
--adam_beta1 0.9 \
--adam_beta2 0.98 \
--adam_epsilon 1e-6 \
--tokenizer_name './tokenizer/{model}' \
--output_dir './{out_dir}' \
--do_train \
--do_eval \
--num_train_epochs 40 \
--overwrite_output_dir {overwrite} \
--ignore_data_skip"
)
After warm-up, the log indicates that the learning rate tops out at 1e-05—a default from somewhere, I guess, but I'm not sure where (and certainly not 6e-4):
{'loss': 3.9821, 'learning_rate': 1e-05, 'epoch': 0.09}

RRDTool graph command insert into variable

I am using sh(1) shell script to generate RRDTool for day, week, month and year. My question is, how to make this code more effective without repeating the same code? I tried it like this but always get errors, use trailing escape character etc:
NEWVAR="-a PNG -v "Interrupts/s" -h 130 -w 576 \
--watermark "`hostname`:`date "+%H:%M:%S - %d/%m/%Y"`" --font TITLE:15: \
--font LEGEND:7: --font UNIT:7: --font AXIS:5: -c CANVAS#000000 -c BACK#D8D8D8 \
DEF:DI="$RRDFDI":DI:AVERAGE \
CDEF:DIx=DI,8000,LT,DI,UNKN,IF \
AREA:DIx#FFDD44:"" \
LINE1.2:DIx#31B404:"Device Interrupts" \
GPRINT:DIx:MIN:"Min\:%6.0lf" \
GPRINT:DIx:AVERAGE:"Avg\:%6.0lf" \
GPRINT:DIx:MAX:"Max\:%6.0lf" \
GPRINT:DIx:LAST:"Cur\:%6.0lf\l""
$RRDBIN graph $NEWVAR -t "Device Interrupts Days Statistic (Interrupts)" --start -1d
$RRDBIN graph $NEWVAR -t "Device Interrupts Weeks Statistic (Interrupts)" --start -1w
$RRDBIN graph $NEWVAR -t "Device Interrupts Months Statistic (Interrupts)" --start -1m
$RRDBIN graph $NEWVAR -t "Device Interrupts Years Statistic (Interrupts)" --start -1y
Thank in advance!
Use a for loop, something along the lines of following.
NEWVAR="-a PNG -v "Interrupts/s" -h 130 -w 576 \
--watermark "`hostname`:`date "+%H:%M:%S - %d/%m/%Y"`" --font TITLE:15: \
--font LEGEND:7: --font UNIT:7: --font AXIS:5: -c CANVAS#000000 -c BACK#D8D8D8 \
DEF:DI="$RRDFDI":DI:AVERAGE \
CDEF:DIx=DI,8000,LT,DI,UNKN,IF \
AREA:DIx#FFDD44:"" \
LINE1.2:DIx#31B404:"Device Interrupts" \
GPRINT:DIx:MIN:"Min\:%6.0lf" \
GPRINT:DIx:AVERAGE:"Avg\:%6.0lf" \
GPRINT:DIx:MAX:"Max\:%6.0lf" \
GPRINT:DIx:LAST:"Cur\:%6.0lf\l""
for c in d w m y ; do
$RRDBIN graph $NEWVAR -t "Device Interrupts Days Statistic (Interrupts)" --start -1$c output.file.$c
done
The most obvious reason for your errors when using this code is that you have failed to escape the embedded double quotes in your declaration of NEWVAR.
You cannot generate all three graphs simulataneously; the way you are coding it is optimal, though you need to fix your use of embedded quotes in the NEWVAR declaration. You will also need to double-escape the colons and \l endof line sequence...
NEWVAR="-a PNG -v \"Interrupts/s\" -h 130 -w 576 \
--watermark \""`hostname`":"`date '+%H:%M:%S - %d/%m/%Y'`"\" --font TITLE:15: \
--font LEGEND:7: --font UNIT:7: --font AXIS:5: -c CANVAS#000000 -c BACK#D8D8D8 \
DEF:DI=$RRDFDI:DI:AVERAGE \
CDEF:DIx=DI,8000,LT,DI,UNKN,IF \
AREA:DIx#FFDD44:\"\" \
LINE1.2:DIx#31B404:'Device Interrupts' \
GPRINT:DIx:MIN:'Min\:%6.0lf' \
GPRINT:DIx:AVERAGE:'Avg\:%6.0lf' \
GPRINT:DIx:MAX:'Max\:%6.0lf' \
GPRINT:DIx:LAST:'Cur\:%6.0lf\l'"

Resources