Write content into a file with bash script - bash

I'm trying to create a bash script in order to daemonized celeryd task. In my bash script, I need to create some files and add content. This content have a variable given by user named $app_name with read method and some others variables in the content.
Issue:
When I copy the content located in my bash script to the given path, it doesn't copy variables already present inside.
Example:
In my bash script I have :
########################
# Get application name #
########################
read -p "Define the application name (lowercase and without spaces): " app_name
echo "You defined the application name: $app_name"
############################################################
# Create service file /usr/local/etc/rc.d/celeryd_app_name #
############################################################
cat > /usr/local/etc/rc.d/celeryd_$app_name << EOF
#!/bin/sh
# =====================================================
# celeryd_$app_name - Starts the Celery worker daemon.
# =====================================================
#
# :Usage: /etc/init.d/celeryd_$app_name {start|stop|force-reload|restart|try-restart|status}
# :Configuration file: /etc/default/celeryd_$app_name
### BEGIN INIT INFO
# Provides: celeryd_$app_name
# Required-Start: $network $local_fs $remote_fs
# Required-Stop: $network $local_fs $remote_fs
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
### END INIT INFO
EOF
But if I open the created file I get:
### BEGIN INIT INFO
# Provides: celeryd_$app_name
# Required-Start:
# Required-Stop:
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
### END INIT INFO
It doesn't copy $network $local_fs $remote_fs which are present in the content.
There is another way to do that ?
Thank you !

What is happening is that the here document is expanding the variables and since they are not declared you are getting empty values, from the wiki:
By default, behavior is largely identical to the contents of double quotes: variable names are replaced by their values, commands within backticks are evaluated, etc
$ cat << EOF
> \$ Working dir "$PWD" `pwd`
> EOF
$ Working dir "/home/user" /home/user
This can be disabled by quoting any part of the label, which is then ended by the unquoted value; the behavior is essentially identical to that if the contents were enclosed in single quotes. Thus for example by setting it in single quotes:
$ cat << 'EOF'
> \$ Working dir "$PWD" `pwd`
> EOF
\$ Working dir "$PWD" `pwd`
So from your example, you could modify your script to be something like:
#!/bin/sh
cat << 'EOF' > /usr/local/etc/rc.d/celeryd_$app_name
#!/bin/sh
# =====================================================
# celeryd_$app_name - Starts the Celery worker daemon.
# =====================================================
#
# :Usage: /etc/init.d/celeryd_$app_name {start|stop|force-reload|restart|try-restart|status}
# :Configuration file: /etc/default/celeryd_$app_name
### BEGIN INIT INFO
# Provides: celeryd_$app_name
# Required-Start: $network $local_fs $remote_fs
# Required-Stop: $network $local_fs $remote_fs
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
### END INIT INFO
EOF
Which then will produce the output:
#!/bin/sh
# =====================================================
# celeryd_$app_name - Starts the Celery worker daemon.
# =====================================================
#
# :Usage: /etc/init.d/celeryd_$app_name {start|stop|force-reload|restart|try-restart|status}
# :Configuration file: /etc/default/celeryd_$app_name
### BEGIN INIT INFO
# Provides: celeryd_$app_name
# Required-Start: $network $local_fs $remote_fs
# Required-Stop: $network $local_fs $remote_fs
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
### END INIT INFO

Related

Ignoring the 'pipelines.yml' file because modules or command line options are specified

I have set up elasticsearch with password protected, and i am successfully able to work with elastic search by entering username=elastic and password=mypassword
but now I am trying to import mysql data into elasticsearch using logstash, when i run logstash using below command it gives error.
am i missing something?
logstash -f mysql.conf
logstash-plain.log
[2019-06-14T18:12:34,410][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-06-14T18:12:34,424][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.1.0"}
[2019-06-14T18:12:35,400][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, } at line 16, column 23 (byte 507) after output {\r\n elasticsearch {\r\n\thosts => \"http://10.42.35.14:9200/\"\r\n user => elastic\r\n password => pharma", :backtrace=>["D:/softwares/ElasticSearch/Version7.1/logstash-7.1.0/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "D:/softwares/ElasticSearch/Version7.1/logstash-7.1.0/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "D:/softwares/ElasticSearch/Version7.1/logstash-7.1.0/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "D:/softwares/ElasticSearch/Version7.1/logstash-7.1.0/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "D:/softwares/ElasticSearch/Version7.1/logstash-7.1.0/logstash-core/lib/logstash/java_pipeline.rb:23:in `initialize'", "D:/softwares/ElasticSearch/Version7.1/logstash-7.1.0/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "D:/softwares/ElasticSearch/Version7.1/logstash-7.1.0/logstash-core/lib/logstash/agent.rb:325:in `block in converge_state'"]}
[2019-06-14T18:12:35,758][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2019-06-14T18:12:40,664][INFO ][logstash.runner ] Logstash shut down.
mysql.conf
# file: contacts-index-logstash.conf
input {
jdbc {
jdbc_connection_string => "jdbc:mysql://52.213.22.96:3306/prbi"
jdbc_user => "myuser"
jdbc_password => "mypassword"
jdbc_driver_library => "mysql-connector-java-6.0.5.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
statement => "SELECT * from tmp_j_summaryreport"
}
}
output {
elasticsearch {
hosts => "http://10.42.35.14:9200/"
user => elastic
password => myelasticpassword
index => "testing123"
}
stdout { codec => json_lines }
}
logstash.yml
# Settings file in YAML
#
# Settings can be specified either in hierarchical form, e.g.:
#
# pipeline:
# batch:
# size: 125
# delay: 5
#
# Or as flat keys:
#
# pipeline.batch.size: 125
# pipeline.batch.delay: 5
#
# ------------ Node identity ------------
#
# Use a descriptive name for the node:
#
# node.name: test
#
# If omitted the node name will default to the machine's host name
#
# ------------ Data path ------------------
#
# Which directory should be used by logstash and its plugins
# for any persistent needs. Defaults to LOGSTASH_HOME/data
#
# path.data:
#
# ------------ Pipeline Settings --------------
#
# The ID of the pipeline.
#
# pipeline.id: main
#
# Set the number of workers that will, in parallel, execute the filters+outputs
# stage of the pipeline.
#
# This defaults to the number of the host's CPU cores.
#
# pipeline.workers: 2
#
# How many events to retrieve from inputs before sending to filters+workers
#
# pipeline.batch.size: 125
#
# How long to wait in milliseconds while polling for the next event
# before dispatching an undersized batch to filters+outputs
#
# pipeline.batch.delay: 50
#
# Force Logstash to exit during shutdown even if there are still inflight
# events in memory. By default, logstash will refuse to quit until all
# received events have been pushed to the outputs.
#
# WARNING: enabling this can lead to data loss during shutdown
#
# pipeline.unsafe_shutdown: false
#
# ------------ Pipeline Configuration Settings --------------
#
# Where to fetch the pipeline configuration for the main pipeline
#
# path.config:
#
# Pipeline configuration string for the main pipeline
#
# config.string:
#
# At startup, test if the configuration is valid and exit (dry run)
#
# config.test_and_exit: false
#
# Periodically check if the configuration has changed and reload the pipeline
# This can also be triggered manually through the SIGHUP signal
#
# config.reload.automatic: false
#
# How often to check if the pipeline configuration has changed (in seconds)
#
# config.reload.interval: 3s
#
# Show fully compiled configuration as debug log message
# NOTE: --log.level must be 'debug'
#
# config.debug: false
#
# When enabled, process escaped characters such as \n and \" in strings in the
# pipeline configuration files.
#
# config.support_escapes: false
#
# ------------ Module Settings ---------------
# Define modules here. Modules definitions must be defined as an array.
# The simple way to see this is to prepend each `name` with a `-`, and keep
# all associated variables under the `name` they are associated with, and
# above the next, like this:
#
# modules:
# - name: MODULE_NAME
# var.PLUGINTYPE1.PLUGINNAME1.KEY1: VALUE
# var.PLUGINTYPE1.PLUGINNAME1.KEY2: VALUE
# var.PLUGINTYPE2.PLUGINNAME1.KEY1: VALUE
# var.PLUGINTYPE3.PLUGINNAME3.KEY1: VALUE
#
# Module variable names must be in the format of
#
# var.PLUGIN_TYPE.PLUGIN_NAME.KEY
#
# modules:
#
# ------------ Cloud Settings ---------------
# Define Elastic Cloud settings here.
# Format of cloud.id is a base64 value e.g. dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRub3RhcmVhbCRpZGVudGlmaWVy
# and it may have an label prefix e.g. staging:dXMtZ...
# This will overwrite 'var.elasticsearch.hosts' and 'var.kibana.host'
# cloud.id: <identifier>
#
# Format of cloud.auth is: <user>:<pass>
# This is optional
# If supplied this will overwrite 'var.elasticsearch.username' and 'var.elasticsearch.password'
# If supplied this will overwrite 'var.kibana.username' and 'var.kibana.password'
# cloud.auth: elastic:<password>
#
# ------------ Queuing Settings --------------
#
# Internal queuing model, "memory" for legacy in-memory based queuing and
# "persisted" for disk-based acked queueing. Defaults is memory
#
# queue.type: memory
#
# If using queue.type: persisted, the directory path where the data files will be stored.
# Default is path.data/queue
#
# path.queue:
#
# If using queue.type: persisted, the page data files size. The queue data consists of
# append-only data files separated into pages. Default is 64mb
#
# queue.page_capacity: 64mb
#
# If using queue.type: persisted, the maximum number of unread events in the queue.
# Default is 0 (unlimited)
#
# queue.max_events: 0
#
# If using queue.type: persisted, the total capacity of the queue in number of bytes.
# If you would like more unacked events to be buffered in Logstash, you can increase the
# capacity using this setting. Please make sure your disk drive has capacity greater than
# the size specified here. If both max_bytes and max_events are specified, Logstash will pick
# whichever criteria is reached first
# Default is 1024mb or 1gb
#
# queue.max_bytes: 1024mb
#
# If using queue.type: persisted, the maximum number of acked events before forcing a checkpoint
# Default is 1024, 0 for unlimited
#
# queue.checkpoint.acks: 1024
#
# If using queue.type: persisted, the maximum number of written events before forcing a checkpoint
# Default is 1024, 0 for unlimited
#
# queue.checkpoint.writes: 1024
#
# If using queue.type: persisted, the interval in milliseconds when a checkpoint is forced on the head page
# Default is 1000, 0 for no periodic checkpoint.
#
# queue.checkpoint.interval: 1000
#
# ------------ Dead-Letter Queue Settings --------------
# Flag to turn on dead-letter queue.
#
# dead_letter_queue.enable: false
# If using dead_letter_queue.enable: true, the maximum size of each dead letter queue. Entries
# will be dropped if they would increase the size of the dead letter queue beyond this setting.
# Default is 1024mb
# dead_letter_queue.max_bytes: 1024mb
# If using dead_letter_queue.enable: true, the directory path where the data files will be stored.
# Default is path.data/dead_letter_queue
#
# path.dead_letter_queue:
#
# ------------ Metrics Settings --------------
#
# Bind address for the metrics REST endpoint
#
# http.host: "127.0.0.1"
#
# Bind port for the metrics REST endpoint, this option also accept a range
# (9600-9700) and logstash will pick up the first available ports.
#
# http.port: 9600-9700
#
# ------------ Debugging Settings --------------
#
# Options for log.level:
# * fatal
# * error
# * warn
# * info (default)
# * debug
# * trace
#
# log.level: info
# path.logs:
#
# ------------ Other Settings --------------
#
# Where to find custom plugins
# path.plugins: []
#
# ------------ X-Pack Settings (not applicable for OSS build)--------------
#
# X-Pack Monitoring
# https://www.elastic.co/guide/en/logstash/current/monitoring-logstash.html
#xpack.monitoring.enabled: false
#xpack.monitoring.elasticsearch.username: logstash_system
#xpack.monitoring.elasticsearch.password: password
#xpack.monitoring.elasticsearch.hosts: ["https://es1:9200", "https://es2:9200"]
#xpack.monitoring.elasticsearch.ssl.certificate_authority: [ "/path/to/ca.crt" ]
#xpack.monitoring.elasticsearch.ssl.truststore.path: path/to/file
#xpack.monitoring.elasticsearch.ssl.truststore.password: password
#xpack.monitoring.elasticsearch.ssl.keystore.path: /path/to/file
#xpack.monitoring.elasticsearch.ssl.keystore.password: password
#xpack.monitoring.elasticsearch.ssl.verification_mode: certificate
#xpack.monitoring.elasticsearch.sniffing: false
#xpack.monitoring.collection.interval: 10s
#xpack.monitoring.collection.pipeline.details.enabled: true
#
# X-Pack Management
# https://www.elastic.co/guide/en/logstash/current/logstash-centralized-pipeline-management.html
#xpack.management.enabled: false
#xpack.management.pipeline.id: ["main", "apache_logs"]
#xpack.management.elasticsearch.username: logstash_admin_user
#xpack.management.elasticsearch.password: password
#xpack.management.elasticsearch.hosts: ["https://es1:9200", "https://es2:9200"]
#xpack.management.elasticsearch.ssl.certificate_authority: [ "/path/to/ca.crt" ]
#xpack.management.elasticsearch.ssl.truststore.path: /path/to/file
#xpack.management.elasticsearch.ssl.truststore.password: password
#xpack.management.elasticsearch.ssl.keystore.path: /path/to/file
#xpack.management.elasticsearch.ssl.keystore.password: password
#xpack.management.elasticsearch.ssl.verification_mode: certificate
#xpack.management.elasticsearch.sniffing: false
#xpack.management.logstash.poll_interval: 5s
#xpack.management.enabled: true
xpack.management.elasticsearch.hosts: "http://10.42.35.14:9200/"
#xpack.management.elasticsearch.username: logstash_system
xpack.management.elasticsearch.password: myelasticpassword
This message on the logstash log indicates that there is something wrong with your config file:
Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError"
The rest of message says that the problem is in your output block
:message=>"Expected one of #, {, } at line 16, column 23 (byte 507) after output {
Double check your output configuration, it needs to be something like this:
output {
elasticsearch {
hosts => ["10.42.35.14:9200"]
user => "elastic"
password => "myelasticpassword"
index => "testing123"
}
stdout { codec => "json_lines" }
}

Debian - InitScript - "Do NOT set -e" in /etc/init.d/skeleton

I was wondering why "Do NOT set -e" was present in /etc/init.d/skeleton as I thought it was always good practice.
#! /bin/sh
### BEGIN INIT INFO
# Provides: skeleton
# Required-Start: $remote_fs $syslog
# Required-Stop: $remote_fs $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Example initscript
# Description: This file should be used to construct scripts to be
# placed in /etc/init.d.
### END INIT INFO
# Author: Foo Bar <foobar#baz.org>
#
# Please remove the "Author" lines above and replace them
# with your own name if you copy and modify this script.
# Do NOT "set -e"
Debian wiki recommends it (https://wiki.debian.org/LSBInitScripts/StatusSupport) so I don't know what I should do.

Ruby: check if a string is formatted in a very specific way

I'm trying to write a piece of code that tells the user if a string is formatted in a specific way. The format looks like this (it's a circle):
######\n
### ###\n
# #\n
# #\n
# #\n
# #\n
# #\n
### ###\n
######\n
\n
The newlines are included for clarity; the # character may be any character from an existing character class.
What I want to do is check if a string (from a file) contains one or more of these "circles". Multiple circles should be seperated like this:
# #\n
### ###\n
######\n
\n
######\n
### ###\n
# #\n
So this is what I've tried:
isCircle = "( #{get_i}{6}\n"
isCircle += " #{get_i}{3} #{get_i}{3}\n"
isCircle += " #{get_i} #{get_i}\n"
isCircle +=("#{get_i} #{get_i}\n"*3)
isCircle += " #{get_i} #{get_i}\n"
isCircle += " #{get_i}{3} #{get_i}{3}\n"
isCircle = " #{get_i}{6}\n\n?)*"
isCircle = Regexp.new(isCircle)
(get_i is a method that returns the aforementioned character class, correctly escaped and everything)
However, when testing this against an incorrect input string, it still tells me there's a match.
What am I doing wrong and how can I correctly perform the match?
Maybe this:
[ ]+#{6}[\n ]+#{3}[ ]+#{3}[\n ]+(?>#[ ]+#[\n\s]+)+#{3}[ ]+#{3}[\n\s]+#{6}\s*
Demo
Is this what you want to do?
Suppose you wish to search for this pattern:
pattern =
%{ ######
### ###
# #
# #
# #
# #
# #
### ###
######
}
in a text file whose contents have been read into this string:
str =
%{###
##
######
### ###
# #
# #
# #
# #
# #
### ###
######
#######
######
######
### ###
# #
# #
# #
# #
# #
### ###
######
###
}
Just use String#scan:
puts str.scan(pattern).join("\n")
######
### ###
# #
# #
# #
# #
# #
### ###
######
######
### ###
# #
# #
# #
# #
# #
### ###
######
If you instead want the line offsets of str where the pattern begins:
sarr = str.lines
parr = pattern.lines
prows = parr.size
(sarr.size-prows+1).times.select { |i| sarr[i,prows] == parr }
#==> [2, 13]

calling each on a string object

This code (by someone else) might have been written using an older version of Ruby because now I'm getting an error calling 'each' on a string object. The maze string below gets passed to the maze_string_to_array method. When it's run, it yields this error in `maze_string_to_array'
NoMethodError: undefined method `each' for #<String:0x00000100854ac0>
Can you explain what the problem is, and how to fix it?
def maze_string_to_array(mazestring)
#maze = []
mazestring.each do |line|
#maze.push line.chomp
end
end
Maze string
MAZE1 = %{#####################################
# # # #A # # #
# # # # # # ####### # ### # ####### #
# # # # # # # # #
# ##### # ################# # #######
# # # # # # # # #
##### ##### ### ### # ### # # # # # #
# # # # # # B# # # # # #
# # ##### ##### # # ### # # ####### #
# # # # # # # # # # # #
# ### ### # # # # ##### # # # ##### #
# # # # # # #
#####################################}
This code is unnecessarily verbose. The whole method can be written with a map and using 1.9 String#lines instead of the old 1.8.x String#each to split lines:
#maze = mazestring.lines.map(&:chomp)
Ruby 1.8 String#each used to iterate through lines. In 1.9, String#each_line does the same thing.
In Ruby 1.9 use each_line instead of each.
But it looks as if you could replace the whole method with mazestring.split(/\n/) anyway.

homebrew brew doctor fail message

I am going through my home brew doctor messages and no matter what I do I can not seem to remove this message:
Warning: You have uncommitted modifications to Homebrew's core.
Unless you know what you are doing, you should run:
cd /usr/local && git reset --hard
Does anyone have any idea on how to get rid of this?
From git status /usr/local
# Library/Aliases/dwarffortress
# Library/Aliases/ultima4
# Library/Aliases/ultima7
# Library/Contributions/examples/
# Library/Formula/abfind.rb
# Library/Formula/abuse.rb
# Library/Formula/akonadi.rb
# Library/Formula/angband.rb
# Library/Formula/apc.rb
# Library/Formula/aqua-less.rb
# Library/Formula/argp-standalone.rb
# Library/Formula/attica.rb
# Library/Formula/bashreduce.rb
# Library/Formula/c10t.rb
# Library/Formula/chocolate-doom.rb
# Library/Formula/cmigemo.rb
# Library/Formula/coffee-script.rb
# Library/Formula/csstidy.rb
# Library/Formula/dosbox.rb
# Library/Formula/dotless.rb
# Library/Formula/dwarf-fortress.rb
# Library/Formula/exult.rb
# Library/Formula/flip.rb
# Library/Formula/frobtads.rb
# Library/Formula/frotz.rb
# Library/Formula/gearman-php.rb
# Library/Formula/gnu-chess.rb
# Library/Formula/gnu-go.rb
# Library/Formula/growlme.rb
# Library/Formula/imagick.rb
# Library/Formula/inform6.rb
# Library/Formula/jnethack.rb
# Library/Formula/jwhois.rb
# Library/Formula/kde-phonon.rb
# Library/Formula/kdebase-runtime.rb
# Library/Formula/kdelibs.rb
# Library/Formula/kdepimlibs.rb
# Library/Formula/libgdiplus.rb
# Library/Formula/libiconv.rb
# Library/Formula/libsgml.rb
# Library/Formula/maatkit.rb
# Library/Formula/mcrypt-php.rb
# Library/Formula/mednafen.rb
# Library/Formula/memcache-php.rb
# Library/Formula/memcached-php.rb
# Library/Formula/memcachedb.rb
# Library/Formula/midgard2-php.rb
# Library/Formula/mongo-php.rb
# Library/Formula/n2n.rb
# Library/Formula/nazghul.rb
# Library/Formula/nethack.rb
# Library/Formula/netris.rb
# Library/Formula/ninja.rb
# Library/Formula/open-tyrian.rb
# Library/Formula/orderly.rb
# Library/Formula/oxygen-icons.rb
# Library/Formula/parsley.rb
# Library/Formula/pbrt.rb
# Library/Formula/pcntl-php.rb
# Library/Formula/phpmyadmin.rb
# Library/Formula/pioneers.rb
# Library/Formula/platypus.rb
# Library/Formula/pspell-php.rb
# Library/Formula/qimageblitz.rb
# Library/Formula/robotfindskitten.rb
# Library/Formula/shared-desktop-ontologies.rb
# Library/Formula/shen.rb
# Library/Formula/slashem.rb
# Library/Formula/solr-php.rb
# Library/Formula/soprano.rb
# Library/Formula/sparse.rb
# Library/Formula/spim.rb
# Library/Formula/stone-soup.rb
# Library/Formula/strigi.rb
# Library/Formula/uggconv.rb
# Library/Formula/unnethack.rb
# Library/Formula/voldemort.rb
# Library/Formula/woof.rb
# Library/Formula/xboard.rb
# Library/Formula/xcache.rb
# Library/Formula/xdebug.rb
# Library/Formula/xmoto.rb
# Library/Formula/xu4.rb
# Library/Homebrew/test/tests
After posting on the homebrew issues log https://github.com/mxcl/homebrew/issues/11761 I was able to clear the error by running git clean -df and once that is finished brew doctor. Once complete you should see:
Your system is raring to brew.

Resources