I'm trying to use pt-query-digest to aggregate slow query logs from Aurora stored in Cloudwatch. I use the awslogs tool to grab them and write them to a file, it looks like they are formatted wrong, even after I strip out the host information. A traditional slow query has the following as a header before the query:
# Time: 2017-08-04T19:24:50.630232Z
# User#Host: root[root] # localhost [] Id: 236
# Schema: Last_errno: 0 Killed: 0
# Query_time: 11.003017 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 0 Rows_affected: 0
# Bytes_sent: 57
where as the logs from Cloudwatch look like this:
# Time: 2021-02-16T20:35:10.179940Z
# User#Host: root[root] # localhost [] Id: 236
# Query_time: 5.290956 Lock_time: 0.001651 Rows_sent: 8 Rows_examined: 1858252
SET timestamp=1613507710;
has anyone found a way to use pt-query-deigest to consume Aurora logs?
Related
I have log files that are broken down into between 1 and 4 "Tasks". In each "Task" there are sections for "WU Name" and "estimated CPU time remaining". Ultimately, I want to the bash script output to look like this 3 Task example;
Task 1 Mini_Protein_binds_COVID-19_boinc_ 0d:7h:44m:28s
Task 2 shapeshift_pair6_msd4X_4_f_e0_161_ 0d:4h:14m:22s
Task 3 rep730_0078_symC_reordered_0002_pr 1d:1h:38m:41s
So far; I can count the Tasks in the log. I can isolate x number of characters I want from the "WU Name". I can convert the "estimated CPU time remaining" in seconds to days:hours:minutes:seconds. And I can output all of that into 'pretty' columns. Problem is that I can only process 1 Task using;
# Initialize counter
counter=1
# Count how many iterations
cnt_wu=`grep -c "WU name:" /mnt/work/sec-conv/bnc-sample3.txt`
# Iterate the loop for cnt-wu times
while [ $counter -le ${cnt_wu} ]
do
core_cnt=$counter
wu=`cat /mnt/work/sec-conv/bnc-sample3.txt | grep -Po 'WU name: \K.*' | cut -c1-34`
sec=`cat /mnt/work/sec-conv/bnc-sample3.txt | grep -Po 'estimated CPU time remaining: \K.*' | cut -f1 -d"."`
dhms=`printf '%dd:%dh:%dm:%ds\n' $(($sec/86400)) $(($sec%86400/3600)) $(($sec%3600/60)) \ $(($sec%60))`
echo "Task ${core_cnt}" $'\t' $wu $'\t' $dhms | column -ts $'\t'
counter=$((counter + 1))
done
Note: /mnt/work/sec-conv/bnc-sample3.txt is a static one Task sample only used for this scripts dev.
What I can't figure out is the next step which is to be able to process x number of multiple Tasks. I can't figure out how to leverage the while/counter combination properly, and can't figure out how to increment through the occurrences of Tasks.
Adding bnc-sample.txt (contains 3 Tasks)
1) -----------
name: Rosetta#home
master URL: https://boinc.bakerlab.org/rosetta/
user_name: XXXXXXX
team_name:
resource share: 100.000000
user_total_credit: 10266.993660
user_expavg_credit: 512.420495
host_total_credit: 10266.993660
host_expavg_credit: 512.603669
nrpc_failures: 0
master_fetch_failures: 0
master fetch pending: no
scheduler RPC pending: no
trickle upload pending: no
attached via Account Manager: no
ended: no
suspended via GUI: no
don't request more work: no
disk usage: 0.000000
last RPC: Wed Jun 10 15:55:29 2020
project files downloaded: 0.000000
GUI URL:
name: Message boards
description: Correspond with other users on the Rosetta#home message boards
URL: https://boinc.bakerlab.org/rosetta/forum_index.php
GUI URL:
name: Your account
description: View your account information
URL: https://boinc.bakerlab.org/rosetta/home.php
GUI URL:
name: Your tasks
description: View the last week or so of computational work
URL: https://boinc.bakerlab.org/rosetta/results.php?userid=XXXXXXX
jobs succeeded: 117
jobs failed: 0
elapsed time: 2892439.609931
cross-project ID: 3538b98e5f16a958a6bdd2XXXXXXXXX
======== Tasks ========
1) -----------
name: shapeshift_pair6_msd4X_4_f_e0_161_X_0001_0001_fragments_abinitio_SAVE_ALL_OUT_946179_730_0
WU name: shapeshift_pair6_msd4X_4_f_e0_161_X_0001_0001_fragments_abinitio_SAVE_ALL_OUT_946179_730
project URL: https://boinc.bakerlab.org/rosetta/
received: Mon Jun 8 09:58:08 2020
report deadline: Thu Jun 11 09:58:08 2020
ready to report: no
state: downloaded
scheduler state: scheduled
active_task_state: EXECUTING
app version num: 420
resources: 1 CPU
estimated CPU time remaining: 26882.771040
slot: 1
PID: 28434
CPU time at last checkpoint: 3925.896000
current CPU time: 4314.761000
fraction done: 0.066570
swap size: 431 MB
working set size: 310 MB
2) -----------
name: rep730_0078_symC_reordered_0002_propagated_0001_0001_0001_A_v9_fold_SAVE_ALL_OUT_946618_54_0
WU name: rep730_0078_symC_reordered_0002_propagated_0001_0001_0001_A_v9_fold_SAVE_ALL_OUT_946618_54
project URL: https://boinc.bakerlab.org/rosetta/
received: Mon Jun 8 09:58:08 2020
report deadline: Thu Jun 11 09:58:08 2020
ready to report: no
state: downloaded
scheduler state: scheduled
active_task_state: EXECUTING
app version num: 420
resources: 1 CPU
estimated CPU time remaining: 26412.937920
slot: 2
PID: 28804
CPU time at last checkpoint: 3829.626000
current CPU time: 3879.975000
fraction done: 0.082884
swap size: 628 MB
working set size: 513 MB
3) -----------
name: Mini_Protein_binds_COVID-19_boinc_site3_2_SAVE_ALL_OUT_IGNORE_THE_REST_0aw6cb3u_944116_2_0
WU name: Mini_Protein_binds_COVID-19_boinc_site3_2_SAVE_ALL_OUT_IGNORE_THE_REST_0aw6cb3u_944116_2
project URL: https://boinc.bakerlab.org/rosetta/
received: Mon Jun 8 09:58:47 2020
report deadline: Thu Jun 11 09:58:46 2020
ready to report: no
state: downloaded
scheduler state: scheduled
active_task_state: EXECUTING
app version num: 420
resources: 1 CPU
estimated CPU time remaining: 27868.559616
slot: 0
PID: 30988
CPU time at last checkpoint: 1265.356000
current CPU time: 1327.603000
fraction done: 0.032342
swap size: 792 MB
working set size: 668 MB
Again, I appreciate any guidance!
I have set up elasticsearch with password protected, and i am successfully able to work with elastic search by entering username=elastic and password=mypassword
but now I am trying to import mysql data into elasticsearch using logstash, when i run logstash using below command it gives error.
am i missing something?
logstash -f mysql.conf
logstash-plain.log
[2019-06-14T18:12:34,410][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-06-14T18:12:34,424][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.1.0"}
[2019-06-14T18:12:35,400][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, } at line 16, column 23 (byte 507) after output {\r\n elasticsearch {\r\n\thosts => \"http://10.42.35.14:9200/\"\r\n user => elastic\r\n password => pharma", :backtrace=>["D:/softwares/ElasticSearch/Version7.1/logstash-7.1.0/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "D:/softwares/ElasticSearch/Version7.1/logstash-7.1.0/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "D:/softwares/ElasticSearch/Version7.1/logstash-7.1.0/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "D:/softwares/ElasticSearch/Version7.1/logstash-7.1.0/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "D:/softwares/ElasticSearch/Version7.1/logstash-7.1.0/logstash-core/lib/logstash/java_pipeline.rb:23:in `initialize'", "D:/softwares/ElasticSearch/Version7.1/logstash-7.1.0/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "D:/softwares/ElasticSearch/Version7.1/logstash-7.1.0/logstash-core/lib/logstash/agent.rb:325:in `block in converge_state'"]}
[2019-06-14T18:12:35,758][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2019-06-14T18:12:40,664][INFO ][logstash.runner ] Logstash shut down.
mysql.conf
# file: contacts-index-logstash.conf
input {
jdbc {
jdbc_connection_string => "jdbc:mysql://52.213.22.96:3306/prbi"
jdbc_user => "myuser"
jdbc_password => "mypassword"
jdbc_driver_library => "mysql-connector-java-6.0.5.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
statement => "SELECT * from tmp_j_summaryreport"
}
}
output {
elasticsearch {
hosts => "http://10.42.35.14:9200/"
user => elastic
password => myelasticpassword
index => "testing123"
}
stdout { codec => json_lines }
}
logstash.yml
# Settings file in YAML
#
# Settings can be specified either in hierarchical form, e.g.:
#
# pipeline:
# batch:
# size: 125
# delay: 5
#
# Or as flat keys:
#
# pipeline.batch.size: 125
# pipeline.batch.delay: 5
#
# ------------ Node identity ------------
#
# Use a descriptive name for the node:
#
# node.name: test
#
# If omitted the node name will default to the machine's host name
#
# ------------ Data path ------------------
#
# Which directory should be used by logstash and its plugins
# for any persistent needs. Defaults to LOGSTASH_HOME/data
#
# path.data:
#
# ------------ Pipeline Settings --------------
#
# The ID of the pipeline.
#
# pipeline.id: main
#
# Set the number of workers that will, in parallel, execute the filters+outputs
# stage of the pipeline.
#
# This defaults to the number of the host's CPU cores.
#
# pipeline.workers: 2
#
# How many events to retrieve from inputs before sending to filters+workers
#
# pipeline.batch.size: 125
#
# How long to wait in milliseconds while polling for the next event
# before dispatching an undersized batch to filters+outputs
#
# pipeline.batch.delay: 50
#
# Force Logstash to exit during shutdown even if there are still inflight
# events in memory. By default, logstash will refuse to quit until all
# received events have been pushed to the outputs.
#
# WARNING: enabling this can lead to data loss during shutdown
#
# pipeline.unsafe_shutdown: false
#
# ------------ Pipeline Configuration Settings --------------
#
# Where to fetch the pipeline configuration for the main pipeline
#
# path.config:
#
# Pipeline configuration string for the main pipeline
#
# config.string:
#
# At startup, test if the configuration is valid and exit (dry run)
#
# config.test_and_exit: false
#
# Periodically check if the configuration has changed and reload the pipeline
# This can also be triggered manually through the SIGHUP signal
#
# config.reload.automatic: false
#
# How often to check if the pipeline configuration has changed (in seconds)
#
# config.reload.interval: 3s
#
# Show fully compiled configuration as debug log message
# NOTE: --log.level must be 'debug'
#
# config.debug: false
#
# When enabled, process escaped characters such as \n and \" in strings in the
# pipeline configuration files.
#
# config.support_escapes: false
#
# ------------ Module Settings ---------------
# Define modules here. Modules definitions must be defined as an array.
# The simple way to see this is to prepend each `name` with a `-`, and keep
# all associated variables under the `name` they are associated with, and
# above the next, like this:
#
# modules:
# - name: MODULE_NAME
# var.PLUGINTYPE1.PLUGINNAME1.KEY1: VALUE
# var.PLUGINTYPE1.PLUGINNAME1.KEY2: VALUE
# var.PLUGINTYPE2.PLUGINNAME1.KEY1: VALUE
# var.PLUGINTYPE3.PLUGINNAME3.KEY1: VALUE
#
# Module variable names must be in the format of
#
# var.PLUGIN_TYPE.PLUGIN_NAME.KEY
#
# modules:
#
# ------------ Cloud Settings ---------------
# Define Elastic Cloud settings here.
# Format of cloud.id is a base64 value e.g. dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRub3RhcmVhbCRpZGVudGlmaWVy
# and it may have an label prefix e.g. staging:dXMtZ...
# This will overwrite 'var.elasticsearch.hosts' and 'var.kibana.host'
# cloud.id: <identifier>
#
# Format of cloud.auth is: <user>:<pass>
# This is optional
# If supplied this will overwrite 'var.elasticsearch.username' and 'var.elasticsearch.password'
# If supplied this will overwrite 'var.kibana.username' and 'var.kibana.password'
# cloud.auth: elastic:<password>
#
# ------------ Queuing Settings --------------
#
# Internal queuing model, "memory" for legacy in-memory based queuing and
# "persisted" for disk-based acked queueing. Defaults is memory
#
# queue.type: memory
#
# If using queue.type: persisted, the directory path where the data files will be stored.
# Default is path.data/queue
#
# path.queue:
#
# If using queue.type: persisted, the page data files size. The queue data consists of
# append-only data files separated into pages. Default is 64mb
#
# queue.page_capacity: 64mb
#
# If using queue.type: persisted, the maximum number of unread events in the queue.
# Default is 0 (unlimited)
#
# queue.max_events: 0
#
# If using queue.type: persisted, the total capacity of the queue in number of bytes.
# If you would like more unacked events to be buffered in Logstash, you can increase the
# capacity using this setting. Please make sure your disk drive has capacity greater than
# the size specified here. If both max_bytes and max_events are specified, Logstash will pick
# whichever criteria is reached first
# Default is 1024mb or 1gb
#
# queue.max_bytes: 1024mb
#
# If using queue.type: persisted, the maximum number of acked events before forcing a checkpoint
# Default is 1024, 0 for unlimited
#
# queue.checkpoint.acks: 1024
#
# If using queue.type: persisted, the maximum number of written events before forcing a checkpoint
# Default is 1024, 0 for unlimited
#
# queue.checkpoint.writes: 1024
#
# If using queue.type: persisted, the interval in milliseconds when a checkpoint is forced on the head page
# Default is 1000, 0 for no periodic checkpoint.
#
# queue.checkpoint.interval: 1000
#
# ------------ Dead-Letter Queue Settings --------------
# Flag to turn on dead-letter queue.
#
# dead_letter_queue.enable: false
# If using dead_letter_queue.enable: true, the maximum size of each dead letter queue. Entries
# will be dropped if they would increase the size of the dead letter queue beyond this setting.
# Default is 1024mb
# dead_letter_queue.max_bytes: 1024mb
# If using dead_letter_queue.enable: true, the directory path where the data files will be stored.
# Default is path.data/dead_letter_queue
#
# path.dead_letter_queue:
#
# ------------ Metrics Settings --------------
#
# Bind address for the metrics REST endpoint
#
# http.host: "127.0.0.1"
#
# Bind port for the metrics REST endpoint, this option also accept a range
# (9600-9700) and logstash will pick up the first available ports.
#
# http.port: 9600-9700
#
# ------------ Debugging Settings --------------
#
# Options for log.level:
# * fatal
# * error
# * warn
# * info (default)
# * debug
# * trace
#
# log.level: info
# path.logs:
#
# ------------ Other Settings --------------
#
# Where to find custom plugins
# path.plugins: []
#
# ------------ X-Pack Settings (not applicable for OSS build)--------------
#
# X-Pack Monitoring
# https://www.elastic.co/guide/en/logstash/current/monitoring-logstash.html
#xpack.monitoring.enabled: false
#xpack.monitoring.elasticsearch.username: logstash_system
#xpack.monitoring.elasticsearch.password: password
#xpack.monitoring.elasticsearch.hosts: ["https://es1:9200", "https://es2:9200"]
#xpack.monitoring.elasticsearch.ssl.certificate_authority: [ "/path/to/ca.crt" ]
#xpack.monitoring.elasticsearch.ssl.truststore.path: path/to/file
#xpack.monitoring.elasticsearch.ssl.truststore.password: password
#xpack.monitoring.elasticsearch.ssl.keystore.path: /path/to/file
#xpack.monitoring.elasticsearch.ssl.keystore.password: password
#xpack.monitoring.elasticsearch.ssl.verification_mode: certificate
#xpack.monitoring.elasticsearch.sniffing: false
#xpack.monitoring.collection.interval: 10s
#xpack.monitoring.collection.pipeline.details.enabled: true
#
# X-Pack Management
# https://www.elastic.co/guide/en/logstash/current/logstash-centralized-pipeline-management.html
#xpack.management.enabled: false
#xpack.management.pipeline.id: ["main", "apache_logs"]
#xpack.management.elasticsearch.username: logstash_admin_user
#xpack.management.elasticsearch.password: password
#xpack.management.elasticsearch.hosts: ["https://es1:9200", "https://es2:9200"]
#xpack.management.elasticsearch.ssl.certificate_authority: [ "/path/to/ca.crt" ]
#xpack.management.elasticsearch.ssl.truststore.path: /path/to/file
#xpack.management.elasticsearch.ssl.truststore.password: password
#xpack.management.elasticsearch.ssl.keystore.path: /path/to/file
#xpack.management.elasticsearch.ssl.keystore.password: password
#xpack.management.elasticsearch.ssl.verification_mode: certificate
#xpack.management.elasticsearch.sniffing: false
#xpack.management.logstash.poll_interval: 5s
#xpack.management.enabled: true
xpack.management.elasticsearch.hosts: "http://10.42.35.14:9200/"
#xpack.management.elasticsearch.username: logstash_system
xpack.management.elasticsearch.password: myelasticpassword
This message on the logstash log indicates that there is something wrong with your config file:
Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError"
The rest of message says that the problem is in your output block
:message=>"Expected one of #, {, } at line 16, column 23 (byte 507) after output {
Double check your output configuration, it needs to be something like this:
output {
elasticsearch {
hosts => ["10.42.35.14:9200"]
user => "elastic"
password => "myelasticpassword"
index => "testing123"
}
stdout { codec => "json_lines" }
}
i have a hard problem, i want to remove some block from a text which have a special word,here is a example , what i want is remove match backup_all and one line before match line ,and three line after match line
# Time: 2018-01-23T03:41:41.454104+08:00
# User#Host: backup_all[backup_all] # [127.0.0.1] Id: 3168695
# Query_time: 0.129250 Lock_time: 0.000062 Rows_sent: 3535 Rows_examined: 3535
SET timestamp=1516650101;
SELECT /*!40001 SQL_NO_CACHE */ * FROM `legend_gather_customer_note_effect`;
# Time: 2018-01-23T03:41:41.527587+08:00
# User#Host: backup_all[backup_all] # [127.0.0.1] Id: 3168695
# Query_time: 0.066378 Lock_time: 0.000059 Rows_sent: 193 Rows_examined: 193
SET timestamp=1516650101;
SELECT /*!40001 SQL_NO_CACHE */ * FROM `legend_gather_performance_config`;
# Time: 2018-01-23T03:41:41.558254+08:00
# User#Host: backup_all[backup_all] # [127.0.0.1] Id: 3168695
# Query_time: 0.025533 Lock_time: 0.000058 Rows_sent: 296 Rows_examined: 296
SET timestamp=1516650101;
SELECT /*!40001 SQL_NO_CACHE */ * FROM `legend_gift`;
# Time: 2018-01-23T03:42:09.536559+08:00
# User#Host: zabbix_agent[zabbix_agent] # [127.0.0.1] Id: 3169056
# Query_time: 0.000304 Lock_time: 0.000162 Rows_sent: 1 Rows_examined: 1
SET timestamp=1516650129;
SELECT SUM(trx_rows_locked) AS rows_locked, SUM(trx_rows_modified) AS rows_modified, SUM(trx_lock_memory_bytes) AS lock_memory FROM information_schema.INNODB_TRX;
after removing ,the text become the following output,how to use sed to get this goal?
# Time: 2018-01-23T03:42:09.536559+08:00
# User#Host: zabbix_agent[zabbix_agent] # [127.0.0.1] Id: 3169056
# Query_time: 0.000304 Lock_time: 0.000162 Rows_sent: 1 Rows_examined: 1
SET timestamp=1516650129;
SELECT SUM(trx_rows_locked) AS rows_locked, SUM(trx_rows_modified) AS rows_modified, SUM(trx_lock_memory_bytes) AS lock_memory FROM information_schema.INNODB_TRX;
update
this text is from mysql slow log ,i want to remove all sql info from user 'backup_all' , as you see ,this text product by user 'backup_all' is definite to be regular ,but other text product by other user may not be same,they will be
# Time: 2018-01-23T03:41:24.490723+08:00
# User#Host: zabbix_agent[zabbix_agent] # [127.0.0.1] Id: 3169038
# Query_time: 0.000669 Lock_time: 0.000334 Rows_sent: 1 Rows_examined: 27
SET timestamp=1516650084;
select count(*) Slownum from information_schema.processlist where COMMAND = 'Query' and info not like '%information_schema.processlist%' and TIME > 0;
# Time: 2018-01-23T03:41:40.628284+08:00
# User#Host: backup_all[backup_all] # [127.0.0.1] Id: 3168695
# Query_time: 78.333179 Lock_time: 0.000073 Rows_sent: 13269064 Rows_examined: 13269064
SET timestamp=1516650100;
SELECT /*!40001 SQL_NO_CACHE */ * FROM `legend_final_inventory`;
# Time: 2018-01-23T03:41:40.925596+08:00
# User#Host: backup_all[backup_all] # [127.0.0.1] Id: 3168695
# Query_time: 0.175956 Lock_time: 0.000065 Rows_sent: 5101 Rows_examined: 5101
SET timestamp=1516650100;
SELECT /*!40001 SQL_NO_CACHE */ * FROM `legend_finance_account`;
or
# Time: 2018-01-23T04:29:26.903048+08:00
# User#Host: yun_rw_legend[yun_rw_legend] # [10.162.86.162] Id: 3167670
# Query_time: 0.000150 Lock_time: 0.000053 Rows_sent: 6 Rows_examined: 32
SET timestamp=1516652966;
select id, value from legend_precheck_value where value_type = 8 and is_deleted = 'N';
# Time: 2018-01-23T04:29:31.825823+08:00
# User#Host: yun_rw_legend[yun_rw_legend] # [10.162.86.162] Id: 3167670
# Query_time: 0.000826 Lock_time: 0.000146 Rows_sent: 0 Rows_examined: 947
SET timestamp=1516652971;
select
id as id,
is_deleted as isDeleted,
gmt_create as gmtCreate,
creator as creator,
gmt_modified as gmtModified,
modifier as modifier,
shop_id as shopId,
customer_id as customerId,
account_id as accountId,
customer_car_id as customerCarId
from legend_customer_car_rel
WHERE is_deleted = 'N'
and shop_id = 3374
and customer_car_id = 1307177;
# Time: 2018-01-23T04:30:01.149529+08:00
# User#Host: backup_all[backup_all] # [127.0.0.1] Id: 3170398
# Query_time: 0.003047 Lock_time: 0.000169 Rows_sent: 0 Rows_examined: 385
SET timestamp=1516653001;
SELECT LOGFILE_GROUP_NAME, FILE_NAME, TOTAL_EXTENTS, INITIAL_SIZE, ENGINE, EXTRA FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = 'UNDO LOG' AND FILE_NAME IS NOT NULL AND LOGFILE_GROUP_NAME IS NOT NULL GROUP BY LOGFILE_GROUP_NAME, FILE_NAME, ENGINE, TOTAL_EXTENTS, INITIAL_SIZE, EXTRA ORDER BY LOGFILE_GROUP_NAME;
# Time: 2018-01-23T04:30:01.151783+08:00
# User#Host: backup_all[backup_all] # [127.0.0.1] Id: 3170398
# Query_time: 0.002082 Lock_time: 0.000119 Rows_sent: 0 Rows_examined: 385
SET timestamp=1516653001;
SELECT DISTINCT TABLESPACE_NAME, FILE_NAME, LOGFILE_GROUP_NAME, EXTENT_SIZE, INITIAL_SIZE, ENGINE FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = 'DATAFILE' ORDER BY TABLESPACE_NAME, LOGFILE_GROUP_NAME;
Obligatory awk solution which seems to work:
awk '
$2!~/backup_all/ {
printf "# Time%s", $0
}' RS='# Time' ORS='' FS='
' example_in
It splits the input into records on every # Time, and fields at \n to give each line as a field. If the second line (field) doesn't have backup_all in it, print
output for first example:
# Time: 2018-01-23T03:42:09.536559+08:00
# User#Host: zabbix_agent[zabbix_agent] # [127.0.0.1] Id: 3169056
# Query_time: 0.000304 Lock_time: 0.000162 Rows_sent: 1 Rows_examined: 1
SET timestamp=1516650129;
SELECT SUM(trx_rows_locked) AS rows_locked, SUM(trx_rows_modified) AS rows_modified, SUM(trx_lock_memory_bytes) AS lock_memory FROM information_schema.INNODB_TRX
and for the second:
# Time: 2018-01-23T04:29:26.903048+08:00
# User#Host: yun_rw_legend[yun_rw_legend] # [10.162.86.162] Id: 3167670
# Query_time: 0.000150 Lock_time: 0.000053 Rows_sent: 6 Rows_examined: 32
SET timestamp=1516652966;
select id, value from legend_precheck_value where value_type = 8 and is_deleted = N;
# Time: 2018-01-23T04:29:31.825823+08:00
# User#Host: yun_rw_legend[yun_rw_legend] # [10.162.86.162] Id: 3167670
# Query_time: 0.000826 Lock_time: 0.000146 Rows_sent: 0 Rows_examined: 947
SET timestamp=1516652971;
select
id as id,
is_deleted as isDeleted,
gmt_create as gmtCreate,
creator as creator,
gmt_modified as gmtModified,
modifier as modifier,
shop_id as shopId,
customer_id as customerId,
account_id as accountId,
customer_car_id as customerCarId
from legend_customer_car_rel
WHERE is_deleted = N
and shop_id = 3374
and customer_car_id = 1307177;
This might work for you (GNU sed):
sed -n '/^#/{:a;N;/^SELECT/M!ba;/backup_all/!p}' file
As this is a reduction/filtering of lines use seds grep-like nature achieved by setting the -n option. Gather up lines that begin with a comment line (#) and end with a line beginning SELECT. If these lines do not contain the word backup_all print them.
N.B. the M flag on regexp invokes a multiline mode, where ^ and $ can be matched on start/end of lines. An alternative regexp would be /\nSELECT/.
I've been playing with capistrano deploy, I managed to connect to my server, but now I am stuck at step 03, but I don't have log errors and I don't know which might be the next thing to do.
Help is highly appreciate. I don't whant you to solve by me of course not. I just want some guidance, thank you.
$ cap production deploy --trace
** Invoke production (first_time)
** Execute production
** Invoke load:defaults (first_time)
** Execute load:defaults
** Invoke bundler:map_bins (first_time)
** Execute bundler:map_bins
** Invoke deploy (first_time)
** Execute deploy
** Invoke deploy:starting (first_time)
** Execute deploy:starting
** Invoke deploy:check (first_time)
** Execute deploy:check
** Invoke git:check (first_time)
** Invoke git:wrapper (first_time)
** Execute git:wrapper
00:00 git:wrapper
01 mkdir -p /tmp
✔ 01 vlmg#vlmg.mx 0.249s
Uploading /tmp/git-ssh-vlmg-forms-production-ubuntu.sh 100.0%
02 chmod 700 /tmp/git-ssh-vlmg-forms-production-ubuntu.sh
✔ 02 vlmg#vlmg.mx 0.055s
This is my deploy.rb
# config valid only for current version of Capistrano
lock '3.6.0'
set :application, 'vlmg-forms'
set :repo_url, 'git#github.com:jose-octomedia/ar-vlmg-contact-forms.git'
# Default branch is :master
# ask :branch, `git rev-parse --abbrev-ref HEAD`.chomp
# Default deploy_to directory is /var/www/my_app_name
#set :deploy_to, '/'
# Default value for :scm is :git
# set :scm, :git
# Default value for :format is :airbrussh.
# set :format, :airbrussh
# You can configure the Airbrussh format using :format_options.
# These are the defaults.
# set :format_options, command_output: true, log_file: 'log/capistrano.log', color: :auto, truncate: :auto
# Default value for :pty is false
set :pty, true
# Default value for :linked_files is []
# append :linked_files, 'config/database.yml', 'config/secrets.yml'
# Default value for linked_dirs is []
# append :linked_dirs, 'log', 'tmp/pids', 'tmp/cache', 'tmp/sockets', 'public/system'
# Default value for default_env is {}
# set :default_env, { path: "/opt/ruby/bin:$PATH" }
# Default value for keep_releases is 5
# set :keep_releases, 5
desc 'Restart application'
task :restart do
on roles(:web), in: :sequence, wait: 5 do
execute "service thin restart" ## -> line you should add
end
end
And this is my production.rb file
# server-based syntax
# ======================
# Defines a single server with a list of roles and multiple properties.
# You can define all roles on a single server, or split them:
server 'vlmg.mx', user: 'vlmg', roles: %w{web}, primary: true, port: 15555
# server 'example.com', user: 'deploy', roles: %w{app web}, other_property: :other_value
# server 'db.example.com', user: 'deploy', roles: %w{db}
set :deploy_to, "/home/vlmg/subdomains/ar.vlmg.mx"
# role-based syntax
# ==================
# Defines a role with one or multiple servers. The primary server in each
# group is considered to be the first unless any hosts have the primary
# property set. Specify the username and a domain or IP for the server.
# Don't use `:all`, it's a meta role.
#role :app, %w{admin#example.com}, my_property: :my_value
role :web, %w{vlmg#vlmg.mx}
# role :db, %w{deploy#example.com}
# Configuration
# =============
# You can set any configuration variable like in config/deploy.rb
# These variables are then only loaded and set in this stage.
# For available Capistrano configuration variables see the documentation page.
# http://capistranorb.com/documentation/getting-started/configuration/
# Feel free to add new variables to customise your setup.
# Custom SSH Options
# ==================
# You may pass any option but keep in mind that net/ssh understands a
# limited set of options, consult the Net::SSH documentation.
# http://net-ssh.github.io/net-ssh/classes/Net/SSH.html#method-c-start
#
# Global options
# --------------
set :ssh_options, {
keys: %w(/home/ubuntu/.ssh/id_rsa),
forward_agent: true
#auth_methods: %w(password)
}
#
# The server-based syntax can be used to override options:
# ------------------------------------
# server 'example.com',
# user: 'user_name',
# roles: %w{web app},
# ssh_options: {
# user: 'user_name', # overrides user setting above
# keys: %w(/home/ubuntu/.ssh/id_rsa),
# forward_agent: false,
## auth_methods: %w(publickey password)
# # password: 'please use keys'
# }
Edit: In response of enabling the logs. This is the results.
Still nothing in the logs that could help to solve.
INFO ---------------------------------------------------------------------------
INFO START 2016-08-10 19:49:49 +0000 cap production deploy
INFO ---------------------------------------------------------------------------
INFO [204569e4] Running /usr/bin/env mkdir -p /tmp as vlmg#vlmg.mx
DEBUG [204569e4] Command: /usr/bin/env mkdir -p /tmp
INFO [9596edc0] Running /usr/bin/env mkdir -p /tmp as vlmg#vlmg.mx
DEBUG [9596edc0] Command: /usr/bin/env mkdir -p /tmp
INFO [9596edc0] Finished in 0.288 seconds with exit status 0 (successful).
DEBUG Uploading /tmp/git-ssh-vlmg-forms-production-ubuntu.sh 0.0%
INFO Uploading /tmp/git-ssh-vlmg-forms-production-ubuntu.sh 100.0%
INFO [1014be51] Running /usr/bin/env chmod 700 /tmp/git-ssh-vlmg-forms-production-ubuntu.sh as vlmg#vlmg.mx
DEBUG [1014be51] Command: /usr/bin/env chmod 700 /tmp/git-ssh-vlmg-forms-production-ubuntu.sh
INFO [1014be51] Finished in 0.057 seconds with exit status 0 (successful).
After last command this are the results, basically the same, hanging for 20-30 seconds then:
00:00 git:wrapper
01 mkdir -p /tmp
✔ 01 vlmg#vlmg.mx 0.265s
Uploading /tmp/git-ssh-vlmg-forms-production-ubuntu.sh 100.0%
02 chmod 700 /tmp/git-ssh-vlmg-forms-production-ubuntu.sh
✔ 02 vlmg#vlmg.mx 0.063s
(Backtrace restricted to imported tasks)
cap aborted!
Net::SSH::ConnectionTimeout: Net::SSH::ConnectionTimeout
Errno::ETIMEDOUT: Connection timed out - connect(2) for 109.73.225.169:22
Tasks: TOP => git:wrapper
(See full trace by running task with --trace)
This step 03 is solved, thanks. Solution was specify port on ssh connection too.
set :ssh_options, {
keys: %w(/home/ubuntu/.ssh/id_rsa),
forward_agent: true,
port: 15555,
#auth_methods: %w(password)
}
Now I have new errors, but I think that should could be in another post.
I have a ruby script that uploads a file via FTP.
I tested this with a LINUX server and the upload was working well.
But when I changed the upload destination to the production server which is a Windows Server 2012 R2 running IIS, I got the following error message:
A Connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. - connect<2> <Errno::ETIMEDOUT>
I do however find the connection in the FTPlog:
2015-01-28 10:48:55 12.34.56.78 - 87.65.43.21 21 ControlChannelOpened - - 0 0 0 0 0
2015-01-28 10:48:55 12.34.56.78 - 87.65.43.21 21 USER .... 331 0 0 23 17 0
2015-01-28 10:48:55 12.34.56.78 WIN-0CFF8VSL25E\.... 87.65.43.21 21 PASS *** 230 0 0 21 15 94
2015-01-28 10:48:55 12.34.56.78 WIN-0CFF8VSL25E\.... 87.65.43.21 21 TYPE I 200 0 0 20 8 0
2015-01-28 10:48:55 12.34.56.78 WIN-0CFF8VSL25E\.... 87.65.43.21 21 PASV - 227 0 0 50 6 0
2015-01-28 10:49:16 - WIN-0CFF8VSL25E\.... 87.65.43.21 49994 DataChannelClosed - - 0 0 0 0 0
2015-01-28 10:49:16 12.34.56.78 WIN-0CFF8VSL25E\.... 87.65.43.21 21 ControlChannelClosed - - 0 0 141 46 21469
I am doing almost the same task with a batch script today, using windows native ftp cli, and this works fine.
Here is the ruby code:
require "net/ftp"
require "rubygems"
require "zip"
puts "Copying data"
FileUtils.cp("...", "...")
folder = "C:\\..."
input_filenames = ['...']
zipfile_name = "C:\\...\\....zip"
puts "Compressing data"
Zip::File.open(zipfile_name, Zip::File::CREATE) do |zipfile|
input_filenames.each do |filename|
zipfile.add(filename, folder + '\\' + filename)
end
end
progress = 0.0;
file_size = File.size(zipfile_name)
NET::FTP.debug_mode = true
puts "Uploading data:"
ftp = Net::FTP.new('12.34.56.78','xxx','xxx')
ftp.debug_mode = true
ftp.read_timeout = 10000
ftp.passive = true
ftp.putbinaryfile('C:\\...\\....zip', '....zip') do |data|
progress = progress.to_f+data.length.to_f
print "\rProgress: " + ((progress/file_size)*100).round(2).to_s + "%"
end
ftp.close()
puts "\nUpload completed!"
I am assuming there is no problem in the script, since it was working fine with a different server. What I am assuming it is, is a difference in how IIS' FTP server handles/responds to connections vs how vsftpd handles them.
Any suggestions to how to fix this?
I set passive to false and that solved my problem.