cat <<EOF | curl --data-binary #- http://localhost:9091/metrics/job/pushgetway/instance/test_instance
http_s_attack_type{hostname="test1",scheme="http",src_ip="192.168.33.86",dst_ip="192.168.33.85",port="15555"} 44
http_s_attack_type{hostname="other",scheme="tcp",src_ip="1.2.3.4",dst_ip="192.168.33.85",port="15557"} 123
EOF
Change data and write again：
cat <<EOF | curl --data-binary #- http://localhost:9091/metrics/job/pushgetway/instance/test_instance
http_s_attack_type{hostname="test2",scheme="http",src_ip="192.168.33.86",dst_ip="192.168.33.85",port="15555"} 55
http_s_attack_type{hostname="other3",scheme="tcp",src_ip="1.2.3.4",dst_ip="192.168.33.85",port="15557"} 14
EOF
View the data on localhost:9091 becomes the last write data, the data written for the first time is overwritten。
Is there a problem with my operation? Please tell me how to continuously introduce new data without being overwritten or replaced
This is working exactly as designed. The pushgateway is meant to hold the results of batch jobs when they exit, so on the next run the results will replace the previous run.
It sounds like you're trying to do event logging. Prometheus is not a suitable tool for that use case, you might want to consider something like the ELK stack instead.
Related
For a project I need to extract data from a lot of different blockchain GitHub profiles to a csv.
After browsing through the GitHub API I was able to achieve some of the necessary data being shown as txt/csv files using bash commands and jq.
Now doing all of this manually would probably take 7 days. I have a list of profiles i need to loop through saved as CSV.
The list looks like this --> https://docs.google.com/spreadsheets/d/1lFsewAYI7F8zSw7WPhI9E9WwR8f4G1clw1yjxY3wz_4/edit#gid=0
My approach so far to get all the repo names looks like this:
sample='[{"name":"0chain"},{"name":"0stateapp"},{"name":"0xcert"}]'
the csv belongs in here, I didn't know how to redirect it to that variable yet, but for testing purposes this was enough. If somebody knows how to, feel free to give a hint.
for row in $(echo "${sample}" | jq -r '.[] | #base64'); do
_jq()
{
echo ${row} | base64 --decode | jq -r ${1}
}
for GHUSER in $( echo $(_jq '.name')); do
curl -s https://api.github.com/users/$GHUSER/repos?per_page=100 | jq -r '.[]|.full_name'
done
done
The output looks like this:
0chain/0chain-token
0chain/client-sdk
0chain/docs
0chain/gorocksdb
0chain/hostadmin
0chain/rocksdb
0stateapp/ZSCoin
0xcert/0xcert
0xcert/conventions
0xcert/docs
0xcert/erc721-validator
0xcert/erc721-validator-api
0xcert/erc721-validator-ui
0xcert/erc721-website
0xcert/ethereum
0xcert/ethereum-crowdsale
0xcert/ethereum-dex
0xcert/ethereum-erc20
0xcert/ethereum-erc721
0xcert/ethereum-minter
0xcert/ethereum-utils
0xcert/ethereum-xcert
0xcert/ethereum-xcert-builder
0xcert/ethereum-zxc
0xcert/framework
0xcert/framework-cert-test
0xcert/nonfungiblealliance-www
0xcert/solidity-style-guide
0xcert/techpaper
0xcert/truffle
0xcert/web3.js
What I need to do is use all of the above values and generate a file that contains:
Github Profile (already stored in the attached sheet)
The Date when accessing this information
All the repositories belonging to that profile (code above but
filtered)
Now the Interesting part:
The commit history
number of commit (ID)
number of commit (ID)
Date of commit
Description of commit
person who commited
checks passed
checks failed
Almost the same needs to be done for closed and open pull requests although I think when solving the "problem" above solving the pull requests is the same strategy.
For the commits I'd do something like this:
for commits in $( $repoarray) do curl -i https://api.github.com/repos/$commits/commits | jq -r '.[]|.author.lgoin (and whatever els is needed)' done
basically this chart here needs to be filled
https://docs.google.com/spreadsheets/d/1mFXiohiWNXNP8CVztFA1PFF41jn3J9sRUhYALZShsPY/edit?usp=sharing
what I need help with:
storing my output from the first loop in a an array
loop through that array to get the number of commits
loop through that array to get the data to closed pull requests
loop through that array to get the data to open pull requests
Excuse my "noobish" question.
I'm using bash/jq and the GitHub API for the time.
I'd appreciate any kind of help.
I'm having issues while trying to insert data using a csv file to clickhouse using CURL, the very first value is adding some characters and looks like follow:
┌─name─ ┬─lastname─┐
│&'Mark'│ Olson │
│ Joe │ Robins │
└────── ┴──────────┘
my CSV file is ok, it is like this:
'Mark','Olson'
'Joe','Robins'
as you can see the table is adding the first value in first record as &'Mark'
This is my code in bash
query="INSERT INTO Myschema.persons FORMAT CSV"
cat ${csv} | curl -X POST -d "$query" $user:$password#localhost:8123 --data-binary #-
Do you know what's the problem?
Thanks
I think you should use following format where query is part of url
cat *.csv |curl http://localhost:8123/?query=INSERT%20INTO%20Myschema.persons%20FORMAT%20CSV' --data-binary #-
I am not sure why your curl is not working but my best guess is that Clickhouse has parsing rules which are not able to consume your specified format, ${query} and ${csv} both being parameters to POST are getting appended by '&' in final http url, while parsing Clickhouse is unable to consider this case.
Quotes from clickhouse documentation -
You can send the query itself either in the POST body, or in the URL
parameter.
and
The POST method of transmitting data is necessary for INSERT queries.
In this case, you can write the beginning of the query in the URL
parameter, and use POST to pass the data to insert. The data to insert
could be, for example, a tab-separated dump from MySQL. In this way,
the INSERT query replaces LOAD DATA LOCAL INFILE from MySQL.
Check here for more details and examples - https://clickhouse.yandex/docs/en/interfaces/http_interface/
Is there a way to make sure that AB gets proper responses from server? For example:
To force it to output the response of a single request to STDOUT OR
To ask it to check that some text fragment is included into the response body
I want to make sure that authentication worked properly and i am measuring response time of the target page, not the login form.
Currently I just replace ab -n 100 -c 1 -C "$MY_COOKIE" $MY_REQUEST with curl -b "$MY_COOKIE" $MY_REQUEST | lynx -stdin .
If it's not possible, is there an alternative more comprehensive tool that can do that?
You can use the -v option as listed in the man doc:
-v verbosity
Set verbosity level - 4 and above prints information on headers, 3 and above prints response codes (404, 200, etc.), 2 and above prints warnings and info.
https://httpd.apache.org/docs/2.4/programs/ab.html
So it would be:
ab -n 100 -c 1 -C "$MY_COOKIE" -v 4 $MY_REQUEST
This will spit out the response headers and HTML content. The 3 value will be enough to check for a redirect header.
I didn't try piping it to Lynx but grep worked fine.
Apache Benchmark is good for a cursory glance at your system but is not very sophisticated. I am currently attempting to tune a web service and am finding that AB does not measure complete response time when considering the transfer of the body. Also as you mention you can not verify what is returned.
My current recommendation is Apache JMeter. http://jmeter.apache.org/
I am having much better success with it. You may find the Response Assertion useful for your situation. http://jmeter.apache.org/usermanual/component_reference.html#Response_Assertion
On the Parse.com cloud-code console, I can see logs, but they only go back maybe 100-200 lines. Is there a way to see or download older logs?
I've searched their website & googled, and don't see anything.
Using the parse command-line tool, you can retrieve an arbitrary number of log lines:
Usage:
parse logs [flags]
Aliases:
logs, log
Flags:
-f, --follow=false: Emulates tail -f and streams new messages from the server
-l, --level="INFO": The log level to restrict to. Can be 'INFO' or 'ERROR'.
-n, --num=10: The number of the messages to display
Not sure if there is a limit, but I've been able to fetch 5000 lines of log with this command:
parse logs prod -n 5000
To add on to Pascal Bourque's answer, you may also wish to filter the logs by a given range of dates. To achieve this, I used the following:
parse logs -n 5000 | sed -n '/2016-01-10/, /2016-01-15/p' > filteredLog.txt
This will get up to 5000 logs, use the sed command to keep all of the logs which are between 2016-01-10 and 2016-01-15, and store the results in filteredLog.txt.
Using a Raspi/Debian - I have a script that parses the results from an iwlist scan and sends them via UDP to a Pure Data patch. This runs fine in gui mode, but now I'm trying to automate the whole process in another script with the following:
pd-extended -nogui /home/pi/patch.pd & /home/pi/libOSC/scan.sh && fg
But when I run this new script, the UDP appears to only send the info to Pure Data once, and then the scanning continues but Pd does not receive the packet. Any help with this would be appreciated.
What happens when you run /home/pi/libOSC/scan.sh? It sends the results only once? Then maybe you need to do it differently, like calling that script from within pd using the 'shell' or 'popen' objects for instance. Or you implement a polling command via UDP that will return the values.
how does your scan.sh script look like?
you probably want to make it something like:
pdhost=localhost
pdport=9999
do_scan() {
## some code here that does the scan and print's the result to stdout
}
doscan | while read line
do
echo "${line};" | pdsend ${pdhost} ${pdport}
done
rather than the following:
doscan | pdsend ${pdhost} ${pdport}