How to save and display Dashing historical values? - ruby

Currently to setup a graph widget, the job should pass all values to be displayed:
data = [
{ "x" => 1980, "y" => 1323 },
{ "x" => 1981, "y" => 53234 },
{ "x" => 1982, "y" => 2344 }
]
I would like to read just current (the latest) value from my server, but previous values should be also displayed.
It looks like I could create a job, which will read the current value from the server, but remaining values to be read from the Redis (or sqlite database, but I would prefer Redis). The current value after that should be saved to the database.
I never worked with Ruby and Dashing before, so the first question I have - is it possible? If I will use Redis, then the question is how to store the data since this is key-value database. I can keep it as widget-id-1, widget-id-2, widget-id-3 ... widget-id-N etc., but in this case I will have to store N value (like widget-id=N). Or, is there any better way?

I came to the following solution:
require 'redis' # https://github.com/redis/redis-rb
redis_uri = URI.parse(ENV["REDISTOGO_URL"])
redis = Redis.new(:host => redis_uri.host, :port => redis_uri.port, :password => redis_uri.password)
if redis.exists('values_x') && redis.exists('values_y')
values_x = redis.lrange('values_x', 0, 9) # get latest 10 records
values_y = redis.lrange('values_y', 0, 9) # get latest 10 records
else
values_x = []
values_y = []
end
SCHEDULER.every '10s', :first_in => 0 do |job|
rand_data = (Date.today-rand(10000)).strftime("%d-%b") # replace this line with the code to get your data
rand_value = rand(50) # replace this line with the code to get your data
values_x << rand_data
values_y << rand_value
redis.multi do # execute as a single transaction
redis.lpush('values_x', rand_data)
redis.lpush('values_y', rand_value)
# feel free to add more datasets values here, if required
end
data = [
{
label: 'dataset-label',
fillColor: 'rgba(220,220,220,0.5)',
strokeColor: 'rgba(220,220,220,0.8)',
highlightFill: 'rgba(220,220,220,0.75)',
highlightStroke: 'rgba(220,220,220,1)',
data: values_y.last(10) # display last 10 values only
}
]
options = { scaleFontColor: '#fff' }
send_event('barchart', { labels: values_x.last(10), datasets: data, options: options })
end
Not sure if everything is implemented correctly here, but it works.

Related

Logstash count events using ruby filter

I have following pipeline (example, not full pipeline) where I need count the number of events for EVERY file I am writing as output. For example, this pipleline creates new file every 30 seconds with the events processed in that 30 seconds. This code is sort of working except two issues:
Writing each trap_id as an entry in out put file (I just want ONE entry with final count) -- Though I am doing just event count, my end goal is, count different log_level events, for example, how many "Error", "Information" windows events etc.,
When I reset the trap_id counter, the first entry in output file starting with ZERO instead of ONE.
Can someone please advise how I can address these issues?
input {
beats {
port => 5045
}
}
filter
{
ruby {
init => '
#trap_id = 0
#lasttimestmp = 0
'
code => '
evnttme = event.get("[#metadata][ms]")
if #lasttimestmp == evnttme
#trap_id += 1
event.set("lsttimestmp", #lasttimestmp)
event.set("trap_id", #trap_id)
else
#trap_id = 0
#lasttimestmp = evnttme
event.set("lsttimestmp", evnttme)
event.set("[#metadata][ms]", evnttme)
event.set("trap_id", #trap_id)
end
'
}
}
output {
file {
path => "output.log"
}
file {
flush_interval => 30
codec => line { format => "%{[#metadata][ts]}, %{[trap_id]}"}
path => "C:/lgstshop/local/csv/output%{[#metadata][ms]}.csv"
}
}

How to get commit history of users in gitlab in rails

I am trying to create an app which will fetch the gitlab/github commit history of the user which i can just show in a side panel and it can be checked on or off depending on my criteria. I want to know if there is a way to fetch the current logged user's gitlab/github commit history. I tried to use the gem
https://github.com/libgit2/rugged
But couldn't find a way to implement my need. If anyone knows how to implement this it would be so much helpful. Thanks.
Update
I have now managed to get the user by using :
Gitlab.endpoint = 'https://gitlab.com/api/v4'
Gitlab.private_token = 'token'
g = Gitlab.client(
endpoint: 'https://gitlab.com/api/v4',
private_token: 'token',
httparty: {
headers: { 'Cookie' => 'gitlab_canary=true' }
}
)
By using the command g.user i am able to get the user but i need to get the commits of the user that he has done in gitlab.
Use this gitlab API GET /projects/:id/repository/commits to fetch all the commits on a repository gitlab api. Check the attached code for more details.
Basically this git log --author="user_name" command can give you git commit history for a specific user, you can even use email address just the first name or last name in the above command.
Once you have authenticated with gitlab you can run the following command from ruby.
cmd = 'git log --author="user_name"'
system(cmd)
Sample code form ruby to connect to gitlab using private token which is not ideal but just an example
require 'json'
require 'curb'
require 'net/http'
begin
def parseCoverageReport(report_text)
coverage_perc = report_text.match /All files\s+\|\s+(\d+\.?\d+).*\n/
if not coverage_perc then
coverage_perc = report_text.match /^TOTAL\s+\d+\s+\d+\s+(\d+)%$/
end
if coverage_perc then
#if we found coverage value in job trace
#puts "coverage_perc[1]: #{coverage_perc[1]}"
coverage_perc[1].to_i
end
end
gen_config = YAML.load_file("config/general.yml")
gitlab_config = YAML.load_file("config/gitlab.yml")
SCHEDULER.every gen_config[:job_trigger_interval], :first_in => 0 do |job|
table = {
title: "Projects",
hrows: Array.new.push({cols: [
{value: "Project name"},
{value: "Open Merge Requests"},
{value: "Code coverage"}
]}),
rows: Array.new
}
instances = gitlab_config['instances']
instances.each do |instance|
gitlab_url = gitlab_config['instances'][instance.first]['url']
# gitlab_token = gitlab_config['instances'][instance.first]['api_key']
gitlab_token = ENV[gitlab_config['instances'][instance.first]['api_key']]
red_threshold = gitlab_config['instances'][instance.first]['red_threshold']
orange_threshold = gitlab_config['instances'][instance.first]['orange_threshold']
cov_red_threshold = gitlab_config['instances'][instance.first]['cov_red_threshold']
cov_orange_threshold = gitlab_config['instances'][instance.first]['cov_orange_threshold']
projects = gitlab_config['instances'][instance.first]['projects']
projects.each do |name, project|
merge_reqs = JSON.parse(Curl.get("#{gitlab_url}/api/v4/projects/#{project['id']}/merge_requests?state=opened&private_token=#{gitlab_token}&per_page=200").body_str)
git_project = JSON.parse(Curl.get("#{gitlab_url}/api/v4/projects/#{project['id']}?private_token=#{gitlab_token}").body_str)
opened_mrs = merge_reqs.select { |merge_reqs| %w[opened].include? merge_reqs['state'] }
repo_name = git_project['name']
repo_url = git_project['web_url']
status = case
when opened_mrs.size >= red_threshold then 'danger'
when opened_mrs.size >= orange_threshold then 'warning'
else
'ok'
end
mrs_count = "#{opened_mrs.size}"
send_event("#{name}_mr", { current: mrs_count, status: status })
color = case
when opened_mrs.size >= red_threshold then 'red'
when opened_mrs.size >= orange_threshold then 'orange'
else
'green'
end
font_color = color == 'orange' ? 'black' : 'white'
cov_color = color
font_cov_color = 'white'
code_coverage = "---"
code_coverage_tag = "---"
cov_job_url = ''
jobs = JSON.parse(Curl.get("#{gitlab_url}/api/v4/projects/#{project['id']}/jobs?scope=success&private_token=#{gitlab_token}&per_page=30").body_str)
code_cov_job = jobs.find { |gitlab_job| !gitlab_job['coverage'].nil? }
if not code_cov_job then
#if no job has 'coverage' feature set up in Gitlab try to parse
#'coverage' from jobs trace manually
jobs.each do |job|
trace_report = Curl.get("#{gitlab_url}/api/v4/projects/#{project['id']}/jobs/#{job['id']}/trace?private_token=#{gitlab_token}").body_str
code_cov_percentage = parseCoverageReport(trace_report)
if code_cov_percentage then
code_cov_job = job
code_cov_job['coverage'] = code_cov_percentage
break
end
end
end
if code_cov_job then
#found code coverage data => process them
code_coverage = code_cov_job['coverage'].to_i
cov_job_url = code_cov_job['web_url'].to_s
#update code covergate SprintProgress widgets at the same job
widget_title = "code_coverage_progress_#{project['id']}"
send_event(widget_title, {
title: "Code Coverage - #{git_project['name']}",
sprintboard_url: cov_job_url,
min: 0,
max: 100,
value: code_coverage,
moreinfo: ''
})
cov_color = case
when code_coverage <= cov_red_threshold then 'red'
when code_coverage <= cov_orange_threshold then 'orange'
else
'green'
end
code_coverage = "#{code_coverage}%"
code_coverage_tag = "<a href='#{cov_job_url}' target='_blank'>#{code_coverage.to_s}</a>"
end
repo_name_a_tag = "<a href='#{repo_url}' target='_blank'>#{repo_name}</a>"
open_mrs_size = "<a href='#{repo_url}/merge_requests' target='_blank'>#{opened_mrs.size}</a>"
table[:rows].push({
cols: [
{ value: repo_name_a_tag, style: "color: #{font_color}; background-color: #{color}" },
{ value: open_mrs_size, style: "color: #{font_color}; background-color: #{color}" },
{ value: code_coverage_tag, style: "color: #{cov_color == 'orange' ? 'black' : 'white'}; background-color: #{cov_color}" }
]
})
end
end
send_event('open_merge_requests_table', table)
end
rescue Errno::ENOENT
puts "No config file found for gitlab - not starting the Gitlab job"
end
In the above ruby example please have a look at the following code snippet
merge_reqs = JSON.parse(Curl.get("#{gitlab_url}/api/v4/projects/#{project['id']}/merge_requests?state=opened&private_token=#{gitlab_token}&per_page=200").body_str)
git_project = JSON.parse(Curl.get("#{gitlab_url}/api/v4/projects/#{project['id']}?private_token=#{gitlab_token}").body_str)
opened_mrs = merge_reqs.select { |merge_reqs| %w[opened].include? merge_reqs['state'] }
repo_name = git_project['name']
repo_url = git_project['web_url']
In here what i am trying to do is connect to our gitlab instance using a private_token and then for a specific project id (which you can get it form the UI of gitlab) check for the open merge request. I also get the git_project from which i get the name and web_url (which was my use case).
For your use case you will have to get the project_id(for gitlab UI) and then use some appropriate method to get the commits.gitlab docs

Parsing a string field

I have these Syslog messages:
N 4000000 PROD 15307 23:58:12.13 JOB78035 00000000 $HASP395 GGIVJS27 ENDED\r
NI0000000 PROD 15307 23:58:13.41 STC81508 00000200 $A J78036 /* CA-JOBTRAC JOB RELEASE */\r
I would like to parse these messages into various fields in a Hash, e.g.:
event['recordtype'] #=> "N"
event['routingcode'] #=> "4000000"
event['systemname'] #=> "PROD"
event['datetime'] #=> "15307 23:58:12.13"
event['jobid'] #=> "JOB78035"
event['flag'] #=> "00000000"
event['messageid'] #=> "$HASP395"
event['logmessage'] #=> "$HASP395 GGIVJS27 ENDED\r"
This is the code I have currently:
message = event["message"];
if message.to_s != "" then
if message[2] == " " then
array = message.split(%Q[ ]);
event[%q[recordtype]] = array[0];
event[%q[routingcode]] = array[1];
event[%q[systemname]] = array[2];
event[%q[datetime]] = array[3] + " " +array[4];
event[%q[jobid]] = message[38,8];
event[%q[flags]] = message[47,8];
event[%q[messageid]] = message[57,8];
event[%q[logmessage]] = message[56..-1];
else
array = message.split(%Q[ ]);
event[%q[recordtype]] = array[0][0,2];
event[%q[routingcode]] = array[0][2..-1];
event[%q[systemname]] = array[1];
event[%q[datetime]] = array[2] + " "+array[3];
event[%q[jobid]] = message[38,8];
event[%q[flags]] = message[47,8];
event[%q[messageid]] = message[57,8];
event[%q[logmessage]] = message[56..-1];
end
end
I'm looking to improve the above code. I think I could use a regular expression, but I don't know how to approach it.
You can't use split(' ') or a default split to process your fields because you are dealing with columnar data that has fields that have no whitespace between them, resulting in your array being off. Instead, you have to pick apart each record by columns.
There are many ways to do that but the simplest and probably fastest, is indexing into a string and grabbing n characters:
'foo'[0, 1] # => "f"
'foo'[1, 2] # => "oo"
The first means "starting at index 0 in the string, grab one character." The second means "starting at index 1 in the string, grab two characters."
Alternately, you could tell Ruby to extract by ranges:
'foo'[0 .. 0] # => "f"
'foo'[1 .. 2] # => "oo"
These are documented in the String class.
This makes writing code that's easily understood:
record_type = message[ 0 .. 1 ].rstrip
routing_code = message[ 2 .. 8 ]
system_name = message[ 10 .. 17 ]
Once you have your fields captured add them to a hash:
{
'recordtype' => record_type,
'routingcode' => routing_code,
'systemname' => system_name,
'datetime' => date_time,
'jobid' => job_id,
'flags' => flags,
'messageid' => message_id,
'logmessage' => log_message,
}
While you could use a regular expression there's not much gained using one, it's just another way of doing it. If you were picking data out of free-form text it'd be more useful, but in columnar data it tends to result in visual noise that makes maintenance more difficult. I'd recommend simply determining your columns then cutting the data you need based on those from each line.

How to I reference an array member in Ruby?

Given this array in Ruby:
myarray = [name: "John", age: 35]
How do I refer to the age?
I tried myarray[:age] but got an error can't convert Symbol into Integer
Update:
I was trying to simplify my question by extracting what I thought my problem is. I may not understand completely.
I'm experimenting with Dashing and trying to send a number to a meter widget. I've created a variable, 'response_raw' and am trying to send it in the third send event. Here's my code:
SCHEDULER.every '1m', :first_in => 0 do
# Get checks
url = "https://#{CGI::escape user}:#{CGI::escape password}#api.pingdom.com/api/2.0/checks"
`enter code here`response = RestClient.get(url, {"App-Key" => api_key})
response = JSON.parse(response.body, :symbolize_names => true)
if response[:checks]
checks = response[:checks].map { |check|
if check[:status] == 'up'
state = 'up'
last_response_time = "#{check[:lastresponsetime]}ms"
response_raw = check[:lastresponsetime]
else
state = 'down'
last_response_time = "DOWN"
response_raw = 0
end
{ name: check[:name], state: state, lastRepsonseTime: last_response_time, pt: response_raw }
}
else
checks = [name: "pingdom", state: "down", lastRepsonseTime: "-", pt: 0]
end
checks.sort_by { |check| check['name'] }
send_event('pingdom', { checks: checks })
send_event('pingdom-meter', { value: checks[:pt] })
end
In CoffeeScript [name: "John", age: 35] is an array containing single object with two properties (name and age).
Here is how it'll look in plain JavaScript:
myarray = [
{
name: "John",
age: 35
}
];
So, answering your question, to access an age you should take the first element of an array and then reference an age property:
myarray[0].age
or
myarray[0]['age']
But, judging from your question, your're probably using wrong data structure. Why don't you want to use a plain object instead of an array?
person = name: "John", age: 35
console.log "#{person.name}'s age is #{person.age}"
Update
It looks like your question is actually about Ruby and not about CoffeeScript. Though, my answer will remain the same.
To access an age you should take the first element of an array and then reference an age property:
myarray[0][:age]
Since myarray is an array, Ruby expects an integer index, but you're giving it symbol :age instead.
I finally figured it out with Leonid's help. Thank you.
I changed:
send_event('pingdom-meter', { value: checks[:pt] })
to
send_event('pingdom-meter', { value: checks[0][:pt] })

capped collection mongodb

I have issues with mongoDB.
Currently i'm working with Ruby mongodb drivers and there r some strange things r going on:
i need to insert 20 documents in the capped collection but when i write the following code, it inserts only 3 docs and i can't get what's going on:
coll = db.create_collection("test",:capped => true, :max=>20)
1024.times{#pad_string +=" "}
20.times{coll.insert({
:HostName => #hostname,
:CommandLine => #cmdline,
:Pid => "1111",
:BlockName => #blockname,
:ExitCode => 0,
:StartTime => Time.now,
:EndTime => Time.utc(2000,"jan",1,00,00,00),
:StdErr => #pad_string,
:Stdout => #pad_string}
)}
actually the point is that i insert #pad_string with 1024 preallocated spaces. As soon as i do that before inserting 1024.times{#pad_string +=" "}, it inserts only 3 docs maximum.
When you cap a collection based on the number of objects you also have to cap it based on size - I wonder what size the ruby driver is sending down.
try this:
coll = db.create_collection("test",:capped => true, :size=>100000, :max=>20)
Then tweak the size to whatever works for you (it's in bytes).

Resources