I am new to cassandra so I followed this guide on how to get Cassandra set up on an EC2 instance. I have gotten it all set up and ready to go but for some reason I'm not able to connect from ruby. Here is what I have been trying:
require 'cassandra'
client = Cassandra.new('PERSON', 'ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com:9160')
# =>
# <Cassandra:0x100cda3b0
# #auto_discover_nodes = true,
# #column_name_class = {},
# #column_name_maker = {},
# #is_super = {},
# #sub_column_name_class = {},
# #sub_column_name_maker = {},
# attr_accessor :keyspace = "PERSON",
# attr_reader :servers = [
# [0] "ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com:9160"
# ],
# attr_reader :thrift_client_class = ThriftClient < AbstractThriftClient,
# attr_reader :thrift_client_options = {
# :transport_wrapper => Thrift::FramedTransport < Thrift::BaseTransport,
# :thrift_client_class => ThriftClient < AbstractThriftClient,
# :protocol => Thrift::BinaryProtocolAccelerated < Thrift::BinaryProtocol
# }
# >
Then when I tried to use the client, I get this error:
client.keyspaces
#=> ThriftClient::NoServersAvailable: No live servers in [ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com:9160].
I am able to connect via SSH so I'm not sure what I'm doing wrong here.
Update:
My security group includes:
port: 9160
protocol: tcp
source: sg-xxxxxxxx
Where is the "source" of the connection into the Cassandra server?
The security group you are using only opens up port 9160 to instances in the group sg-xxxxxxxx. If you are trying to connect from anywhere else (like the outside world) you will not be successful.
There are two things to check:
do you have a firewall running on the client and/or the server?
do you allow access in your EC2 security group from your client to your server?
Related
I'm working to create a Logstash Input plugin to utilize ADAL for integration with the Office 365 Management Activity API's. I've written the individual components to get a token, use that token to subscribe, and to pull activity log data.
Now I'm working to integrate into the Logstash framework, and running into issues where Logstash is complaining that it's doesn't know what ADAL is, even though I have it required.
All the same code works independently outside of Logstash, just not within the plugin class.
This is my first foray into Ruby, so I'm pretty stumped. Any help?
Error message from Logstash:
[2018-09-16T00:51:32,816][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-09-16T00:51:33,921][INFO ][logstash.inputs.office365managementapi] Starting Office 365 Management API input...
[2018-09-16T00:51:34,246][ERROR][logstash.pipeline ] Error registering plugin {:pipeline_id=>"main", :plugin=>"<LogStash::Inputs::Office365ManagementApi client_id=>\"redacted\", tenant_id=>\"redacted\", tenant_domain=>\"redacted\", private_key=>\"/tmp/o365.pfx\", subscriptions=>[\"Audit.AzureActiveDirectory\", \"Audit.Exchange\", \"Audit.SharePoint\", \"Audit.General\", \"DLP.All\"], id=>\"fb61b83b76494f098a0a7e24391779ee1212f0d9adf8ef8dedae4424e8dedb57\", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>\"plain_c7c9d514-5d23-459d-98ea-87d250e7a00c\", enable_metric=>true, charset=>\"UTF-8\">, resource=>\"https://manage.office.com\">", :error=>"uninitialized constant LogStash::Inputs::Office365ManagementApi::ADAL::Logging\nDid you mean? LogStash::Logging", :thread=>"#<Thread:0xca2e135 run>"}
[2018-09-16T00:51:34,367][ERROR][logstash.pipeline ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<NameError: uninitialized constant LogStash::Inputs::Office365ManagementApi::ADAL::Logging
Did you mean? LogStash::Logging>, :backtrace=>["org/jruby/RubyModule.java:3343:in `const_missing'", "/usr/local/Cellar/logstash/6.2.4/libexec/vendor/local_gems/82bdbf8d/logstash-input-office365_management_api-1.0.0/lib/logstash/inputs/office365_management_api.rb:70:in `register'", "/usr/local/Cellar/logstash/6.2.4/libexec/logstash-core/lib/logstash/pipeline.rb:342:in `register_plugin'", "/usr/local/Cellar/logstash/6.2.4/libexec/logstash-core/lib/logstash/pipeline.rb:353:in `block in register_plugins'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/local/Cellar/logstash/6.2.4/libexec/logstash-core/lib/logstash/pipeline.rb:353:in `register_plugins'", "/usr/local/Cellar/logstash/6.2.4/libexec/logstash-core/lib/logstash/pipeline.rb:500:in `start_inputs'", "/usr/local/Cellar/logstash/6.2.4/libexec/logstash-core/lib/logstash/pipeline.rb:394:in `start_workers'", "/usr/local/Cellar/logstash/6.2.4/libexec/logstash-core/lib/logstash/pipeline.rb:290:in `run'", "/usr/local/Cellar/logstash/6.2.4/libexec/logstash-core/lib/logstash/pipeline.rb:250:in `block in start'"], :thread=>"#<Thread:0xca2e135 run>"}
[2018-09-16T00:51:34,418][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: LogStash::PipelineAction::Create/pipeline_id:main, action_result: false", :backtrace=>nil}
Code is below:
# encoding: utf-8
require "logstash/inputs/base"
require "logstash/namespace"
require "stud/interval"
require "socket" # for Socket.gethostname
require "json"
require 'net/http'
require 'uri'
# Using this input you can receive activities from the Office 365 Management API
# ==== Security
# This plugin utilizes certificate authentication with the Office 365 Management API
# to generate an access token, which is then used for all subsequent API activities.
# If the token expires, the plugin will request a new token automatically.
# All communication for this plugin is encrypted by SSL/TLS communication.
class LogStash::Inputs::Office365ManagementApi < LogStash::Inputs::Base
config_name "office365_management_api"
# Codec used to decode the incoming data.
# This codec will be used as a fall-back if the content-type
# is not found in the "additional_codecs" hash
default :codec, "plain"
# Fix for broken ruby ADAL
module ADAL
class TokenRequest
module GrantType
JWT_BEARER = 'urn:ietf:params:oauth:client-assertion-type:jwt-bearer'
end
end
end
# Client ID generated through your custom application in Azure AD
# https://portal.azure.com/#blade/Microsoft_AAD_IAM/ActiveDirectoryMenuBlade/RegisteredApps
config :client_id, :validate => :string, :required => true
# Tenant ID/Directory ID of your Office 365 tenant
# https://portal.azure.com/#blade/Microsoft_AAD_IAM/ActiveDirectoryMenuBlade/Properties
config :tenant_id, :validate => :string, :required => true
# Your Office 365 tenant domain, ie. yourdomain.onmicrosoft.com
config :tenant_domain, :validate => :string, :required => true
# Resource you are requesting access to. This defaults to https://manage.office.com and shouldn't change unless necessary.
config :resource, :validate => :string, :default => 'https://manage.office.com'
# PFX Private key for your Application Certificate you created
config :private_key, :validate => :path
# Private key password if one was used
config :private_key_password, :validate => :string, :default => nil
# Activity subscriptions you want to monitor
# These can be one or many of:
# Audit.AzureActiveDirectory
# Audit.Exchange
# Audit.Sharepoint
# Audit.General
# DLP.All
config :subscriptions, :validate => :array, :default => ["Audit.AzureActiveDirectory", "Audit.Exchange", "Audit.SharePoint", "Audit.General", "DLP.All"]
public
def register
require "adal"
#logger.info("Starting Office 365 Management API input...")
#host = Socket.gethostname
# ADAL supports four logging options: VERBOSE, INFO, WARN and ERROR.
ADAL::Logging.log_level = ADAL::Logger::VERBOSE
end # def register
def get_token
#logger.info("Generating access token...")
if #private_key_password.nil?
pfx = OpenSSL::PKCS12.new(File.read(#private_key))
else
pfx = OpenSSL::PKCS12.new(File.read(#private_key), #private_key_password)
end
authority = ADAL::Authority.new("login.microsoftonline.com", #tenant_domain)
client_cred = ADAL::ClientAssertionCertificate.new(authority, #client_id, pfx)
result = ADAL::AuthenticationContext
.new("login.microsoftonline.com", #tenant_domain)
.acquire_token_for_client(#resource, client_cred)
case result
when ADAL::SuccessResponse
puts 'Successfully authenticated with client credentials. Received access ' "token: #{result.access_token}."
# Create class variable for reuse of Access Token
#access_token = result.access_token
#http_headers = {
'Authorization' => "Bearer #{#access_token}",
'Content-Type' => 'application/x-www-form-urlencoded'
}
when ADAL::FailureResponse
puts 'Failed to authenticate with client credentials. Received error: ' "#{result.error} and error description: #{result.error_description}."
exit 1
end
end #def get_token
def check_subscription
#logger.info("Checking for proper subscriptions...")
#subscriptions.each do |sub|
sub_uri = URI("https://manage.office.com/api/v1.0/#{#tenant_id}/activity/feed/subscriptions/start?contentType=#{sub}")
sub_http = Net::HTTP.new(sub_uri.host, sub_uri.port)
sub_http.use_ssl = true
sub_resp = http.post(sub_uri.request_uri, data = "", #http_headers)
case sub_resp
when Net::HTTPSuccess
puts "Created subscription to #{sub} in tenant #{#tenant_id}..."
when Net::HTTPUnauthorized
puts "Authentication Error Encountered: #{sub_resp.message}"
when Net::HTTPServerError
puts "Server Error Encountered: #{sub_resp.message}"
else
puts "Unknown Error Encountered: #{sub_resp.message}"
end
end
end #def check_subscription
def run(queue)
# we can abort the loop if stop? becomes true
while !stop?
#event = LogStash::Event.new("message" => #message, "host" => #host)
#decorate(event)
#queue << event
raise 'Error getting token' unless get_token().status == 0
# because the sleep interval can be big, when shutdown happens
# we want to be able to abort the sleep
# Stud.stoppable_sleep will frequently evaluate the given block
# and abort the sleep(#interval) if the return value is true
Stud.stoppable_sleep(#interval) { stop? }
end # loop
end # def run
def stop
# nothing to do in this case so it is not necessary to define stop
# examples of common "stop" tasks:
# * close sockets (unblocking blocking reads/accepts)
# * cleanup temporary files
# * terminate spawned threads
end
end # class LogStash::Inputs::Office365ManagementApi
There appears to be a bug when using empty passwords with logstash.
The code responsible for this comes from the calculate_property method in HttpClient , that treats empty strings as nil:
default = nil if default.is_a?(String) && default.empty? # Blanks are as good as nil
uri_value = nil if uri_value.is_a?(String) && uri_value.empty?
One way to fix this is by upgrading to the latest Logstash, if you are currently using an older version.
I try to test a networking application. BUT I might have a knot in my brain.
As far as I know the tests in minitest run parallel. On this assumption I think it is clear that when I allocate a Port in setup() it fails when several tests are run:
RuntimeError: Address already in use - bind(2) for nil port 2000 TCPServer new failed
So what is the best practise to do several tests on a server which listens on a port?
class ServerTest < Minitest::Test
def setup
# -- set env variable
ENV["conf"] = "./tc_gpio.config"
Thread::abort_on_exception = true
#my_thr = Thread.start() do
#server = Server.new
#server.start
assert #server.hi() != nil
end
end
def teardown
Thread.kill(#my_thr) # sends exit() to thr
end
def test_connect_and_disconnect
sleep 1
hostname = 'localhost'
port = 2000
s = TCPSocket.open(hostname, port)
my_s = s.recvmsg()
s.sendmsg(:set.to_s, 0) # Failes since a serialized object is expected
my_s2 = s.recvmsg()
assert_equal( "READY" , my_s[0] )
assert_equal( "eeFIN" , my_s2[0])
end
def test_send_command
# fill command
com = Command.new
com.type = :set
com.device_name = 'pump0'
com.device_address = 'resource0'
com.value = 2
serial_com = YAML::dump(com)
sleep 1
hostname = 'localhost'
port = 2000
s = TCPSocket.open(hostname, port)
my_s = s.recvmsg()
s.sendmsg(serial_com, 0)
my_s2 = s.recvmsg()
assert_equal( "READY" , my_s[0] )
assert_equal( "FIN" , my_s2[0])
end
end
When testing a TCP server in parallel, each instance of the server should be started with a distinct port. That can be done by specifying port number 0 when creating a socket. When port number 0 is given, the socket will be bound to a random unused port:
interface = "0.0.0.0"
port = 0
tcp_server = TCPServer.new(interface, port)
You can find out which port the TCP server socket was bound to:
bound_port = #server_socket.addr[1]
One way to use these facts is to have a server something like this:
class Server
# Create a server instance. If the port is unspecified,
# or 0, then a random ephemeral port is bound to.
def initialize(interface: "127.0.0.1", port: 0)
#server_socket = TCPServer.new(interface, port)
...
end
# Return the port the server socket is bound to.
def bound_port
#server_socket.addr[1]
end
...
end
The test then creates server instances using port 0:
server = Server.new(port: 0)
When making a connection to the server, the test uses the #bound_port accessor to find out what port to connect to:
client = TCPSocket.open("localhost", server.bound_port)
and then carries on normally.
I'm attempting to use ruby to connect to mongodb. But first I need to connect to the server hosting mongo.
There is a bastion host that is able to be connected to publicly. The database host is only able to be connected to through the bastion.
Here is my code:
require 'net/ssh/gateway'
require 'mongo'
include Mongo
ops_host = '<PUBLIC IP/DNS>'
db_host = '<PRIVATE IP>'
port = '27017'
user = 'ubuntu'
key = File.read("<PATHTOKEY>")
gateway = Net::SSH::Gateway.new( ops_host, user, :key_data => key, :keys_only => TRUE)
gateway.ssh(db_host, user, :key_data => key, :keys_only => TRUE) do |ssh|
puts ssh.exec!("hostname")
client = MongoClient.new(db_host, port)
db = client.db('<DBNAME>')
coll = db.collection('profiles')
puts coll.find().count()
end
I'm able to connect to the database host as the puts ssh.exec!("hostname") command returns the correct hostname, but I cannot get mongo to connect as I keep recieving Failed to connect to a master node errors.
I'm using this config to connect to MongoDB with MongoMapper in my Sinatra application:
MongoMapper.connection = Mongo::Connection.new('localhost', 27017)
Now I have a replica set with 2 mongos on separate servers, 10.5.5.5, and 10.5.5.6. How do I setup the connection with both mongos? How do I add authentication to this connection?
I ended up doing this:
MongoMapper.connection = Mongo::MongoReplicaSetClient.new(
['10.5.5.5:27017', '10.5.5.6:27017'],
:read => :primary, :rs_name => 'name', :connect_timeout => 30, :op_timeout => 30
)
MongoMapper.database = "db_name"
MongoMapper.database.authenticate("user", "test123")
Works beautifully.
You should be able to set a different connection per model. But I guess this is not exactly what you trying to do.
class MyModel
include MongoMapper::Document
connection(Mongo::Connection.new('localhost', 27017))
set_database_name "my_database"
# ...
end
Or there is ReplSetConnection with this you can set your replications sets:
MongoMapper.connection = Mongo::ReplicaSetConnection.new(['10.5.5.5', 30000], [' 10.5.5.6', 30000])
And the authentication is simple:
MongoMapper.connection = Mongo::Connection.new('localhost', 27017)
MongoMapper.database = "DBNAME"
MongoMapper.database.authenticate("USERNAME", "PASSWORD")
I am trying to connect via Net::FTPTLS to a Microsoft-based file server (IIS) which is configured to use FTP on port 22 and requires SSL.
I connect via:
require 'net/ftptls'
ftp = Net::FTPTLS.new()
ftp.connect('host.com', port_number)
ftp.login('Username', 'Password')
ftp.puttextfile('somefile.txt', 'where/to/save/somefile.txt')
ftp.close
Problem is, I get the following error:
hostname does not match the server certificate
It seems that I have to disable the openssl peer verification: OpenSSL::SSL::VERIFY_PEER should become OpenSSL::SSL::VERIFY_NONE.
Any ideas on how to monkey-patch the Net::FTPTLS class? Has anyone done this successfully?
Instead using Net::FTPTLS, use Ruby 2.4+ with the following code:
require 'net/ftp'
ftp = Net::FTP.new(nil, ssl: {:verify_mode => OpenSSL::SSL::VERIFY_NONE})
ftp.connect('host.com', port_number)
ftp.login('Username', 'Password')
ftp.puttextfile('somefile.txt', 'where/to/save/somefile.txt')
ftp.close
What I did, rather than monkeypatching ruby itself, was bring a copy of this into /lib of my project.
module Net
class FTPTLS < FTP
def connect(host, port=FTP_PORT)
#hostname = host
super
end
def login(user = "anonymous", params = {:password => nil, :acct => nil, :ignore_cert => false})
store = OpenSSL::X509::Store.new
store.set_default_paths
ctx = OpenSSL::SSL::SSLContext.new('SSLv23')
ctx.cert_store = store
ctx.verify_mode = params[:ignore_cert] ? OpenSSL::SSL::VERIFY_NONE : OpenSSL::SSL::VERIFY_PEER
ctx.key = nil
ctx.cert = nil
voidcmd("AUTH TLS")
#sock = OpenSSL::SSL::SSLSocket.new(#sock, ctx)
#sock.connect
#sock.post_connection_check(#hostname) unless params[:ignore_cert]
super(user, params[:password], params[:acct])
voidcmd("PBSZ 0")
end
end
end
I also cleaned up the param passing a bit. You would use this like so:
require 'ftptls' # Use my local version, not net/ftptls
#ftp_connection = Net::FTPTLS.new()
#ftp_connection.passive = true
#ftp_connection.connect(host, 21)
#ftp_connection.login('user', :password => 'pass', :ignore_cert => true)
HTH
This works fine for me. #ROR
ftp = Net::FTP.new("ftps.host.com", ftp_options)
open("where/is/your/file/somefile.txt") do |file_data|
ftp.putbinaryfile(file_data, 'where/to/save/somefile.txt')
end
ftp.puttextfile('somefile.txt', 'where/to/save/somefile.txt')
def ftp_options
{
port: FTP_PORT,
username: 'ftp_user',
password: 'password',
passive: true,
ssl: { verify_mode: 0 }
}
end
Remember that you have to provide ftps.hostname.com.