Apache flume Regex Extractor Interceptor - hadoop

I am trying to append the header using Regex Extractor Interceptor but it is not working.
I am not sure where I am going wrong. Below is my config file content :
agent.channels = CH
agent.channels.CH.type = memory
agent.channels.CH.capacity = 1000
agent.sources = SRC
agent.sources.SRC.type = avro
agent.sources.SRC.bind = localhost.localdomain
agent.sources.SRC.port = 2222
agent.sources.SRC.channels = CH
agent.sources.SRC.interceptors = i1
agent.sources.SRC.interceptors.i1.type = regex_extractor
agent.sources.SRC.interceptors.i1.regex = ^.*(aa).*$
agent.sources.SRC.interceptors.i1.serializers = t1
agent.sources.SRC.interceptors.i1.serializers.t1.name = A:
agent.sinks = SNK
agent.sinks.SNK.type = hdfs
agent.sinks.SNK.hdfs.fileType = DataStream
agent.sinks.SNK.channel = CH
agent.sinks.SNK.hdfs.path = hdfs://localhost.localdomain:8020/user/cloudera/test/fl1/
When i tried Regex Extractor Interceptor with multiplexing channel selector,the events containing "aa" were routed to "/user/cloudera/test/fl1/" and events containing "bb" were routed to "/user/cloudera/test/fl2/" but headers were not appended in this case too. Below is my config file content :
agent.channels = CH1 CH2
agent.channels.CH1.type = CHory
agent.channels.CH1.capacity = 100
agent.channels.CH2.type = CHory
agent.channels.CH2.capacity = 100
agent.sources = SRC
agent.sources.SRC.channels = ch1
agent.sources.SRC.type = avro
agent.sources.SRC.bind = localhost.localdomain
agent.sources.SRC.port = 2222
agent.sources.SRC.channels = CH1 CH2
agent.sources.SRC.interceptors = i1
agent.sources.SRC.interceptors.i1.type = regex_extractor
agent.sources.SRC.interceptors.i1.regex = ^.*(aa|bb).*$
agent.sources.SRC.interceptors.i1.serializers = t1
agent.sources.SRC.interceptors.i1.serializers.t1.name = type1
agent.sources.SRC.selector.type = multiplexing
agent.sources.SRC.selector.header = type1
agent.sources.SRC.selector.mapping.aa = CH1
agent.sources.SRC.selector.default = CH2
agent.sinks = SNK1 SNK2
agent.sinks.SNK1.type = hdfs
agent.sinks.SNK1.hdfs.fileType = DataStream
agent.sinks.SNK1.channel = CH1
agent.sinks.SNK1.hdfs.path = hdfs://localhost.localdomain:8020/user/cloudera/test/fl1/
agent.sinks.SNK2.type = hdfs
agent.sinks.SNK2.hdfs.fileType = DataStream
agent.sinks.SNK2.channel = CH2
agent.sinks.SNK2.hdfs.path = hdfs://localhost.localdomain:8020/user/cloudera/test/fl2/
Any help is appreciated.
With Regards

Related

Error with the message "Cannot cast (inet::physicallayer::Ieee80211DimensionalTransmission*) to type 'const inet::physicallayer::IScalarSignal *'"

I use Omnet++ and Inet 4.4, I want to simulate a scenario to investigate the effect of interference, which is as follows:
In a network consisting of two pairs of nodes including a node called source and a node called destination, in which node source sends packets to node destination and is tuned in channel 2 of Ieee802.11b/g, let's examine the second pair consisting of node Node1 and node Node2, which node Node1 sends packets to node Node 4 and is tuned in channel 4.
But recently a problem appear very often and the simulations are stopped, the error is this:
check_and_cast(): Cannot cast (inet::physicallayer::Ieee80211DimensionalTransmission*) to type 'const inet::physicallayer::IScalarSignal *' -- in module (inet::physicallayer::Ieee80211Radio) AnalogModelShowcaseDistanceNetworkRM.source.wlan[1].radio (id=200), at t=0.001s, event #24
My omnetpp.ini is as:
[Config Distance]
network = AnalogModelShowcaseDistanceNetworkRM
sim-time-limit = 5s
# Maryam **.radio.packetErrorRate.result-recording-modes = +vector
# Maryam **.radio.bitErrorRate.result-recording-modes = +vector
# application parameters
*.source.numApps = 1
*.source.app[0].typename = "UdpBasicApp"
*.source.app[*].destAddresses = "destination"
*.source.app[*].destPort = 1000
*.source.app[*].messageLength = 1000byte
*.source.app[*].sendInterval = 1ms
*.destination.numApps = 1
*.destination.app[0].typename = "UdpSink"
*.destination.app[*].localPort = 1000
*.Node1.numApps = 1
*.Node1.app[0].typename = "UdpBasicApp"
*.Node1.app[*].destAddresses = "Node2"
*.Node1.app[*].destPort = 1001
*.Node1.app[*].messageLength = 1000byte
*.Node1.app[*].sendInterval = 1ms
*.Node2.numApps = 1
*.Node2.app[0].typename = "UdpSink"
*.Node2.app[*].localPort = 1001
*.source.numWlanInterfaces = 2
*.destination.numWlanInterfaces = 2
*.Node1.numWlanInterfaces = 2
*.Node2.numWlanInterfaces = 2
*.source.wlan[*].radio.typename = "Ieee80211DimensionalRadio"
*.destination.wlan[*].radio.typename = "Ieee80211DimensionalRadio"
*.Node*.wlan[*].radio.typename = "Ieee80211DimensionalRadio"
*.source.wlan[*].radio.centerFrequency = 2.412GHz
*.source.wlan[*].radio.bandwidth = 2MHz
*.source.wlan[*].radio.transmitter.power = 2mW
*.source.wlan[*].radio.transmitter.bitrate = 2Mbps
*.source.wlan[*].radio.transmitter.preambleDuration = 0s
*.source.wlan[*].radio.transmitter.headerLength = 96b
*.source.wlan[*].radio.transmitter.modulation = "BPSK"
*.source.wlan[*].radio.receiver.sensitivity = -85dBm
*.source.wlan[*].radio.receiver.energyDetection = -85dBm
*.source.wlan[*].radio.receiver.snirThreshold = 4dB
*.destination.wlan[*].radio.centerFrequency = 2.412GHz
*.destination.wlan[*].radio.bandwidth = 2MHz
*.destination.wlan[*].radio.transmitter.power = 2mW
*.destination.wlan[*].radio.transmitter.bitrate = 2Mbps
*.destination.wlan[*].radio.transmitter.preambleDuration = 0s
*.destination.wlan[*].radio.transmitter.headerLength = 96b
*.destination.wlan[*].radio.transmitter.modulation = "BPSK"
*.destination.wlan[*].radio.receiver.sensitivity = -85dBm
*.destination.wlan[*].radio.receiver.energyDetection = -85dBm
*.destination.wlan[*].radio.receiver.snirThreshold = 4dB
*.Node*.wlan[*].radio.centerFrequency = 2.412GHz
*.Node*.wlan[*].radio.bandwidth = 2MHz
*.Node*.wlan[*].radio.transmitter.power = 2mW
*.Node*.wlan[*].radio.transmitter.bitrate = 2Mbps
*.Node*.wlan[*].radio.transmitter.preambleDuration = 0s
*.Node*.wlan[*].radio.transmitter.headerLength = 96b
*.Node*.wlan[*].radio.transmitter.modulation = "BPSK"
*.Node*.wlan[*].radio.receiver.sensitivity = -85dBm
*.Node*.wlan[*].radio.receiver.energyDetection = -85dBm
*.Node*.wlan[*].radio.receiver.snirThreshold = 4dB
*.source.wlan[0].radio.channelNumber = 2
*.destination.wlan[0].radio.channelNumber = 2
*.Node1.wlan[0].radio.channelNumber = 4
*.Node2.wlan[0].radio.channelNumber = 4
# mobility parameters
*.destination.mobility.typename = "LinearMobility"
*.destination.mobility.initialMovementHeading = 0deg
*.destination.mobility.speed = 200mps
*.destination.mobility.constraintAreaMinX = 500m
*.destination.mobility.constraintAreaMaxX = 1200m
# wlan
*.source.**.transmitter.power = 12mW
*.source.**.displayCommunicationRange = true
**.backgroundNoise.power = -105dBm
**.wlan*.mac.*.rateSelection.dataFrameBitrate = 54Mbps
**.wlan*.mac.dcf.channelAccess.pendingQueue.packetCapacity = 14
# visualizer parameters
*.visualizer.*.numStatisticVisualizers = 2
*.visualizer.*.statisticVisualizer[0].signalName = "packetSentToUpper"
*.visualizer.*.statisticVisualizer[0].statisticExpression = "packetErrorRate"
*.visualizer.*.statisticVisualizer[0].sourceFilter = "*.destination.wlan[*].radio"
*.visualizer.*.statisticVisualizer[0].format = "packetErrorRate(Maryam): %v"
*.visualizer.*.statisticVisualizer[1].signalName = "packetSentToUpper"
*.visualizer.*.statisticVisualizer[1].statisticExpression = "minimumSnir"
*.visualizer.*.statisticVisualizer[1].sourceFilter = "*.destination.wlan[*].radio"
*.visualizer.*.statisticVisualizer[1].format = "SNIR(Maryam): %v"
*.visualizer.*.statisticVisualizer[1].placementHint = "topLeft"
*.visualizer.*.dataLinkVisualizer[0].displayLinks = true
*.visualizer.*.packetDropVisualizer[0].displayPacketDrops = true
*.visualizer.*.packetDropVisualizer[0].nodeFilter = "destination"
*.visualizer.*.packetDropVisualizer[0].labelFormat = "(Maryam) %r"
*.visualizer.*.infoVisualizer[0].displayInfos = true
*.visualizer.*.infoVisualizer[0].modules = "*.destination.app[0]"
How do I solve this?
You need to set the radioMedium type to dimensional as well
radioMedium: Ieee80211DimensionalRadioMedium {
parameters:
#display("p=62.247997,287.14398");
}
In the NED file.

Flume agent is not transferring the data into source to target folder using file_roll

hi am using below conf file for agent.
when i run the agent i am not getting anything in sink folder. could anyone advice on this?
# Define a memory channel called ch1 on agent1
agent1.channels.ch1.type = memory
agent1.sources.spool.type = spooldir
agent1.sources.spool.channels = ch1
agent1.sources.spool.spoolDir = /logs/source
agent1.sources.spool.fileHeader = true
agent1.sinks.fr1.type = file_roll
agent1.sinks.fr1.channel = ch1
agent1.sinks.fr1.sink.directory = /logs/sink
agent1.channels = ch1
agent1.sources = spool
agent1.sinks = fr1

Postfix server stopped sending external email

We have a postfix server running on MAC. Sometime yesterday our server stopped sending external email. We can send and receive internally, and we can receive from external. When we sent emails to other domains, though, they just disappear...
We've been working on this all day with no luck... Anyone have any thoughts?
Thanks!
--Charles
Here is our postconf -n
biff = no
command_directory = /usr/sbin
config_directory = /Library/Server/Mail/Config/postfix
content_filter =
daemon_directory = /usr/libexec/postfix
data_directory = /Library/Server/Mail/Data/mta
debug_peer_level = 2
debugger_command = PATH=/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin xxgdb $daemon_directory/$process_name $process_id & sleep 5
dovecot_destination_recipient_limit = 1
enable_server_options = yes
header_checks =
html_directory = /usr/share/doc/postfix/html
imap_submit_cred_file = /Library/Server/Mail/Config/postfix/submit.cred
inet_interfaces = all
inet_protocols = all
local_recipient_maps =
mail_owner = _postfix
mailbox_size_limit = 0
mailbox_transport = dovecot
mailq_path = /usr/bin/mailq
manpage_directory = /usr/share/man
message_size_limit = 20971520
mydestination =
mydomain = ourdomain.com
mydomain_fallback = localhost
myhostname = relay.ourdomain.com
mynetworks = 127.0.0.0/8 192.168.0.0/16 [::1]/128 [2001:240:587::]/64
myorigin = $mydomain
newaliases_path = /usr/bin/newaliases
queue_directory = /Library/Server/Mail/Data/spool
readme_directory = /usr/share/doc/postfix
recipient_canonical_maps = hash:/Library/Server/Mail/Config/postfix/system_user_maps
recipient_delimiter = +
relayhost =
sample_directory = /usr/share/doc/postfix/examples
sendmail_path = /usr/sbin/sendmail
setgid_group = _postdrop
smtp_sasl_auth_enable = no
smtp_sasl_password_maps =
smtpd_client_restrictions = permit_mynetworks permit_sasl_authenticated permit
smtpd_enforce_tls = no
smtpd_helo_required = no
smtpd_helo_restrictions =
smtpd_pw_server_security_options = cram-md5,digest-md5,login,plain
smtpd_recipient_limit = 1000
smtpd_recipient_restrictions = permit_sasl_authenticated permit_mynetworks reject_unauth_destination permit
smtpd_sasl_auth_enable = yes
smtpd_tls_CAfile = /etc/certificates/relay.ourdomain.com.028E0388CA5957E50A13248BAD1711F41C84F596.chain.pem
smtpd_tls_cert_file = /etc/certificates/relay.ourdomain.com.028E0388CA5957E50A13248BAD1711F41C84F596.cert.pem
smtpd_tls_ciphers = medium
smtpd_tls_exclude_ciphers = SSLv2, aNULL, ADH, eNULL
smtpd_tls_key_file = /etc/certificates/relay.ourdomain.com.028E0388CA5957E50A13248BAD1711F41C84F596.key.pem
smtpd_use_pw_server = yes
smtpd_use_tls = yes
tls_random_source = dev:/dev/urandom
unknown_local_recipient_reject_code = 550
use_sacl_cache = yes
virtual_alias_domains = $virtual_alias_maps hash:/Library/Server/Mail/Config/postfix/virtual_domains
virtual_alias_maps = $virtual_maps hash:/Library/Server/Mail/Config/postfix/virtual_users
postconf: warning: /Library/Server/Mail/Config/postfix//main.cf: unused parameter: default_destination_recipient_limit:=1000
relay:~ localadmin$
Looks like the issue was being caused by a typo...
I kept getting the error:
unused parameter: default_destination_recipient_limit
Upon looking at the master.cf file, I found that it had:
default_destination_recipient_limit: = 1000
Note the colon in that line. Fixed an the issue has gone away.

connecting new kannel box to opensmppbox

i have been trying to connect a new box (17.x.x.x)which i just installed kannel on it. i already have a running kannel on another box (19.x.x.x) which can send sms and recieve. i installed opensmppbox on the 19.x.x.x . when i start opensmppbox it shows waiting for incomming connections. then when i start kannel on the 17.x.x.x it connects to the smsbox but gives an error connecting to the opensmppbox. i have spend too ling a time on this.
how can i solve this problem. here is my kannel.conf for the 17.x.x.x
# Core
group = core
admin-port=9000
wapbox-port=9001
smsbox-port=9002
admin-password=Wass
status-password=Wing
dlr-storage=mysql
log-file="/var/log/kannel/bearerbox.log"
log-level=0
access-log="/var/log/kannel/access.log"
access-log-clean=true
access-log-format="%t %l [SMSC:%i] [SVC:%n] [ACT:%A] [BINF:%B] [FID:%F][from:%p] [to:%P] [flags:%m:%c:%M:%C:%d] [msg:%L:%b] [udh:%U:%u]"
box-allow-ip="*.*.*.*"
unified-prefix="0234,+234,00234"
#######################################
# WAPBOX box
group = wapbox
bearerbox-host = 127.0.0.1
log-file = "/var/log/kannel/wapbox.log"
#######################################
#SMSC group - For 38276
group = smsc
smsc = smpp
smsc-id = smppbox
host = 192.101.17.30
port = 2775
transceiver-mode = true
smsc-username = "hot"
smsc-password = "kkkk"
interface-version = 34
system-type =
my-number=38260
msg-id-type = 0x03
alt-charset = GSM
#denied-smsc-id =
#source-addr-ton = 0
#source-addr-npi = 0
#dest-addr-ton = 0
#dest-addr-npi = 0
allowed-prefix-regex = ^\+?(0|234)((8[01]2)|(808)|(7[01]8))
#######################################
# SMSBOX
group = smsbox
bearerbox-host = localhost
smsbox-id = vsmsc
sendsms-port = 9009
sendsms-url = "/cgi-bin/sendsms"
global-sender = cingular
reply-couldnotfetch = "Could not temporarily get content...please try again"
sendsms-chars = "0123456789 +-"
http-request-retry = 1
reply-emptymessage = ""
http-queue-delay = 30
max-pending-requests = 1000
log-file = "/var/log/kannel/smsbox.log"
log-level = 0
access-log = "/var/log/kannel/smsbox-access.log"
#######################################
# DATABASE
group = mysql-connection
id=mydlr
host=localhost
username=root
password=Wind
database=inm
max-connections=1
group = dlr-db
id = mydlr
table = dlr
field-smsc = smsc
field-timestamp = ts
field-destination = destination
field-source = source
field-service = service
field-url = url
field-mask = mask
field-status = status
field-boxc-id = boxc
below is the opensmppbox.conf
#
group = core
dlr-storage = internal
group = opensmppbox
opensmppbox-id = smppbox
opensmppbox-port = 2775
bearerbox-host = localhost
bearerbox-port = 9000
log-level = 0
log-file = /var/log/kannel/opensmppbox/opensmppbox.log
our-system-id =
use-systemid-as-smsboxid = true
route-to-smsc =Singtel
smpp-logins = "/etc/kannel/smpplogins.txt"
#route-to-smsc = emitsms
group = mysql-connection
id = mydlr
host = localhost
username = root
password = Wing
database = inm
max-connections = 1
group = dlr-db
id = mydlr
table = dlr
field-smsc = smsc
field-timestamp = ts
field-destination = destination
field-source = source
field-service = service
field-url = url
field-mask = mask
field-status = status
field-boxc-id = boxcid
below is the smpplogins.txt
system-id-01 password-01 system-type-01
client-02 password-02 vma 127.0.0.1
client-03 password-03 vma *.*.*.*
hot cingul12 vma 17.x.x.x
in opensmppbox change bearerbox-port to smsbox-port of core group in kannel conf ie change 9000 to 9002 in opensmppbox conf

Need help trying to get gateway running ( mx.rpc.events::FaultEvent #0 )

I followed the setup for http://www.zamfbrowser.org/documentation.html
When I access http://www.zamfbrowser.org/zamf_demo/gateway.php
When I access the http://localhost/gateway.php I get this error message which
There was an error loading the server's info. Error: (mx.rpc.events::FaultEvent)#0
bubbles = false
cancelable = true
currentTarget = (mx.rpc.remoting.mxml::RemoteObject)#1
channelSet = (mx.messaging::ChannelSet)#2
authenticated = false
channelIds = (Array)#3
[0] (null)
channels = (Array)#4
[0] (mx.messaging.channels::AMFChannel)#5
authenticated = false
channelSets = (Array)#6
connected = false
connectTimeout = -1
enableSmallMessages = true
endpoint = "http://localhot/gateway.php"
failoverURIs = (Array)#7
id = (null)
mpiEnabled = false
netConnection = (flash.net::NetConnection)#8
client = (mx.messaging.channels::AMFChannel)#5
connected = false
maxPeerConnections = 8
objectEncoding = 3
proxyType = "none"
uri = (null)
piggybackingEnabled = false
polling = false
pollingEnabled = true
pollingInterval = 3000
protocol = "http"
reconnecting = false
recordMessageSizes = false
recordMessageTimes = false
requestTimeout = -1
uri = "http://localhot/gateway.php"
url = "http://localhot/gateway.php"
useSmallMessages = false
clustered = false
connected = false
currentChannel = (mx.messaging.channels::AMFChannel)#5
heartbeatInterval = 0
initialDestinationId = (null)
messageAgents = (Array)#9
[0] (mx.rpc::AsyncRequest)#10
authenticated = false
autoConnect = true
channelSet = (mx.messaging::ChannelSet)#2
clientId = (null)
connected = false
defaultHeaders = (null)
destination = "AMF"
id = "1D3472F3-161B-0099-5187-B4A799F5FC4C"
priority = -1
reconnectAttempts = 0
reconnectInterval = 0
requestTimeout = -1
subtopic = ""
concurrency = "multiple"
convertParametersHandler = (null)
convertResultHandler = (null)
destination = "AMF"
endpoint = "http://localhot/gateway.php"
getServices = (mx.rpc.remoting.mxml::Operation)#11
argumentNames = (Array)#12
arguments = (Object)#13
concurrency = "multiple"
lastResult = (null)
makeObjectsBindable = true
name = "getServices"
operationManager = (null)
properties = (null)
resultElementType = (null)
resultType = (null)
service = (mx.rpc.remoting.mxml::RemoteObject)#1
showBusyCursor = true
makeObjectsBindable = true
managers = (null)
operations = (Object)#14
getServices = (mx.rpc.remoting.mxml::Operation)#11
requestTimeout = -1
showBusyCursor = true
source = "ZendAmfServiceBrowser"
eventPhase = 2
fault = (mx.rpc::Fault)#15
content = (Object)#16
errorID = 0
faultCode = "Client.Error.DeliveryInDoubt"
faultDetail = "Channel disconnected before an acknowledgement was received"
faultString = "Channel disconnected"
message = "faultCode:Client.Error.DeliveryInDoubt faultString:'Channel disconnected' faultDetail:'Channel disconnected before an acknowledgement was received'"
name = "Error"
rootCause = (mx.messaging.events::ChannelEvent)#17
bubbles = false
cancelable = false
channel = (mx.messaging.channels::AMFChannel)#5
channelId = (null)
connected = false
currentTarget = (mx.messaging.channels::AMFChannel)#5
eventPhase = 2
reconnecting = true
rejected = false
target = (mx.messaging.channels::AMFChannel)#5
type = "channelDisconnect"
headers = (null)
message = (mx.messaging.messages::ErrorMessage)#18
body = (Object)#16
clientId = (null)
correlationId = "3407DB24-F4AD-C634-0E23-B4A799F60FC9"
destination = ""
extendedData = (null)
faultCode = "Client.Error.DeliveryInDoubt"
faultDetail = "Channel disconnected before an acknowledgement was received"
faultString = "Channel disconnected"
headers = (Object)#19
messageId = "FB5C929F-69E4-8573-F628-B4A79AFE514B"
rootCause = (mx.messaging.events::ChannelEvent)#17
timestamp = 0
timeToLive = 0
messageId = "FB5C929F-69E4-8573-F628-B4A79AFE514B"
statusCode = 0
target = (mx.rpc.remoting.mxml::RemoteObject)#1
token = (mx.rpc::AsyncToken)#20
message = (mx.messaging.messages::RemotingMessage)#21
body = (Array)#22
clientId = (null)
destination = "AMF"
headers = (Object)#23
DSEndpoint = (null)
DSId = "nil"
messageId = "3407DB24-F4AD-C634-0E23-B4A799F60FC9"
operation = "getServices"
source = "ZendAmfServiceBrowser"
timestamp = 0
timeToLive = 0
responders = (null)
result = (null)
type = "fault"
A lot of information and I don't really know what any of it means
When accessing http://localhost/gateway.php I get the message Zend Amf Endpoint
This seemed to be a windows 7 issue, the same code worked on my linux server, although I didn't solve it I'll just move on.
thanks M$

Resources