I'm new to pig and am trying to perform some basic analysis on a file containing events that look like the below:
1345477765 2012-08-20 08:49:24 servername 12.34.56.78 192.168.1.4 joebloggs ManageSystem Here's your message
I attempt to load the file as below:
logs = LOAD '/path/to/file' using PigStorage AS (loggedtime:long, serverdate:chararray, servertime:chararray, servername:chararray, externalip:chararray, internalip:chararray, username:chararray, systemtype:chararray, message:chararray);
When I illustrate logs everything looks ok:
Illustrate logs
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| logs | loggedtime:long | serverdate:chararray | servertime:chararray | servername:chararray | externalip:chararray | internalip:chararray | username:chararray | systemtype:chararray | message:chararray |
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| | 1345477765 | 2012-08-20 | 08:49:24 | servername | 12.34.56.78 | 192.168.1.4 | joebloggs | ManageSystem | Here's your message |
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Also, when a describe them everything is as I would expect:
logs: {loggedtime: long,serverdate: chararray,servertime: chararray,servername: chararray,externalip: chararray,internalip: chararray,username: chararray,systemtype: chararray,message: chararray}
However, when I dump logs, the loggedtime is not included.
dump logs;
(,2012-08-20,08:49:24,servername,12.34.56.78,192.168.1.4,joebloggs,ManageSystem,Here's your message)
Presumably as a result of this, my filter returns no events:
specificlog = FILTER logs BY loggedtime == 1345477765;
Hopefully I'm missing something easy here.
I eventually figured this out myself. To parse to a long I had to put an "L" at the end of the number.
e.g. by changing my source data to the below I was able to get this working.
1345477765L 2012-08-20 08:49:24 servername 12.34.56.78 192.168.1.4 joebloggs ManageSystem Here's your message
Hopefully this will help someone with the same problem.
Related
I am still new in learning Cassandra, and I am doing some measurements, regarding memory and processor resources for each of my query. Does Cassandra has her own way to show performance of query or should I use some third party tool?
You can use the tracing switched on to see the internal steps.
TRACING ON
For the query below
INSERT INTO cycling.cyclist_name (
id,
lastname,
firstname
)
VALUES (
e7ae5cf3-d358-4d99-b900-85902fda9bb0,
'FRAME',
'Alex'
);
The below is the trace log
Tracing session: 9b378c70-b114-11e6-89b5-b7fad52e1885
activity | timestamp | source | source_elapsed | client
-----------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------+-----------+----------------+-----------
Execute CQL3 query | 2016-11-22 16:34:34.300000 | 127.0.0.1 | 0 | 127.0.0.1
Parsing INSERT INTO cycling.cyclist_name (id, lastname, firstname) VALUES (e7ae5cf3-d358-4d99-b900-85902fda9bb0, 'FRAME','Alex'); [Native-Transport-Requests-1] | 2016-11-22 16:34:34.305000 | 127.0.0.1 | 5935 | 127.0.0.1
Preparing statement [Native-Transport-Requests-1] | 2016-11-22 16:34:34.308000 | 127.0.0.1 | 9199 | 127.0.0.1
Determining replicas for mutation [Native-Transport-Requests-1] | 2016-11-22 16:34:34.330000 | 127.0.0.1 | 30530 | 127.0.0.1
Appending to commitlog [MutationStage-3] | 2016-11-22 16:34:34.330000 | 127.0.0.1 | 30979 | 127.0.0.1
Adding to cyclist_name memtable [MutationStage-3] | 2016-11-22 16:34:34.330000 | 127.0.0.1 | 31510 | 127.0.0.1
Request complete | 2016-11-22 16:34:34.333633 | 127.0.0.1 | 33633 | 127.0.0.1
Reference link: https://docs.datastax.com/en/cql-oss/3.3/cql/cql_reference/cqlshTracing.html
I have a list of ip addresses in a text file that I wish to use in a script.
Here is the code outputting the ip addresses in the text file
openstack server list | grep agent | awk '{print \$9}' >> ${STACK}_list.txt
I would like to retrieve the ip addresses and use in a loop by ssh'ing in to them but not sure how to do that
Please refer this post.
script to read a file with IP addresses and login
Might be helpful for you.
Thanks.
Subhadeep
You can use a regex to filter all ip addresses from the server list-output:
openstack server list | grep -o '[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*'
You could pipe this output into a file if you need or to use this within a bash-script you could make something like this, without writing it into a file:
#!/bin/bash
#
ADDRESSES=$(openstack server list | grep -o '[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*')
for ADDRESS in $ADDRESSES
do
echo "ip: $ADDRESS"
done
It reads all ip-addresses from ther server-list output and iterate within the for-loop over this output and prints each ip separate on the terminal. Instead of the echo you could insert your ssh-command.
Example-server on my deployment:
root#m1r1:~# openstack server list
+--------------------------------------+-----------------------+--------+--------------------------+----------------+--------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+-----------------------+--------+--------------------------+----------------+--------+
| 46d04a77-4d33-4bb3-8214-b1444eed33a3 | server1 | ACTIVE | l2-network=192.168.4.131 | cirros | XS |
| e9489aca-00c3-4fc9-afc5-515c08b17406 | server2 | ACTIVE | l2-network=192.168.4.61 | | XS |
| ea8cec6a-a8d5-4bbb-970e-aaf65d7374b2 | server3 | ACTIVE | l2-network=192.168.4.163 | cirros | S |
| 7d934ec4-1d53-467b-9220-d67b4b68a832 | server4 | ACTIVE | l2-network=192.168.4.184 | | XS |
| 74d3036e-372a-4566-8ba2-10a0760c5562 | server5 | ACTIVE | l2-network=192.168.4.232 | cirros | XS |
| e08e1637-f4df-478d-a478-6578d038cb22 | server6 | ACTIVE | l2-network=192.168.4.190 | | XS |
| 8307a481-679e-4df0-a64e-3a497b13ac81 | server7 | ACTIVE | l2-network=192.168.4.202 | | XS |
| 38d10b12-daa5-483e-b9a5-9a16ba14d841 | server8 | ACTIVE | l2-network=192.168.4.250 | cirros | XS |
+--------------------------------------+-----------------------+--------+--------------------------+----------------+--------+
Output of this example:
ip: 192.168.4.131
ip: 192.168.4.61
ip: 192.168.4.163
ip: 192.168.4.184
ip: 192.168.4.232
ip: 192.168.4.190
ip: 192.168.4.202
ip: 192.168.4.250
#!/bin/sh
openstack server list | grep -o '[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*' > stack
while $(wc -l stack | cut -d' ' -f1) -gt 0 ]
do
ipnumber=$(sed -n '1p' stack)
echo "${ipnumber}"
sed -i '1d' stack
done
The echo command there is just a placeholder. You can replace it with ssh, or whatever else you want to do, with the IP number in the variable.
I am trying to save a file on a library inside a Iseries database using the GxFtpPut on genexus 10 V3 with .net but when sending the file genexus tries to send it to a windows directory instead of sending it to the library which works using the ftp command on the cmd
I've already tried to changing the route is using to no avail and trying to find another way of sending the file through genexus.
for example when using the cmd I just put this :
put C:\FILES\Filename.txt Library/Filename
And it works on sending the file inside the library,
but when doing this on genexus:
Call("GxFtpPut", &FileDirectory , 'Library/'+&FileName,'B' )
Does not work and tries to find a directory with that name inside the windows files of the server
I just want to be able to send it to the server library without issue.
IBM i has two distinct name formats depending on the file system you are trying to use. NAMEFMT 0 is the library/filename format, and is likely unknown to PC FTP clients. NAMEFMT 1 is the typical hierarchical directory path used by non-IBM i computers, and also works with IBM i if you want to put a file anywhere in the IFS (Integrated File System).
Fun fact, the native library file system is also accessible from the IFS. But to address it you need to use a format that might be a little unfamiliar. /QSYS.lib/library.lib/filename.file/membername.mbr You may be able to drop the member name.
To change name format, you can issue the SITE sub-command on your remote host like this:
QUOTE SITE NAMEFMT 0 -- This sets name format 0 (library/filename)
QUITE SITE NAMEFMT 1 -- This sets name format 1 (directory path)
I did some testing with a plain Windows FTP client. The test file on the PC was a text file created in Notepad++. Turns out that we start out in NAMEFMT 0 unless it is changed. It looks like genexus only supports a limited set of commands. So here is the limited FTP script that works:
ascii
put test.txt mylib/testpf
I can now pull up testpf on the greenscreen utilities and read it. I can also read testpf in my GUI SQL client. The ASCII text has been converted properly to EBCDIC.
|TESTPF |
|--------------------------------------------------------------------------------|
| |
|// ------------------------------------ |
|// Sweep |
|// |
|// Performs the sweep logic |
|// ------------------------------------ |
|dcl-proc Sweep; |
| |
| |
| exec sql |
| update atty a |
| set ymglsb = (select ymglsb from glaty |
| where atty = a.atty) |
| where atty in (select atty from glaty where atty = a.atty); |
|// where ymglsb in (select ymglsb from glaty where atty = a.atty); |
| if %subst(sqlstate: 1: 2) < '00' or |
| %subst(sqlstate: 1: 2) > '02'; |
| exec sql get diagnostics condition 1 |
| :message = message_text; |
| SendSqlMsg('02: ' + message); |
| endif; |
| |
| exec sql |
| update atty a |
| set ymglsb = '000' |
| where not exists (select * from glaty where atty = a.atty); |
| if %subst(sqlstate: 1: 2) < '00' or |
| %subst(sqlstate: 1: 2) > '02'; |
| exec sql get diagnostics condition 1 |
| :message = message_text; |
| SendSqlMsg('03: ' + message); |
| endif; |
| |
|end-proc; |
However, if I try to transfer in binary mode, the resulting data in the file looks like this:
|TESTPF |
|--------------------------------------------------------------------------------|
|ëÏÁÁø&ÁÊÃ?Ê_ËÈÇÁËÏÁÁø% |
|?ÅÑÄÀÄ%øÊ?ÄëÏÁÁøÁÌÁÄËÉ% |
|ÍøÀ/ÈÁ/ÈÈ`/ËÁÈ`_Å%ËÂËÁ%ÁÄÈ`_Å%ËÂÃÊ?_Å%/È` |
|ÏÇÁÊÁ/ÈÈ`//ÈÈ`ÏÇÁÊÁ/ÈÈ`Ñ>ËÁ%ÁÄÈ/ÈÈ`ÃÊ?_Å%/È`ÏÇÁÊÁ/ÈÈ |
|`//ÈÈ`ÏÇÁÊÁ`_Å%ËÂÑ>ËÁ%ÁÄÈ`_Å%ËÂÃÊ?_Å%/È`ÏÇÁÊÁ/ÈÈ`//ÈÈ |
|`ÑöËÍÂËÈËÉ%ËÈ/ÈÁ?ʶËÍÂËÈËÉ%ËÈ/ÈÁ |
|ÁÌÁÄËÉ%ÅÁÈÀÑ/Å>?ËÈÑÄËÄ?>ÀÑÈÑ?>_ÁËË/ÅÁ_ÁËË/ÅÁ¬ÈÁÌÈ |
|ëÁ>ÀëÉ%(ËÅ_ÁËË/ÅÁÁ>ÀÑÃÁÌÁÄËÉ%ÍøÀ/ÈÁ/ÈÈ`/ |
|ËÁÈ`_Å%ËÂÏÇÁÊÁ>?ÈÁÌÑËÈËËÁ%ÁÄÈÃÊ?_Å%/È`ÏÇÁÊÁ/ÈÈ`// |
|ÈÈ`ÑöËÍÂËÈËÉ%ËÈ/ÈÁ?ʶËÍÂËÈËÉ%ËÈ/ÈÁ |
|ÁÌÁÄËÉ%ÅÁÈÀÑ/Å>?ËÈÑÄËÄ?>ÀÑÈÑ?>_ÁËË/ÅÁ_ÁËË/ÅÁ¬ÈÁÌÈ |
|ëÁ>ÀëÉ%(ËÅ_ÁËË/ÅÁÁ>ÀÑÃÁ>ÀøÊ?Ä |
This has not been converted because we have told IBM i FTP server not to convert to EBCDIC because it is binary.
So try ASCII mode, use the library/filename format. The target file does not need to pre-exist.
I am not sure it this is intended to be so, but I am confused by the behavior.
When I have the following Scenario Outline:
Scenario Outline: outline1
Given url
And query parameters <query_params>
When method
Then status is
Examples:
| method | endpoint | query_params | status |
| GET | /endpoint1 | ?a=1&b=1 | 200 |
| GET | /endpoint1 | ?a=1&b=1&c=3 | 200 |
I see the following snippet generated.
func FeatureContext(s *godog.Suite) {
s.Step(^method GET$, methodGET)
s.Step(^query parameters \?a=(\d+)&b=(\d+)$, queryParametersAB)
s.Step(^query parameters \?a=(\d+)&b=(\d+)&c=(\d+)$, queryParametersABC)
}
As you can see 2 lines of "query parameters" produces 2 different functions. Why is godog parsing this text? This is a little different from cucumber gherkin parsing.
One side effect of this is that if I have 100 lines in the data table, I am forced to implement all of them.
Is there a way I can ask godog to not do this parsing?
The solution to the problem is to use double quotes around as given below.
Scenario Outline: outline1
Given url
And query parameters "<query_params>"
When method
Then status is
Examples:
| method | endpoint | query_params | status |
| GET | /endpoint1 | ?a=1&b=1 | 200 |
| GET | /endpoint1 | ?a=1&b=1&c=3 | 200 |
Then the following will be generated:
s.Step(`^query parameters "([^"]*)"$`, queryParameters)
I am using a MongoDB on mLab to store a basic collection of boardgames which I wish to show in my Ruby app. I have completed a tutorial that uses Mongoid to implement this locally, but so far I can't get it working with the mLab instance of the DB.
I add this to my mongoid.yml file
development:
clients:
default:
uri: 'mongodb://user:password#ds141232.mlab.com:41232/boardgame_banter'
The other options automatically generated in the config file, I have left blank (as default).
I want to understand these 2 lines from the Terminal:
MONGODB | ds141232-a.mlab.com:41232 | boardgame_banter.find | STARTED | {"find"=>"boardgames", "filter"=>{}}
MONGODB | ds141232-a.mlab.com:41232 | boardgame_banter.find | SUCCEEDED | 0.037816999999999996s
I get no errors, but also no documents returned and the generated index.html is blank...
Can anyone explain the first of the two lines MONGODB | ... to me, or at least confirm if my assumptions below are correct? Particularly the last part of the chain, is this telling me that the filtered results are empty?
MONGODB | <<hostname>> | <<database.find()>> | <<STATUS>> | {"find"=><<collection>>, "filter"=>{<<no results??>>}}
UPDATE after suggestion from #tfogo in the comments
In my controller:
# GET /boardgames
# GET /boardgames.json
def index
#boardgames = Boardgame.all
#log = Boardgame.all.to_a
puts "LOG: #{#log}"
end
Which produces the following empty Log statement in the console:
Started GET "/boardgames" for 127.0.0.1 at 2018-03-02 11:25:00 +0100
Processing by BoardgamesController#index as HTML
D, [2018-03-02T11:25:00.186878 #12983] DEBUG -- : MONGODB | ds141232-a.mlab.com:41232 | boardgame_banter.find | STARTED | {"find"=>"boardgames", "filter"=>{}}
D, [2018-03-02T11:25:00.223330 #12983] DEBUG -- : MONGODB | ds141232-a.mlab.com:41232 | boardgame_banter.find | SUCCEEDED | 0.035911000000000005s
LOG: [#<Boardgame _id: 5a984b439de90b3769420f2d, name: nil, rating: nil, minplayer: nil, maxplayer: nil, duration: nil, owner: nil>]
Rendering boardgames/index.html.erb within layouts/application
D, [2018-03-02T11:25:00.235908 #12983] DEBUG -- : MONGODB | ds141232-a.mlab.com:41232 | boardgame_banter.find | STARTED | {"find"=>"boardgames", "filter"=>{}}
D, [2018-03-02T11:25:00.274734 #12983] DEBUG -- : MONGODB | ds141232-a.mlab.com:41232 | boardgame_banter.find | SUCCEEDED | 0.038311s
Rendered boardgames/index.html.erb within layouts/application (42.3ms)
Completed 200 OK in 127ms (Views: 76.9ms)