I'm trying to read a log file like that one:
199.72.81.55 - - [01/Jul/1995:00:00:01 -0400] "GET /history/apollo/ HTTP/1.0" 200 6245
unicomp6.unicomp.net - - [01/Jul/1995:00:00:06 -0400] "GET /shuttle/countdown/ HTTP/1.0" 200 3985
199.120.110.21 - - [01/Jul/1995:00:00:09 -0400] "GET /shuttle/missions/sts-73/mission-sts-73.html HTTP/1.0" 200 4085
burger.letters.com - - [01/Jul/1995:00:00:11 -0400] "GET /shuttle/countdown/liftoff.html HTTP/1.0" 304 0
199.120.110.21 - - [01/Jul/1995:00:00:11 -0400] "GET /shuttle/missions/sts-73/sts-73-patch-small.gif HTTP/1.0" 200 4179
I'm sending 1000 lines each time I run this exercise, and I'm using a splitText processor, and in the extractText processor I use this regex:
successCode -> ^[0-9A-Z\-a-z\.]* - - \[[0-9A-Za-z\/\:]* -[0-9]*\] \"[A-Z]* [0-9A-Za-z\/\.\- ]*\" ([0-9]*) [0-9]*
tiemStamp -> ^[0-9A-Z\-a-z\.]* - - \[([0-9A-Za-z\/\:]*) -[0-9]*\] \"[A-Z]* [0-9A-Za-z\/\.\- ]*\" [0-9]* [0-9]*
important -> ^([0-9A-Z\-a-z\.]*) - - \[[0-9A-Za-z\/\:]* -[0-9]*\] \"[A-Z]* [0-9A-Za-z\/\.\- ]*\" [0-9]* [0-9]*
It can be a mistake on it. Surely here is my problem.
Then, I tryed to send different logs to different routes. If successCode == 200 then I tried to put it on route /user//success/%{tiemStamp}/, but all my lines go to the third way: "unmatched"
On the RouteOnContent processor I've tryed:
successCode -> ${successCode:equals("200")}
successCode -> ${successCode:contains(2)}
successCode -> ${successCode:contains("2")}
Has anyone worked with "RouteOnContent" processor?
According to the documentation, the ExtractText Processor "Evaluates one or more Regular Expressions against the content of a FlowFile. The results of those Regular Expressions are assigned to FlowFile Attributes [...]"
So you should not use a RouteOnContent but a RouteOnAttribute processor in the next step.
(If you stop your RouteOnXXX processor in order to keep the messages in the queue, you can see the content of the flowfiles. On the "Attributes" tab of a flowfile, you can see the values of the different attributes. And I confirm that with your regexp, I have successCode=200. )
Basically you can use both RouteOnAttribute or RouteOnText, but each uses different parameters.
If you chose to use ExtractText, the properties you defined are populated for each row (after the original file was split by SplitText processor).
Now, you have two options:
Route based on the attributes that have been extracted (RouteOnAttribute).
Route based on the content (RouteOnContent). In this case, you don't really need to use Extract Text.
Each processor routes the FlowFile differently:
RouteOnAttribute queries the attributes of the FlowFile (a NiFi Expression Language query). For example, let's say I defined the property 'name', routing based on its value can be:
On the other hand, RouteOnContext queries the content of the FlowFile based on a regex expression. For example:
After defining these parameters, you can continue to route based on these dynamic relationships:
My idea is using Rack to load a web application separated in two parts. This is the structure:
my_app
|______api
| |______poros
|
|______fe
| |______build
| |______node_modules
| |______package.json
| |______public
| | |______ index.html
| | ...
| |______README.md
| |______src
| |______index.js
| ...
|______config.ru
The fe part (which stands for front-end) is a React application created with create-react-app. I've built it with
npm run build
and then its static optimized build is under my_app/fe/build directory.
This is my my_app/config.ru:
#\ -w -p 3000
require 'emeraldfw'
use Rack::Reloader, 0
use Rack::ContentLength
run EmeraldFW::Loader.new
And this is my EmeraldFW::Loader class, which is part of a gem installed and running fine.
module EmeraldFW
class Loader
def call(env)
path_info = (env['PATH_INFO'] == '/') ? '/index.html' : env['PATH_INFO']
extension = path_info.split('/').last.split('.').last
file = File.read("fe/build#{path_info}")
[200,{'Content-Type' => content_type[extension]},[file]]
end
def content_type
{
'html' => 'text/html',
'js' => 'text/javascript',
'css' => 'text/css',
'svg' => 'image/svg+xm',
'ico' => 'image/x-icon',
'map' => 'application/octet-stream'
}
end
end
end
As you may see, this is all quite simple. I capture the request with in my EmeraldFW::Loader class and transform its path_info it a bit. If the request is '/' I rewrite it to '/index.html' before doing anything else. In all cases I prepend fe/build to make it load from the static build of the React application.
When I run
rackup config.ru
and load the application at http://localhost:3000 the result is completely fine:
[2017-03-15 21:28:23] INFO WEBrick 1.3.1
[2017-03-15 21:28:23] INFO ruby 2.3.3 (2016-11-21) [x86_64-linux]
[2017-03-15 21:28:23] INFO WEBrick::HTTPServer#start: pid=11728 port=3000
::1 - - [15/Mar/2017:21:28:27 -0300] "GET / HTTP/1.1" 200 386 0.0088
::1 - - [15/Mar/2017:21:28:27 -0300] "GET /static/css/main.9a0fe4f1.css HTTP/1.1" 200 623 0.0105
::1 - - [15/Mar/2017:21:28:27 -0300] "GET /static/js/main.91328589.js HTTP/1.1" 200 153643 0.0086
::1 - - [15/Mar/2017:21:28:28 -0300] "GET /static/media/logo.5d5d9eef.svg HTTP/1.1" 200 2671 0.0036
::1 - - [15/Mar/2017:21:28:28 -0300] "GET /static/js/main.91328589.js.map HTTP/1.1" 200 1705922 0.1079
::1 - - [15/Mar/2017:21:28:28 -0300] "GET /static/css/main.9a0fe4f1.css.map HTTP/1.1" 200 105 0.0021
As you may see, all resources are loading correctly, with the correct mime types. But it happens that my default React logo, which should be spinning in my app frontpage, is not there! As if it wasn't loaded.
The big picture of all this is having this rack middleware loading the React front-end of the app and, at the same time, redirecting correctly the api requests made with fetch from the front-end to the poros (Plain Old Ruby Objects) which are the API part.
The concept is quite simple. But I just can't understand why this specific resource, the svg logo, is not loading.
It was all a matter of wrong mime type, caused by a typo.
It may be clearly seen in my question that I wrote:
def content_type
{
'html' => 'text/html',
'js' => 'text/javascript',
'css' => 'text/css',
'svg' => 'image/svg+xm', # <== Here is the error!!!
'ico' => 'image/x-icon',
'map' => 'application/octet-stream'
}
end
when the correct mime type for SVG files is 'image/svg-xml', with an 'l'...
This was making the browser ignore the file received and so it won't display it correctly.
How would you convert the Grape::Request object to a string in order to store it's contents in the database? I tried
request.to_s and only get #<Grape::Request:0x007f7fd34605c8>
I also tried
request.to_json and get this IOError Exception: not opened for reading
Perhaps the Grape::Endpoint#request method is what you're looking for, although there are others like env that may have more or less of what you need.
I tried it by putting it in the Alongside Sinatra (or other frameworks) example given in the docs:
require 'grape'
require 'sinatra/base'
class API < Grape::API
get :hello do
{req: request}
end
end
class Web < Sinatra::Base
get '/' do
"Hello world."
end
end
run Rack::Cascade.new [API, Web]
and on calling it:
$ [2015-07-22 08:21:17] INFO WEBrick 1.3.1
[2015-07-22 08:21:17] INFO ruby 2.2.2 (2015-04-13) [x86_64-darwin13]
[2015-07-22 08:21:17] INFO WEBrick::HTTPServer#start: pid=30420 port=9292
curl http://localhost:9292/hello
127.0.0.1 - - [22/Jul/2015:08:21:19 +0100] "GET /hello HTTP/1.1" 200 3665 0.0389
{:req=>#<Grape::Request:0x007f9971425d08 #env={"GATEWAY_INTERFACE"=>"CGI/1.1", "PATH_INFO"=>"/hello", "QUERY_STRING"=>"", "REMOTE_ADDR"=>"127.0.0.1", "REMOTE_HOST"=>"localhost", "REQUEST_METHOD"=>"GET", "REQUEST_URI"=>"http://localhost:9292/hello", "SCRIPT_NAME"=>"", "SERVER_NAME"=>"localhost", "SERVER_PORT"=>"9292", "SERVER_PROTOCOL"=>"HTTP/1.1", "SERVER_SOFTWARE"=>"WEBrick/1.3.1 (Ruby/2.2.2/2015-04-13)", "HTTP_USER_AGENT"=>"curl/7.30.0", "HTTP_HOST"=>"localhost:9292", "HTTP_ACCEPT"=>"*/*", "rack.version"=>[1, 3], "rack.input"=>#<Rack::Lint::InputWrapper:0x007f99711d6458 #input=#<StringIO:0x007f99711e52f0>>, "rack.errors"=>#<Rack::Lint::ErrorWrapper:0x007f99711d6430 #error=#<IO:<STDERR>>>, "rack.multithread"=>true, "rack.multiprocess"=>false, "rack.run_once"=>false, "rack.url_scheme"=>"http", "rack.hijack?"=>true, "rack.hijack"=>#<Proc:0x007f99711dc3d0#/Users/iainb/Projects/Test/31551252/vendor.noindex/ruby/2.2.0/gems/rack-1.6.4/lib/rack/lint.rb:525>, "rack.hijack_io"=>nil, "HTTP_VERSION"=>"HTTP/1.1", "REQUEST_PATH"=>"/hello", "sinatra.commonlogger"=>true, "rack.tempfiles"=>[], "rack.routing_args"=>{:route_info=>#<Grape::Route:0x007f997106fcb8 #options={:params=>{}, :prefix=>nil, :version=>nil, :namespace=>"/", :method=>"GET", :path=>"/hello(.:format)", :compiled=>/\A\/hello(?:\.(?<format>[^\/.?]+))?\Z/, :settings=>{}}>}, "api.endpoint"=>#<Grape::Endpoint:0x007f9971497750 #inheritable_setting=#<Grape::Util::InheritableSetting:0x007f9971157040 #route={:saved_declared_params=>[], :saved_validations=>[]}, #api_class={}, #namespace=#<Grape::Util::InheritableValues:0x007f9971156820 #inherited_values={}, #new_values={}>, #namespace_inheritable=#<Grape::Util::InheritableValues:0x007f9971156780 #inherited_values=#<Grape::Util::InheritableValues:0x007f99711c5860 #inherited_values={}, #new_values={}>, #new_values={:default_error_status=>500}>, #namespace_stackable=#<Grape::Util::StackableValues:0x007f9971156618 #inherited_values=#<Grape::Util::StackableValues:0x007f99711c57c0 #inherited_values={}, #new_values={}, #froozen_values={}>, #new_values={}, #froozen_values={}>, #point_in_time_copies=[], #parent=#<Grape::Util::InheritableSetting:0x007f99711c5950 #route={}, #api_class={}, #namespace=#<Grape::Util::InheritableValues:0x007f99711c58d8 #inherited_values={}, #new_values={}>, #namespace_inheritable=#<Grape::Util::InheritableValues:0x007f99711c5860 #inherited_values={}, #new_values={}>, #namespace_stackable=#<Grape::Util::StackableValues:0x007f99711c57c0 #inherited_values={}, #new_values={}, #froozen_values={}>, #point_in_time_copies=[], #parent=nil>>, #options={:method=>["GET"], :path=>[:hello], :for=>API, :route_options=>{:params=>{}}}, #source=#<Proc:0x007f99711c5450#/Users/iainb/Projects/Test/31551252/config.ru:5>, #block=#<Proc:0x007f9971143cc0#/Users/iainb/Projects/Test/31551252/vendor.noindex/ruby/2.2.0/gems/grape-0.12.0/lib/grape/endpoint.rb:47>, #namespace="/", #routes=[#<Grape::Route:0x007f997106fcb8 #options={:params=>{}, :prefix=>nil, :version=>nil, :namespace=>"/", :method=>"GET", :path=>"/hello(.:format)", :compiled=>/\A\/hello(?:\.(?<format>[^\/.?]+))?\Z/, :settings=>{}}>], #env={...}, #header={}, #request=#<Grape::Request:0x007f9971425d08 ...>, #params=#<Hashie::Mash>, #headers={"User-Agent"=>"curl/7.30.0", "Host"=>"localhost:9292", "Accept"=>"*/*", "Version"=>"HTTP/1.1"}, #cookies=#<Grape::Cookies:0x007f99713be090 #cookies={}, #send_cookies={}>>, "api.format"=>:txt, "rack.request.query_string"=>"", "rack.request.query_hash"=>{}, "rack.request.cookie_hash"=>{}}, #params=#<Hashie::Mash>, #headers={"User-Agent"=>"curl/7.30.0", "Host"=>"localhost:9292", "Accept"=>"*/*", "Version"=>"HTTP/1.1"}>}%
In my controller I have the following
post "/buy_item" do
redirect '/login' unless session[:name]
#owner = Market::User.user_by_name(params[:owner])
#item = #owner.item_by_id(params[:item].to_i)
#current_user = Market::User.user_by_name(session[:name])
if #current_user.buy_item?(#item)
#current_user.buy_item(#item)
else
redirect '/error'
end
end
when I'm forcing an "else" I get correctly redirected to /error, but directly after that I get redirected to another page (/?loggedin=true). redirect "/?loggedin=true" is the last line of the post "/login" method. So, it seems that it is calling POST /login somehow..
The route for /error looks like this:
get "/error" do
redirect '/login' unless session[:name]
template = ERB.new File.new($VIEWS_FOLDER + "/error.erb").read, nil, "%"
template.result(binding)
end
Nothing in /error.erb is redirecting, when I call localhost:4567/error directly it doesn't get redirected.
Here's the log:
127.0.0.1 - - [03/Oct/2012 17:15:03] "POST /login HTTP/1.1" 303 - 0.0012
localhost - - [03/Oct/2012:17:15:03 CEST] "POST /login HTTP/1.1" 303 0
localhost:4567/login -> /login
127.0.0.1 - - [03/Oct/2012 17:15:03] "GET /?loggedin=true HTTP/1.1" 200 3916 0.0055
localhost - - [03/Oct/2012:17:15:03 CEST] "GET /?loggedin=true
HTTP/1.1" 200 3916
localhost:4567/login -> /?loggedin=true
127.0.0.1 - - [03/Oct/2012 17:15:05] "POST /buy_item HTTP/1.1" 303 - 0.0030
localhost - - [03/Oct/2012:17:15:05 CEST] "POST /buy_item HTTP/1.1"
303 0
localhost:4567/?loggedin=true -> /buy_item
127.0.0.1 - - [03/Oct/2012 17:15:05] "GET /error HTTP/1.1" 200 1609 0.0039
localhost - - [03/Oct/2012:17:15:05 CEST] "GET /error HTTP/1.1" 200
1609
localhost:4567/?loggedin=true -> /error
127.0.0.1 - - [03/Oct/2012 17:15:05] "GET /?loggedin=true HTTP/1.1" 200 3916 0.0063
localhost - - [03/Oct/2012:17:15:05 CEST] "GET /?loggedin=true
HTTP/1.1" 200 3916
localhost:4567/login -> /?loggedin=true
You have two redirects inside your route '/buy_item'; and in '/error' you do not return after the redirect. redirect does something to your HTTP header and it is good practice to return after the call, i.e. in both routes /buy_item and /error:
-redirect '/login' unless session[:name]
+unless session[:name]
+ redirect '/login'
+ return nil
+end
I have worked on a large Sinatra app and there were a lot of problems with redirect, the above patch should help. Moreover I recommend you to take a look at the HTTP RFC and make yourself familiar with the Post-Redirect-Get scheme. To to redirects with custom status code, you can do redirect '/foo', 303.
How can I disable messages from webrick echoed on to the terminal? For the INFO messages that appear at the beginning, I was able to disable it by setting the Logger parameter so as:
s = WEBrick::HTTPServer.new(
Port: 3000,
BindAddress: "localhost",
Logger: WEBrick::Log.new("/dev/null"),
)
But I further want to disable the messages that look like:
localhost - - [17/Jun/2011:10:01:38
EDT] "GET .... HTTP/1.1" 200 0
http://localhost:3000/ -> .....
when a request is made from the web browser.
Following the link to the source and suggestion provied by Yet Another Geek, I was able to figure out a way. Set the AccessLog parameter to [nil, nil] [] (Changed following suggestion by Robert Watkins).
s = WEBrick::HTTPServer.new(
Port: 3000,
BindAddress: "localhost",
Logger: WEBrick::Log.new("/dev/null"),
AccessLog: [],
)