Well, I'll keep it simple, PURGE requests (or so I thought?) were pretty much all handled by literally:
acl purge {
"localhost";
"127.0.0.1";
}
and then
if (req.method == "PURGE") {
if (!client.ip ~ purge) {
return(synth(405, "This IP is not allowed to send PURGE requests."));
}
return (purge);
}
I fairly certain both of the above statements are "correct", the thing I'm hung-up on is that if I sent a
curl -X PURGE http://domain.com/
or
curl -X PURGE http://domain.com/.*
and Varnish sends back 200 Purged well... the cache is purged? Even if it's just the homepage and not the entire cache (swear it was all using the above .* method) is and the above snippets of code are correct is there any particular reason http://domain.com (as in the actual homepage) isn't purged?
varnishncsa shows:
MYIP - - [16/Feb/2015:23:23:10 -0600] "PURGE http://domain.com/ HTTP/1.1" 200 241 "-" "curl/7.29.0"
I know I gotta be missing something silly but I can't figure it out?
The official documentation says basically what you've done, but keep in mind PURGE != BAN, you can use regular expressions in BAN, but not in PURGE. With PURGE you delete all the Vary defined copies of one particular url.
In my tests with 4.0.2 it works as advised, I have the homepage cached, I do a curl -X PURGE http://localhost:8080/ and in varnishlog I see (among other things):
* << Request >> 65542
- Begin req 65541 rxreq
- Timestamp Start: 1424188792.108573 0.000000 0.000000
- Timestamp Req: 1424188792.108573 0.000000 0.000000
- ReqStart ::1 60496
- ReqMethod PURGE
- ReqURL /
- VCL_acl MATCH purge "localhost"
- VCL_return purge
- VCL_call HASH
- VCL_return lookup
- VCL_call PURGE
And reloading I see a MISS & backend request (because it's not in the cache):
* << BeReq >> 65548
- Begin bereq 65547 fetch
- Timestamp Start: 1424188815.112540 0.000000 0.000000
- BereqMethod GET
- BereqURL /
- BereqProtocol HTTP/1.1
- VCL_call BACKEND_RESPONSE
- TTL VCL 120 21600 0 1424188815
- VCL_return deliver
* << Request >> 65547
- Begin req 65546 rxreq
- ReqMethod GET
- ReqURL /
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- Debug "XXXX MISS"
- VCL_call MISS
- VCL_return fetch
- Link bereq 65548 fetch
BTW Add "::1" to the list of ips in the purge acl just in case you are using ipv6. It would have returned a 405, but who knows.
Related
I have one app that is a Sinatra app. I would like to have the log of each request on one line, Currently only 1 line per request is shown(the request itself). But I want to append custom info to it.
This is my current line:
[02/Feb/2018:11:52:32 +0000] "GET /feed/name.csv HTTP/1.0" 200 - 0.1620
2400:gh654:32:6532:0:0:a43e:404f, 0.0.0.0:2017 - - [05/Feb/2018:13:32:48 +0000] "HEAD / HTTP/1.0" 404 18 0.0676
2400:gh654:32:6532:0:0:a43e:414d, 0.0.0.0:2017 - - [05/Feb/2018:13:32:48 +0000] "HEAD / HTTP/1.0" 404 18 0.0041
I want to transform it in something like:
INFO 2018-03-02T14:03:36Z [main] [066843] [8a6e846f-45b7-433d-9d98-21c8c0766ad7] method=GET path= /some/path.csv format=csv action=action_name status=200 duration=459.97 view=154.94 db=172.96 user=user_id ip=user_id on=protocol_used agent=agent_used params=params_used
In the past to do this in rails applications I have used lograge but it seems it is not supported by sinatra apps.
Have you guys ever had to append custom info to the sinatra logs?
This is my configuration for logging:
configure do
enable :logging
file = File.new("#{RAILS_ROOT}/log/#{settings.environment}.log", 'a+')
file.sync = true
use Rack::CommonLogger, file
set :show_exceptions, true
set :logging, true
ActiveRecord::Base.logger = Logger.new("#{RAILS_ROOT}/log/#{settings.environment}.log")
end
My idea is using Rack to load a web application separated in two parts. This is the structure:
my_app
|______api
| |______poros
|
|______fe
| |______build
| |______node_modules
| |______package.json
| |______public
| | |______ index.html
| | ...
| |______README.md
| |______src
| |______index.js
| ...
|______config.ru
The fe part (which stands for front-end) is a React application created with create-react-app. I've built it with
npm run build
and then its static optimized build is under my_app/fe/build directory.
This is my my_app/config.ru:
#\ -w -p 3000
require 'emeraldfw'
use Rack::Reloader, 0
use Rack::ContentLength
run EmeraldFW::Loader.new
And this is my EmeraldFW::Loader class, which is part of a gem installed and running fine.
module EmeraldFW
class Loader
def call(env)
path_info = (env['PATH_INFO'] == '/') ? '/index.html' : env['PATH_INFO']
extension = path_info.split('/').last.split('.').last
file = File.read("fe/build#{path_info}")
[200,{'Content-Type' => content_type[extension]},[file]]
end
def content_type
{
'html' => 'text/html',
'js' => 'text/javascript',
'css' => 'text/css',
'svg' => 'image/svg+xm',
'ico' => 'image/x-icon',
'map' => 'application/octet-stream'
}
end
end
end
As you may see, this is all quite simple. I capture the request with in my EmeraldFW::Loader class and transform its path_info it a bit. If the request is '/' I rewrite it to '/index.html' before doing anything else. In all cases I prepend fe/build to make it load from the static build of the React application.
When I run
rackup config.ru
and load the application at http://localhost:3000 the result is completely fine:
[2017-03-15 21:28:23] INFO WEBrick 1.3.1
[2017-03-15 21:28:23] INFO ruby 2.3.3 (2016-11-21) [x86_64-linux]
[2017-03-15 21:28:23] INFO WEBrick::HTTPServer#start: pid=11728 port=3000
::1 - - [15/Mar/2017:21:28:27 -0300] "GET / HTTP/1.1" 200 386 0.0088
::1 - - [15/Mar/2017:21:28:27 -0300] "GET /static/css/main.9a0fe4f1.css HTTP/1.1" 200 623 0.0105
::1 - - [15/Mar/2017:21:28:27 -0300] "GET /static/js/main.91328589.js HTTP/1.1" 200 153643 0.0086
::1 - - [15/Mar/2017:21:28:28 -0300] "GET /static/media/logo.5d5d9eef.svg HTTP/1.1" 200 2671 0.0036
::1 - - [15/Mar/2017:21:28:28 -0300] "GET /static/js/main.91328589.js.map HTTP/1.1" 200 1705922 0.1079
::1 - - [15/Mar/2017:21:28:28 -0300] "GET /static/css/main.9a0fe4f1.css.map HTTP/1.1" 200 105 0.0021
As you may see, all resources are loading correctly, with the correct mime types. But it happens that my default React logo, which should be spinning in my app frontpage, is not there! As if it wasn't loaded.
The big picture of all this is having this rack middleware loading the React front-end of the app and, at the same time, redirecting correctly the api requests made with fetch from the front-end to the poros (Plain Old Ruby Objects) which are the API part.
The concept is quite simple. But I just can't understand why this specific resource, the svg logo, is not loading.
It was all a matter of wrong mime type, caused by a typo.
It may be clearly seen in my question that I wrote:
def content_type
{
'html' => 'text/html',
'js' => 'text/javascript',
'css' => 'text/css',
'svg' => 'image/svg+xm', # <== Here is the error!!!
'ico' => 'image/x-icon',
'map' => 'application/octet-stream'
}
end
when the correct mime type for SVG files is 'image/svg-xml', with an 'l'...
This was making the browser ignore the file received and so it won't display it correctly.
I'm having issues getting data from GitHub Archive.
The main issue is my problem with encoding {} and .. in my URL. Maybe I am misreading the Github API or not understanding encoding correctly.
require 'open-uri'
require 'faraday'
conn = Faraday.new(:url => 'http://data.githubarchive.org/') do |faraday|
faraday.request :url_encoded # form-encode POST params
faraday.response :logger # log requests to STDOUT
faraday.adapter Faraday.default_adapter # make requests with Net::HTTP
end
#query = '2015-01-01-15.json.gz' #this one works!!
query = '2015-01-01-{0..23}.json.gz' #this one doesn't work
encoded_query = URI.encode(query)
response = conn.get(encoded_query)
p response.body
The GitHub Archive example for retrieving a range of files is:
wget http://data.githubarchive.org/2015-01-01-{0..23}.json.gz
The {0..23} part is being interpreted by wget itself as a range of 0 .. 23. You can test this by executing that command with the -v flag which returns:
wget -v http://data.githubarchive.org/2015-01-01-{0..1}.json.gz
--2015-06-11 13:31:07-- http://data.githubarchive.org/2015-01-01-0.json.gz
Resolving data.githubarchive.org... 74.125.25.128, 2607:f8b0:400e:c03::80
Connecting to data.githubarchive.org|74.125.25.128|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2615399 (2.5M) [application/x-gzip]
Saving to: '2015-01-01-0.json.gz'
2015-01-01-0.json.gz 100%[===========================================================================================================================================>] 2.49M 3.03MB/s in 0.8s
2015-06-11 13:31:09 (3.03 MB/s) - '2015-01-01-0.json.gz' saved [2615399/2615399]
--2015-06-11 13:31:09-- http://data.githubarchive.org/2015-01-01-1.json.gz
Reusing existing connection to data.githubarchive.org:80.
HTTP request sent, awaiting response... 200 OK
Length: 2535599 (2.4M) [application/x-gzip]
Saving to: '2015-01-01-1.json.gz'
2015-01-01-1.json.gz 100%[===========================================================================================================================================>] 2.42M 867KB/s in 2.9s
2015-06-11 13:31:11 (867 KB/s) - '2015-01-01-1.json.gz' saved [2535599/2535599]
FINISHED --2015-06-11 13:31:11--
Total wall clock time: 4.3s
Downloaded: 2 files, 4.9M in 3.7s (1.33 MB/s)
In other words, wget is substituting values into the URL and then getting that new URL. This isn't obvious behavior, nor is it well documented, but you can find mention of it "out there". For instance in "All the Wget Commands You Should Know":
7. Download a list of sequentially numbered files from a server
wget http://example.com/images/{1..20}.jpg
To do what you want, you need to iterate over the range in Ruby using something like this untested code:
0.upto(23) do |i|
response = conn.get("/2015-01-01-#{ i }.json.gz")
p response.body
end
To get a better idea of what's going wrong, let's start with the example given in the GitHub documentation:
wget http://data.githubarchive.org/2015-01-01-{0..23}.json.gz
The thing to note here is that {0..23} is automagically getting expanded by bash. You can see this by running the following command:
echo {0..23}
> 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
This means wget doesn't get called just once, but instead gets called a total of 24 times. The problem you're having is that Ruby doesn't automagically expand {0..23} like bash does, and instead you're making a literal call to http://data.githubarchive.org/2015-01-01-{0..23}.json.gz, which doesn't exist.
Instead you will need to loop through 0..23 yourself and make a single call every time:
(0..23).each do |n|
query = "2015-01-01-#{n}.json.gz"
encoded_query = URI.encode(query)
response = conn.get(encoded_query)
p response.body
end
In Sinatra I have a route setup similar to the following:
put '/test' do
begin
logger.info 'In put begin block'
write_to_file(request.body.read)
[200, '']
rescue RuntimeError => e
[500, 'some error']
end
end
def write_to_file(data)
logger.info "writing data with size #{data.size}"
# Code to save file...
end
When I send data that is < ~500 MBytes it everything seems to work correctly but when I attempt to send data that is >= 500 MBytes I get some weird log output and then the client eventually errors out with the following error: Excon::Errors::SocketError: EOFError (EOFError)
The logs from the server (Sinatra) look like the following:
For data < 500 MBytes:
I, [2013-01-07T21:33:59.386768 #17380] INFO -- : In put begin block
I, [2013-01-07T21:34:01.279922 #17380] INFO -- : writing data with size 209715200
xxx.xxx.xxx.xxx - - [07/Jan/2013 21:34:22] "PUT /test " 200 - 22.7917
For data > 500 MBytes:
I, [2013-01-07T21:47:37.434022 #17386] INFO -- : In put begin block
I, [2013-01-07T21:47:41.152932 #17386] INFO -- : writing data with size 524288000
I, [2013-01-07T21:48:16.093683 #17380] INFO -- : In put begin block
I, [2013-01-07T21:48:20.300391 #17380] INFO -- : writing data with size 524288000
xxx.xxx.xxx.xxx - - [07/Jan/2013 21:48:39] "PUT /test " 200 - 62.4515
I, [2013-01-07T21:48:54.718971 #17386] INFO -- : In put begin block
I, [2013-01-07T21:49:00.381725 #17386] INFO -- : writing data with size 524288000
I, [2013-01-07T21:49:33.980043 #17267] INFO -- : In put begin block
I, [2013-01-07T21:49:41.990671 #17267] INFO -- : writing data with size 524288000
xxx.xxx.xxx.xxx - - [07/Jan/2013 21:50:06] "PUT /test " 200 - 110.2076
xxx.xxx.xxx.xxx - - [07/Jan/2013 21:51:22] "PUT /test " 200 - 108.5339
Not entirely sure whats going on here so I guess my question is two fold, A. What is fundamentally different between these two cases that would cause them to behave this way? B. Is there a better way to handle data to mitigate against this?
The problem turned out to be with apache/passenger. Running the server with WEBrick alleviated the issue.
In my controller I have the following
post "/buy_item" do
redirect '/login' unless session[:name]
#owner = Market::User.user_by_name(params[:owner])
#item = #owner.item_by_id(params[:item].to_i)
#current_user = Market::User.user_by_name(session[:name])
if #current_user.buy_item?(#item)
#current_user.buy_item(#item)
else
redirect '/error'
end
end
when I'm forcing an "else" I get correctly redirected to /error, but directly after that I get redirected to another page (/?loggedin=true). redirect "/?loggedin=true" is the last line of the post "/login" method. So, it seems that it is calling POST /login somehow..
The route for /error looks like this:
get "/error" do
redirect '/login' unless session[:name]
template = ERB.new File.new($VIEWS_FOLDER + "/error.erb").read, nil, "%"
template.result(binding)
end
Nothing in /error.erb is redirecting, when I call localhost:4567/error directly it doesn't get redirected.
Here's the log:
127.0.0.1 - - [03/Oct/2012 17:15:03] "POST /login HTTP/1.1" 303 - 0.0012
localhost - - [03/Oct/2012:17:15:03 CEST] "POST /login HTTP/1.1" 303 0
localhost:4567/login -> /login
127.0.0.1 - - [03/Oct/2012 17:15:03] "GET /?loggedin=true HTTP/1.1" 200 3916 0.0055
localhost - - [03/Oct/2012:17:15:03 CEST] "GET /?loggedin=true
HTTP/1.1" 200 3916
localhost:4567/login -> /?loggedin=true
127.0.0.1 - - [03/Oct/2012 17:15:05] "POST /buy_item HTTP/1.1" 303 - 0.0030
localhost - - [03/Oct/2012:17:15:05 CEST] "POST /buy_item HTTP/1.1"
303 0
localhost:4567/?loggedin=true -> /buy_item
127.0.0.1 - - [03/Oct/2012 17:15:05] "GET /error HTTP/1.1" 200 1609 0.0039
localhost - - [03/Oct/2012:17:15:05 CEST] "GET /error HTTP/1.1" 200
1609
localhost:4567/?loggedin=true -> /error
127.0.0.1 - - [03/Oct/2012 17:15:05] "GET /?loggedin=true HTTP/1.1" 200 3916 0.0063
localhost - - [03/Oct/2012:17:15:05 CEST] "GET /?loggedin=true
HTTP/1.1" 200 3916
localhost:4567/login -> /?loggedin=true
You have two redirects inside your route '/buy_item'; and in '/error' you do not return after the redirect. redirect does something to your HTTP header and it is good practice to return after the call, i.e. in both routes /buy_item and /error:
-redirect '/login' unless session[:name]
+unless session[:name]
+ redirect '/login'
+ return nil
+end
I have worked on a large Sinatra app and there were a lot of problems with redirect, the above patch should help. Moreover I recommend you to take a look at the HTTP RFC and make yourself familiar with the Post-Redirect-Get scheme. To to redirects with custom status code, you can do redirect '/foo', 303.