I have many record in table name AE_MKT (7500 record and more in the future)
and I use Query Builder
$query = DB::table('AE_MKT')->get();
so I get error see on this image:
and another config at the .htaccess
Header set Cache-Control "max-age=290304000, public"
Header append Cache-Control s-maxage=600 "expr=%{REQUEST_STATUS} == 200"
Header set Connection keep-alive
but still error when I need query with large records
Related
I've in a NiFi attribute the full INSERT command that works with CURL:
echo "INSERT INTO default.PERFTEST_BUFFER VALUES (1, '2020-04-09 19:06:02', 48.8644, 'A')" \
| curl 'http://xxx.xxx.xxx.xxx:8123/?' --data-binary #-
Insert in Attribut :
INSERT INTO default.PERFTEST_BUFFER
VALUES (1, '2020-04-09 19:06:02', 48.8644, 'A')
I can't figure out what should be all the parameters of the InvokeHTTP processor.
I've used for the first two :
POST
http://xxx.xxx.xxx.xxx:8123/
then I'm lost.
Any idea how to configure it ?
If your query is static you can pass it in the query string in the encoded format:
# GET
curl 'http://xxx.xxx.xxx.xxx:8123/?query=INSERT%20INTO%20default.PERFTEST_BUFFER%20VALUES%20%281%2C%20%272020-04-09%2019%3A06%3A02%27%2C%2048.8644%2C%20%27A%27%29'
# POST
curl -d '' 'http://xxx.xxx.xxx.xxx:8123/?query=INSERT%20INTO%20default.PERFTEST_BUFFER%20VALUES%20%281%2C%20%272020-04-09%2019%3A06%3A02%27%2C%2048.8644%2C%20%27A%27%29'
It needs to pass the required values to just two params: HTTP Method and Remote URL of InvokeHTTP-processor.
The way I made it work thanks to daggett and and example here :
https://github.com/zezutom/NiFiByExample
Use before the InvokeHTTP a processor ReplaceText that replace the flowfile content by the variable "Insert" value that have the full Insert command
Search Value (?s)(^.*$)
Replacement Value ${Insert}
So the flow file content is
INSERT INTO default.PERFTEST_BUFFER VALUES (1, '2020-04-09 19:06:02', 48.8644, 'A')
Follow by a processor InvokeHTTP the following parameters
HTTP Method POST
Remote URL http://xxx.xxx.xxx.xxx:8123/
SSL Context Service No value set
Connection Timeout 5 secs
Read Timeout 15 secs
Include Date Header True
Follow Redirects True
Attributes to Send No value set
Basic Authentication Username No value set
Basic Authentication Password No value set
Proxy Configuration Service No value set
Proxy Host No value set
Proxy Port No value set
Proxy Type http
Proxy Username No value set
Proxy Password No value set
Put Response Body In Attribute No value set
Max Length To Put In Attribute 256
Use Digest Authentication false
Always Output Response true
Add Response Headers to Request false
Content-Type ${mime.type}
Send Message Body true
Use Chunked Encoding false
Penalize on "No Retry" false
Use HTTP ETag false
Maximum ETag Cache Size 10MB
Final Edit: This works with no semantic errors:
+ Request
+ Headers
Accept: application/json
Content-Type: application/json
X-Auth-Client: Your Client Id
X-Auth-Token: Your Token
+ Body
+ Attributes (ProductPost)
+ Response 200
+ Headers
Content-Encoding: Entity header is used to compress the media-type.
Content-Type: application/json
Date: The date the response was sent.
Transfer-Encoding: Header specifies the form of encoding used to safely transfer the entity to the user.
Vary: HTTP response header determines how to match future request headers to decide whether a cached response can be used rather than requesting a fresh one from the origin server. We use Accept Encoding
X-Rate-Limit-Requests-Left: Header details how many remaining requests your client can make in the current window before being rate-limited.
X-Rate-Limit-Requests-Quota: Header shows how many API requests are allowed in the current window for your client
X-Rate-Limit-Time-Reset-Ms: Header shows how many milliseconds are remaining in the window.
X-Rate-Limit-Time-Window-Ms: Header shows the size of your current rate-limiting window
+ Body
+ Attributes (ProductResponse)
Edit: The header section is rendering, but now the Body section is just showing the text " + Attributes (ProductPost)"
+ Request
+ Headers
Accept: application/json
Content-Type: application/json
X-Auth-Client: Your Client Id
X-Auth-Token: Your Token
+ Response 200
+ Headers
Content-Encoding: Entity header is used to compress the media-type.
Content-Type: application/json
Date: The date the response was sent.
Transfer-Encoding: Header specifies the form of encoding used to safely transfer the entity to the user.
Vary: HTTP response header determines how to match future request headers to decide whether a cached response can be used rather than requesting a fresh one from the origin server. We use Accept Encoding
X-Rate-Limit-Requests-Left: Header details how many remaining requests your client can make in the current window before being rate-limited.
X-Rate-Limit-Requests-Quota: Header shows how many API requests are allowed in the current window for your client
X-Rate-Limit-Time-Reset-Ms: Header shows how many milliseconds are remaining in the window.
X-Rate-Limit-Time-Window-Ms: Header shows the size of your current rate-limiting window
+ Body
+ Attributes (ProductCollectionResponse)
I am trying to define the Request Body and after reading this:
https://help.apiary.io/api_101/apib-authentication/ &
https://github.com/apiaryio/api-blueprint/blob/master/API%20Blueprint%20Specification.md#def-headers-section
It seemed like I could split them into sections. But the Attributes section is not being recognized. This is a /GET request.
Any ideas why?
+ Request (application/json)
+ Headers
+ Attributes (RequestHeaders)
+ Body
+ Attributes (ProductPost)
Headers section can't contain Attributes, you need to define them explicitly. Just replace:
+ Attributes (RequestHeaders)
with definition of RequestHeaders.
Also try to align Body and Attributes at the same column:
+ Body
+ Attributes (ProductPost)
We are trying to build an incoming request validation platform using HAProxy+Lua.
Our use-case is to create a LUA scripts that will essentially make a socket call to a Validation API, and based on the response
from Validation API we want to redirect the request to a backend API, and if the validation fails we would want to return
the request right from the LUA script. For example, for 200 response we would want to redirect the request to backend api, and for 404 we would want
to return the request. From the documentation, I understand that there are various default functions available
with Lua-Haproxy integration.
core.register_action() --> I'm using this. Take TXN as input
core.register_converters() --> Essentially used for string manipulations.
core.register_fetches() --> Takes TXN as input and returns string; Mainly used for representing dynamic backend profiles in haproxy config
core.register_init() --> Used for initialization
core.register_service() --> You have to return the response mandatorily while using this function, which doesn't satisfy our requirements
core.register_task() --> For using normal functions. No mandatory input class. TXN is required to fetch header details from request
I have tried all of the functions from above list, I understand that core.register_service is basically to return a response from the
Lua script. However, what is problematic is, we must send the response from the LUA script and it will not redirect the request to BACKEND.
Currently, I am using core.register_action to interrupt the requests, but I'm not able to return the request using this function. Here's
what my code looks like:
local http_socket = require("socket.http")
local pretty_print = require("pl.pretty")
function add_http_request_header(txn, name, value)
local headerName = name
local headerValue = value
txn.http:req_add_header(headerName, headerValue)
end
function call_validation_api()
local request, code, header = http_socket.request {
method = "GET", -- Validation API Method
url = "http://www.google.com/" -- Validation API URL
}
-- Using core.log; Print in some cases is a blocking operation http://www.arpalert.org/haproxy-lua.html#h203
core.Info( "Validation API Response Code: " .. code )
pretty_print.dump( header )
return code
end
function failure_response(txn)
local response = "Validation Failed"
core.Info(response)
txn.res:send(response)
-- txn:close()
end
core.register_action("validation_action", { "http-req", "http-res" }, function(txn)
local validation_api_code = call_validation_api()
if validation_api_code == 200 then
core.Info("Validation Successful")
add_http_request_header(txn, "test-header", "abcdefg")
pretty_print.dump( txn.http:req_get_headers() )
else
failure_response(txn) --->>> **HERE I WANT TO RETURN THE RESPONSE**
end
end)
Following is the configuration file entry:
frontend http-in
bind :8000
mode http
http-request lua.validation_action
#Capturing header of the incoming request
capture request header test-header len 64
#use_backend %[lua.fetch_req_params]
default_backend app
backend app
balance roundrobin
server app1 127.0.0.1:9999 check
Any help is much appreciated in achieving this functionality. Also, I understand that SOCKET call from Lua script is a blocking call, which is opposite to HAProxy's default nature of keep-alive connection. Please feel free to suggest any other utility to achieve this functionality, if you have already used it.
Ok I have figured out the answer to this question:
I created 2 backends for success and failure of requests, and based on the response I am returning 2 different strings. In "failure_backend", I have called a different service, which essentially is a core.register_service and can return the response. I'm pasting code for both the configuration file and lua script
HAProxy conf file:
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2
maxconn 4000
user haproxy
group haproxy
daemon
#lua file load
lua-load /home/aman/coding/haproxy/http_header.lua
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
retries 3
timeout http-request 90s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend http-in
bind :8000
mode http
use_backend %[lua.validation_fetch]
default_backend failure_backend
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend success_backend
balance roundrobin
server app1 172.23.12.94:9999 check
backend failure_backend
http-request use-service lua.failure_service
# For displaying HAProxy statistics.
frontend stats
bind :8888
default_backend stats
backend stats
stats enable
stats hide-version
stats realm Haproxy Statistics
stats uri /haproxy/stats
stats auth aman:rjil#123
Lua script:
local http_socket = require("socket.http")
local pretty_print = require("pl.pretty")
function add_http_request_header(txn, name, value)
local headerName = name
local headerValue = value
txn.http:req_add_header(headerName, headerValue)
end
function call_validation_api()
local request, code, header = http_socket.request {
method = "GET", -- Validation API Method
url = "http://www.google.com/" -- Validation API URL
}
-- Using core.log; Print in some cases is a blocking operation http://www.arpalert.org/haproxy-lua.html#h203
core.Info( "Validation API Response Code: " .. code )
pretty_print.dump( header )
return code
end
function failure_response(txn)
local response = "Validation Failed"
core.Info(response)
return "failure_backend"
end
-- Decides back-end based on Success and Failure received from validation API
core.register_fetches("validation_fetch", function(txn)
local validation_api_code = call_validation_api()
if validation_api_code == 200 then
core.Info("Validation Successful")
add_http_request_header(txn, "test_header", "abcdefg")
pretty_print.dump( txn.http:req_get_headers() )
return "success_backend"
else
failure_response(txn)
end
end)
-- Failure service
core.register_service("failure_service", "http", function(applet)
local response = "Validation Failed"
core.Info(response)
applet:set_status(400)
applet:add_header("content-length", string.len(response))
applet:add_header("content-type", "text/plain")
applet:start_response()
applet:send(response)
end)
I am completely stuck on this one and hours of searching hasn't turned up a result. I'm getting a 406 error when I try to do an ajax request. Here is my code..
Tiers controller
def index
series = Series.find_by_id(params[:series_id])
respond_to do |format|
format.json { render :json => series.tiers }
end
end
Javascript
$('#series_id').change ->
series_id = $('#series_id').val()
$.get '/admin/tiers', {series_id:series_id}, (result) ->
element = $('select[name *= "tier_id"]')
element.empty()
$.each result, (index, item) ->
element.append $('<option/>', value:item.id, text: item.title)
And here is the error I am getting from the rails server
Started GET "/admin/tiers/1" for 127.0.0.1 at 2013-01-21 14:19:38 +1100
Processing by Admin::TiersController#index as JSON
Parameters: {"series_id"=>"1"}
User Load (0.4ms) SELECT "users".* FROM "users" WHERE "users"."id" = 1 LIMIT 1
(0.1ms) begin transaction
(0.5ms) UPDATE "users" SET "last_request_at" = '2013-01-21 14:19:38.827810', "perishable_token" = 'y0TMicsW2l4zgSiQcJx', "updated_at" = '2013-01-21 14:19:38.829452' WHERE "users"."id" = 1
(2.5ms) commit transaction
Tier Load (0.3ms) SELECT "tiers".* FROM "tiers" WHERE "tiers"."id" IS NULL LIMIT 1
Tier Load (0.3ms) SELECT "tiers".* FROM "tiers"
Series Load (0.2ms) SELECT "series".* FROM "series"
Series Load (0.2ms) SELECT "series".* FROM "series" WHERE "series"."id" = 1 LIMIT 1
Completed 406 Not Acceptable in 30ms (ActiveRecord: 4.4ms)
Really don't know what is stuffing it up, the controller looks fine and it looks like the content-type is json.
I'm also using rails 3.2.11
So changing
respond_to do |format|
format.json { render :json => series.tiers }
end
to
render :json => series.tiers
fix the problem but I am still not sure why :( If anyone has any idea why your input would be greatly appreciated.
I can't tell you why that fixed the problem, but I think a better solution may be to use the .getJSON call rather than .get.
Rails generates a 406 when it can't find the proper response type for the requested type in the Accept header. When I use Fiddler2 to snoop the request from a .get, I see the request header:
Accept: */*
This doesn't give Rails any information to work with. If there's an extension on the requested URL, such as .js in the case of javascript, it can intuit from that as well, but that's not available for your JSON request.
When I make a .getJSON call, I see the request header:
Accept: application/json, text/javascript, */*; q=0.01
This is probably enough information for Rails to match the requested type to your json handler.
I have the following Business Process defined within a Production on an Intersystems Cache Installation
/// Makes a call to Merlin based on the message sent to it from the pre-processor
Class sgh.Process.MerlinProcessor Extends Ens.BusinessProcess [ ClassType = persistent, ProcedureBlock ]
{
Property WorkingDirectory As %String;
Property WebServer As %String;
Property CacheServer As %String;
Property Port As %String;
Property Location As %String;
Parameter SETTINGS = "WorkingDirectory,WebServer,Location,Port,CacheServer";
Method OnRequest(pRequest As sgh.Message.MerlinTransmissionRequest, Output pResponse As Ens.Response) As %Status
{
Set tSC=$$$OK
Do ##class(sgh.Utils.Debug).LogDebugMsg("Packaging an HTTP request for Saved form "_pRequest.DateTimeSaved)
Set dateTimeSaved = pRequest.DateTimeSaved
Set patientId = pRequest.PatientId
Set latestDateTimeSaved = pRequest.LatestDateTimeSaved
Set formName = pRequest.FormName
Set formId = pRequest.FormId
Set episodeNumber = pRequest.EpisodeNumber
Set sentElectronically = pRequest.SentElectronically
Set styleSheet = pRequest.PrintName
Do ##class(sgh.Utils.Debug).LogDebugMsg("Creating HTTP Request Class")
set HTTPReq = ##class(%Net.HttpRequest).%New()
Set HTTPReq.Server = ..WebServer
Set HTTPReq.Port = ..Port
do HTTPReq.InsertParam("DateTimeSaved",dateTimeSaved)
do HTTPReq.InsertParam("HospitalNumber",patientId)
do HTTPReq.InsertParam("Episode",episodeNumber)
do HTTPReq.InsertParam("Stylesheet",styleSheet)
do HTTPReq.InsertParam("Server",..CacheServer)
Set Status = HTTPReq.Post(..Location,0) Quit:$$$ISERR(tSC)
Do ##class(sgh.Utils.Debug).LogDebugMsg("Sent the following request: "_Status)
Quit tSC
}
}
The thing is when I check the debug value (which is defined as a global) all I get is the number '1' - I have no idea therefore if the request has succeeded or even what is wrong (if it has not)
What do I need to do to find out
A) What is the actual web call being made?
B) What the response is?
There is a really slick way to get the answer the two questions you've asked, regardless of where you're using the code. Check the documentation out on the %Net.HttpRequest object here: http://docs.intersystems.com/ens20102/csp/docbook/DocBook.UI.Page.cls?KEY=GNET_http and the class reference here: http://docs.intersystems.com/ens20102/csp/documatic/%25CSP.Documatic.cls?APP=1&LIBRARY=ENSLIB&CLASSNAME=%25Net.HttpRequest
The class reference for the Post method has a parameter called test, that will do what you're looking for. Here's the excerpt:
method Post(location As %String = "", test As %Integer = 0, reset As %Boolean = 1) as %Status
Issue the Http 'post' request, this is used to send data to the web server such as the results of a form, or upload a file. If this completes correctly the response to this request will be in the HttpResponse. The location is the url to request, e.g. '/test.html'. This can contain parameters which are assumed to be already URL escaped, e.g. '/test.html?PARAM=%25VALUE' sets PARAM to %VALUE. If test is 1 then instead of connecting to a remote machine it will just output what it would have send to the web server to the current device, if test is 2 then it will output the response to the current device after the Post. This can be used to check that it will send what you are expecting. This calls Reset automatically after reading the response, except in test=1 mode or if reset=0.
I recommend moving this code to a test routine to view the output properly in terminal. It would look something like this:
// To view the REQUEST you are sending
Set sc = request.Post("/someserver/servlet/webmethod",1)
// To view the RESPONSE you are receiving
Set sc = request.Post("/someserver/servlet/webmethod",2)
// You could also do something like this to parse your RESPONSE stream
Write request.HttpResponse.Data.Read()
I believe the answer you want to A) is in the Server and Location properties of your %Net.HttpRequest object (e.g., HTTPReq.Server and HTTPReq.Location).
For B), the response information should be in the %Net.HttpResponse object stored in the HttpResponse property (e.g. HTTPReq.HttpResponse) after your call is completed.
I hope this helps!
-Derek
(edited for formatting)
From that code sample it looks like you're using Ensemble, not straight-up Cache.
In that case you should be doing this HTTP call in a Business Operation that uses the HTTP Outbound Adapter, not in your Business Process.
See this link for more info on HTTP Adapters:
http://docs.intersystems.com/ens20102/csp/docbook/DocBook.UI.Page.cls?KEY=EHTP
You should also look into how to use the Ensemble Message Browser. That should help with your logging needs.