I'm making a simple web app that looks for color words in a text, and plots statistics about them. You can test it at colors.jonreeve.com if it's not too busy. I'm using the Scotty web framework to handle the web stuff. It works OK for short texts, but longer texts, like full novels, take so long that the browser normally times out. So I'm guessing what I need here is to send the form via Jquery AJAX or something, and then have the server send JSON every so often with its status ("now loading file," "now counting colors," etc) and then when it receives a "success" signal, then redirect to some other URL?
This is my first time trying to do something like this, so forgive me if this all sounds uninformed. I also noticed that there are some similar questions out there, but I have a feeling that Scotty handles things a little differently than most setups. I noticed that there are a few functions for setting raw output, setting headers and so forth. Do I try to emit certain signals at each stage in the analysis? And how would I do that, given Haskell's handling of side-effects? I'm struggling to even think of the best approach, here.
Instead of a single long-running GET request, I would perhaps set up an endpoint accepting POST requests. The POST would return immediately with two links in the response body:
one link to a new resource representing the task result, which wouldn't be immediately available. Until then, GET requests to the result could return 409 (Conflict).
one link to a related, immediately available resource representing notifications emitted while performing the task.
Once the client has made a successful GET of the task result resource, it could DELETE it. That should delete both the task result resource and the associated notification resource.
For each POST request, you would need to spawn a background worker thread. You would also need a background thread for deleting task results that grew old (because the clients could be lazy and not invoke DELETE). These threads would communicate with MVars, TVars, channels or similar methods.
Now the question is: how to best handle the notifications emitted by the server? There are several options:
Just poll periodically the notification resource from the client. Disadvantages: potentially many HTTP requests, notifications are not received promptly.
long polling. A sequence of GET requests which are kept open until the server wants to emit some notification, or until a timeout.
server-sent events. wai-extra has support for this, but I don't know how to hook a raw wai Application back into Scotty.
websockets. Not sure how to integrate with Scotty though.
Here's the server-side skeleton of a long polling mechanism. Some preliminary imports:
{-# LANGUAGE NumDecimals #-}
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE TypeApplications #-}
import Control.Concurrent (threadDelay)
import Control.Concurrent.Async (concurrently_) -- from async
import Control.Concurrent.STM -- from stm
import Control.Concurrent.STM.TMChan -- from stm-chans
import Control.Monad.IO.Class (liftIO)
import Data.Aeson (ToJSON) -- from aeson
import Data.Foldable (for_)
import Data.Text (Text)
import Web.Scotty
And here is the main code.
main :: IO ()
main =
do
chan <- atomically $ newTMChan #Text
concurrently_
( do
for_
["starting", "working on it", "finishing"]
( \msg -> do
threadDelay 10e6
atomically $ writeTMChan chan msg
)
atomically $ closeTMChan chan
)
( scotty 3000
$ get "/notifications"
$ do
mmsg <- liftIO $ atomically $ readTMChan chan
json $
case mmsg of
Nothing -> ["closed!"]
Just msg -> [msg]
)
There are two concurrent threads. One feeds messages into a closeable channel at 10 second intervals, the other runs a Scotty server, where each GET invocation hangs until a new message arrives in the channel.
Testing it from bash using curl, we should see a succession of messages:
bash$ for run in {1..4}; do curl -s localhost:3000/notifications ; done
["starting"]["working on it"]["finishing"]["closed!"]
For comparison, here's the skeleton of a solution based on server-sent events. It uses yesod instead of scotty though, because Yesod offers a way to hook as a handler the wai-extra Application that manages the events.
The Haskell code
{-# LANGUAGE NumDecimals #-}
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE QuasiQuotes #-}
{-# LANGUAGE TemplateHaskell #-}
{-# LANGUAGE TypeFamilies #-}
import Control.Concurrent (threadDelay)
import Control.Concurrent.Async (concurrently_) -- from async
import Control.Concurrent.STM -- from stm
import Control.Concurrent.STM.TMChan -- from stm-chans
import Control.Monad.IO.Class (liftIO)
import Data.Binary.Builder -- from binary
import Data.Foldable (for_)
import Network.Wai.EventSource -- from wai-extra
import Network.Wai.Middleware.AddHeaders -- from wai-extra
import Yesod -- from yesod
data HelloWorld = HelloWorld (TMChan ServerEvent)
mkYesod
"HelloWorld"
[parseRoutes|
/foo FooR GET
|]
instance Yesod HelloWorld
getFooR :: Handler ()
getFooR = do
HelloWorld chan <- getYesod
sendWaiApplication
. addHeaders [("Access-Control-Allow-Origin", "*")]
. eventStreamAppRaw
$ \send flush ->
let go = do
mevent <- liftIO $ atomically $ readTMChan chan
case mevent of
Nothing -> do
send CloseEvent
flush
Just event -> do
send event
flush
go
in go
main :: IO ()
main =
do
chan <- atomically $ newTMChan
concurrently_
( do
for_
[ ServerEvent
(Just (fromByteString "ev"))
(Just (fromByteString "id1"))
[fromByteString "payload1"],
ServerEvent
(Just (fromByteString "ev"))
(Just (fromByteString "id2"))
[fromByteString "payload2"],
ServerEvent
(Just (fromByteString "ev"))
(Just (fromByteString "eof"))
[fromByteString "payload3"]
]
( \msg -> do
threadDelay 10e6
atomically $ writeTMChan chan msg
)
atomically $ closeTMChan chan
)
( warp 3000 (HelloWorld chan)
)
And a small blank page to test the server-sent events. The messages appear on the browser console:
<!DOCTYPE html>
<html lang="en">
<body>
</body>
<script>
window.onload = function() {
var source = new EventSource('http://localhost:3000/foo');
source.onopen = function () { console.log('opened'); };
source.onerror = function (e) { console.error(e); };
source.addEventListener('ev', (e) => {
console.log(e);
if (e.lastEventId === 'eof') {
source.close();
}
});
}
</script>
</html>
I am trying to upgrade from version 0.18 to 0.19 of Elm. My project depends on elm-lang/websocket in 0.18? I cannot seem to find the equivalent package in 0.19. What am I missing?
Here is a minimal working example of an interactive form to echo input from echo.websocket.org using 2 simple input/output ports for communicating with a JavaScript WebSocket object external to the elm 0.19 module:
File: echo.elm. Compile with: elm make echo.elm --output=echo.js
port module Main exposing (main)
import Browser
import Html exposing (Html)
import Html.Attributes as HA
import Html.Events as HE
import Json.Encode as JE
-- JavaScript usage: app.ports.websocketIn.send(response);
port websocketIn : (String -> msg) -> Sub msg
-- JavaScript usage: app.ports.websocketOut.subscribe(handler);
port websocketOut : String -> Cmd msg
main = Browser.element
{ init = init
, update = update
, view = view
, subscriptions = subscriptions
}
{- MODEL -}
type alias Model =
{ responses : List String
, input : String
}
init : () -> (Model, Cmd Msg)
init _ =
( { responses = []
, input = ""
}
, Cmd.none
)
{- UPDATE -}
type Msg = Change String
| Submit String
| WebsocketIn String
update : Msg -> Model -> (Model, Cmd Msg)
update msg model =
case msg of
Change input ->
( { model | input = input }
, Cmd.none
)
Submit value ->
( model
, websocketOut value
)
WebsocketIn value ->
( { model | responses = value :: model.responses }
, Cmd.none
)
{- SUBSCRIPTIONS -}
subscriptions : Model -> Sub Msg
subscriptions model =
websocketIn WebsocketIn
{- VIEW -}
li : String -> Html Msg
li string = Html.li [] [Html.text string]
view : Model -> Html Msg
view model = Html.div []
--[ Html.form [HE.onSubmit (WebsocketIn model.input)] -- Short circuit to test without ports
[ Html.form [HE.onSubmit (Submit model.input)]
[ Html.input [HA.placeholder "Enter some text.", HA.value model.input, HE.onInput Change] []
, model.responses |> List.map li |> Html.ol []
]
]
Embed the compiled echo.js into echo.html:
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>Echo</title>
<script src="echo.js"></script>
</head>
<body>
<div id="elm-node"></div>
<script>
var app = Elm.Main.init({node: document.getElementById("elm-node")});
var ws = new WebSocket("wss://echo.websocket.org");
ws.onmessage = function(message)
{
console.log(message);
app.ports.websocketIn.send(JSON.stringify({data:message.data,timeStamp:message.timeStamp}));
};
app.ports.websocketOut.subscribe(function(msg) { ws.send(msg); });
</script>
</body>
</html>
This works on Firefox 60.2.0esr on Linux but has not been tested on other platforms.
Again, this is only a minimal example to demonstrate how to use ports with WebSockets for Elm 0.19. It does not included closing the WebSocket, error handling, etc. but hopefully this example can help you get started in that direction. It is expected that WebSockets will be directly supported by Elm again soon, so this is only a temporary work-around. If you don't need to upgrade to 0.19, then consider staying with 0.18 instead.
The websocket package is currently redesigned for Elm 0.19, see this issue:
This package has not been updated for 0.19 yet. I have heard lots of folks saying they need more features from this package, so I'd rather take that into consideration in the update rather than just doing the same stuff. I recommend using ports or 0.18 if you absolutely need this right this second.
EDIT: April 15, 2020 update
The package has been archived and the Readme file updated as
follows:
The recommended way to use WebSockets with Elm for now is through
ports. You can see a minimal example in the js-integration-examples
repo [IMAGE CLIPPED]
History
We had a bare bones version of WebSockets in within Elm in versions
0.17 and 0.18, part of the introduction of subscriptions to Elm. But users found that the API was not able to cover a lot of situations
they faced in practice. How can this work with Elixir Pheonix?
Firebase? How can I use a different backoff strategy for reconnecting?
How can I hear about when the connection goes down or comes back? How
about sub-protocols?
In trying to expand the API to cover all the cases people were facing
in practice, I came to think that it may not be possible with the
current subscriptions infrastructure. (My feeling is that effect
managers may not be a great fit for web sockets because they do not
have great mechanisms for uniquely identifying resources. Do we have
one connections or two? How do we tell the difference? If it requires
function pointer equality, how can we make that reliable when an
anonymous function is used?) I did not understand this problem as well
in 2016, and I think it manifested most clearly with web sockets.
So facing the prospect of either (1) having an API that many
eventually had to leave behind for ports or (2) recommending that
people go with ports from the start, we figured that (2) was probably
the best for someone new coming to Elm. That way they would hook up to
their preferred web socket manager without the intermediate step of
learning a promising but incomplete API.
I have an application that listens for a cell click in a DT in any cell and then updates a plot accordingly. The program works perfectly when I runApp() locally. However when I depoloy the app on a shiny server, the click no longer triggers any actions. This discepancy does not exists for other action listeners such as a simple refresh button, as I have demonstrated in the code below. You can see how the discrepancy between remote and local does not exist for the refresh button condition input$refreshButton!=0, but there is a discrepancy using the length(input$table_cell_clicked)>0 trigger condition.
I have done some research into this error and this is what I know so far:
1) I am getting the warning "Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user's experience." in the console when the app is deployed remotely. I am told this has something to do with a setting in a file in my shiny server called javascript.min.js and jquery.min.js that says c.async="false" I have searched for every file on my serer with that name or containing the string async="false", and changed the setting to sync="true". However I did not find any files with the exact string c.async="true". I can see the file with this string in the browser console, which gives a location relative to server::port/, but I do not know where that file actually lives on my system, and I suspect it is just a file made on the fly by shiny services.
2) It is possible that this could be fixed with something related to the selectize functionality in some shiny inputs, which may cause the code to execute asynchronously(?). I have tried a few different things but couldn't get any to solve the problem.
3) There is a commonly known annoyance with shiny that it is generally hard to debug. In my case, it would be extremely helpful if I could see the output of the server.R functions as I would when using runApp() locally. Using a call to browser, options(shiny.trace = T) were both recommended, but when I add them to the code below, nothing apprears in the console output. I even tried using sink in order to save to output to some file on the remote server, and it rusn without error, but I do not see any file in the location indicated. If I could at least see the output of this file, or the request/response messages between the server and the client it would go a long way towards debugging this.
So the two questions are: how can I see these messages/output when the app is deployed remotely? And more importantly, how can I implement a fix so that all of my hard work on this project (unfortunately can't disclose any details) will not be a waste.
The basic code for my shiny app is below.
server.R:
server <- function(input, output, session) {
options(shiny.trace = T)
browser()
sink("~/outputfile.txt",append = T,type = "output",split = T)
end_date=as.character(as.Date(Sys.Date()-10))
library(DT)
library(data.table)
library(xtable)
library(zoo)
library(lattice)
library(RSQLite)
output$table = DT::renderDataTable({
thisTable = head(iris)
return(thisTable)
},server = T,options = list(target = 'cell'))
output$plot1 <- renderPlot({
cell= as.numeric(input$table_cell_clicked)
print(cell)
row = as.numeric(input$transtable_row_last_clicked)
print(paste0("last row clicked: ",row))
print(paste0("timestamp: ",Sys.time()))
(cell, file = "/home/plintilhac/cell_file.txt") ## causes error that dumps SND and REC messages to javascript console
# if (length(input$row_last_clicked)>0){ ##works remotely and locally
# if (input$refreshButton!=0){ ##works remotely and locally
if (length(cell)>0){ #works locally, but doesn't work remotely
plot(0,0,xlim = c(-1,1),ylim = c(-1,1))
}
else{return(plot(0,1,xlim = c(-1,1),ylim = c(-1,1)))}}
)
output$text1 <- renderText({
if (input$refreshButton!=0){
"clicked"
}
else{"unclicked"}
})
}
ui.R
shinyUI(
fluidPage(
fluidRow(plotOutput("plot1",click = "plot_click"),theme = "bootstrap.css"),
mainPanel(
DT::dataTableOutput('table'),
fluidRow(
actionButton("refreshButton", "refresh")
)
)
))
EDIT:
I was able to get some output by placing an erroneous line of code right after the cell variable is defined, causing the shiny server to dump output to the javascript console. At this time this is the only way I know how to capture any output. However, the output is quite informative, as it shows that the table_cell_clicked attribute is not being exported on the remote server at all, whereas other attributes such as row_last_clicked are.
here is the output I get when the server is run locally ithout the erroneous line (note it includes table_cell_clicked as a feature):
SEND
{"config":{"workerId":"","sessionId":"ef292cd0c98baee4afa504aa8330b49e"}}
RECV
{"method":"init","data":{"refreshButton:shiny.action":0,".clientdata_output_plot1_width":873,".clientdata_output_plot1_height":400,".clientdata_output_plot1_hidden":false,".clientdata_output_table_hidden":false,".clientdata_pixelratio":1.100000023841858,".clientdata_url_protocol":"http:",".clientdata_url_hostname":"d2rm01",".clientdata_url_port":"8787",".clientdata_url_pathname":"/p/4944/",".clientdata_url_search":"",".clientdata_url_hash_initial":"",".clientdata_singletons":"",".clientdata_allowDataUriScheme":true}}
SEND
{"errors":[],"values":{"table":{"x":{"filter":"none","container":"<table
class=\"display\">\n <thead>\n <tr>\n <th> </th>\n
<th>Sepal.Length</th>\n <th>Sepal.Width</th>\n
<th>Petal.Length</th>\n <th>Petal.Width</th>\n
<th>Species</th>\n </tr>\n
</thead>\n</table>","options":{"target":"cell","selectize":true,"columnDefs":[{"className":"dt-right","targets":[1,2,3,4]},{"orderable":false,"targets":0}],"order":[],"autoWidth":false,"orderClasses":false,"ajax":{"url":"session/ef292cd0c98baee4afa504aa8330b49e/dataobj/table?w=","type":"POST","data":"function(d)
{\nd.search.caseInsensitive = true;\nd.escape = true;\nvar encodeAmp
= function(x) { x.value = x.value.replace(/&/g, \"%26\"); }\nencodeAmp(d.search);\n$.each(d.columns, function(i, v)
{encodeAmp(v.search);});\n}"},"serverSide":true,"processing":true},"selection":{"mode":"multiple","selected":null,"target":"row"}},"evals":["options.ajax.data"],"deps":[{"name":"datatables","version":"1.10.7","src":{"file":"/home/plintilhac/R/x86_64-pc-linux-gnu-library/3.2/DT/htmlwidgets/lib/datatables/js","href":"datatables-1.10.7"},"meta":null,"script":"jquery.dataTables.min.js","stylesheet":null,"head":null,"attachment":null},{"name":"datatables-default","version":"1.10.7","src":{"file":"/home/plintilhac/R/x86_64-pc-linux-gnu-library/3.2/DT/htmlwidgets/lib/datatables/css/default","href":"datatables-default-1.10.7"},"meta":null,"script":[],"stylesheet":["dataTables.extra.css","jquery.dataTables.min.css"],"head":null,"attachment":null}]},"plot1":{"src":"data:image/png;[base64
data]","width":873,"height":400,"coordmap":[{"domain":{"left":-1.08,"right":1.08,"bottom":-1.08,"top":1.08},"range":{"left":58.9093125,"right":842.8269375,"bottom":325.745454545455,"top":57.8909090909091},"log":{"x":null,"y":null},"mapping":{}}]}},"inputMessages":[]}
RECV
{"method":"update","data":{"table_rows_selected":[],"table_rows_current":[],"table_rows_all":[],"table_state":null,"table_search":"","table_cell_clicked":{}}}
SEND {"progress":{"type":"binding","message":{"id":"plot1"}}} SEND
{"errors":[],"values":{"plot1":{"src":"data:image/png;[base64
data]","width":873,"height":400,"coordmap":[{"domain":{"left":-1.08,"right":1.08,"bottom":-1.08,"top":1.08},"range":{"left":58.9093125,"right":842.8269375,"bottom":325.745454545455,"top":57.8909090909091},"log":{"x":null,"y":null},"mapping":{}}]}},"inputMessages":[]}
RECV
{"method":"update","data":{"table_rows_current":[1,2,3,4,5,6],"table_rows_all":[1,2,3,4,5,6]}}
RECV {"method":"update","data":{"plot_click":null}} RECV
{"method":"update","data":{"table_cell_clicked":{"row":1,"col":2,"value":3.5},"table_rows_selected":[1],"table_row_last_clicked":1}}
SEND {"progress":{"type":"binding","message":{"id":"plot1"}}} SEND
{"errors":[],"values":{"plot1":{"src":"data:image/png;[base64
data]","width":873,"height":400,"coordmap":[{"domain":{"left":-1.08,"right":1.08,"bottom":-1.08,"top":1.08},"range":{"left":58.9093125,"right":842.8269375,"bottom":325.745454545455,"top":57.8909090909091},"log":{"x":null,"y":null},"mapping":{}}]}},"inputMessages":[]}
RECV {"method":"update","data":{"table_rows_selected":[]}}
while this is the output when it is run remotely with the erroneous line (note table_cell_clicked is not being received):
Loading required package: DBI
SEND {"errors":[],"values":{"table":{"x":{"container":"<table class=\"display\">\n <thead>\n <tr>\n <th> </th>\n <th>Sepal.Length</th>\n <th>Sepal.Width</th>\n <th>Petal.Length</th>\n <th>Petal.Width</th>\n <th>Species</th>\n </tr>\n </thead>\n</table>","options":{"target":"cell","selectize":true,"columnDefs":[{"className":"dt-right","targets":[1,2,3,4]},{"orderable":false,"targets":0}],"order":[],"autoWidth":false,"orderClasses":false,"ajax":{"url":"session/07190712bb533d7cf1929522b19e436a/dataobj/table?w=","type":"POST","data":"function(d) {\nd.search.caseInsensitive = true;\nd.escape = true;\n}"},"serverSide":true,"processing":true},"callback":null,"filter":"none","selection":"multiple"},"evals":["options.ajax.data"],"deps":[{"name":"datatables","version":"1.10.7","src":{"file":"/usr/local/lib/R/site-library/DT/htmlwidgets/lib/datatables/js","href":"datatables-1.10.7"},"meta":null,"script":"jquery.dataTables.min.js","stylesheet":null,"head":null,"attachment":null},{"name":"datatables-default","version":"1.10.7","src":{"file":"/usr/local/lib/R/site-library/DT/htmlwidgets/lib/datatables/css/default","href":"datatables-default-1.10.7"},"meta":null,"script":[],"stylesheet":["dataTables.extra.css","jquery.dataTables.min.css"],"head":null,"attachment":null}]},"plot1":{"src":"data:image/png;[base64 data]","width":1745,"height":400,"coordmap":[{"domain":{"left":-1.08,"right":1.08,"bottom":-1.08,"top":1.08},"range":{"left":58.9062532569046,"right":1714.82850442939,"bottom":325.745454545455,"top":57.8909090909091},"log":{"x":null,"y":null},"mapping":{}}]}},"inputMessages":[]}
RECV {"method":"update","data":{"table_rows_selected":[],"table_rows_current":[],"table_rows_all":[],"table_state":null,"table_search":""}}
RECV {"method":"update","data":{"table_rows_current":["1","2","3","4","5","6"],"table_rows_all":["1","2","3","4","5","6"]}}
RECV {"method":"update","data":{"table_rows_selected":["3"],"table_row_last_clicked":"3"}}
RECV {"method":"update","data":{".clientdata_output_plot1_width":463}}
SEND {"progress":{"type":"binding","message":{"id":"plot1"}}}
SEND {"errors":[],"values":{"plot1":{"src":"data:image/png;[base64 data]","width":463,"height":400,"coordmap":[{"domain":{"left":-1.08,"right":1.08,"bottom":-1.08,"top":1.08},"range":{"left":58.9256188605108,"right":432.81858546169,"bottom":325.745454545455,"top":57.8909090909091},"log":{"x":null,"y":null},"mapping":{}}]}},"inputMessages":[]}
RECV {"method":"update","data":{"plot_click":null}}
SEND {"config":{"workerId":"","sessionId":"7b20c500ee810e198324a75b6512a353"}}
RECV {"method":"init","data":{"refreshButton:shiny.action":0,"ss-net-opt-websocket":true,"ss-net-opt-xdr-streaming":true,"ss-net-opt-xhr-streaming":true,"ss-net-opt-iframe-eventsource":true,"ss-net-opt-iframe-htmlfile":true,"ss-net-opt-xdr-polling":true,"ss-net-opt-xhr-polling":true,"ss-net-opt-iframe-xhr-polling":true,"ss-net-opt-jsonp-polling":true,".clientdata_output_plot1_width":463,".clientdata_output_plot1_height":400,".clientdata_output_plot1_hidden":false,".clientdata_output_table_hidden":false,".clientdata_pixelratio":1.100000023841858,".clientdata_url_protocol":"http:",".clientdata_url_hostname":"d2rm01",".clientdata_url_port":"3838",".clientdata_url_pathname":"/testFunnel/",".clientdata_url_search":"",".clientdata_url_hash_initial":"",".clientdata_singletons":"",".clientdata_allowDataUriScheme":true}}
Error in source(file, ..., keep.source = TRUE, encoding = checkEncoding(file)) :
/srv/shiny-server/testFunnel/server.R:38:10: unexpected ','
37: #print(paste0("timestamp: ",Sys.time()))
38: (cell,