Can Vertica call http get function?
IF yes, how?
I have a link i need to access from Vertica.
It appears that Vertica may be able to use cURL to open and read a file over HTTP using a subclass of UDSource (C++):
The UDSource class is responsible for acquiring the data from an external source and producing that data in a streaming manner. A wrapper is also provided for UDSource called ContinuousUDSource. ContinuousUDSource provides an abstraction that allows you to treat the input data as a continuous stream of data. This allows you to write the data from the source "at will" instead of having to create an iterator to use the base UDSource method.
Example UDSource
class CurlSource : public UDSource {
private:
URL_FILE *handle;
std::string url;
virtual StreamState process(ServerInterface &srvInterface, DataBuffer &output) {
output.offset = url_fread(output.buf, 1, output.size, handle);
return url_feof(handle) ? DONE : OUTPUT_NEEDED;
}
public:
CurlSource(std::string url) : url(url) {}
void setup(ServerInterface &srvInterface) {
handle = url_fopen(url.c_str(),"r");
}
void destroy(ServerInterface &srvInterface) {
url_fclose(handle);
}
};
Additionally, it appears that you can use the standard HDFS connector:
COPY testTable SOURCE Hdfs(url='http://hadoop:50070/webhdfs/v1/tmp/test.txt', username='hadoopUser');
Related
Protobuf provides service keyword that defines rpc-interface of one application.
I also want to use concept of entity which means that is part of service (one service contains multiple entities). Each entity type has own unique identifier that gives possibility to address different entities in service.
I would like to use proto like this
message UserReq {
string username = 1;
string password = 2;
}
message RegReq {
uint8 result_code = 1;
}
message RemoteEntityInterface
{
MyEntity entity = 1;
}
message GiveItemResult
{
uint8 result_code = 1;
}
service MyService {
rpc RegisterUser (UserReq) returns (RegReq) {}
rpc Login(UserReq) returns (RemoteEntityInterface) {}
}
entity MyEntity
{
rpc GiveItem (GiveItemReq) returns (GiveItemResult) {}
}
As you can see in example, I used unknown for protobuf keyword entity, this keyword means that MyService can return the interface to some remote object (MyEntity) by using Login remote method.
What are the ways to do this? (maybe write plugin or known way to modify source code of protobuf). Or maybe there are more flexible solutions than protobuf?
I also would like to use multiple parameters per one rpc; adding java-like attributes to rpc; service and entity; and data-model for entity (variables/fields) to add real-time replication support from entity to another service.
I think it is very flexible for services in game-development.
The only official way to extend .proto syntax is to define custom options.
For example, you could have something like:
extend google.protobuf.ServiceOptions {
optional bool is_entity = 123456;
}
service MyEntity
{
option (is_entity) = true;
rpc GiveItem (GiveItemReq) returns (GiveItemResult) {}
}
The default code generator will not do anything special with this option, but you can access it from your own code and from a protoc plugin if you write one.
Referring to the following example:
public static void Run([CosmosDBTrigger(
databaseName: "ToDoItems",
collectionName: "Items",
ConnectionStringSetting = "CosmosDBConnection",
LeaseCollectionName = "leases",
CreateLeaseCollectionIfNotExists = true)]IReadOnlyList<Document> documents,
ILogger log)
I understand, the connectionStringSetting isn't the connection string to use, rather it's name of the setting to look up containing the ConnectionString.
Will this also work for CollectionName and databasename as well? I understand I can experiment and figure out, but I am confused as to how this is even resolved at build time/deployment time?
I see several properties being assigned values while others are taking them from configuration? Is it the underlying constructor for CosmosDBTrigger which takes care of using appropriate value?
Binding to a function is a way of declaratively connecting another resource to the function; bindings may be connected as input bindings, output bindings, or both. Data from bindings is provided to the function as parameters.
here is small sample of Azure function using CosmosDB trigger that is invoked when there are inserts or updates in the specified database and collection.
using Microsoft.Azure.Documents;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Host;
using System.Collections.Generic;
using Microsoft.Extensions.Logging;
namespace CosmosDBSamplesV2
{
public static class CosmosTrigger
{
[FunctionName("CosmosTrigger")]
public static void Run([CosmosDBTrigger(
databaseName: "ToDoItems",
collectionName: "Items",
ConnectionStringSetting = "CosmosDBConnection",
LeaseCollectionName = "leases",
CreateLeaseCollectionIfNotExists = true)]IReadOnlyList<Document> documents,
ILogger log)
{
if (documents != null && documents.Count > 0)
{
log.LogInformation($"Documents modified: {documents.Count}");
log.LogInformation($"First document Id: {documents[0].Id}");
}
}
}
}
and here is the binding information of same azure function which is used to pass the param value to function
Cosmos DB trigger binding in a function.json file
{
"type": "cosmosDBTrigger",
"name": "documents",
"direction": "in",
"leaseCollectionName": "leases",
"connectionStringSetting": "<connection-app-setting>",
"databaseName": "Tasks",
"collectionName": "Items",
"createLeaseCollectionIfNotExists": true
}
To answer your question how this is even resolved at build time/deployment time" :- To use it locally we pass the same binding information in host.json file and local.settings.json file.
That's how it bind the information internally by checking param name.
Hope it helps.
I have a straightforward use case. This is to make a rest call, query mongo and then return an arbitrarily large stream of data back to the client, all with reactive streams type back pressure management.
This was quite easy to achieve using Spring WebFlux and Reactor. I am now trying to achieve the same goal using vert.x, as a comparison of ease of implementation.
Having found the vert.x mongo client to be lacking any support for managing back pressure, I am now attempting to use the WebFlux mongo client and then pump the data back through the vert.x HttpResponse, as shown in the following code:
public class MyMongoVerticle extends AbstractVerticle {
ReactiveMongoOperations operations;
public void start() throws Exception {
final Router router = Router.router(vertx);
router.route().handler(BodyHandler.create());
router.get("/myUrl").handler(ctx -> {
// WebFlux mongo operations returns a ReactiveStreams compatible entity
Flux<Document> mongoStream = operations.findAll(Document.class, "myCollection");
ReactiveReadStream rrs = ReactiveReadStream.readStream();
// rrs is ReactiveStream streams subscriber
mongoStream.subscribe(rrs);
// Pump pumps the rrs (ReactiveReadStream) to the HttpServerResponse (ReactiveWriteStream)
Pump pump = Pump.pump(rrs, ctx.response());
pump.start();
});
vertx.createHttpServer().requestHandler(router::accept).listen(8777);
}
}
The issue I have encountered is that the HttpServerResponse implements ReactiveWriteStream<Buffer> so is expecting a Buffer rather than a stream of Document's. The result is a ClassCaseException.
The question I have is how can I convert this stream of Documents into a into a ReactiveWriteStream<Buffer>? There may be another better way to do this, so I'm open to other suggestions on how to achieve this.
Pump won't work for you, as it doesn't support transformations currently. You'll have to implement pump by yourself. Luckily, this shouldn't be too hard:
Flux<Document> mongoStream = operations.findAll(Document.class, "myCollection");
ReactiveReadStream<Document> rrs = ReactiveReadStream.readStream();
mongoStream.subscribe(rrs);
HttpServerResponse outStream = ctx.response();
// Changes start here
rrs.handler(d -> {
if (outStream.writeQueueFull()) {
outStream.drainHandler((s) -> {
rrs.resume();
});
rrs.pause();
}
else {
outStream.write(d.toJson());
}
}).endHandler(h -> {
outStream.end();
});
Note that I wouldn't expect this to be more effective than "native" WebFlux implementation.
Also, JSON in this example will be mangled, as I don't wrap it in proper JSON Array
I'd like to provide read access for certain properties to all classes/structs that implement a protocol while client classes of the protocol are allowed read+write access. Is there a way to do this in Swift?
protocol WheelsProtocol {
var count: Int {get set}
}
struct Car: WheelsProtocol {
var count: Int = 0
func checkTirePressure() {
// Here, we will iterate over the count of wheels but we should
// not allow the number of wheels to be changed
}
}
struct CarFactory {
var wheels: WheelsProtocol
init(wheels: WheelsProtocol) {
self.wheels = wheels
}
mutating func configureVehicle() {
self.wheels.count = 4
}
}
What about a protocol for car makers and a different one for cars, something like
protocol MakesCars {
var wheelCount: Int {get set}
}
protocol HasWheels{
var wheelCount: Int { get }
}
struct Car: HasWheels {
let wheelCount: Int
init(wheelCount: Int) {
self.wheelCount = wheelCount
}
func checkTirePressure() {
// Here, we will iterate over the count of wheels but we should
// not allow the number of wheels to be changed
wheelCount = 5 //COMPILER ERROR
}
}
struct CarFactory: MakesCars {
...
}
The key is that you have to define a read-only property in the protocol as a var with { get } but in the object that adopts that protocol you have to put let and set it in an initializer.
There is not a way to limit access to a particular method in the way you're describing. From the documentation on Access Control:
Swift provides three different access levels for entities within your
code. These access levels are relative to the source file in which an
entity is defined, and also relative to the module that source file
belongs to.
Public access enables entities to be used within any source file from
their defining module, and also in a source file from another module
that imports the defining module. You typically use public access when
specifying the public interface to a framework.
Internal access enables entities to be used within any source file
from their defining module, but not in any source file outside of that
module. You typically use internal access when defining an app’s or a
framework’s internal structure.
Private access restricts the use of an entity to its own defining
source file. Use private access to hide the implementation details of
a specific piece of functionality.
To accomplish this, you would need to create a separate module (i.e., a Framework) and limit the writes to the internal scope and make the reads of the public scope.
I've used WebAPI for a while, and generally set it to use camel case json serialization, which is now rather common and well documented everywhere.
Recently however, working on a much larger project, I came across a more specific requirement: we need to use camel case json serialization, but because of backward compatibility issues with our client scripts, I only want it to happen for specific actions, to avoid breaking other parts of the (extremely large) website.
I figure one option is to have a custom content type, but that then requires client code to specify it.
Is there any other option?
Thanks!
Try this:
public class CamelCasingFilterAttribute : ActionFilterAttribute
{
private JsonMediaTypeFormatter _camelCasingFormatter = new JsonMediaTypeFormatter();
public CamelCasingFilterAttribute()
{
_camelCasingFormatter.SerializerSettings.ContractResolver = new CamelCasePropertyNamesContractResolver();
}
public override void OnActionExecuted(HttpActionExecutedContext actionExecutedContext)
{
ObjectContent content = actionExecutedContext.Response.Content as ObjectContent;
if (content != null)
{
if (content.Formatter is JsonMediaTypeFormatter)
{
actionExecutedContext.Response.Content = new ObjectContent(content.ObjectType, content.Value, _camelCasingFormatter);
}
}
}
}
Apply this [CamelCasingFilter] attribute to any action you want to camel-case. It will take any JSON response you were about to send back and convert it to use camel casing for the property names instead.