I'm new to clojure and I am trying to figure out what Jepsen does, a software used to check consistency of distributed system.
My questions are:
What does extend-protocol do?
To be more specific, In Jepsen.Generator, what is op in the Protocol.
What does mix mean in the context
Kind Regards
What does extend-protocol do?
A protocol is an abstract thing that looks like an interface in Java. It does nothing, but some other entities may implement it. In Java, you declare a class that implements an interface. In Clojure, you extend a
particular protocol with a custom type declared either with deftype or defrecord calling extend-protocol on it.
When extending a protocol with a type, you need to specify implementation for signatures mentioned in that protocol.
A good example might be JSON serialization. Say, the protocol would look like as follows:
(defprotocol JSON
(to-json [obj]))
If you call (to-json ...) on any value, you'll have an error saying that there is no to-json implementation for that type. You need to extend it:
(extend-protocol JSON
Integer
(to-json [obj] (str obj))
Boolean
(to-json [obj]
(if obj "true" "false")))
Now that, calling (to-json 42) and (to-json false) will work. You may extend that protocol for the rest of types: floats, array, maps and so on.
what is op in the Protocol
Protocols do not have implementations, only signatures. os is a signature of some function that just takes three arguments: [gen test process]. It's up to what should it return. As an example, you may refer the line #46 where its behavior is implemented for the clojure.lang.AFunction type. Since I'm not familiar with jepsen, I cannot say more on that.
What does mix mean in the context
I think its docstring is pretty clear as well as the code is. I takes a collection of gens. If it's empty, the result would be a special Generator instance named void. It's an anonymous type that extends Generator protocol returning just nil when calling op without any computations.
It the gens are not empty, the code returns an instance of Generator type with such op implementation that takes a random gen when executing op.
Hope that will help.
I'm trying to marshal an interface to another thread.
Windows provides the convenient CoMarshalInterThreadInterfaceInStream helper function to take care of the boilerplate code associated with using CoMarshalInterface directly.
const Guid CLSID_Widget = "{F8383852-FCD3-11d1-A6B9-006097DF5BD4}";
const Guid IID_IWidget = "{EBBC7C04-315E-11D2-B62F-006097DF5BD4}";
//Create our widget
HRESULT hr = CoCreateInstance(CLSID_Widget, null,
CLSCTX_INPROC_SERVER | CLSCTX_LOCAL_SERVER,
IID_IWidget, out widget);
OleCheck(hr);
//Marshall the interface into an IStream
IStream stm;
hr = CoMarshalInterThreadInterfaceInStream(IID_IWidget, widget, out stm);
OleCheck(hr);
Except that the call to CoMarshalThreadInterfaceInStream fails with:
REGDB_E_IIDNOTREG (0x80040155)
Interface not registered
Go directly to CoMarshalInterface
The COM API function CoMarshalInterThreadInterfaceInStream provides a simple wrapper around CreateStreamOnHGlobal and CoMarshalInterface, as shown here:
// from OLE32.DLL (approx.)
HRESULT CoMarsha1InterThreadInterfaceInStream(
REFIID riid, IUnknown *pItf, IStream **ppStm)
{
HRESULT hr = CreateStreamOnHGlobal(0, TRUE, ppStm);
if (SUCCEEDED(hr))
hr = CoMarshalInterface(*ppStm, riid, pItf,
MSHCTX_INPROC, 0, MSHLFLAGS_NORMAL);
return hr;
}
So we can try ourselves.
IStream stm = new Stream()
hr = CoMarshallInterface(stm, IID_IWidget, widget,
MSHCTX_INPROC, // destination context is in-process/same host
NULL, // reserved, must be null
MSHLFLAGS_NORMAL // marshal once, unmarshal once
);
OleCheck(hr);
But that fails with:
REGDB_E_IIDNOTREG (0x80040155)
Interface not registered
Use standard marshaling
My class does not implement IMarhsal interface. This is right and normal.
By default, when CoMarshalInterface is first called on an object, the object is asked whether it wishes to handle its own cross-apartment communications. This question comes in the form of a QueryInterface request for the IMarshal interface. Most objects do not implement the IMarshal interface and fail this
QueryInterface request, indicating that they are perfectly happy to let COM
handle all communications via ORPC calls. Objects that do implement the
IMarshal interface are indicating that ORPC is inappropriate and that the object implementor would prefer to handle all cross-apartment communications
via a custom proxy. When an object implements the IMarshalinterface all references to the object will be custom marshaled.
When an object does not implement the IMarshal interface, all references to the object will be standard marshaled. Most objects elect to use standard marshaling.
So the the question becomes why is the standard COM marshaler having so much problems? What is the source of the error Interface not registered?
The interface is, in fact, not registered
The requires for COM are undocumented, but i can tell you that my interface GUID does not exist in:
HKEY_CLASSES_ROOT\Interface\{EBBC7C04-315E-11D2-B62F-006097DF5BD4}
The reason for this will be explained at the end.
I know that Windows provides you the CoRegisterPSClsid function that allows you to register an interface inside your process, so that the standard marshaler will be able to marshal it:
Enables a downloaded DLL to register its custom interfaces within its running process so that the marshaling code will be able to marshal those interfaces.
HRESULT CoRegisterPSClsid(
_In_ REFIID riid,
_In_ REFCLSID rclsid
);
Parameters:
riid [in]: A pointer to the IID of the interface to be registered.
rclsid [in]: A pointer to the CLSID of the DLL that contains the proxy/stub code for the custom interface specified by riid.
Which i can try calling, but what clsid do i use?
CoRegisterPSClsid(IID_IWidget, ???);
What is the CLSID of the DLL that contains the proxy/stub code for the custom interface specified by riid? Do i use my class itself?
CoRegisterPSClsid(IID_IWidget, CLSID_Widget);
That doesn't sound right; but i don't understand the COM standard marshaler well enough. Doesn't that CLSID have to be one of the standard COM marsharling classes; implementing IPSFactoryBuffer?
Either way, it doesn't work. I still get the error "Interface not registered".
Register the interface
I of course can register my interface in the registry:
HKEY_CURRENT_USER\Software\Classes\Interface\{EBBC7C04-315E-11D2-B62F-006097DF5BD4}
(default) = "IWidget"
But that doesn't fix it. Spelunking through the Interface registry keys, i notice that many specify a ProxyStubClsid32 entry.
When a new interface is requested on an object, the proxy and stub managers
must resolve the requested IID onto the CLSID of the interface marshaler.
Under Windows NT 5.0, the class store maintains these mappings in the NT
directory, and they are cached at each host machine in the local registry. The
machine-wide IID-to-CLSID mappings are cached at
HKEY_CLASSES_ROOT\Interface
and the per-user mappings are cached at
HKEY_CURRENT_USER\Software\Classes\Interface
One or both of these keys will contain a subkey for each known interface. If the interface has an interface marshaler installed, there will be an additional
subkey (ProxyStubClsid32) that indicates the CLSID of the interface
marshaler.
Except what class implements marshaling? I don't have a marshaler.
Can COM automatically marshal based on a TypeLibrary
Is it possible that if i register a Type Library with my Interface, that COM's standard marshaler will be able to bootstrap a proxy class on the fly?
I registered my interface above. Now i manually include the TypeLibrary:
HKEY_CURRENT_USER\Software\Classes\Interface\{EBBC7C04-315E-11D2-B62F-006097DF5BD4}\TypeLib
(default) = "{38D528BD-4948-4F28-8E5E-141A51090580}"
And if i monitor the registry during the call to CoMarshalInterface i see that it attempts, and does find, my Interface IID:
Operation: RegOpenKey
Path: HKCR\WOW6432Node\Interface\{EBBC7C04-315E-11D2-B62F-006097DF5BD4}
Result: SUCCESS
It then tries to look for a ProxyStubClsid32, and fails:
Operation: RegOpenKey
Path: HKCR\WOW6432Node\Interface\{668790E3-83CC-47E0-907F-A44BA9A99C8D}\ProxyStubClsid32
Result: NAME NOT FOUND
My hope would then be that the standard COM marshaler attempts to look for:
Operation: RegOpenKey
Path: HKCR\WOW6432Node\Interface\{668790E3-83CC-47E0-907F-A44BA9A99C8D}\TypeLib
But it doesn't.
The OLE Automation marshaler
According to Don Box, the Ole Automation marshaler (PSOAInterface - {00020424-0000-0000-C000-000000000046}) is able to build a stub/proxy out of a type library:
The Type Library Marshaler
When these specially annotated interfaces are encountered by RegisterTypeLib (or LoadTypeLib in legacy mode), COM adds ProxyStubClsid32 entries for the interface with the value {00020424-0000-0000-C0000-000000000046}. This GUID corresponds to the class PSOAInterface that is registered as living in OLEAUT32.DLL, the OLE automation DLL. Because of the DLL that it lives in, this marshaler is sometimes called the [oleautomation] marshaler, although it is also called the type library marshaler or the universal marshaler. I'll refer to it as the type library marshaler, since it really has very little to do with IDispatch. (In fact, it is common to use the [oleautomation] attribute on interfaces that don't derive from IDispatch directly or indirectly.)
The class factory for the type library marshaler does something very tricky in its CreateProxy and CreateStub routines. Rather than return a statically compiled vtable (which is impossible given the fact that the requested interface didn't exist when OLEAUT32.DLL was built as part of the OS), the type library marshaler actually builds an /Oicf-style proxy and stub based on the type library for the interface. Because there is no efficient way to find the ITypeInfo for an arbitrary interface, the LIBID and version of the interface's type library must be stored under the:
HKCR\Interface\{XXX}\TypeLib
registry key.
I tried to set PSOAInterface as my interface's ProxyStub class:
Register standard COM class PSOAInterface as our proxy stub clsid
HKEY_CURRENT_USER\Software\Classes\Interface\{EBBC7C04-315E-11D2-B62F-006097DF5BD4}\ProxyStubClsid32
(default) = "{00020424-0000-0000-C0000-000000000046}"
Register our type library for our interface
HKEY_CURRENT_USER\Software\Classes\Interface\{EBBC7C04-315E-11D2-B62F-006097DF5BD4}\TypeLib
(default) = "{38D528BD-4948-4F28-8E5E-141A51090580}"
Version = "1.0"
The type library itself is already registered:
HKEY_CURRENT_USER\Software\TypeLib\{38D528BD-4948-4F28-8E5E-141A51090580}\1.0\0\win32
(default) = "D:\Junk\ComLibraryProxyTest\Win32\Project1.dll"
But it still fails.
it reads HKCR\Interface\[IID_IWidget]
it reads HKCR\Interface\[IID_IWidget]\ProxyStubClsid32
But it:
never reads: HKCR\Interface\[IID_IWidget]\TypeLib
So it fails to proxy the object.
Questions
Is it possible for the standard COM "Ole Automation" marshaler to build a proxy class out of a Type Library at runtime?
Is it possible for me to build a proxy class out of a Type Library at runtime?
Background
CoMarshalInterface takes an interface pointer on input and writes the
serialized representation of the pointer to a caller-provided byte stream. This byte stream can then be passed to another apartment, where the
CoUnmarshalInterface API function uses the byte stream to return an interface
pointer that is semantically equivalent to the original object yet can be
legally accessed in the apartment that executes the CoUnmarshalInterface
call. When calling CoMarshalInterface, the caller must indicate how far away
the importing apartment is expected to be. COM defines an enumeration for
expressing this distance:
enum MSHCTX {
MSHCTX_INPROC = 4, // in-process/same host
MSHCTX_LOCAL = 0, // out-of-process/same host
MSHCTX_NOSHAREDMEM = 1, //16/32 bit/same host
MSHCTX_DIFFERENTMACHINE = 2, // off-host
};
It is legal to specify a greater distance than required, but it is more efficient
to use the correct MSHCTX when possible. CoMarshalInterface also allows
the caller to specify the marshaling semantics using the following marshal flags:
enum MSHLFLAGS {
MSHLFLAGS_NORMAL, // marshal once, unmarshal once
MSHLFLAGS_TABLESTRONG, // marshal once, unmarshal many
MSHLFLAGS_TABLEWEAK, // marshal once, unmarshal many
MSHLFlAGS_NOPING = 4 // suppress dist. garbage collection
Normal marshaling (sometimes called call marshaling) indicates that the
marshaled object reference must be unmarshaled only once, and if additional
proxies are needed, additional calls to CoMarshalInterface are required. Table
marshaling indicates that the marshaled object reference may be unmarshaled
zero or more times without requiring additional calls to CoMarshalInterface.
I think that all COM object type libraries, when compiled by MIDL, can automatically create a proxy/stub factory. But in my case:
if the COM standard marshaler can't find a proxy/stub factory for an interface, it returns the error REGDB_E_IIDNOTREG.
It may be the case that i have to either:
use CreateProxyFromTypeInfo and CreateStubFromTypeInfo to create my own proxy
let the standard COM marshaler automatically create a proxy/stub if there's a typeinfo chunk associated with the interface GUID.
Bonus Reading
Old New Thing: What are the rules for CoMarshalInterThreadInterfaceInStream and CoGetInterfaceAndReleaseStream?
Old New Thing: Why do I get the error REGDB_E_IIDNOTREG when I call a method that returns an interface?
Old New Thing: Why do I get E_NOINTERFACE when creating an object that supports that interface?
Don Box - MSJ: Standard Marshalling in COM (archive)
Mysterious disappearing acts by samples from Microsoft SDKs
What CoMarshalInterface actually needs is IMarshal implementation (pointer) for the given interface, so that API could request the marshaling magic from it and, in particular, request IMarshal::GetUnmarshalClass in order to obtain information who is going to do the reverse magic afterwards:
This method is called indirectly, in a call to CoMarshalInterface, by whatever code in the server process is responsible for marshaling a pointer to an interface on an object. This marshaling code is usually a stub generated by COM for one of several interfaces that can marshal a pointer to an interface implemented on an entirely different object.
You don't have IMarshal implemented on your widget, so you are going to get it somewhere from.
As you started your question mentioning that you "want to marshal an interface to another thread" and the code comment says "Create our widget" there is a chance that you can utilize IMarhsal implementation of free threaded marshaler. The question does not provide information to tell whether it is possible and/or acceptable solution.
Back to the challenge of obtaining a marshaler, you are trying to work this around by utilizing a "standard" marshaler by "registering the interface":
why is the standard COM marshaler having so much problems
[...]
I of course can register my interface in the registry
Well, this is not how things actually work.
Interface registration is not just a registry key that "okay, this IID has its own key in the registry". The purpose of such registration is to point where to look for proxy/stub pair for this interface. Your manual creating of registry entries cannot help here if you don't have a proxy/stub DLL for the interface in question. If you had it, you would just regsvr32 it the usual way to create the registry keys.
So called standard marshaler, your next try, is not supposed to marshal any interface and your IWidget in particular. OLE supplies so called "PSOAInterface" marshaler, which is capable of supplying proxy/stub pairs for OLE Automation compatible interfaces. Just for them! The marshaler does not have so much problems, it actually has just one: your IWidget is unlikely to be compatible or you would not have the problem in first place.
If your IWidget was compatible, had an associated type library, where it was marked as [oleautomation], the type library registration process would have automatically created the registry keys referencing PSOAInterface and supplying ProxyStubClsid32. Then marshaling API would have picked PSOAInterface for your widget, it would have picked up the registered type library, load details of the interface, then provided standard proxy/stub pair for it and that's it. Standard marshaler works just within these contraints and not just for any interface pointed to.
That is, your options are:
implement IMarshal on the widget server
make IWidget OLE Automation compatible, with type library, so that type library registration process activates "standard marshaler" PSOAInterface for your interface
build, if possible and applicable, proxy/stub implementation for your interface automatically generated by MIDL complier (you can check standard ATL DLL project, created from Visual Studio template on how it can be done - it creates additional project with "PS" suffix, for a supplementary proxy/stub DLL)
implement separately IMarshal in a standalone DLL, and register it with the registry and your IWidget interface, so that marshaling API would pick it up when it attempts to marshal the interface pointer (I suppose it's the only option here if your IWidget is OLE incompatible and you have reasons to not alter the original implementation)
use free threaded marshaler in the widget server if its restrictions are acceptable
Oh wait, hold on - there is another weird one. If you don't want or cannot afford, or you are reluctant to change the COM server (widget, that is) but you can modify client side code as you like, you can create a thin wrapper that implements two interfaces IWidget (all methods forward calls to real server) and IMarshal on client side, and pass its IWidget to the CoMarshalInterThreadInterfaceInStream API. This will force COM to use your marshaling without altering the original server. You are on your own, of course, to do the actual marshaling afterwards. It is unlikely that it matches your actual need, and it is just an abstract note to the discussion (which mostly consists of attempts to do impossible without details on interface itself and available options on modification of server implementation).
TL;DR: Actual questions:
Is it possible for the standard COM "Ole Automation" marshaler to build a proxy class out of a Type Library at runtime?
Short answer: yes.
Is it possible for me to build a proxy class out of a Type Library at runtime?
Short answer: yes, with the type library marshaler and IPSFactoryBuffer. Or if you're willing to use the undocumented CreateProxyFromTypeInfo and CreateStubFromTypeInfo.
I wonder why you'd want to this.
This question is riddled with red herrings.
I'm trying to marshal an interface to another thread.
(...) the call to CoMarshalThreadInterfaceInStream fails with:
REGDB_E_IIDNOTREG (0x80040155)
Interface not registered
A useful error code, in this case.
Go directly to CoMarshalInterface
(...) that fails with:
REGDB_E_IIDNOTREG (0x80040155)
Interface not registered
It's not by switching API calls that do essentially the same thing that you could solve your issue.
Use standard marshaling
My class does not implement IMarhsal interface.
Can you confirm it doesn't implement INoMarshal, IStdMarshalInfo and IAgileObject either, just to be exhaustive?
The interface is, in fact, not registered
This was expected.
I know that Windows provides you the CoRegisterPSClsid
(...)
Which i can try calling, but what clsid do i use?
CoRegisterPSClsid(IID_IWidget, ???);
If you don't have a proxy/stub, why would you go through this?
But to answer this question, a standard marshalers' CLSID is usually the same GUID as the first IID found in an IDL file.
Register the interface
I of course can register my interface in the registry:
HKEY_CURRENT_USER\Software\Classes\Interface\{EBBC7C04-315E-11D2-B62F-006097DF5BD4}
(default) = "IWidget"
But that doesn't fix it. Spelunking through the Interface registry keys, i notice that many specify a ProxyStubClsid32 entry.
You should read the documentation, instead of relying on what other registry entries contain.
(...) I don't have a marshaler.
This should be the actual problem. If this object is yours, how do you intend to marshal it? Please see the bottom section of my answer.
Can COM automatically marshal based on a TypeLibrary
This is clearly rhetoric. Why would you think of a type library just now, instead of to begin with? The rest of your question corroborates.
Now i manually include the TypeLibrary:
HKEY_CURRENT_USER\Software\Classes\Interface\{EBBC7C04-315E-11D2-B62F-006097DF5BD4}\TypeLib
(default) = "{38D528BD-4948-4F28-8E5E-141A51090580}"
You mention this twice.
But it:
never reads: HKCR\Interface[IID_IWidget]\TypeLib
The type library marshaler has its own cache of interface->type library. To clear the cache, try recreating the registry entries, logging off and logging on, or ultimately, reboot.
The object may implement IProvideClassInfo, I don't know if the type library marshaler actually cares to QueryInterface for this to fetch a runtime ITypeInfo to work with.
These are the main types of marshalers:
Standard marshalers, which are usually compiled into a DLL from C/C++ source code generated by MIDL (which consists mainly of declarations on how to marshal types)
The type library marshaler, which can marshal types at runtime based on automation-compatible type information
The free-threaded marshaler aggregated with CoCreateFreeThreadedMarshaler, which avoids marshaling between threads within the same process
Custom marshalers, which do whatever the developer wants, most commonly to implement marshal-by-value, or a free-threaded marshaler back when CoCreateFreeThreadedMarshaler didn't exist
The standard marshalers generated by MIDL consist mainly of declarations on how to marshal types in and out, so the way they work is in the same vein as the type library marshaler. According to Don Box, the result is very similar.
However, the actual declarations are very different. The type library marshaler works on type information that is meant to work in VB6 and (excluding certain things, such as user-defined types) scripting languages (mainly VBScript and JScript) and is intended to be consumed by IDEs (e.g. VB6, VBA, Delphi, OleView) and interop code generators (e.g. VC++ #import, tlbimp.exe, Delphi's "Import Type Library", Lisp 1 2), etc.
I am having a problem when trying to match a password of a user using spring-security-core:2.0-RC4 with Grails 2.3.3.
I'm getting the following error when doing passwordEncoder.matches(rawPassword, encodedPassword)
No signature of method grails.plugin.springsecurity.authentication.encoding.BCryptPasswordEncoder.matches() is applicable for argument types: (java.lang.String, java.lang.String)
I've checked the BCryptPasswordEncoder source to search for clues and ended up in PasswordEncoder class definition.
The import in BCryptPasswordEncoder looks wrong though as the new PasswordEncoder is in org.springframework.security.crypto.password.
Doing passwordEncoder.isPasswordValid(rawPassword, encodedPassword, null) works (as in, there are no errors), but I don't know how to get the salt.
Is this working properly? If so, how do I get the salt?
EDIT:
I tried using NullSaltSource too but it gives me the error:
Salt value must be null when used with crypto module PasswordEncoder
As the error says, you can't use a salt with bcrypt. That's fine though - the algorithm is very robust and acts as if it's using a salt already.
The point of a salt is to ensure that if you and I have the same password, we don't have the same hashed password (assuming we each have our own salt value). Unsalted passwords cannot be de-hashed, but it's possible to create a table of hashes for all combination of passwords up to a certain length and use that as a lookup to find the cleartext password given a hash. Do an internet search for "rainbow table" and you'll find sites that have lookup tables for MD5, SHA-1 and other algorithms.
If you run encodePassword with a null salt value using bcrypt you'll get a different hash string for each run. The isPasswordValid method implementation with simpler algorithms usually hashes the cleartext password with the provided salt if there is one, and checks that this value is the same as the stored hash. But with bcrypt that's not sufficient, so it has the logic to verify that they are equivalent, but not necessarily equal.
The plugin uses a mix of implementations of the two interfaces for backwards compatibility, and will drop support for the old interface in a future release.
I am consuming a predefined wsdl with svcutil a la:
svcutil some_service.wsdl
one of the methods generated has the following signature:
[System.ServiceModel.OperationContractAttribute(Action="http://ws.example.org/SubmitData", ReplyAction="*")]
SubmitDataResponse SubmitData( SubmitDataRequest request )
While scvutil from VS2010/.net35 generates only the above and VS has no problem lanuching the service,
the svcutil program that is part of VS2012/.net45 also generates a method with the signature
[System.ServiceModel.OperationContractAttribute(Action="http://ws.example.org/SubmitData", ReplyAction="*")]
Task<SubmitDataResponse> SubmitDataAsync( SubmitDataRequest request );
This causes a run-time exception:
System.InvalidOperationException: Cannot have two operations in the
same contract with the same name, methods SubmitDataAsync and
SubmitData in type MyType violate this rule. You can change the name
of one of the operations by changing the method name or by using the
Name property of OperationContractAttribute.
I can work around this by deleting the Async appended methods or simply using svcutil from VS2010. But I am wondering why svcutil generates an interface that causes a runtime exception (is this a bug?), and whether there is something additional I am supposed to do to make it work.
The default behaviour appears to have been changed. If you provide the /syncOnly parameter it preserved the old behaviour for me.
/syncOnly - Generate only synchronous method
signature. Default: generate synchronous
and task-based asynchronous method
signatures.
I've been looking through the documentation for Go's openpgp package, and I think I must be missing some obvious points. For example, there's a ReadKeyRing function, but no WriteKeyRing. I can, on the other hand, Serialize an Entity, but I have no way to read it back. What's going on here? Does anyone actually use this package?
An entity represents public+private GPG key information. The ReadKeyRing function allows you to read a list of GPG keys.
The Entity.Serialize function documentation states:
Serialize writes the public part of the given Entity to w. (No private key material will be output).
As it is only the public part of the entity, you can create a new entity with the serialized data as the public key.
A WriteKeyRing does indeed not exist. It would go through the list of entities and extract the public keys into an array.
I was also struggeling quite a lot with this - in the end I just learned it by example:
Encryption and Decryption example: https://gist.github.com/jyap808/8250124
Decryption Example: https://gist.github.com/jyap808/8250067
The thinking behind this is not made for a user, but seems to come strongly out of the actual way pgp is technically implemented.
I would suggest to generate the keys not via the package but just with a pgp command line tool.