How do you join a Liberty collective on Linux? - websphere-liberty

Question

I am trying to set up a Liberty collective using Docker hosts running linux. The videos they have about setting up Liberty collectives at the moment use Windows and are all on the same machine.
To join the collective so it appears on adminCenter isn't too hard it is just a matter of collective join --host=...
The problem is the administration part i.e. changing the configuration file or stopping and starting the servers is not working.
I tried various ways of passing in hostInfo in server.xml or --sshPrivateKey Hard coding root passwords and none of them work.
According to the instructions all you needed was an openssh-server which I have already enabled and running I have already exposed the ports and verify I can connect to them using a certificate from the controller container as well.
In addition based on the REST API it uses a stringified SSH Private Key itself rather than a file and that should be sent through the collective registerHost but it does not appear to work and there is nothing in the command line logs even with .level=ALL and ...consolelogger...=ALL that show what the hostAuthInfo is.
The one of the commands I ran for collective join is
collective join defaultServer \
--host=controller \
--port=9443 \
--user=adminUser \
--password=adminPassword \
--autoAcceptCertificates \
--rpcUser=root \
--sshPrivateKey=$HOME/.ssh/id_rsa \
--keystorePassword=$PASSWORD \
--createConfigFile=/config/collective-join-include.xml
I say one of because I tried various combinations where I removed or changed --rpcuser, --sshPrivateKey and other authInfo related items.
server.xml of member is at this point ...
<?xml version="1.0" encoding="UTF-8"?>
<server description="Application Server">
<featureManager>
<feature>javaee-7.0</feature>
<feature>clusterMember-1.0</feature>
<!--<feature>scalingMember-1.0</feature>-->
</featureManager>
<remoteFileAccess>
<writeDir>${server.config.dir}</writeDir>
</remoteFileAccess>
<httpEndpoint id="defaultHttpEndpoint" httpPort="9080" httpsPort="9443" host="*"/>
<!--<hostSingleton name="ScalingMemberSingletonService" port="5164" />-->
<applicationManager autoExpand="true"/>
<!--<hostAuthInfo rpcUser="root" sshPublicKeyPath="/root/.ssh/id_rsa.pub" sshPrivateKeyPath="/root/.ssh/id_rsa"/>-->
<include location="${server.config.dir}/collective-join-include.xml"/>
<dataSource id="myds" jndiName="jdbc/sample" type="javax.sql.XADataSource">
<jdbcDriver javax.sql.ConnectionPoolDataSource="org.mariadb.jdbc.MariaDbDataSource" javax.sql.DataSource="org.mariadb.jdbc.MariaDbDataSource" javax.sql.XADataSource="org.mariadb.jdbc.MariaDbDataSource">
<library>
<file name="${server.config.dir}/mariadb-java-client-1.5.9.jar"/>
</library>
</jdbcDriver>
<properties databaseName="jeesample" password="password" serverName="database" user="jeeuser"/>
</dataSource>
<basicRegistry id="basic" realm="BasicRealm">
<user name="websphere" password="{xor}KDo9LC83Oi06"/>
</basicRegistry>
<ejbContainer>
<timerService>
<persistentExecutor taskStoreRef="mystore"/>
</timerService>
</ejbContainer>
<databaseStore dataSourceRef="myds" id="mystore"/>
</server>
Controller side
<?xml version="1.0" encoding="UTF-8"?>
<server description="Collective Controller">
<variable name="defaultHostName" value="controller"/>
<httpEndpoint id="defaultHttpEndpoint" host="*" httpPort="9080" httpsPort="9443"/>
<featureManager>
<!--<feature>scalingController-1.0</feature>-->
<feature>adminCenter-1.0</feature>
<feature>dynamicRouting-1.0</feature>
</featureManager>
<remoteFileAccess>
<writeDir>${server.config.dir}</writeDir>
</remoteFileAccess>
<!--<scalingDefinitions>
<defaultScalingPolicy enabled="true" min="2" max="2"/>
</scalingDefinitions>-->
<include location="${server.config.dir}/resources/collective/collective-create-include.xml"/>
<collectiveController user="adminUser" password="adminPassword"/>
</server>

Answer 1

By default, when ssh is properly configured and running on the linux machines (controller's and member's host machine), you only need to run the 'collective join' command from the member's wlp/bin dir. You should not need to specify hostInfo nor --sshPrivateKey via server.xml nor the collective updateHost/registHost commands. This flow will use the ssh keys generated by the collective.
The useHostCredentials flag is generally meant to be used with rpcUser and rpcUserPassword (provided via registerHost, updateHost, or server.xml) instead of ssh, especially useful for systems that do not have ssh configured (like windows by default). However, it can also be used to specify custom ssh keys.
If you're still having trouble, provide the collective join command that was ran from the member's wlp/bin, as well as the server.xml of the controller and member.