I want to connect a cluster edge to a node. Please find a simple Graphviz code below. Here is the online demo.
digraph G {
compound=true;
node1
subgraph cluster0 {
label="cluster"
node2;
}
node2 -> node1 [ltail="cluster0"]
}
It produces this output:
So, in the above method the cluster0 is connected to a node1. node2 is inside the cluster.
To connect the cluster itself to node we need compound=true and [ltail="cluster0"]. So we are just connecting node2 to node1 behind the scene and generating the edge from behind the cluster.
node2 -> node1 [ltail="cluster0"]
What I really want is to connect the cluster itself in the code like:
digraph G {
compound=true;
node1
subgraph cluster0 {
label="cluster"
node2;
}
cluster0 -> node1 [ltail="cluster0"]
}
But unfortunately, it produces this:
Well, the first one produces the image as desired but not the right approach. I'm working with State machines and generating the Graphviz code programmatically. Logically speaking, there is only one transition in the state digram:
cluster -> node1
But to do so we're trasitioning from node2 to node1 which is logically not right. I am curious that there is a way to achieve this and connect the cluster directly to a node instead of using a node inside the cluster.
Thanks in advance!
The ltail attribute is what is connecting it with the cluster edge.
(It seems like what you say is correct, that it's connecting the nodes on the back end but covering the edge when it gets to the cluster.)
Removing the ltail attribute should fix it:
digraph G {
compound=true;
node1
subgraph cluster0 {
label="cluster"
node2;
}
node2 -> node1
}
Related
Is it possible to define a message which has a field of its own type as in linked lists with protocol buffers? For example the value of next field could be null at the end of the list.
message Node {
string name = 1;
Node next = 2;
}
I know you asked how to make a custom type. The documentation mentions to try to avoid that. I find that you can model almost anything with the parts / types that are built in. To make a list of nodes with the built in types do this:
message ResponseOfNodes {
repeated Node nodes = 1;
}
message Node {
string name = 1;
string someOtherData = 2;
}
Here's code that uses the generated code / list.
// make a node so it can be added to the list.
Node node1 = Node.newBuilder().setName("a name").setSomeOtherData("etc").build();
// Build a response (ResponseOfNodes) that will get nodes added to its list. Add two Nodes.
ResponseOfNodes.Builder responseBuilder = ResponseOfNodes.newBuilder().addNodes(node1);
responseBuilder.addNodes(node1.toBuilder().setName("name 2").build());
// build the ResponseOfNodes and print it out.
ResponseOfNodes responseOfNodes = responseBuilder.build();
log.info("the list: {}", responseOfNodes.getNodesList());
Console Output;
[INFO ] [main] - Hello I am a client.
[INFO ] [main] - the list: [name: "a name"
someOtherData: "etc"
, name: "name 2"
someOtherData: "etc"
I am relatively new to Omnet++. Currently I am trying to encapsulate a flooding (simple) module inside a compound module. This is what I came up with:
Node.ned:
module Node
{
gates:
inout g[];
submodules:
floodingModule: FloodingModule;
connections allowunconnected:
g++ <--> floodingModule.g++;
}
FloodingModule.ned
simple FloodingNode
{
parameters:
bool sendInitialMessage = default(false);
gates:
inout g[] #loose;
}
Network.ned
network FloodingNetworkSmall {
submodules:
node0 : Node;
node1 : Node;
node2 : Node;
node3 : Node;
node4 : Node;
node5 : Node;
node6 : Node;
node7 : Node;
node8 : Node;
node9 : Node;
connections:
node0.g++ <--> Link <--> node1.g++;
node0.g++ <--> Link <--> node3.g++;
node0.g++ <--> Link <--> node5.g++;
node1.g++ <--> Link <--> node2.g++;
node1.g++ <--> Link <--> node4.g++;
node1.g++ <--> Link <--> node6.g++;
node1.g++ <--> Link <--> node7.g++;
node1.g++ <--> Link <--> node9.g++;
node2.g++ <--> Link <--> node5.g++;
node2.g++ <--> Link <--> node8.g++;
node3.g++ <--> Link <--> node4.g++;
node3.g++ <--> Link <--> node6.g++;
node4.g++ <--> Link <--> node6.g++;
node4.g++ <--> Link <--> node8.g++;
node6.g++ <--> Link <--> node7.g++;
node7.g++ <--> Link <--> node8.g++;
}
I am calling the flooding in the initialize method of the floodingModule like this:
for (GateIterator i(this); !i.end(); i++) {
cGate *gate = *i;
if (gate->getType() == cGate::OUTPUT) {
SimpleMessage *csmsg = smsg->dup();
send(csmsg, gate);
}
}
The problem I am facing is, that the flooding only happens on the first link that is created. This means, there must be a problem mapping the compound module gates to the submodule gates, or not? Am I doing something wrong here which is obvious? Do you need more code?
Thanks for any advice!
I couldn't even start the simulation with your code unmodified, because there were unconnected gates. (And there are some mildly annoying inconsistencies between file and module names, with the words Node and Module, but that is beside the point.)
To solve this, you need to change the connections segment of your Node module to something similar to what's in the Node module of the routing sample simulation included with OMNeT++:
module Node
{
gates:
inout g[];
submodules:
floodingModule: FloodingNode;
connections:
for i=0..sizeof(g)-1 { // <= this is the important part
g++ <--> floodingModule.g++;
}
}
The point is that for every "outside" connection between Nodes in the network, there has to be a corresponding connection (to continue the path) inside the Node module (at each end), between it and the floodingNode submodule. There is no "automatic merging/diverging" of connection paths at the gate vectors on compound module boundaries.
Yes, this means that if any given node has, say, five other nodes connected to it on the same gate vector, then there has to be five "parallel" connections inside that node, all leading to the gate vector of the submodule - in this case.
And there is no need for the allowunconnected specifier, nor the #loose property anywhere, in this case they do more harm by allowing "invalid" networks, than good. They are mostly useful for wireless simulations anyway.
Also, you should consider only scheduling a simple "timer" self-message (even if it's at T=0) in initialize(), and sending the "real" messages in the handleMessage() method when receiving said timer, this way the visualization of the graphical environments work better, and it is also arguably better design.
I want to monitor Hadoop (Hadoop version-0.20.2) multi node cluster using ganglia. My Hadoop is working properly.I have installed Ganglia after reading following blogs---
http://hakunamapdata.com/ganglia-configuration-for-a-small-hadoop-cluster-and-some-troubleshooting/
http://hokamblogs.blogspot.in/2013/06/ganglia-overview-and-installation-on.html
I have also studied Monitoring with Ganglia.pdf(APPENDIX B
Ganglia and Hadoop/HBase ).
I have modified only the following lines in **Hadoop-metrics.properties**(same on all Hadoop Nodes)==>
// Configuration of the "dfs" context for ganglia
dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
dfs.period=10
dfs.servers=192.168.1.182:8649
// Configuration of the "mapred" context for ganglia
mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
mapred.period=10
mapred.servers=192.168.1.182:8649:8649
// Configuration of the "jvm" context for ganglia
jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
jvm.period=10
jvm.servers=192.168.1.182:8649
**gmetad.conf** (Only on Hadoop master Node )
data_source "Hadoop-slaves" 5 192.168.1.182:8649
RRAs "RRA:AVERAGE:0.5:1:302400" //Because i want to analyse one week data.
**gmond.conf** (on all the Hadoop Slave nodes and Hadoop Master)
globals {
daemonize = yes
setuid = yes
user = ganglia
debug_level = 0
max_udp_msg_len = 1472
mute = no
deaf = no
allow_extra_data = yes
host_dmax = 0 /*secs */
cleanup_threshold = 300 /*secs */
gexec = no
send_metadata_interval = 0
}
cluster {
name = "Hadoop-slaves"
owner = "Sandeep Priyank"
latlong = "unspecified"
url = "unspecified"
}
/* The host section describes attributes of the host, like the location */
host {
location = "CASL"
}
/* Feel free to specify as many udp_send_channels as you like. Gmond
used to only support having a single channel */
udp_send_channel {
host = 192.168.1.182
port = 8649
ttl = 1
}
/* You can specify as many udp_recv_channels as you like as well. */
udp_recv_channel {
port = 8649
}
/* You can specify as many tcp_accept_channels as you like to share
an xml description of the state of the cluster */
tcp_accept_channel {
port = 8649
}
Now Ganglia is only giving system metrics(mem , disk etc.) for all the nodes. But it is not showing the Hadoop metrics( like jvm, mapred metrics
etc. ) on the web interface. how can i fix this problem ?
I do work Hadoop with Ganglia, and yes, I see on Ganglia a lot of metrics of Hadoop (Containers, map task, vmem). In fact, Hadoop specific report to Ganglio more of hundred metrics.
The hokamblogs Post was enough for this.
I edit hadoop-metrics2.properties on the master node and the content is:
namenode.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31
namenode.sink.ganglia.period=10
namenode.sink.ganglia.servers=gmetad_hostname_or_ip:8649
resourcemanager.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31
resourcemanager.sink.ganglia.period=10
resourcemanager.sink.ganglia.servers=gmetad_hostname_or_ip:8649
and I also edit the same files on the slaves:
datanode.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31
datanode.sink.ganglia.period=10
datanode.sink.ganglia.servers=gmetad_hostname_or_ip:8649
nodemanager.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31
nodemanager.sink.ganglia.period=10
nodemanager.sink.ganglia.servers=gmetad_hostname_or_ip:8649
Your remember restart Hadoop and Ganglia after change the files.
I hope this help you.
Thanks to everyone, If you are using older version of Hadoop then put following files( from new version of Hadoop) ==>
GangliaContext31.java
GangliaContext.java
In path ==> hadoop/src/core/org/apache/hadoop/metrics/ganglia
From the new version of Hadoop.
Compile your Hadoop using ant ( and set proper proxy while compiling).
If it gives error like function definition is missing then put that function definition( from new version) in proper java file and then compile Hadoop again. It will work.
I just started learning Hadoop, but don't understand how a datanode becomes a reducer node.
Once the map task completes, the content of its sort buffer is flushed to the local disk
after the KV pairs are sorted and partitioned
Then the jobtracker is notified about the spilled partitions.
After then the reducers start asking the data from a particular partition.
But how the jobtracker decides which node becomes a reducer node? I'm reading the Hadoop Definitive guide but this step is not mentioned in the book.
Thanks,
Bruckwald
Pretty much first-come, first-serve. Tasks are assigned by heartbeats, so if a Tasktracker pings the Jobtracker that it is alive, it will get a response that might contain a new task to run:
List<Task> tasks = getSetupAndCleanupTasks(taskTrackerStatus);
if (tasks == null ) {
tasks = taskScheduler.assignTasks(taskTrackerStatus);
}
if (tasks != null) {
for (Task task : tasks) {
expireLaunchingTasks.addNewTask(task.getTaskID());
LOG.debug(trackerName + " -> LaunchTask: " + task.getTaskID());
actions.add(new LaunchTaskAction(task));
}
}
Here's the relevant source code of the Jobtracker. So besides which tasktracker comes first, the taskscheduler will check for resource conditions (e.g. if there is a free slot, or a single node is not overloaded).
The relevant code can be found here (which isn't particular exciting):
//
// Same thing, but for reduce tasks
// However we _never_ assign more than 1 reduce task per heartbeat
//
final int trackerCurrentReduceCapacity =
Math.min((int)Math.ceil(reduceLoadFactor * trackerReduceCapacity),
trackerReduceCapacity);
final int availableReduceSlots =
Math.min((trackerCurrentReduceCapacity - trackerRunningReduces), 1);
boolean exceededReducePadding = false;
if (availableReduceSlots > 0) {
exceededReducePadding = exceededPadding(false, clusterStatus,
trackerReduceCapacity);
synchronized (jobQueue) {
for (JobInProgress job : jobQueue) {
if (job.getStatus().getRunState() != JobStatus.RUNNING ||
job.numReduceTasks == 0) {
continue;
}
Task t = job.obtainNewReduceTask(taskTracker, numTaskTrackers, taskTrackerManager.getNumberOfUniqueHosts());
if (t != null) {
assignedTasks.add(t);
break;
}
// Don't assign reduce tasks to the hilt!
// Leave some free slots in the cluster for future task-failures,
// speculative tasks etc. beyond the highest priority job
if (exceededReducePadding) {
break;
}
}
}
Basically, the first tasktracker that heartbeats to the Jobtracker and has enough slots available will get a reduce tasks.
I read from hadoop operations that if a datanode fails during writing process,
A new replication pipeline containing the remaining datanodes is
opened and the write resumes. At this point, things are mostly back to
normal and the write operation continues until the file is closed. The
namenode will notice that one of the blocks in the file is
under-replicated and will arrange for a new replica to be created
asynchronously. A client can recover from multiple failed datanodes
provided at least a minimum number of replicas are written (by
default, this is one).
But what happens if all the datanodes fail? i.e., minimum number of replicas are not written?
Will client ask namenode to give new list of datanodes? or will the job fail?
Note : My question is NOT what happens when all the data nodes fails in the cluster. Question is what happens if all the datanodes to which the client was supposed to write, fails, during the write operation
Suppose namenode told the client to write BLOCK B1 to datanodes D1 in Rack1, D2 in Rack2 and D3 in Rack1. There might be other racks also in the cluster(Rack 4,5,6,...). If Rack1 and 2 failed during the write process, client knows that the data was not written successfully since it didn't receive the ACK from the datanodes, At this point, will it ask Namenode to give new set of datanodes? may be in the still alive Racks ?
OK I got what you are asking. DFSClient will get a list of datanodes from the namenode where it is supposed to write a block (say A) of a file. DFSClient will iterate over that list of Datanodes and write the block A in those locations. If block write fails in the first datanodes, it'll abandon the block write and ask namenode a new set of datanodes where it can attempt to write again.
Here the sample code from DFSClient that explains that -
private DatanodeInfo[] nextBlockOutputStream(String client) throws IOException {
//----- other code ------
do {
hasError = false;
lastException = null;
errorIndex = 0;
retry = false;
nodes = null;
success = false;
long startTime = System.currentTimeMillis();
lb = locateFollowingBlock(startTime);
block = lb.getBlock();
accessToken = lb.getBlockToken();
nodes = lb.getLocations();
//
// Connect to first DataNode in the list.
//
success = createBlockOutputStream(nodes, clientName, false);
if (!success) {
LOG.info("Abandoning block " + block);
namenode.abandonBlock(block, src, clientName);
// Connection failed. Let's wait a little bit and retry
retry = true;
try {
if (System.currentTimeMillis() - startTime > 5000) {
LOG.info("Waiting to find target node: " + nodes[0].getName());
}
Thread.sleep(6000);
} catch (InterruptedException iex) {
}
}
} while (retry && --count >= 0);
if (!success) {
throw new IOException("Unable to create new block.");
}
return nodes;
}