We are trying to implement the onem2m specification but we have some misunderstanding what makes a sensor an AE or an Container ?
For example, I have a ardunio board and I connected to it 10 simple leds. Then I am confused with two options. First one is I have one AE (Ardunio) and 10 containers (leds) for each leds. The other one is I have an AE (Ardunio), it is parent of another 10 AEs (leds) these AEs have their own container.
So what makes a resource an AE or a Container ? What should I look for making that decision ?
Perhaps the oneM2M resource structure is easier to understand when you look at the general architecture of a oneM2M device. Your Arduino most likely does not host a CSE, so it could be an Application Dedicated Node (ADN), that connects to a remote CSE.
Application is the keyword here: Your Arduino hosts an application, that, for example, reads and processes some sensor data. It implements the functional logic for that device. So, that application in oneM2M terms is an Application Entity (AE). In oneM2M an AE has certain properties, such as an application ID etc. It also holds the access rights to be connected to a remote CSE and access resources on that remote CSE.
An AE is also the root for the actual data and data structures, which this AE manages. This structure can be built from Containers. It is a common way to have one Container for each sensor and its data, for example a Temperature Container, an Air Pressure Container etc. It is also possible to have Containers within a Container if you want to structure your data a bit more.
The actual sensor data is managed within those Containers. These data entities are called ContentInstances. You usually only add ContentInstances to a container, but never remove or update them individually. This way you get a kind of small history of your data values. This behaviour, for example, how long this history should get, is a property of a Container and can be set individually. The Container offers two special data points, latest (la) and oldest (ol) which conveniently allows you to access the newest or oldest stored value in a Container, respectively.
Please note, that is also possible that a device may host more than one AE. The AE is an abstract concept. Your Arduino application might actually implement two AEs: one for storing the sensor data, and another one to read data from an CSE, for example to set operation parameters on the Arduino.
Also note, that there is a Node resource, if you want to represent the actual hardware device (node is the term for a connected device in oneM2M, as the "N" in the "ADN" explained above). Here, you can create a sub-structure that represents the device, get information about the firmware, battery status, network information etc, but also information about the AEs that are hosted on the device. Nodes, as AEs, are located at the root of a CSE.
I hope this helps you to structure the resource tree for your application.
Update
The following drawing visualises this resource tree.
CSEBase ─┬─ AE ─┬─ Container ─┬─ ContentInstance ◀═══ oldest
│ │ │
│ │ ├─ ContentInstance
│ │ │
│ │ ├── ...
│ │ │
│ │ └─ ContentInstance ◀═══ latest
│ │
│ └─ Container ─┬─ Container ─┬─ ContentInstance
│ │ │
│ │ └─ ...
│ │
│ └─ Container ─┬─ ContentInstance
│ │
│ └─ ...
│
├─ AE ─── ...
│
└─ Node ─── ...
Related
I have a structured streaming job that reads in a bunch of json.gz files under the following directory and writes to a delta table
headFolder
|- 00
|-- file1.json.gz
|- 01
|-- file2.json.gz
...
|- 23
|-- file24.json.gz
The structured streaming I'm running is as follows
spark.readStream
.format('cloudFiles')
.options({"cloudFiles.format": "json", "cloudFiles.schemaEvolutionMode": "rescue"})
.schema(schema_predefined)
.load("./headFolder/")
.withColumn("input_file_path", input_file_name())
.writeStream
.format("delta")
.outputMode("append")
.options({'checkpointLocation': checkpoint_path, 'path': output_path})
.trigger({'once': True})
.queryName("query_name")
.start()
I omitted some details in the query above, please take all undeclared parameters as pre-defined. After I run the job, all the 24 files were processed and I can validate that data was correct. However, the input_file_name() function didn't work as I was expecting.
When I check the input_file_name column, I was expecting 24 distinct records since their key names are different. However, I see only around 5 filenames, which varies based on file size. After I looked into the documentation here, indeed it returns the file name of the TASK instead of the individual files, thus since I'm reading from the top level, Spark automatically divides the 24 hours into several tasks and picked one name from the files read.
My question is, is there still a way to accurately record the filename for the file processed under the current framework? I don't want to change the file path or force it to run one task per file for runtime reasons.
Thank you!
When I am building a native image of my application with quarkus, I get an executable that has a size of 150MB (the associated jar has a size of 12MB I doubt that Substrate VM make up for all that space). Is this normal or am I doing something wrong ? Is there a way to investigate like logging what is loaded in the native image ?
$ du -sh target/placeholder-1.0.0-SNAPSHOT-runner 1.9m mar. 06 juil. 2021 00:13:58
150M target/placeholder-1.0.0-SNAPSHOT-runner
$ du -sh target/placeholder-1.0.0-SNAPSHOT.jar mar. 06 juil. 2021 00:14:32
12M target/placeholder-1.0.0-SNAPSHOT.jar
Yes, the size of the binary can be larger than the jar file.
It mostly consists of 2 parts: code of your application compiled to binary and the "image heap" the preinitialized data structures and components of your application.
For example, if your application initializes any classes at build time, then the values of the static fields of these classes are serialized into the image heap. This can help startup of the app, because it doesn't need to initialize these again, but it can also make the binary larger.
There are reporting options you can enable for the native image build process and tools that help you make sense of the contents of the image.
From the article linked above, you can use the -H:+DashboardAll option and the dashboard tool here (it is hosted on github and works offline): https://www.graalvm.org/docs/tools/dashboard/?ojr=dashboard
And then it can visualize what takes space for example like this:
https://miro.medium.com/max/2400/1*mWIhq53ALPiI2GP-IQkYoQ.png
Since docker images are made of layers and every image created by a DockerFile basically is a collection of layers (one layer per DockerFile line), I wonder if it is possible to extract relevant layers, transmit those and reconstruct the image using these layers along with a base image.
This way securely creating individual images on the server, transmitting only the relevant changes and reconstructing the resulting image would be a very good option for a variety of scenarios - including the one we are contemplating.
Docker's file system and layer management is smart enough to do this for you.
Consider this scenario, for example... create a Dockerfile FROM node:6.9.5 and add some new configuration or libraries or whatever. publish the resulting image to Dockerhub... call it "my-user/my-image" just for an example.
Now go to another computer that already has the node:6.9.5 image on it. Create a new Dockerfile that builds FROM my-user/my-image. When you build this image, Docker will not download the node:6.9.5 image again, if you already have it downloaded.
Here's the thing, though... Docker image layers are explicitly tied to each other.
If you have 3 layers in an image (layer-a, layer-b, layer-c), you can't extract layer-b alone. that doesn't make sense, because layer-b relies on the exact state of layer-a for it to work. similarly, layer-c relies on layer-b to work.
each successive layer is only a difference between the previous layer and the result of the next instruction from the Dockerfile.
so the idea of what you're trying to do - to extract layers - both doesn't work the way you might be thinking, and is already handled by Docker itself.
Perhaps docker history is what you are looking for. It gives you each layer, change by change with the commands that have triggered them:
docker history myimagename
be51b77efb42 8 days ago /bin/sh -c apt-get update && apt-get install 338.3 MB
4b137612be55 6 weeks ago /bin/sh -c #(nop) ADD jessie.tar.xz in / 121 MB
see: docker history
The simple straight version I used is to export the docker container / image which basically collapses all layers into a single layer. By comparing two different images (base image + current image) one can easily compute the file-system diff and create a file-system "patch" that can be distributed and used based on the base image to recreate the current image. All left is to commit the current container and we are done.
I started using the zap log library for my Go project. I want to print different colors to tty console based on the log level.
I find the zap/internal/color package can display different colors for strings, but I want to change the log level with different color.
I also want to write the log to some log files with different log level.
How to init and config the zap logger?
Just met the same issue, and here are some code snippets for enable colors:
config := zap.NewDevelopmentConfig()
config.EncoderConfig.EncodeLevel = zapcore.CapitalColorLevelEncoder
logger, _ := config.Build()
logger.Info("Now logs should be colored")
reference: https://github.com/uber-go/zap/pull/307
I am writing a tool which spawns multiple child processes. In fact 3 levels of child processes to speed up the entire logic/process.
To display the output in terminal I have chosen Curses::UI. The curses ui objects/widgets are created at each level of parent/child relationship and manipulated in the last level of child processes. This multiple levels of child processes seem to be causing issues with the curses display.
I thought it would be stable if I shared just one curses ui object across all child/parent processes.
To achieve this sharing, I am trying to use Storable/Shareable module but not able to get it to run due to errors like these:
quicode sub { │
│ exit; │
│ } caused an error: 'exit' trapped by operation mask at (eval 99) line 2, at my_curser.pl line 147 │
code sub {──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
exit;
} caused an error: 'exit' trapped by operation mask at (eval 99) line 2, at my_curser.pl line 147
│ode sub { │
│ exit;
Is it possible to share curses ui object across mutliple processes ?
curses relies on C and terminal or terminal emulator state which is not reliably shareable between processes even from C, and is not visible to Perl wrappers such as UI::Curses. (A terminal has a single "current position"/cursor position; consider what happens if different subprocesses try to update widgets in different parts of the display at the same time.) As such, there is no way you can share these widgets between subprocesses.
In general, a better design is to dedicate a thread or process to the UI and distribute other aspects of processing to other threads/processes.