My question is fairly simple:
I am coding on a single-device small laptop and I am using jax.pmap because my code will run on multiple TPUs. I would like to "fake" having multiple devices to test my code and try different things.
Is there any way to do it? I doubt the solution will be within Jax though. Thanks!
You can spoof multiple XLA devices backed by a single device by setting the following environment variable:
$ set XLA_FLAGS="--xla_force_host_platform_device_count=8"
In Python, you could do it like this
# Note: must set this env variable before jax is imported
import os
os.environ['XLA_FLAGS'] = "--xla_force_host_platform_device_count=8"
import jax
print(jax.devices())
# [CpuDevice(id=0), CpuDevice(id=1), CpuDevice(id=2), CpuDevice(id=3),
# CpuDevice(id=4), CpuDevice(id=5), CpuDevice(id=6), CpuDevice(id=7)]
import jax.numpy as jnp
out = jax.pmap(lambda x: x ** 2)(jnp.arange(8))
print(out)
# [ 0 1 4 9 16 25 36 49]
Note that when a only a single physical device is present, all the "devices" here will be backed by the same threadpool. This will not improve performance of the code, but it can be useful for testing the semantics of parallel implementations on a single-device machine.
Related
I have an linux embedded system. I can manage I/Os with shell commands. This commands change out status of GPIO #48 :
/sys/class/gpio# echo 48 > /sys/class/gpio/export
/sys/class/gpio# echo out > /sys/class/gpio/gpio48/direction
/sys/class/gpio# echo high > /sys/class/gpio/gpio48/direction
/sys/class/gpio# echo low > /sys/class/gpio/gpio48/direction
How can I manage I/Os with Goland efficiently ? Is it possible to manage them without going through the shell commands ?
On Linux GPIO interface is exported via sys filesystem in /sys/class/gpio hierarchy so as in your shell example you just need to write data into these files, something like:
// To export pin 48 (same as echo 48 > /sys/class/gpio/export)
ioutil.WriteFile("/sys/class/gpio/export", []byte("48"), 0666)
...
Depending on your platform and needs you may want to consider some pre-existing package (e.g. go-rpio for Raspberry Pi or periph which is more general and supports much more than GPIO).
If you want more efficient/faster solution than writing sysfs files you might also consider memory-mapped GPIO access where you basically directly access GPIO periphery via memory range given to it by the kernel. This requires somewhat deeper knowledge of your target platform (understanding its GPIO registers and their mapping). You can read about that approach in detail in this blogpost.
EDIT: As #0andriy pointed out in his comment gpio syssfs is deprecated. That applies for both your Bash example above and my answer how to do same thing in Go. Instead a new ABI was introduced and libgpiod to interact with it. Go port is available here https://github.com/warthog618/gpiod.
I have 2 B210 radios on USB3 on a Windows 10 system using the UHD USRP C API latest release and Python 3.6 as the programming environment. I can "sort of" run them simultaneously in separate processes but would like to know if it is possible to run them in a single thread? How?
1 Happy to move to Linux if it makes things easier, I'm just more familiar with Windows.
2 "sort of" = I sometimes get errors which might be the two processes colliding somewhere down the stack.
The code below illustrates the race condition, sometimes one or both processes fail with error code 40 (UHD_ERROR_ASSERTION) or occasionally code 11 ( UHD_ERROR_KEY )
from ctypes import (windll, byref, c_void_p, c_char_p)
from multiprocessing import Process, current_process
def pread(argstring):
# get handle for device
usrp = c_void_p(0)
uhdapi = windll.uhd
p_str=c_char_p(argstring.encode("UTF8"))
errNo = uhdapi.uhd_usrp_make(byref(usrp),p_str)
if errNo != 0:
print("\r*****************************************************************")
print("ERROR: ",errNo," IN: ", current_process())
print("=================================================================")
if usrp.value != 0:
uhdapi.uhd_usrp_free(byref(usrp))
return
if __name__ == '__main__':
while True:
p2 = Process(target=pread, args=("",))
p1 = Process(target=pread, args=("",))
p1.start()
p2.start()
p1.join()
p2.join()
print("end")
Yes, you can have multiple multi_usrp handles.
By the way, note that UHD is natively C++, and the C API is just a wrapper around that. It's designed for generating scripting interfaces like the Python thing you're using (don't know which interface between Python and the C API you're using – something self-written?).
While it's possible, there's no good reason to call the recv and send functions from the same thread – most modern machines are multi-threaded, and you should make use of that. Real-time SDR is a CPU-intensive task and you should use all the CPU resources available to get data to and from the driver, as to avoid overflowing buffers.
We have built a system where videos are stored in mongodb. The videos are each a couple of hundred megabytes in size. The system is built in python3 using mongoengine. The c extensions of pymongo and bson are installed.
The definition of the mongoengine documents is:
class VideoStore(Document, GeneralMixin):
video = EmbeddedDocumentListField(SingleFrame)
mutdat = DateTimeField()
_collection = 'VideoStore'
def gen_video(self):
for one_frame in self.video:
yield self._get_single_frame(one_frame)
def _get_single_frame(self, one_frame):
if one_frame.frame.tell() != 0:
one_frame.frame.seek(0)
return pickle.loads(one_frame.frame.read())
class SingleFrame(EmbeddedDocument):
frame = FileField()
Reading a video in Linux takes about 3 to 4 seconds. However running the same code in Windows takes 13 to 17 seconds.
Does anyone out there have any experience with this problem and any kind of solution?
I have thought of and tested (to no avail):
increasing the chunksize
reading the video as a single blob without using yield
Storing the file as a single blob (so no storing of separate frames)
Use Linux, Windows is poorly supported. The use of "infinite" virtual memory among other things causes issues with Windows variants. This thread elaborates further:
Why Mongodb performance better on Linux than on Windows?
I would like to disable c-states on my computer.
I disabled c-state on BIOS but I don't obtain any result. However, I found an explanation :
"Most newer Linux distributions, on systems with Intel processors, use the “intel_idle” driver (probably compiled into your kernel and not a separate module) to use C-states. This driver uses knowledge of the various CPUs to control C-states without input from system firmware (BIOS). This driver will mostly ignore any other BIOS setting and kernel parameters"
I found two solutions to solve this problem but I don't know how to apply:
1) " so if you want control over C-states, you should use kernel parameter “intel_idle.max_cstate=0” to disable this driver."
I don't know neither how I can check the value (of intel_idle.max_cstate ) and neither how I can change its value.
2) "To dynamically control C-states, open the file /dev/cpu_dma_latency and write the maximum allowable latency to it. This will prevent C-states with transition latencies higher than the specified value from being used, as long as the file /dev/cpu_dma_latency is kept open. Writing a maximum allowable latency of 0 will keep the processors in C0"
I can't read the file cpu_dma_latency.
Thanks for your help.
Computer:
Intel Xeon CPU E5-2620
Gnome 2.28.2
Linux 2.6.32-358
To alter the value at boot time, you can modify the GRUB configuration or edit it on the fly -- the method to modify that varies by distribution. This is the Ubuntu documentation to change kernel parameters either for a single boot, or permanently. For a RHEL-derived distribution, I don't see docs that are quite as clear, but you directly modify /boot/grub/grub.conf to include the parameter on the "kernel" lines for each bootable stanza.
For the second part of the question, many device files are read-only or write-only. You could use a small perl script like this (untested and not very clean, but should work) to keep the file open:
#!/usr/bin/perl
use FileHandle;
my $fd = open (">/dev/cpu_dma_latency");
print $fd "0";
print "Press CTRL-C to end.\n";
while (1) {
sleep 5;
}
Redhat has a C snippet in a KB article here as well and more description of the parameter.
I'm in a senior design class for my college's computer engineering program. I'm not very good with Embedded Systems. I need to create a program that can send instruction to a microcontroller and it should have a GUI. Someone else is programming the microcontroller so I'm mostly just responding to what they will output to me. I really have no idea where to start with this, I'm a little swamped/overwhelmed on my final semester. Would someone mind pointing me in the right direction?
Additional info: We are using a TI-msp430f2274 for the device and it's really just outputting the readings of 3 gyroscopes. I'm thinking about using GTK to create the GUI but I am looking to see if there's something better.
Edit: My project is a roller alignment tool that uses gyroscopes to compare two rollers. The program would send an instruction to "zero" the tool to one roller and receive a stream of data that says the offset difference of the next roller.
We are using a TTL-232R cable with a UART interface and the program should work on Windows. I have no idea how you talk to it. College has crushed me.
If you're using a windows machine then Visual Studio and a Windows Forms Application could do the trick. I've used it previously to interface with temperature sensors and other control equipment via USB and other projects involving reading from USB Human Interface Devices. So without understanding the specifics of your project, as an IDE Visual Studio, I think, is the best and most intuitive and using the Windows API you can achieve a lot with your code.
Visual Studio 2012
HTH.
p.s. I write most of my code in VB using VS but it will support other languages too...
Edit: some example code of reading from a com port:
Sub ReceiveSerialData()
' Receive strings from a com port
Dim comport as string = "COM 6" 'or whatever com you want...
Dim baud as integer = 9600 'or whatever baud you want...
Try
com = My.Computer.Ports.OpenSerialPort(comport)
com.ReadTimeout = 10000
com.BaudRate = baud
While (com.IsOpen)
Dim Incoming As String = com.ReadLine()
If Incoming Is Nothing Then
Exit While
Else
'do something here with your com data
'i.e. display it in a rich text box, or whatever...
End If
End While
Catch ex As Exception
' display your errors here if you wish....
End Try
End Sub
I think the other answers though, esp about using tcl probably make more sense from an embedded systems point of view. I simply have no experience with them whatsoever hence my VB'ness. :-)
Since you mentioned that you've used tcl before, I'm giving my answer in tcl since IMHO it's the easiest way to do this and therefor the right tool for the right job.
The core advantage of tcl (and indeed one of the reasons it's loved by its fans) is that it's very, very cross-platform. While in other languages cross platform simply means "can run on multiple platforms" but still requires you to use different API depending on the platform, tcl smooths out platform differences providing a unified API across platforms.
Tcl treats serial ports like files so you just open it to talk to it. Granted, different platforms provides different ways of naming the ports. So on Windows, to talk to the serial port you simply do:
set rs232 [open COM1 w+]
On unixen (Linux, MacOSX etc.) you'd do:
set rs232 [open /dev/ttyS0 w+]
To set the baud rate and parity bits you can now do
fconfigure $rs232 -mode "9600,n,8,1"
Be sure to also configure the serial port to binary mode otherwise tcl will re-interpret "\n" for you depending on your OS:
fconfigure $rs232 -mode "9600,n,8,1" -translation binary -blocking 0
The "blocking" bit is to set it to non-blocking mode so that we can write event oriented code which is critical for UI apps since we don't want the IO to block our UI.
So now for a simple program that reads data from the serial port:
package require Tk ;# in case we're running in tclsh
if {[catch {set rs232 [open COM1 w+] err}]} {
tk_dialog .error Error "Could not open COM1: $err" error 0 OK
}
fconfigure $rs232 -mode "9600,n,8,1" -translation binary -blocking 0
# Draw a simple UI do dump data to:
pack [label .l -text "Gyroscope outputs:"]
pack [label .g1 -textvariable gyro1]
pack [label .g2 -textvariable gyro2]
pack [label .g3 -textvariable gyro3]
# Now the fun part, read from the serial continuously with fileevent:
set buffer ""
fileevent $rs232 readable {
append buffer [read $rs232]
# Here, you need to parse the data returned.
# Due to the byte-wise nature of serial ports the data read may not be
# complete so we need to check if it's complete. "Completeness" depends
# on you. For example the data packet may end with a newline. Or it may
# simply be a timeout between data packets.
if {[message_is_complete $buffer]} {
set gyros [parse_message $buffer]
set buffer ""
# Update the UI:
foreach x $gyros y {gyro1 gyro2 gyro2} {
set $y $x
}
}
}
Minus the comments and empty lines that's just 16 lines of code. You can of course go further and implement a more fancy UI rather than just text labels.
If you need to send data to the microcontroller simply puts to it. Just remember to override tcl's automatic newline termination:
puts -nonewline $rs232 $somedata
Use tcl's binary command if you need to format to and from binary.
I personally still prefer tcl for this because you can get a prototype up and running to read the output from your hardware in under 5 minutes. But you also mentioned interest in a node.js solution. So here's an alternative starting point for node.js
First you need to install node.js. This may or may not be trivial depending on your OS. You will need Python 2.x (not 3.0) installed on your system in order to compile node.js modules writen in C (and even node.js itself). Even if you're installing a binary distribution of node you will still have to compile the serial port library since it's not included in the core.
Next you will need to install the serialport module. The instructions vary depending on your OS. See the npm page for the module: https://npmjs.org/package/serialport
Once you've got things up and running you can simply communicate with your serial port using the serialport module:
var SerialPort = require("serialport").SerialPort
var rs232 = new SerialPort("COM1", {
baudrate: 57600
}, false);
Again, like the tcl case, the name of the serial port depends on your OS. The above is an example for Windows. For unixen you'd do:
var rs232 = new SerialPort("/dev/ttyS0", {
baudrate: 57600
}, false);
For UI, node doesn't have a core toolkit like tcl. There are many libraries on npm available allowing you to use QT or GTK or some other widget set. But the most common way of doing UI on node is to simply run it as a web server and connect to it with your web browser. The downside of this is a little additional complexity and the need to mess with HTML. The upside is HTML & CSS! Which means you can leverage tools like CSS3 effects and jQuery to create a really slick UI with very little effort.
Unfortunately, writing an interactive webapp (even a simple one) is fairly involved so I won't be providing a "simple" example implementation. I'd only suggest you read up on the more popular frameworks like express.js and templating libraries like dust or handlebars. You may also want to take a look at socket.io to simplify the implementation of real-time communications between your server and the web browser.