Closed. This question is not about programming or software development. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 3 months ago.
Improve this question
I'm doing a project that needs to generate a vm image file which will then be used as qemu bootable disk image. Previously, our product is like an modified linux system and is made into a USB installation drive and then boot and install into an bare metal machine. But now we want to get rid of the hardware and run it in virtual machines, that's why we need an image file.
Instead of using the existing USB drive to install the system in qemu then shutdown the vm and get the image, we were asked to make an out-of-box image file directly, and skip all the booting and installing on real machine or virtual machine but still get a installed image, so that we can just deliver this image and people can load this image as a ready-to-use virtual machine image.
But during all the procedure, I can NOT use any command that requires root privilege! Don't ask why, there's a whole bunch of restrictions to our project, I just can't use root privilege, no sudo, no su, just anything only a regular user can do....
The part I already achieved is using the latest version of mke2fs -d command to populate a tree of folders and files into different partitions of this image file like this
suppose after the image is booted, we have these folder structure
$ ls /
$ bin dev boot home lib32 mnt proc run srv tmp var boot data etc lib lib64 opt root sbin sys usr
some of the folders are mounted by different partitions
extract a single partition from image
dd if=image of=partitionN skip=offset_of_partition_N count=size_of_partition_N bs=512 conv=sparse
populate a folder into the partition
mke2fs -d root_dir/etc partitionN
put the partition back into image
dd if=partitionN of=image seek=offset_of_partition_N count=size_of_partition_N bs=512 conv=sparse,notrunc
We have the first partition in the image as the boot partition, which contains the 'boot' folder and will be mounted under /boot once it is booted.
And this boot partition is an EFI compatible partition(which actually seems to be a FAT32 format), since our project needs it to be this way.
BUT
After get all partitions successfully populated into the image, I can not find a way to install grub for this bootable image. And that's the most damn important step that needed to make this image bootable.
All solutions I found on the web suggest loop mount the image's boot partition, which I can not do because without root privilege I can't loop mount the image.
So does any one have any idea how to do this?
I tried to understand how grub write raw values into mbr, and how to find stage1 and stage2 from the values inside mbr, and how to figure out the sector list at the end of stage2's first sector, but that's so crazy and I eventually failed to get this trick work.
Disclaimer: if facing this problem myself I would attack the problem directly by making a grub2-mbr-image installer myself. It's a true attack on the problem and would make a better answer, and more on topic for this site; however it's more hours of research than I'm willing to put into it for a stackverflow answer.
There's an extra trick here. We can get success/error code back by using a virtual floppy disk. There's no need to ship the floppy disk driver. You can build it as a module and include it only in the cpio image.
If we are willing to forgo the kqemu component and pay the 10x slowdown price, we can start qemu with -hda image.img, -kernel bzImage and -initrd initrd.cpio.gz and it will boot that. You need an X server (which you can provide with Xvnc) but no privileges a normal user doesn't have. Assuming / and /boot are the same, /linuxrc looks like this:
#!/bin/sh
insmod /lib/modules/kernel/floppy.ko
mount /dev/hda1 -t ext2 /mnt
PATH=/bin:/usr/bin:/usr/sbin:/sbin chroot /mnt grub2-install
RESULT=$?
umount /dev/hda1
mount /dev/fda -t fat /mnt
echo -n $RESULT > /mnt/errorcode
umount /mnt
poweroff
And you can get your error code back with mcopy to read the floppy disk image.
If qemu is not available, you can build a i586 compatible kernel and use dosbox-x instead and start the kernel using loadlin.exe. This actually works. If you try it with modern stuff it just dies because stuff demands i686 now; but you can build the grub install tool itself targeting i586 and just use an old kernel to boot to do the install. https://www.vogons.org/viewtopic.php?t=53531
Caveat: This is not a complete solution, however I'll post since nobody else answered.
I have never tried to do this and don't have a couple of hours spare to test it, however The OpenWrt project has standard x64 disk image files including Grub and kernel which you can find here:
https://downloads.openwrt.org/chaos_calmer/15.05.1/x86/64/openwrt-15.05.1-x86-64-combined-ext4.img.gz
Instructions tell you how to convert the images for VMWare, for Qemu it must be similar:
https://wiki.openwrt.org/doc/howto/vmware
The thing is, OpenWrt philosophy has always been that builds shouldn't be done as root, and it generally refuses to build as root, so I think you'll find they have ways of creating EXT4 filesystem images complete with MBR and Grub. I have only tested embedded platforms and never actually built from source for x86, but this is where you should start if you're stuck.
Of course, the OpenWrt disk has only a single partition, I'm unsure how you'd create a virtual disk with a more complex partition table, but perhaps there are some options in the tools used by OpenWrt.
Related
I'm trying to use docker beta on OSX, mainly for Symfony development but the mounted volumes are incredible slow. Even for a vanilla Symfony project I get 6s page load time. That's unbearable! Has anyone found a solution to this issue? Trying to move away from vagrant but I just can't find any reasonable way to work with docker instead.
Okay the user Spiil gave a solution but I wanted to elaborate on the exact steps to take since I went through 12 hours trying to figure it out, but once you know how its super easy and fixes all the slow down issues!
The key here is to understand this solution creates NFS (Network File System) drives as the means of communication from the Docker Containers to your Mac instead of the standard OSX File System which is very slow currently either due to bugs or the way it works*
Follow these steps exactly.
1.) Clone this repo here (https://github.com/IFSight/d4m-nfs) in your home directory. To do this open up terminal and type cd ~
Then type git clone https://github.com/IFSight/d4m-nfs
Alternatively you can also do this in a one liner git clone https://github.com/IFSight/d4m-nfs ~/d4m-nfs
2.) Next go into the d4m-nfs folder and create a new file in the /etc folder and title it d4m-nfs-mounts.txt
3.) Add the following lines of code to this.
/Users/yourusername:/Users/yourusername:0:0
What the above does is allows you to still use relative folders with docker-compose and allows all ports to connect on it hence the 0:0.
EDIT
Do not put /Volumes here!!
4.) Go to your docker preferences and do the following
Make sure only /tmp is showing and NOTHING ELSE. I mean nothing else it won't work if there is anything else since it will create conflicts with the NFS systems that the script will make for you later. Restart docker and docker-compose down any containers as well.
5.) Finally navigate to the d4m-nfs directory we created in step 1 and type the following command, /bin/bash d4m-nfs.sh
edit The correct way to type the command above is this as another user from the github (if-kenn) pointed out, ./d4m-nfs.sh which uses the Shebang for what shell should run it.
If done correctly there should be no errors and this should work. Please note DO NOT RUN as sh d4m-nfs.sh this will create errors and you will have to delete your exports file to start over. In fact anytime you make any changes you will have to clear your exports file.
This is what mine looks like.
EDIT:: IMPORTANT -- Remove the /private and volumes! This should only be users/username now!
If you see anything other than this you were not running with bash. You can quickly get to the exports file like this in Mac if you make any errors and just clear it out to start over.
Just select go to folder
and then type /etc/exports
This is a nice shortcut to quickly get to it and clear it out in your favorite text editor.
Also make sure no containers are running or you will get the ........ loop of death. If this loop of death continues make sure you upgrade docker and then restart your computer. Yes restart... it seemed to be the only way to get it to work on my friends computer. Refer to this (https://github.com/IFSight/d4m-nfs/issues/3)
Note to .... loop. I recently found another solution. Make sure you are NOT logged as root, and make sure you pulled the git repo into your users ~ folder not the root ~ folder. In otherwords, it should be in Users/username.
Also, make sure /tmp folder has full write permissions since the script needs to write here or this won't work either. chmod 777 -R /tmp
6.) If you did it right when running the script it will look like this.
Then simply run your docker-compose up -d as usual in your symfony project folder (or whatever project you are using with docker) and everything should work... except NO MORE slow downs!
You will need to run this anytime you restart your computer or docker.
Also note if you get mounting errors showing up, you probably don't have your project stored in your Users/username directory. Remember that is where we mounted it. If your project is somewhere other than there you will need to modify the d4m-nfs-mounts.txt file accordingly.
Other Info:
For people reading this now, maybe it's better to wait for Docker to fix this issue. A pull request has already been accepted to improve performance(https://github.com/docker/docker/pull/31047).
This will be release somewhere in April 2017 and should be a big improvement.
I've tried some workarounds for Docker for Mac, but all of them had some pretty big disadvantages, mostly in usability. A good source for alternatives of the OSXFS can be found at: https://github.com/EugenMayer/docker-sync/wiki/Alternatives-to-docker-sync. Credits for Eugen Mayer for setting this up.
EDIT:
First improvement is implemented in the edge release. https://github.com/docker/for-mac/issues/77 has more info on this.
There's a long thread with explanation from Docker Team and various workarounds.
Currently, the issue is being tracked on GitHub.
While some workarounds may be better than others, I'm afraid the ideal option for now is to switch to Linux.
I spent a lot of my time in searching viable solution. And I found.
d4m-nfs
allow you use docker volumes via nfs.
In my case it gave increase performance 16 times! (1.8sec vs ~30sec)
Also d4m-nfs has quite a intricate manual, so here is another link with detailed example: https://github.com/laradock/laradock/issues/353#issuecomment-262897619
I just leave this here for other googlers.
Normaly volumes should be fast.
But you can not change anything to make them faster if you dont want to change the format of your disk.
But maybe the bottleneck is the CPU or RAM.
You can check that with the command docker stats. These are by default set to 2 cores and 2 GB RAM. You can change this in the Docker for Mac GUI.
I had exactly the same thing. For me using docker-bg-sync (see on GitHub) made a dramatic improvement in speed and CPU usage.
Not as nice as just mounting the volume as you have to start a new container for every sync but it does the job.
In latest docker 17.06.0-ce-mac18 volumes mounted with :cached seems to run quite decent.
I've found that creating a CoreOS VM under Parallels, then using the Docker that is inside CoreOS is far faster than Docker for Mac (currently running Version 17.12.0-ce-mac49 (21995)).
I'm doing Linux code builds using CMAKE/Ninja/GCC and it's almost twice as fast as the exact same build from Docker for Mac.
In my case, I have a ton of library sources that are part of the container (e.g. Boost, OpenSSL), and a decent amount of C++ code that I keep local to my Mac.
This seems to be a recent development. Docker/Mac has become much slower than I remember it being a month or two ago. Maybe it's just me...
We overcame this issue by synchronizing the local and the docker for mac filesystem using syncthing. We built an open source tool that follows this approach, in case it helps:
https://github.com/okteto/cnd
I need to provide a single update file to a customer in order to update an embedded system via USB. The system is built using Yocto. I'm curious if the plan I have to implement USB updating is viable, or if I'm missing something that should be obvious.
opkg exists on the system, but in order to use opkg update it needs to have a repo to pull from. Since I have no network capabilities I will need to put the entire repo on a USB drive. Since I need to provide a single file to the customer the repo will then need to be a tar file.
Procedure
Plug USB drive in
The udev rule calls a script and pushes it to the background since this will be a long process (see this)
Un-tar the repo update file
opkg update
Notify user they may remove USB drive
At least from a high-level point of view does this sound like it is a good way to update an embedded system via USB? What pitfalls might exist?
Your use case is covered by SWUpdate - it is maybe worth to take a look at my project (github.com/sbabic/swupdate). Upgrading from USB with a single image file is one of the use case, and you can use meta-swupdate (listed on openembedded) to generate the single image with all artifacts.
Well, regarding what pitfalls might exist, one of the biggest pitfall would likely be a power outage in the middle of the process. How do you recover in that case? (The answer might depend on what type of embedded device your making). (Personally, I'm in favour of complete image based upgrades and not package based upgrades).
Regarding your scenario with the tarball, do you have enough space on your device to unpack out? It might be wiser to instead of distributing a tarball with your opkg repository, to distribute eg a ext2 or squashfs image of the repository. That would allow you to mount it from the USB drive using the loopback device.
Apart from that, as long as you have a good way to communicate work the user, your approach should work. The main issue is what do you do in case of an interrupted upgrade process. That is something you need to think about in advance.
When deciding on the update format and design there are several things to consider; including what you want to update (e.g. kernel, applications, bootloader); there is a paper on the most popular designs here: https://mender.io/user/pages/04.resources/_white-papers/Software%20Updates.pdf
In your case (no network, lots of storage available on USB) the easiest approach is probably a full rootfs update. I am involved in an open source project Mender.io which does a dual A/B rootfs update and integrates with the Yocto Project to make it easy and fast to enable updates on devices without custom low-level coding.
Here is my situation, when I download Gentoo and start to run it and downloaded the stage III Tarball from links and then tried to extract it a stream of white sentences flows down my screen really fast for about a minute just like in the YouTube tutorial I was viewing. However, after that instead of going to the correct stage it says cannot right not enough space on device and I tried repartitioning it but I'm not sure what device it is talkingaboutHowever, after that instead of going to the correct stage it says cannot right not enough space on device and I tried repartitioning it but I'm not sure what device it is talking about. Please help
Sorry you're having this issue, though in general, I truly believe Gentoo Handbook is quite well written and even a newbie can follow it... Here are some advices that I hope I can give you (most important is, digest the handbook and follow it carefully, not that I'm saying "RTFM", it's just that for Gentoo, handbook is essential and without it, we can get lost if you're just starting).
From my experiences, the "stream of white sentences" I'd presume would be from verbose un-tar'ing your stage3. Usually, I only want to see the errors so my suggestions is to remove the "v" (i.e. from "tar xjvpf" to "tar xjpf") so that only errors would appear when un-tar'ing. The caveat to this is that you'll be wondering if it hung or is busy un-tar'ing. Use Alt-F1 and Alt-F2 (if on console/tty mode back-and-forth) to log in on another TTY and do 'ps -auxf' to see if it's still tar'ing. If you're using GUI Terminal, just open another tab and 'ps auxf', you get the picture...
Also, learn the commands "df", it'll come in handy. If you're running out of disk space, perhaps you're trying to install/untar stage3 to your ramdisk (grin) rather than your mounted (i.e. "/mnt/gentoo"). Mount your root '/' device to '/mnt/gentoo' and cd to that mounted path then try it (don't forget to mount your '/boot' as well as your proc, dev, sys, etc before you chroot - again, follow the handbook as carefully as you can - oh also, distro such as Debian hybrids including Ubuntu uses symlink to shm, so read that part about 'rm /dev/shm' and follow it carefully; if you're using Gentoo LiveCD, you can ignore that part).
Other useful commands if you're confused (or new to) mounting devices would be to learn to experiment with commands such as 'lsblk' and 'mount' (by itself) to inspect the sizes of your partition (again, use of 'df' comes in handy as well) as well as what is your device (i.e. /dev/sda1 versus /dev/sdb1). Hint: when you do 'mkfs', use "-L" (or for some file system, it's "-N") to label/name your devices, so that when you use commands such as 'mount' or 'lsblk', you can spot them easier. If you're using GUI/desktop versions of some distro, hopefully there are tools such as "gparted" which can give you visual information in GUI of your devices which can be helpful. One think I'd advise you to stay away from if you're just starting, is to avoid RAID (i.e. mdadm) until you're comfortable with how grub/lilo works. Get your kernel (Gentoo-sources) compiled and MBR written (i.e. grub-install), try booting and have fun first (oh also, if you can avoid GUI like installing Gnome/KDE from the get-go, avoid it as well - you'll get into issues such as "should I use SystemD or OpenRC" and then get hit by the obstacle of some gnome parts needs you to use systemd but you've chosen openrc, and so on).
If I may add my opinions, in my opinion, Gentoo (also Arch and FreeBSD) is an excellent place to start if you want to learn the inside of Linux application workings (library dependencies, why packages are important rather than downloading each libs manually and compile them one by one, etc). I hope this won't discourage you from switching to another distro, but if it does frustrate you on installation and all you want to do is test-drive Linux, there are much easier distro that you'd not have to understand USE and other compilation mechanisms (if you have an old i586, it makes sense to build it with pick-and-chose libraries so that leaner can be faster, but if you have fast machine, why compile binaries when somebody who is expert at it already have done it for you?). SUSE and Fedora/RedHat/CentOS used to be the least frustrating for it was able to find/detect hardwares (legacy and new) but these days, I usually tell people, "if you know how to install Windows, you can install Ubuntu" so that too may be a good way to wet your feet. Good luck!
0_o wow, well.. how about some 411 like size of your hdd and exactly how you partioned it? Linux will look for specific directorys and if missing will instead start to install into the root dir. How you partion is an importent first step. Once you got a generally good partion setup most linux installs will go fine. Most basic tables include /, /home,/var and a swap.
When I boot up Linux Mint from a Live CD, I am able to save files to the "File System". But where are these files being saved to? Can't be the disc, since it's a CDR. I don't think it's stored in the RAM, because it can only hold so much data and isn't really intended to be used as a "hard drive". The only other option is the hard drive... but it's certainly not saving to any partition on the hard drive I know about, since none of them are mounted. Then where are my files being saved to??
Believe it or not, it's a ramdisk :)
All live distros mount a temporary hard disk in RAM memory. The process is completely user-transparent and is all because of the magic of Linux kernel.
The OS, in fact, first allocates an area of your RAM memory into a virtual device, then mounts it as a regular hard drive in your file system.
Once you reboot, you lose all your data from that ramdrive.
Ramdrive is needed by almost all software running on Live CDs. In fact, almost all programs, in particular desktop managers, are designed in order to write files, even temporary, during their execution.
As an example, there are two ways to run KDE on a Live CD: either modify its code deeply in order to disallow you to change wallpaper etc. (the desktop settings are stored inside ~/.kde) or redeploy it onto a writable file system such as a ramdrive in order to avoid write fails on read-only file systems.
Obviously, you can mount your real HDD or any USB drive into your virtual file system and make all writes to them permanent, but by default no live distro mounts your drives into the root file system, instead they usually mount into specific subdirectories like /mnt, /media, /windows
Hope to have been of help.
It does indeed emulate a disk using RAM; from Wikipedia:
It is able to run without permanent
installation by placing the files that
typically would be stored on a hard
drive into RAM, typically in a RAM
disk, though this does cut down on the
RAM available to applications.
RAM. In Linux, and indeed most unix systems, any kind of device is seen as a file system.
For example, to get memory info on linux you use cat /proc/meminfo, where cat is used to read files. Then, there's all sorts of strange stuff like /dev/random (to read random crap) and /dev/null (to throw away crap). ;-)
To make it persistent - use a USB device - properly formatted and with a special name. See here:
https://help.ubuntu.com/community/LiveCD/Persistence
I installed openembedded and tried building a couple of images for Zaurus SL-6000 "Tosa", basically, helloworld-image and console-image. And I ended up with an angstrom-dev/deploy/glibc/images/tosa directory that contains files like this (slightly truncated from a forum post I made elsewhere):
Angstrom-helloworld-image-glibc-ipk-2009.X-test-20090529-tosa-installkit.tgz
Angstrom-helloworld-image-glibc-ipk-2009.X-test-20090529-tosa.rootfs.jffs2
Angstrom-helloworld-image-glibc-ipk-2009.X-test-20090529-tosa.rootfs.tar.bz2
Angstrom-helloworld-image-glibc-ipk-2009.X-test-20090529-tosa.rootfs.tar.gz
helloworld-image-tosa.tar.bz2
helloworld-image-tosa.tar.gz
initramfs-kexecboot-image-tosa.cpio.gz
initramfs-kexecboot-image-tosa.jffs2
initramfs-kexecboot-image-tosa.tar.bz2
initramfs-kexecboot-image-tosa.tar.gz
modules-2.6.29-r0-tosa.tgz
updater.sh.tosa
zImage-2.6.29-r0-tosa.bin
zImage-kexecboot-2.6.24-r0-tosa.bin
zImage-kexecboot-tosa.bin
zImage-tosa.bin
I have no idea what all these do or how to install them properly. What I did try is various combinations of flashing a zImage.bin and initrd.bin using option 4 of the maintenance menu (as specified per earlier instructions). The flashing usually works alright but then when it boots, it loads a bootloader that cannot find any bootable devices. On a hunch, I tried unpacking one of the tar.gz images to an ext2 formatted SD card and tried booting with that plugged in and it was detected by the bootloader. Booting it sort of worked but it quickly exited back to the bootloader (I assume that was just a problem with the image I unpacked).
My questions are:
What is the correct usage for all of these file types, i.e. should the .jffs2 files be renamed initrd.bin and included in the flashing process? What am I supposed to do with the bz2 and gz files? Are they only for unpacking to external media?
How do I install to the internal flash? It used to work with the stable Angstrom 2007-12 build and instructions.
Is there a newer version of updater.sh (that one was not built by oe and I added it myself having picked it up from elsewhere)? The reason I ask is that when trying to flash zImage-2.6.29-r0-tosa.bin it fails during the update program with the error that the file is too big. That kernel is approximately 1.3mb while the others are 1.2mb. Is this a constraint of the SL-6000 itself? I thought it has 32mb of internal memory.
Unfortunately, none of the available documentation that I could find online talks about installing these files. I did find a small entry in the "Angstrom Manual" which talks about what they are but not how to use them as they are all device specific. Unfortunately the tosa documentation only talks about copying the files from an installkit and flashing the device from the maintenance menu.
Okay, "ant" over at OE forums was able to answer my questions ^^ Just recording the answer here for posterity.
installkit-tosa.tar.gz, contains updater.sh and zImage (the kexecboot-kernel). This kexecboot-kernel can be and is likely different from the kernel you will have on the rootfs after the machine boots. Unpack the installkit on a formatted card and follow the flashing procedure for the device.
Regarding the also be various image-rootfs.tar.gz, .bz2, and .jffs2 files. These are the root file systems that will be be booted by kexecboot. The tar.gz or .bz2 archive should be unpacked onto an ext2 (or possibly ext3) formatted SD or CF card. It will be detected by kexecboot at boot time and appear in the kexecboot menu.
If you want a rootfs in nand (installed internally), rename your-image-rootfs.jffs2 to initrd.bin and copy it on the card with updater.sh (then flash).