I have a problem with GitLab Runner on 32-bit Windows. The runners are at version 14.4.0 and our GitLab instance is at version 14.4.1-ee. The runners are tied to specific machines running 32-bit Windows 10 Pro (10.0.19043), use shell executors (PowerShell), and run with full administrative privileges (i.e., as the local system user). This is outside my control.
Sporadically, and for no discernable reason, the runners stop sending log traffic to our GitLab instance. They should be uploading several MB worth of logs. I don't see failed attempts to upload logs in debug mode. I don't see any of the network traffic I expect in WireShark. This might correlate with issues loading a custom driver, but I can't say for sure.
The workaround is even more perplexing. The following protocol fixes the issue: remove all the runners using the GitLab CI interface; uninstall the malfunctioning runner; download a new runner binary, register and install it. If I repeat the same steps, except without downloading a new binary, the issue persists. The files are identical when I run a binary diff on them.
I haven't been able to extract any relevant information from the system event logs or network traffic. The issue only affects our runners on 32-bit Windows. It doesn't affect 64-bit Windows or runners on Linux, regardless of architecture. It seems to happen sporadically, in the sense that I can't correlate it with anything interesting happening on the affected machines.
Clearly, something about our 32-bit Windows environments is different and causing the runners to malfunction. I just don't know what it is. I would appreciate any direction figuring out the source of this problem. The fact that downloading new binaries makes the difference has me worried, but I don't have any reason to suspect our machines have been compromised.
This problem was resolved by running tests remotely over SSH. It's almost certainly a bug with the 32-bit Windows distribution of gitlab-runner.
Related
I'm trying to introduce continuous integration in an old project, and we've got quite specific situation - it's possible to put the CI server only on our test server that runs on CentOS. The server has quite a lot of unused RAM and CPU capability.
However, we need to run Ant builds on Windows (this also used to be how the project did packaging before), however it turned out that not the same output (after binary compare) is produced by just using Unix versions of Java and Ant.
I drew up a diagram of how in my mind it could work, but I'm really wondering whether that is even possible (with already given tools).
The black part is implemented, I'm curious whether the red part could be possible. Could the Jenkins slave communicate with master on different OS?
It should be possible. I have a feeling you will need to play with your network settings. But if before you start changing anything see if you can start a headless slave by following these directions: https://wiki.jenkins-ci.org/display/JENKINS/Step+by+step+guide+to+set+up+master+and+slave+machine
Using VirtualBox for CentOS, it will possible to run a Windows VM on your CentOS host.
I'm not sure you need Docker to launch your Jenkins slave.
It maybe better to use a standard JNLP Windows service to connect your Windows slave to Dockerised Jenkins master.
If the master is not able to view the Windows node using this method, you may have to tweak your network configuration on the Windows VM.
But I'm not sure it's necessary.
I am trying to build and deploy a gaia build from git repo in Windows. I am trying to deploy it in অ Flame.
I am trying to do it in a windows 7 with cygwin installed. After installing everything this is the error I am getting
This works just fine in a Linux machine, but I need to do this in Windows since right now I have access to it only.
Any pointers to what I am doing wrong here?
I'm afraid it's not going to work without significant effort for several reasons. Much better to use a VM with Linux on as even if it did work it will be really slow. Windows is slow at handling lots of file access and Cygwin slows it down even more.
For example in making a simple change to config.sh (full stack build) so it works on Cygwin I found it took hours to run (on a decent PC). And then I had a couple of corrupt git repos I had to hand fix.
I also looked at getting gaia's make to work, but stopped after the problem just got bigger.
Here's what I found for future reference
The build is not really portable, it expects a linux like environment
While cygwin gives good linux emulation most of the tools run are win32 native and handling path conversion for them requires not trivial changes due to assumptions. For example you can switch to the Win32 XPCshell and hack the command line paths to use cygpath, but environment variable are an extra source of dependency in the JS scripts and are all unix paths. ( I did manage this part).
these path and environment dependencies get magnified with the C build chain and other tools.
You need to change the mount to use noacl or else cygwin attaches ACLs to simulate file properties, thus breaking things. It's might even be a little faster without ACLS
I also tried MinGW which provides native versions without the emulation so should be faster. However it falls short of the requirements and its automatic path conversion heuristics get in the way.
you need to turn of any antivirus prog as they slow it down. in fact the very first time I used the old FIrefox WIndows build it would crash after a long time. Turned out to be a mem leak in the AV :(
So all-in-all it's too much hassle in terms of dev time to convert and probably maintain. A true Windows build would be better but then it's so easy these days to run a VM. You can even share directories between the guest and host so could flash from Windows.
I also tried with cygwin, but was unable to build the gaia source code on windows.
It's not straightforward to build the gaia source code on windows. Please follow these steps:-
Download Mozilla Build from MozillaBuild - Mozilla Wiki and install the tools in c:/mozilla-build (preferred). It includes everything (make, wget, python etc) you need to build gaia source code.
Run start-shell.bat. If build process failed with this batch file then run start-shell-msvc2013.bat if you have Visual Studio 2013 or start-shell-msvc2015.bat if you have Visual Studio 2015. (You need Visual Studio for the second step).
Browse to the gaia source code directory using the command cd Mozilla/gaia.
Run DEVICE_DEBUG=1 make command. Don't run DEVICE=1 make or make command (because you won't be able to debug the apps, I was able to connect to the Firefox OS 2.2 but was not able to debug the apps when I ran these commands).
If you are running this command for the first time, it will download the b2g_sdk otherwise it will create a folder profile with your custom profile.
Open the WEBIDE using Firefox (Nightly preferred) and point to the profile folder you just created.
Links for your reference:-
https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Developing_Gaia
https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/Different_ways_to_run_Gaia
https://developer.mozilla.org/en-US/docs/Tools/WebIDE/Troubleshooting
https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/Making_Gaia_code_changes
https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Windows_Prerequisites
I would like to run Teamcity (with a build agent) in a Linux VM to handle our none-.net projects. But in the same breath I'd like to have a BuildAgent setup on a Windows server to handle all of the .net projects.
I can't think of any reasons why this wouldn't work but has anyone any experience and any ideas about the problems I might encounter before I spend too much real time on this?
Ta
It's fully supported. TeamCity also knows which agents to route builds to.
This is a very normal scenario and many project I know do this without any problems. Just make sure that for the builds' Agent Requirements, you properly direct the appropriate job to the appropriate agent. One criterion can be that agent.os.name should contain Windows or Linux etc.
First with the background...
We have a Linux server that supports multiple projects.
The Clearcase server and repository are installed on this Linux server.
Different projects require different cross-compilers and libraries, and all of them are installed on the server.
User can choose different tool sets by running different scripts, which exports different environment variable values such as include paths and compilers.
User needs to run cleartool to mount the repository.
Developers develop in Eclipse and have two options:
SSH into the server and run Eclipse through with X11 tunneling.
Install Eclipse locally on their Windows machine and invoke builds from the SSH terminal.
Now:
Problem with #1 is that Eclipse operations (typing, content assist, etc) are extremely laggy.
Problem with #2 is that the developers need to go through extra hoops to build their code.
This is what I have tried:
Set up Remote System Explorer, which allows remote editing of files and remote running of the compiler:
How to build a c++ project on a remote computer in Eclipse?
This approach works perfectly for files that do not need special environment variable values and mounting of Clearcase repository, but I could not figure out how to get all of these things to integrate.
It would be great if someone can let me know how I can direct RSE to run a script (may be different per project) to set the environment variables and to run the cleartool commands to mount the repository so that it can locate the files.
The cleartool command arguments would be different per user for setting up a particular view.
Some extra info that may help:
I have root access to the development server
The Clearcase filesystem is mapped to a drive on the Windows machine
Thanks in advance for saving me hours of frustration dealing with a slow network!
==================
Additional detail per comments:
- The VOB storage is located locally on the Linux server. We would SSH to the server and start Eclipse there, therefore the delay should not be due to dynamic vs snapshot view and GUI performance seems to be the real problem.
- We also mount the same view on Windows by using Region Synchronizer. When running the local copy of Eclipse installed on Windows, there is no performance problems.
So this question can probably be solved by answering either question:
1. How to improve X11 performance such that development on Linux will suffice?
2. How to set up Windows Eclipse to perform all the steps mentioned above when building projects?
I came here a similar question to your part two, but alas, no one has answered it. However, I have an answer to your part one: https://www.nomachine.com/. It speeds up X11 forwarding considerably.
I know that I can code on one machine and have it build on a different machine (ie. a build server). Now I have also heard that you can have visual studio run a build on a virtual machine (i think it requires Virtual PC). Now my question is if anyone has been able to code on machine A, have it compile on machine B and run a debugging sesion on machine C?
This is pretty common in enterprise development and just about the de facto standard way of doing things.
Typically, a dev works locally. Once s/he is happy with their changes, they'll check it into a source control system.
From that point there are a couple of options ranging from automated building to having someone push the button to cause the remote build.
Once the build is complete there are a host of options available for deploying the app to one or more other servers. And yet other options for kicking off automated test suites.
Concerning remote debugging, you can do that independently of whether you are using a build/deployment/automated testing. It's just a matter of getting the right stuff installed and configured (see ho1's answer for a link).
All of that said, I highly recommend you never enable remote debugging on a production server. Some people might disagree with me but I personally think it's dangerous for security reasons and can certainly lead to site outages.
Finally, the only reasons you would need a virtual machine is if the servers aren't available or if you just want to sandbox everything.
You can do remote debugging, so if you had an automated process to copy the compiled code from B to C, I suppose you could do what you're asking.
See this MSDN article for more details: How to: Set Up Remote Debugging