I'm using GitLab self-hosted at work and I am using the CI/CD they offer with a custom macOS runner for iOS projects. I can't change that, this is company policy and we've got our runners in our own datacenter.
I've got a private pods repo on GitLab, so whenever I update one of our modules, that gets pushed to GitLab and the Cocoapods repository. In the Podspec, I define the GitLab repo URL with SSH. However, this doesn't work with GitLab CI/CD, because I somehow need to use a CI job token, otherwise Git will complain that I have no access. The same goes for Xcode/SPM.
Is there a way to leave my pod specs or SPM definitions as is, without hardcoded need for CI job tokens in my developer environment or am I only going to be able to use a mono repo on GitLab?
Related
I am having this GitLab CI/CD which builds then tests and pushes my projects container to GitLab container register successfully. But now I am wondering how I can do the deployment stage automated too. currently, I am doing it manually and after each successful pipeline, I SSH to my server and run several commands to pull the latest images from the GitLab.com container registry and then run them. But I would like to make this step automated as well. Yet, I don't know how?
Actually I have seen some examples of opening an ssh session from CI/CD pipeline but it doesn't feel secure enough. So I was wondering is there a better way for this or I have to actually do this.
Not that I am using gitlab.com so the GitLab server is not installed on my machine and I can't share assets between them directly
There are many ways to achieve this, depending on your setup, other requirements, scale etc.
I'll just give you two options.
I. Kubernetes
create cluster (ie control plane) somewhere
add your cluster to GitLab (now GitLab can even create cluster for you in AWS and GCP, check this page)
attach your target machine as a worker node to the cluster
create Kubernetes YAML files \ Helm chart for your application and deploy via usual ways, e.g. kubectl apply -f ... or helm install ..., or rely on Auto DevOps to do this step for you
This is quite complex but sort of "right" way of doing things.
II. Private GitLab runner
go to Settings > CI/CD > Runners of your GitLab project or group
obtain the registration token
install your own GitLab runner right on the target machine, and register it on the GitLab server using registration token, see example
give runner some specific tag
use that tag in your .gitlab-ci.yml file, see documentation
then deployment process is just a local process of docker pull... and docker run ... for your image
This is a lot simpler, but is a "wrong" way, as you are mixing CI\CD infrastructure with target environment.
I am using gitlab as repository and want to push my code on ec2 whenever any commit is done on gitlab. The gitlab CD/CI documentation states that I have to add a file .gitlab-ci.yml at the root directory of my repo. This is actually a problem for me because, I want project repo to have only code and not any configuration related info like build and deploy etc. Also when anybody clones the repo, they would have access to location where my code is pushed/deployed on ec2. Is there any work around for this problem ?
You'll need to use a gitlab-ci.yml filke to deploy your application. The file provides instructions and a pipeline "infrastructure" which, if properly configured, will build, test and automatically deploy your code.
If you are worried about leaking credentials, you should use the built-in instance variables to mask your important bits, like a "$SERVERNAME" or "$DB_PASSWORD" for instance.
Lastly, you can use the power of gitignore, in order to not publish all of your credentials or sensitive bits to your projects' servers or instances.
I need to configure CI pipeline in my TFS with source repo in GitLab, I have updated the Trigger section as below and checked in the code, but the build was not triggered, is there any configuration required in the GitLab repo? Please advice hot to achieve this.
I would troubleshoot your problem by checking the following in this order:
when triggering the build manually, does the source code downloading from the GitLab repository successfully work?
if yes, then:
did you created and pushed the develop branch on the repository?
You need to publish your locally created 'develop' branch on the remote repository, e.g. >git push -u origin develop
if no, then there are issues on the "Services" entry you created on TFS for accessing your GitLab repository:
check whether you are using HTTPS in the URL;
to enable HTTPS access to your repository, remember to set a password on your GitLab account, then use that password along with your username (email address) on the "Services" endpoint set up on TFS;
p.s. also, check which 'Connection' is selected in the Repository section of your build definition: it should be the one pointing to your GitLab repo :-)
I have changed the Poling interval to 180 seconds and it worked.
Background: I have a project deployed to heroku. The heroku app is connected with github so I can press the "deploy" button from heroku's web api to manually deploy a branch on github to heroku.
What I'm trying to do is build a slack bot that will let me accomplish that via a slack command. Ideally, there'd be some function on heroku's platform api like .deploy('my_app', 'some_branch_on_github), but I can't seem to find it.
The platform api's build feature is close. That function lets you provide the public url of a tarball that heroku will then deploy. However, my github repo isn't public, so that doesn't work. A private repo shouldn't be a problem, though, since heroku's already connected to my github repo.
TLDR: How can I programmatically tell Heroku to deploy my app from a private github it's connected to?
I got a response from Heroku's support team asking pretty much the same question. Their answer was that what I'm trying to do is not possible, but will be at some point (although not in the next few months, anyway).
They suggested that I could just use the undocumented web api used by heroku's own web console (a POST to an endpoint on kolkrabbi.heroku.com). They did warn that, as a private api, that's likely to change without warning.
As of April 2020, Heroku has integrated GitHub natively without recourse to any janky undocumented Icelandic endpoints.
When configured, Heroku can automatically build and release (if the build is successful) pushes to the specified GitHub repo.
Enabling GitHub integration
You can configure GitHub integration in the Deploy tab of apps in
the Heroku Dashboard.
To configure GitHub integration, you have to authenticate with GitHub.
You only have to do this once per Heroku account.
GitHub repo admin access is required for you to configure automatic
GitHub deploys. This is because Heroku has to register a service hook on
the GitHub repo, and this action requires admin access. For GitHub
organizations, your GitHub account will also need to be a member of the
organization and not an outside collaborator.
If your repo is in a GitHub organization that has third-party
application restrictions
enabled,
an organization admin needs to approve Heroku for use with the
organization. More details are available on
GitHub.
After you link your Heroku app to a GitHub repo, you can selectively
deploy from branches or configure auto-deploys.
Manual deploys
With manual deploys, you can create an immediate deployment of any
branch from the GitHub repo that’s connected to your app. Use manual
deploys if you want to control when changes are deployed to Heroku.
You can also use manual deploys to temporarily deploy a branch other
than the one that’s configured for automatic deployment. For example,
you might have a development app synced to the development GitHub
branch, but you temporarily want to test a feature branch. Simply
trigger a manual deploy of the feature branch to test it on the Heroku
app. Note that release of the feature branch is overwritten on the next
successful GitHub push to the development branch.
Automatic deploys
When you enable automatic deploys for a GitHub branch, Heroku builds and
deploys all pushes to that branch. If, for example, you have a
development app on Heroku, you can configure pushes to your GitHub
development branch to be automatically built and deployed to that app.
If you’ve configured your GitHub repo to use automated Continuous
Integration (with Travis CI, for example), you can check the “Wait for
CI to pass before deploy” checkbox. When enabled, Heroku will only
auto-deploy after all the commit statuses of the relevant commit show
success.
This commit won’t auto-deploy because one of the checks shows a
pending status:
This commit will auto-deploy because all of the checks show a status of
success:
Review apps
With review apps enabled for a Heroku app, Heroku will create temporary
test apps for each pull request that’s opened on the GitHub repo that’s
connected to the parent app. Review apps are great if you’re using
GitHub Flow to propose,
discuss, and merge changes to your code base. Because pull request
branches are deployed to new apps on Heroku, it’s very simple for you
and your collaborators to test and debug code branches. You can also run
automated integration tests on the Heroku app representing a GitHub
branch.
See the Review apps
article
for details.
Heroku CI
Once you’ve connected your GitHub repo to your Pipeline, you can turn on
Heroku CI, our
visual, low-configuration test runner that integrates easily with Heroku
Pipelines (and so complements Review apps, existing Heroku apps, and our
GitHub integrations). Any Heroku Pipeline is already Heroku CI ready –
just turn it on in the Pipeline’s Settings tab.
Links to diffs
For apps that are linked to GitHub repos, releases in the Dashboard
Activity tab will include a “View Diff” link. Following the link
will take you to the GitHub comparison view, showing the changes made
since the last release.
Disconnecting from GitHub
Terminate an obsolete GitHub connection, if necessary.
Disconnecting individual apps
Individual apps can be disconnected in the GitHub pane of the
Deploy tab for the app.
Disconnecting account
You can disconnect your Heroku and GitHub accounts in the Applications
pane on your Dashboard account
page.
My Google-fu is failing me for what seems obvious if I can only find the right manual.
I have a Gitlab server which was installed by our hosting provider
The Gitlab server has many projects.
For some of these projects, I want that Gitlab automatically pushes to a remote repository (in this case Github) every time there is a push from a local client to Gitlab.
Like this: client --> gitlab --> github
Any tags and branches should also be pushed.
AFAICT I have 3 options:
Configure the local client with two remotes, and push simultaneous to Gitlab and Github. I want to avoid this because developers.
Add a git post-receive hook in the repository on the Gitlab server. This would be most flexible (I have sufficient Linux experience to write shell scripts as git hooks) and I have found documentation on how to do this, but I want to avoid this too because then the hosting provider will need to give me shell access.
I use webhooks in Gitlab. I am unfamiliar with what the very basics of webhooks are, and I am unable to locate understandable documentation or even a simple step-by-step example. This is the documentation from Gitlab that I found and I do not understand it: http://demo.gitlab.com/help/web_hooks/web_hooks
I would appreciate good pointers, and I will summarize and document a solution when I find it.
EDIT
I'm using this Ruby code for a web hook:
class PewPewPew < Sinatra::Base
post '/pew' do
push = JSON.parse(request.body.read)
puts "I got some JSON: #{push.inspect}"
end
end
Next: find out how to tell the gitlab server that it has to push a repository. I am going back to the GitLab API.
EDIT
I think I have an idea. On the server where I run the webhook, I pull from GitLab and then I push to Github. I can even do some "magic" (running tests, building jars, deploying to Artifactory,...) before I push to GitHub. In fact it would be great if Jenkins were able to push to a remote repository after a succesful build, then I wouldn't need to write my own webhook, because I'm pretty sure Jenkins already provides a webhook for Gitlab, either native or via a plugin. But I don't know. Yet.
EDIT
I solved it in Jenkins.
You can set more than one git remote in an Jenkins job. I used Git Publisher as a Post-Build Action and it worked like a charm, exactly what I wanted.
would work of course.
is possible but dangerous because GitLab shell automatically symlinks hooks into repositories for you, and those are necessary for permission checks: https://github.com/gitlabhq/gitlab-shell/tree/823aba63e444afa2f45477819770fec3cb5f0159/hooks so I'd rather stay away from it.
Web hooks are not suitable directly: they make an HTTP request with fixed format on certain events, in your case push, not Git protocol requests.
Of course, you could write a server that consumes the hook, clones and pushes, but a service (single push and no deployment) or GitLab CI (already implements hook management) would be strictly better solutions.
services are a the best option if someone implements it: live in the source tree, would do a single push, and require no extra deployment overhead.
GitLab CI or othe CIs like Jenkins are the best option currently available. They are essentially already implemented server for the webhooks, which automatically clone for you: all you have to do then is to push from them.
The keywords you want to Google for are "gitlab mirror github". That has led me to: Gitlab repository mirroring for instance. There seems to be no perfect, easy solution today.
Also this has already been proposed at the feature request forum at: http://feedback.gitlab.com/forums/176466-general/suggestions/4614663-automatic-push-to-remote-mirror-repo-after-push-to Always check there ;) Go and upvote the request.
The key difficulty now is how to store the push credentials.
I solved it in Jenkins. You can set more than one git remote in an Jenkins job. I used Git Publisher as a Post-Build Action and it worked like a charm, exactly what I wanted.
I added "-publisher" jobs that run after "" is built successfully. I could have done it in one job, but I decided to split it up. The build jobs are triggered by a web hook in GitLab; the publisher jobs are using a #daily schedule from the BuildResultTrigger plugin.