I am developing a Magento site.
I have access to a local host and a remote host and would like to
somehow configure development and production environments. On the
remote host I restore the database data that was backed up on the
local host, but when I do so, I overwrite the host's base name and
this causes the site to be redirected to a nonexistent URL when
the page is loaded. How can I avoid this clash:
I want to be able to develop either (a) on http:// remotehost/foobardev
and back up my data to http:// remotehost/foobar or otherwise (b) develop
on http:// localhost/foobar and deploy on http:// remotehost/foobar . I
want to know how to transfer the database data back and forth without
overwriting the values found in Magento Admin Panel -> System
-> Configuration -> Web -> Unsecure Base URL / Secure Base URL
when I run mysql and use the mysql command source to reinstate
the database entries found on the development site onto the
production site.
So, I would like an easier way to restore the database contents without
overwriting the base url configured in magento admin panel as doing so
would cause a redirect to a nonexisting or wrong place on each page load
and thus render the system unusable.
Not exactly a SO type of question. Magento EE has staging built in and can merge your data as well. You have to understand that syncing data from dev to live is not easily possible without some serious sync framework that keeps track on state of every row and column and knows what data is new and what is old and solve syncing conflicts.
Here's your flow based on assumption that you are using CE and does not have data migration tools bundled.
setup live database and count that data will move only from live to dev and never from dev to live as you don't have data migrations. Every config you need to make and preserve in database level do it on live database (test them out in dev environment and then create in live)
make a shell script , fabric script whatever deployment script you are comfortable with that will export live db dump , deletes dev database if exists and create a new database and import live database to it, run a pre or post sql script that will change/delete config values that are environment dependant (like base_url, secure_base_url etc)
to avoid double data entry always create all attributes and config values that you need to preserve with magento setup scripts.
Same goes about code and here's a common setup scenario with live, stage and development environments
one master version control (preferably bare just to avoid that someone will change files there) repository based on clean magento versions tree
separate branches for each environment (live, stage, dev(n)) and a verified code flow from dev (where you develop and can have broken codebase state) to stage (where release candidate resides and is ready for testing and does not change) from stage to live (where your live code is in stable state)
every developer works on a checkout from dev branch and commits to it's own dev branch and then pushes changes to dev where they can be evaluated and decided if changes are mature enough for staging
stage is a place where release candidate lives and client can test (or automated tests) and diagnose if it's ready enough to be released, no one ever changes code here and code comes from dev branch
live is live and running version where no one ever changes any code directly . If tests are passed code can come here from stage only
so to visualise it better imagine your codebase residing in git.
myproject_magento_se (your project git repository on bitbucket.org or in github or wherever you can host)
--> master (branch with all clean magento versions from your current to latest)
--> dev (git checkout -b master (or by specific version from master)
--> stage (while on dev: git checkout -b stage)
--> live (while on stage: git checkout -b live)
and imagine your hosts setup like this:
www.mylivesite.com = git clone yourgitrepo; git checkout live;
stage.mylivesite.com = git clone yourgitrepo; git checkout stage;
dev.mylivesite.com = git clone yourgitrepo; git checkout dev;
For all this you better have deployment scripts that do switching and code and database lifting between environments with a push of the button.
Here's a few common actions that you need to perform daily with every software project
move/reset data from live to stage from live to dev (have obfuscation calls if needed to scramble or change client related data)
move code from dev to stage
move code from stage to live
reset/create any dev with live state (Data and code)
have fun :) and go through this thread as well https://superuser.com/questions/90301/sync-two-mysql-databases and all other you can find searching on SO in similar matter.
Related
We would like to use Nifi registry with git as storage engine. In that case, i modified providers.xml and i was able to save the flows there.
Challenges:
There is no 2 way sync. We can only save the flows modified by Nifi user but if we modify the flow directly in git location, it will not be reflected on nifi registry
There is no review or approval process for Nifi registry. A user has to login to nifi-registry server, create a branch and issue a pull request.
As a workaround, we can delete the database file ( H2) and restart the nifi resgistry.
Lastly, everything should be automated in CI/CD like what we do for regular maven project.
Any suggestions ?
The purpose of the git storage is mostly to let user visualize the differences through tools like git hub, or any other tools that can support diffs, plus by pushing to a remote you also get a remote backup of the flow content. It is not meant to be modified outside of the application, just like you wouldn't bypass an application and go right into it's database and start changing data.
I am investigating adding an app.json file to my heroku pipeline to enable review apps.
Heroku offers the ability to generate one from your existing app setup, but I do not see any way to prevent it from automatically committing it to our repository's master branch.
I need to be able to see it before it gets committed to the master branch because we require at least two staff members to review all changes to the master branch (which triggers an automatic staging build) for SOC-2 security compliance.
Is there a way that I can see what it would generate without committing it to the repository?
I tried forking the repo and connecting the fork to it's own pipeline, but because it did not have any of our heroku add-ons or environment, it would not work for our production pipeline.
I am hesitant to just build the app.json file manually - it seems more prone to error. I would much prefer to get the automatically generated file and selectively remove items.
As a punchline to this story, I ended up investing enough time into the forked repository on it's own pipeline to demonstrate a POC
When you generate your app.json file, it should take you to a secondary screen that has the full app.json in plaintext at the bottom.
Why not open a PR with its contents in your project root. Once it's detected on the repository Heroku shouldn't ask you to regenerate it again.
[FYI - I am not looking to upgrade our Heroku account to Enterprise to take advantage of app grouping. We are a small team]
I am a collaborator on a heroku app. I was tasked with setting up dev-staging-production environments for the app and I am getting confused on how to setup Auto Deploy correctly with the GitHub-Heroku integration guide. I have been already given admin access to the original git repo.
From the 'owner' account on the production app I created two separate apps (dev and staging).
Now, I have 3 apps that I am a collaborator of:
(1) app.domain.com
(2) app-dev.domain.com
(3) app-staging.domain.com
I have created 3 remotes pointing to the 3 generated heroku-git repos:
live
dev
staging
I can push to the 3 Heroku apps easily by:
git push live master
git push dev master
git push staging master
BUT I want to setup auto-deploy for dev-staging apps.
I have pushed an "auto" branch onto dev and staging remote. My intention is for Heroku to detect pushes to that branch and "auto" deploy the code. But, heroku only detects pushes to "master" branch afaik.
git push dev release auto
git push staging release auto
But when I go to administer the Heroku dev-staging apps I only see the "master" branch in the Auto-deploy section. Shouldn't I also see the "auto" branch? Is the reason I am not seeing it because I didn't push auto as a remote branch on the ORIGINAL git repo? In which case, how do I structure this stuff as code lives in 2 separate environments (Heroku repo and GitHub repo)?
Ultimately, the plan is to setup some form of Continuous Integration and get tests run before releases happen across the dev-staging-production zones.
Am I overthinking this or is there a better way to get this stuff working? Appreciate it!
Heroku only tracks changes on the remote master Branch.
https://devcenter.heroku.com/articles/git#deploying-from-a-branch-besides-master
To push your dev branch to the Heroku master remote branch:
git push dev release:master
You may need to use the -f flag to force push (be careful!)
I think auto deploy on Heroku works only on default branch (usually named master), and you probably want to look into Heroku feature called pipelines. I've not used it myself yet and it's in beta stage, but it sounds like exactly what you're looking for. Not sure if I read it right, but it sounds like you created 3 individual projects, instead of branches in 1 project? How do you manage moving changes from one stage to another?
As for the CI setup I guess it depends on the kind of testing you're doing and the workflow. I'm in the middle of creating such process in my workplace, and what I will be proposing is as follows:
Devs develop and test on their local machines on feature branches before commiting and merging onto DEV branch
Commiting or merging onto the DEV branch triggers CI server, runs the build and perform PHPUnit and Behat tests
Successful build is being deployed to a server (possibly Heroku or any hosting provider that will support automated deployments) for QA.
Once tested and approved we can move the code onto STAGING branch repeat CI proccess and on success move to MASTER and go live.
We're using gilab.com to host our project and perform tests (CI server is included as standard even with free account), deploy dev and staging state for QA to Heroku, and finally host live version with a dedicated enterprise level provider.
How best should I accomplish the following deployment objectives with Git deployment for Azure?
Easily switch when working locally to either use fake in-memory data or (eventually) non-production snapshot of real data
Deploy to staging environment on Azure such that at first I could use fake in-memory data and eventually move to non-production snapshot of real data.
Deploy to production with real data
I currently deploy using Github and a staging branch to a staging Azure website. Since I deploy to a public repo, the web.config file is ignored by git. (EDIT: I just learned that ignoring web.config actually causes deployment error on azure)
Any help/suggestion is appreciated.
It's actually supposed to be simpler than that. Please see this page. Basically, the idea is that you set some AppSettings in the Azure portal to override the default values that are committed to your repo.
Well... Here's what I did that works for me right now.
To quickly switch between fake in-memory data locally, I use a compilation symbol LOCAL and a preprocessor directive #if LOCAL.
Same compilation symbol works when you deploy to Azure, so I can work on fake data until I'm ready to switch to real db. I can also use the app settings if I really want to make to switch it more easily.
The challenge was to keep a web.config with "secrets" (like connection string) locally and not expose it to Github. I added it to .gitignore, but then my deployments started failing on Azure because it could not find the web.config. Just copying it to wwwroot via ftp did not help - Azure was looking for web.config in the repository.
So, to make this work I "slightly" altered the deployment process by first copying the Web.config from wwwroot to the repository before running the default deploy.cmd. This was simple - this is what you do:
Create a .deployment file in the root of your repository with the following:
[config]
command = deploy.my.cmd
Create deploy.my.cmd with the following script:
xcopy %DEPLOYMENT_TARGET%\Web.config %DEPLOYMENT_SOURCE%\\ /Y
deploy.cmd
Now, I have web.config with secrets locally. Git ignores this file. I uploaded the correct web.config to Azure via FTP, and it gets used whenever I deploy.
My office is growing and ive been tasked to build out the IT for our web development.
Whats the best tool/setup for doing web development in a group setting? The requirements are a centralized code repository, a location to test development code on, and finally a way to push tagged code out to a staging server. What im thinking is svn/redmine for code repo, each user has an account on a central development machine to allow for ssh access(eclipse over ssh) and their own virtual host on the dev server which gives everyone a centralized development sandbox. Code is written and tested on this dev box then checked back into svn and later tagged and pushed out to the staging server. Yeah? Thoughts comments or recommendations?
*Also, in a dev environment what is the best way to handle databases? Is it wise to pull from the production database? Also should each developer have his/her own db or work off a master db?
**We are building a magento application and also have some custom backoffice tools that run on cakePHP.
Although this subject is off-topic in StackOverflow and flagged so then you need to concentrate on following areas:
VERSION-CONTROL
GIT has all the glory and you don't need your own box for this as https://bitbucket.org/ offers unlimited data and private/public repos and you can set your codebase there. http://github.com is also powerful and de facto most popular version-control oriented tool out there although it comes for a small price
so your master branches live in your version control and your devs will checkout frpom there and commit to it as well
your deployment tools will deploy data to your live and staging environments from your master
ENVIRONMENTS
usually three are used LIVE, STAGE, DEV
LIVE is well live and only approved code gets deployed there
STAGE is pre-live environment and should be exact replica environment according to LIVE so all things can be tested there by merchant
DEV is cool to have exact replica but can as well be on developers local env and is ment for loose testing and experimenting
DATABASES AND DEPLOYMENT
mysql databases are pain in the ass to sync so you better have a script for it that syncs from live to others and prevent syncing from other environments to LIVE. This limitation also requires that all the configuration and content will be added from LIVE only and only then synced down the line. Every change to schema or permanent setting should be handled by update scripts (As we are talking MAGENTO CE , MAGENTO EE has migration built in)
for deployment I also suggest you to build a fabric or capistrano script that resets dev and staging environments, handles database reset and pull from LIVE DB, and imports code from central repository.
it's also a good idea to target the following everyday tasks:
clients needs to reset the stage for it's tests
project manager, developer or testers need to test so spawning a test clone should be oneclick action (take current db and code and make it live in some subfolder for specific test only) as well as deleting the test
3rd party devs might need access to specific test or dev environment (this is actual with magento as in average there are at least 10 external extensions installed in every magento store)