I am trying to setup Heroku Pipelines and would like to configure my enviroment and build process using app.json. But my app.json is being ignored.
That's how my repo looks like:
- Dir1
- Dir2
- ...
- app.json
- some other files
I made a simple app.json but no buildpack is being installed, no database provisioned and so on.
{
"formation": {
"web": {
"quantity": 1,
"size": "free"
},
"worker": {
"quantity": 1,
"size": "free"
}
},
"addons": [
{
"plan": "heroku-postgresql:hobby-dev"
},
{
"plan": "heroku-redis:hobby-dev"
}
],
"buildpacks": [
{
"url": "heroku/nodejs"
}
]
}
Does anyone have an idea why the configuration from app.json is not being used?
So, here is the solution the problem: The configuration in app.json will only be used when the heroku app is created the first time. So you need to delete the app and create a new one to see the changes taking place.
Related
The documentation on Review Apps in Heroku says:
The app.json file has a scripts section that lets you specify a postdeploy command. Use this to run any one-time setup tasks that make the app, and any databases, ready and useful for testing
Does that mean it never runs on production? what about staging? Does it run on production if I create a new production app from scratch?
I'm trying to find out how to generate sample data for my review apps, a post-deploy hook that seems to apply to my whole pipeline, and not just review apps, feels like the wrong place.
It was as easy as specifying it in environments -> review, like this:
{
"env": {
"RAILS_MASTER_KEY": {
"description": "Rails master encryption key.",
"required": true,
"generator": "secret"
},
},
"formation": {
"web": {
"quantity": 1,
"size": "hobby"
},
"worker": {
"quantity": 1,
"size": "hobby"
}
},
"addons": [
{
"plan": "heroku-postgresql"
}
],
"environments": {
"review": {
"scripts": {
"postdeploy": "bundle exec rails db:seed"
}
}
}
}
So I've run into this issue with a web app I've made:
it gets a file path as input
if the file exists on a bucket, it uses a python client api to create a compute engine instance
it passes the file path to the instance in the startup script
When I ran it locally, I created a python virtual environment and then ran the app. When I make the input on the web browser, the virtual machine is created by the api call. I assumed it used my personal account. I changed to the service account in the command line with this command 'gcloud config set account', it ran fine once more.
When I simply go to the source code directory deploy it as is, the application can create the virtual machine instances as well.
When I use Google cloud build and deploy to cloud run, it doesn't create the vm instance.
the web app itself is not throwing any errors, but when I check compute engine's logs, there is an error in the logs:
`{
"protoPayload": {
"#type": "type.googleapis.com/google.cloud.audit.AuditLog",
"status": {
"code": 3,
"message": "INVALID_PARAMETER"
},
"authenticationInfo": {
"principalEmail": "####"
},
"requestMetadata": {
"callerIp": "#####",
"callerSuppliedUserAgent": "(gzip),gzip(gfe)"
},
"serviceName": "compute.googleapis.com",
"methodName": "v1.compute.instances.insert",
"resourceName": "projects/someproject/zones/somezone/instances/nameofinstance",
"request": {
"#type": "type.googleapis.com/compute.instances.insert"
}
},
"insertId": "######",
"resource": {
"type": "gce_instance",
"labels": {
"instance_id": "#####",
"project_id": "someproject",
"zone": "somezone"
}
},
"timestamp": "2021-06-16T12:18:21.253551Z",
"severity": "ERROR",
"logName": "projects/someproject/logs/cloudaudit.googleapis.com%2Factivity",
"operation": {
"id": "operation-#####",
"producer": "compute.googleapis.com",
"last": true
},
"receiveTimestamp": "2021-06-16T12:18:21.253551Z"
}`
In theory, it is the same exact code that worked from my laptop and on app engine. I'm baffled why it only does this for cloud run.
App engines default service account was stripped of all its roles and given a custom role tailored to the web apps function.
The cloud run is using a different service account, but was given that exact same custom role.
Here is the method I use to call the api.
def create_instance(path):
compute = googleapiclient.discovery.build('compute', 'v1')
vmname = "piinnuclei" + date.today().strftime("%Y%m%d%H%M%S")
startup_script = "#! /bin/bash\napt update\npip3 install pg8000\nexport BUCKET_PATH=my-bucket/{}\ngsutil -m cp -r gs://$BUCKET_PATH /home/connor\ncd /home/connor\n./cloud_sql_proxy -dir=cloudsql -instances=sql-connection-name=unix:sql-connection-name &\npython3 run_analysis_upload.py\nexport ZONE=$(curl -X GET http://metadata.google.internal/computeMetadata/v1/instance/zone -H 'Metadata-Flavor: Google')\nexport NAME=$(curl -X GET http://metadata.google.internal/computeMetadata/v1/instance/name -H 'Metadata-Flavor: Google')\ngcloud --quiet compute instances delete $NAME --zone=$ZONE".format(path)
config = {
"kind": "compute#instance",
"name": vmname,
"zone": "projects/my-project/zones/northamerica-northeast1-a",
"machineType": "projects/my-project/zones/northamerica-northeast1-a/machineTypes/e2-standard-4",
"displayDevice": {
"enableDisplay": False
},
"metadata": {
"kind": "compute#metadata",
"items": [
{
"key": "startup-script",
"value": startup_script
}
]
},
"tags": {
"items": []
},
"disks": [
{
"kind": "compute#attachedDisk",
"type": "PERSISTENT",
"boot": True,
"mode": "READ_WRITE",
"autoDelete": True,
"deviceName": vmname,
"initializeParams": {
"sourceImage": "projects/my-project/global/images/my-image",
"diskType": "projects/my-project/zones/northamerica-northeast1-a/diskTypes/pd-balanced",
"diskSizeGb": "100"
},
"diskEncryptionKey": {}
}
],
"canIpForward": False,
"networkInterfaces": [
{
"kind": "compute#networkInterface",
"subnetwork": "projects/my-project/regions/northamerica-northeast1/subnetworks/default",
"accessConfigs": [
{
"kind": "compute#accessConfig",
"name": "External NAT",
"type": "ONE_TO_ONE_NAT",
"networkTier": "PREMIUM"
}
],
"aliasIpRanges": []
}
],
"description": "",
"labels": {},
"scheduling": {
"preemptible": False,
"onHostMaintenance": "MIGRATE",
"automaticRestart": True,
"nodeAffinities": []
},
"deletionProtection": False,
"reservationAffinity": {
"consumeReservationType": "ANY_RESERVATION"
},
"serviceAccounts": [
{
"email": "batch-service-accountg#my-project.iam.gserviceaccount.com",
"scopes": [
"https://www.googleapis.com/auth/cloud-platform"
]
}
],
"shieldedInstanceConfig": {
"enableSecureBoot": False,
"enableVtpm": True,
"enableIntegrityMonitoring": True
},
"confidentialInstanceConfig": {
"enableConfidentialCompute": False
}
}
return compute.instances().insert(
project="my-project",
zone="northamerica-northeast1",
body=config).execute()
The issue was with the zone. For some reason, when it was ran on cloud run, the code below was the culprit.
return compute.instances().insert(
project="my-project",
zone="northamerica-northeast1",
body=config).execute()
"northamerica-northeast1" should have been "northamerica-northeast1-a"
EDIT:
I made a new virtual machine image and quickly ran into the same problem, it would work locally and break down in the cloud run environment. After letting it sit for some time, it began to work again. This is leading me to the conclusion that there is also some sort of delay before it can be called by cloud run.
I've inherited an app hosted on Heroku which uses review apps. Right up until the day before I took over responsibility for the system, the review apps were working absolutely fine, pulling in the branch, building it, then using a postdeploy command to pull in a database backup from the staging server.
Then I started, and all of a sudden, it's not working. I don't know if this is related to something I've done (which at this point is very little), or maybe an actual permissions issue (I've been set-up as an admin on everything, although the other developers, who this was working for before, are also unable to complete it) but the final step of pulling in the database is failing:
I'm at a complete loss as to what's going wrong here.
Below is the app.json file being used, and the $HEROKU_DATABASE_RESTORE is set to clixifix-staging-eu::b530 (which is the staging server::backup file).
{
"buildpacks": [
{ "url": "heroku/nodejs" },
{ "url": "heroku/ruby" },
{ "url": "heroku-community/nginx" }
],
"environments": {
"review": {
"addons": [
{
"plan": "heroku-postgresql:hobby-basic",
"options": {
"version": "9.6"
}
},
{ "plan": "memcachedcloud:30" },
{ "plan": "mailtrap:unpaid" }
],
"buildpacks": [
{ "url": "heroku/nodejs" },
{ "url": "heroku/ruby" },
{ "url": "heroku-community/nginx" },
{ "url": "heroku-community/cli" }
],
"env": {
"SECRET_KEY_BASE": {
"generator": "secret"
}
},
"formation": {
"web": {
"quantity": 1,
"size": "hobby"
},
"generalworker": {
"quantity": 1,
"size": "hobby"
},
"reportworker": {
"quantity": 1,
"size": "hobby"
}
},
"scripts": {
"postdeploy": "heroku pg:backups:restore $HEROKU_DATABASE_RESTORE DATABASE_URL -a $HEROKU_APP_NAME --confirm $HEROKU_APP_NAME"
}
}
}
}
I reached out to Heroku, who gave me the answer I needed:
What the issue is most likely for the error in the postdeploy, is that to run:
heroku pg:backups:restore $HEROKU_DATABASE_RESTORE DATABASE_URL -a $HEROKU_APP_NAME --confirm $HEROKU_APP_NAME
You will need a platform API key stored somewhere withing your pipeline review app config vars so the CLI can log in. The user who this API key belongs to has most likely lost access to your team and doesn't have permissions to access your review apps. You should generate a new API key using heroku authorizations:create and update it on your pipeline.
Basically, when the old guy left, his permissions were revoked, causing the error. I generated a new key using the command above, set the token as the HEROKU_API_KEY value within the envars in the review app settings, and it worked.
I have a Django app, which uses a postgres database hosted on heroku. I'm trying to create a review app based on this Django app. However, I can't seem to access my DB on the review app. I've added a postdeploy script as shown in the app.json below but it doesn't seem to solve it:
{
"name": "bookapp",
"scripts": {
"postdeploy": "pg_dump $DATABASE_URL | psql $DATABASE_URL && python
./manage.py migrate"
},
"env": {
"DATABASE_URL": {
"required": true
},
"QV_CLOUDINARY_API_KEY": {
"required": true
},
"QV_CLOUDINARY_API_SECRET": {
"required": true
},
"QV_CLOUDINARY_CLOUD_NAME": {
"required": true
},
"QV_DATABASE_NAME": {
"required": true
},
"QV_DATABASE_PASSWORD": {
"required": true
},
"QV_DATABASE_USER": {
"required": true
},
"QV_SECRET_KEY": {
"required": true
}
},
"formation": {
"web": {
"quantity": 1
}
},
"addons": [
"heroku-postgresql"
],
"buildpacks": [
{
"url": "heroku/python"
}
]
}
Found the solution!.
The app.json file above is correct. Apparently the post-deploy script only runs the first time the app is created. I had to close the Pull Request for the review app and re-open a new one so that a new review app is created. Doing that executed the post-deploy script and i was able to access my DB.
I am using DocFx to automatically generate documentation in Visual Studio 2015.
Per the Getting Started instructions, I added an empty ASP.NET 4 Web Application and used the Nuget Console to install the DocFx build package (e.g. Install-Package docfx.msbuild). I built the site and it it generated documentation for code within the project.
I was wondering how to configure docfx.json to get DocFx to document code in other projects within the solution.
In docfx.json, there is an array of metadata. The example metadata has a src object with files and exclude properties.
To point to another project in your solution, add a cwd property to metadata and change folders (i.e. "../Another.Project").
{
"metadata": [
{
"src": [
{
"files": [ "**/*.csproj" ],
"exclude": [ "**/bin/**", "**/obj/**", "_site/**" ],
"cwd": "../Another.Project"
}
],
"dest": "obj/api"
}
],
"build": ...
}
This worked for me.
directory structure
+---ClassLibrary
| \---ClassLibrary.csproj
\---DocFxProject
\---docfx.json
docfx.json contents
cwd and src are synonyms for the same property
{
"metadata":
[
{
"src":
[
{
"files": [ "**/ClassLibrary.csproj" ],
"src": "..",
"exclude": [ "**/obj/**", "**/bin/**" ]
}
],
"dest": "obj/api"
}
],
"build": { ... }
}