ConversationHandler not working when deployed on Heroku but is working good when running on local machine - python-telegram-bot

I am using Python Telegram Bot library to make a bot that has a conversation which let's users name their project, payment plans etc. The conversation consists of MessageHandlers and CallBackQueryHandlers. Thr problem I'm facing is, this conversation was working fine when I was using this bot from my local machine, but when I deployed it to heroku, I can't have the full conversation, it would become unresponsive at some point and I would have to cancel the conversation(which surprisingly, works at lightning bolt speed). All other handlers of my application work as expected and respond quickly.
I am using webhook in this application.
The ConversationHandler:
add_project_handler = ConversationHandler(
entry_points=[CallbackQueryHandler(project_name, pattern="add_project")],
states={
PROJECT_NAME: [MessageHandler(Filters.regex(".*"), store_name_maybe_project_type)],
PROJECT_TYPE: [CallbackQueryHandler(store_type_maybe_admin)],
ADMIN:[CallbackQueryHandler(connect_channel)],
CONNECT:[MessageHandler(Filters.regex(".*"), connect_channel_add_currency)],
CURRENCY:[CallbackQueryHandler(store_currency_maybe_frequency)],
FREQUENCY:[CallbackQueryHandler(store_frequency_maybe_amount)],
MONEY:[MessageHandler(Filters.regex(".*"), store_amount_maybe_bot)],
TOKEN:[MessageHandler(Filters.regex(".*"), store_bot_show_links)],
PAY_SETUP:[CallbackQueryHandler(setup_payment)],
PAYPAL:[MessageHandler(Filters.regex(".*"), handle_client_id)],
CLIENT_ID:[MessageHandler(Filters.regex(".*"), handle_client_secret)],
},
fallbacks=[CommandHandler('cancel', cancel)],
)
handlers.append(add_project_handler)
EDIT:
So I checked out the logs, and I saw some really weird behavior, so it recieves an update, and right now, the state of the conversation should have been 1, but it says 'selecting conversation with state None'. That too it does three times.
2021-10-01T16:55:59.411864+00:00 heroku[router]: at=info method=POST path="/1967487217:AAGDmzoxPPlB386VuEYr4s8dPM50fr49d84" host=invitememberbot.herokuapp.com request_id=6d10ee36-ebe4-40de-91b3-48c5e33d6172 fwd="91.108.6.63" dyno=web.1 connect=0ms service=2ms status=200 bytes=154 protocol=https
2021-10-01T16:55:59.415201+00:00 app[web.1]: 2021-10-01 16:55:59,414 - telegram.ext.dispatcher - DEBUG - Processing Update: {'callback_query': {'chat_instance': '1523769404808794079', 'id': '4756348607585474313', 'data': 'channel', 'message': {'delete_chat_photo': False, 'chat': {'id': 1107423707, 'first_name': 'Palash', 'type': 'private', 'username': 'raghu_palash', 'last_name': 'Raghuwanshi'}, 'channel_chat_created': False, 'date': 1633106830, 'supergroup_chat_created': False, 'reply_markup': {'inline_keyboard': [[{'callback_data': 'channel', 'text': 'Paid Telegram Channel'}]]}, 'group_chat_created': False, 'photo': [], 'caption_entities': [], 'new_chat_members': [], 'message_id': 2657, 'entities': [], 'text': 'What do you want to make?', 'new_chat_photo': [], 'from': {'id': 1967487217, 'first_name': 'InviteMemberBot', 'username': 'InviteMemberFiverrBot', 'is_bot': True}}, 'from': {'language_code': 'en', 'last_name': 'Raghuwanshi', 'id': 1107423707, 'first_name': 'Palash', 'username': 'raghu_palash', 'is_bot': False}}, 'update_id': 334393256}
2021-10-01T16:55:59.415374+00:00 app[web.1]: 2021-10-01 16:55:59,415 - telegram.ext.conversationhandler - DEBUG - selecting conversation (1107423707, 1107423707) with state None
2021-10-01T16:55:59.415584+00:00 app[web.1]: 10.1.90.119 - - [01/Oct/2021:16:55:59 +0000] "POST /1967487217:AAGDmzoxPPlB386VuEYr4s8dPM50fr49d84 HTTP/1.1" 200 2 "-" "-"
2021-10-01T16:55:59.415742+00:00 app[web.1]: 2021-10-01 16:55:59,415 - telegram.ext.conversationhandler - DEBUG - selecting conversation (1107423707, 1107423707) with state None
2021-10-01T16:55:59.415867+00:00 app[web.1]: 2021-10-01 16:55:59,415 - telegram.ext.conversationhandler - DEBUG - selecting conversation (1107423707, 1107423707) with state None
And here's something more weird, when I click the button twice rapidly, it takes me to the next state.
2021-10-01T17:15:24.421256+00:00 heroku[router]: at=info method=POST path="/1967487217:AAGDmzoxPPlB386VuEYr4s8dPM50fr49d84" host=invitememberbot.herokuapp.com request_id=ea099aff-e9dd-46fa-8516-958677290171 fwd="91.108.6.63" dyno=web.1 connect=0ms service=6ms status=200 bytes=154 protocol=https
2021-10-01T17:15:24.420031+00:00 app[web.1]: 2021-10-01 17:15:24,419 - telegram.ext.dispatcher - DEBUG - Processing Update: {'callback_query': {'chat_instance': '1523769404808794079', 'id': '4756348606350994438', 'data': 'channel_done', 'message': {'delete_chat_photo': False, 'chat': {'id': 1107423707, 'first_name': 'Palash', 'type': 'private', 'username': 'raghu_palash', 'last_name': 'Raghuwanshi'}, 'channel_chat_created': False, 'date': 1633107558, 'supergroup_chat_created': False, 'reply_markup': {'inline_keyboard': [[{'callback_data': 'channel_done', 'text': 'Done'}]]}, 'group_chat_created': False, 'photo': [], 'caption_entities': [], 'new_chat_members': [], 'message_id': 2659, 'entities': [], 'text': "Please create a private Telegram channel and click Done when it's ready:", 'new_chat_photo': [], 'from': {'id': 1967487217,
'first_name': 'InviteMemberBot', 'username': 'InviteMemberFiverrBot', 'is_bot': True}}, 'from': {'language_code': 'en',
'last_name': 'Raghuwanshi', 'id': 1107423707, 'first_name': 'Palash', 'username': 'raghu_palash', 'is_bot': False}}, 'update_id': 334393260}
2021-10-01T17:15:24.420209+00:00 app[web.1]: 2021-10-01 17:15:24,420 - telegram.ext.conversationhandler - DEBUG - selecting conversation (1107423707, 1107423707) with state 1
I don't understand why it worked fine when I ran it on my own machine, but does this on heroku.
Additional Information -
These are the functions that are involved in this log:
def store_type_maybe_admin(update, context):
# Open db connection
conn = DB_POOL.getconn()
cur = conn.cursor()
# stores project type and conditonally asks for making admin
query = update.callback_query
query.answer()
keyboard = [[InlineKeyboardButton("Done", callback_data="channel_done")]]
reply_markup = InlineKeyboardMarkup(keyboard)
if query.data != "channel_done":
context.user_data["project_type"] = query.data
q = "UPDATE project SET project_type = %s WHERE id = %s"
cur.execute(q, (query.data, context.user_data["project_id"]))
conn.commit()
message = f"Please create a private Telegram {query.data} and click Done when it's ready:"
video = "BAACAgUAAxkDAAICCGFK8ud7FDZ2qbzOkQfrDId-abgXAAKqAwAC-KBYVquzwMps2x7GIQQ"
return_value = PROJECT_TYPE
else:
message = "Please add this bot (#InviteMemberFiverrBot) to the channel admins. It needs Add Subscribers permission.\n\nPress Done when it's ready:"
video = "BAACAgUAAxkDAAICFmFK9GB185nfUvTeE6P3ZTnEp1YEAAKxAwAC-KBYVtfAgpeMwSB0IQQ"
return_value = ADMIN
# Send a video
context.bot.send_video(chat_id=update.effective_chat.id, video=video)
context.bot.send_message(chat_id=update.effective_chat.id ,text=message, reply_markup=reply_markup, parse_mode="markdown")
DB_POOL.putconn(conn)
return return_value # will be either PROJECT_TYPE = 1 or ADMIN = 2
def connect_channel(update, context):
query = update.callback_query
query.answer()
video = "BAACAgUAAxkDAAICGWFK9ShTxKcIN32o9njkIQMCXcwoAAKyAwAC-KBYVi7hYk0auE7IIQQ"
context.bot.send_video(chat_id=update.effective_chat.id, video=video)
context.bot.send_message(
chat_id=update.effective_chat.id,
text=f"To connect the channel, please forward a message (any) from the channel to this bot."
)
return CONNECT
This is how I have setup my bot:
bot = telegram.Bot(token=TOKEN)
dispatcher = setup(bot, db_pool)
#app.route(f"/{TOKEN}", methods=["POST"])
def respond():
"""Run the bot."""
update = telegram.Update.de_json(request.get_json(force=True), bot)
dispatcher.process_update(update)
return "ok"
the setup function:
def setup(bot, db_pool):
# Create bot, update queue and dispatcher instances
dispatcher = Dispatcher(bot, None, workers=0)
##### Register handlers here #####
initialize_bot_handlers = initialize_bot(db_pool) # initialize_bot returns a list of handlers
for handler in initialize_bot_handlers:
dispatcher.add_handler(handler)
return dispatcher
The conversation handler is inside the initialize_bot function.

Related

Playing a stream by my discord bot on heroku doesnt work

I want to make a Bot, that plays a livestream of an online radio. I use Discord JS v13.
On Heroku I have installed the following buildpacks:
heroku/nodejs
https://github.com/jonathanong/heroku-buildpack-ffmpeg-latest.git
https://github.com/xrisk/heroku-opus.git
https://github.com/OnlyNoob/heroku-buildpack-libsodium.git
My code is the following:
let voiceChn = message.member.voice.channel;
const connection = joinVoiceChannel({
channelId: message.member.voice.channel.id,
guildId: message.member.voice.channel.guildId,
adapterCreator: message.guild.voiceAdapterCreator,
selfDeaf: true
});
const player = createAudioPlayer();
let resource = createAudioResource(STREAM_URL);
connection.subscribe(player);
connection.on(VoiceConnectionStatus.Ready, () => {
player.play(resource);
});
It does work running on my pc but it does not run on Heroku.
These are the packages I have installed:
"#discordjs/opus": "^0.5.3"
"#discordjs/rest": "^0.5.0"
"#discordjs/voice": "^0.10.0"
"discord-api-types": "^0.36.0"
"discord.js": "^13.8.1"
"ffmpeg-static": "^4.4.1"
"libsodium-wrappers": "^0.7.10"
I do get the following error: The player immediately emits the idle event and thats logged:
{
status: 'playing',
missedFrames: 0,
playbackDuration: 120,
resource: AudioResource {
playStream: OggDemuxer {
_readableState: [ReadableState],
_events: [Object: null prototype],
_eventsCount: 5,
_maxListeners: undefined,
_writableState: [WritableState],
allowHalfOpen: true,
_remainder: null,
_head: null,
_bitstream: null,
[Symbol(kCapture)]: false,
[Symbol(kCallback)]: null
},
edges: [ [Object], [Object] ],
metadata: null,
volume: undefined,
encoder: undefined,
audioPlayer: undefined,
playbackDuration: 0,
started: true,
silencePaddingFrames: 5,
silenceRemaining: 0
},
onStreamError: [Function: onStreamError]
}
Not sure about the error, but I believe I had the same problem and fixed it.
Didn't test if all the changes are needed, but here goes:
Instead of your heroku-buildpack-libsodium package, I use:
https://github.com/Crazycatz00/heroku-buildpack-libopus
Changed every URLs 'https' with 'http'
Use a DNS lookup tool to change domain name with IPV4 form
'https://stream.skymedia.ee/live/NRJdnb' becomes 'http://185.31.240.229:8888/NRJdnb'
Deploy slash commands after you change their code

discord.py FFMPEG Option reconnect not found

I'm working on a music bot using discord.py, I've looked online and found that I needed to pass in ffmpeg options into discord.FFmpegPCMAudio() for the bot to not stop half way through the song. However it returns an error "Option reconnect not found."
vc = await ctx.message.author.voice.channel.connect()
ydl_opts = {
'format': 'bestaudio/best',
'outtmpl': 'C:/Luke/YoutubeDLAudio/audio.mp3',
'postprocessors': [{
'key': 'FFmpegExtractAudio',
'preferredcodec': 'mp3',
'preferredquality': '192'
}],
'prefer_ffmpeg': True,
'keepvideo': True
}
ffmpeg_opts = {'before_options': '-reconnect 1 -reconnect_streamed 1 -reconnect_delay_max 5','options': '-vn'} # <---------------
with youtube_dl.YoutubeDL(ydl_opts) as ydl:
ydl.download([url])
vc.play(discord.FFmpegPCMAudio(executable="C:/Luke/ffmpeg/bin/ffmpeg.exe", source="C:/Luke/YoutubeDLAudio/audio.mp3", **ffmpeg_opts)) <--------------
Does anyone know the problem? Thanks for any help.

How to deploy Strapi backend to Heroku with Postgres addon?

I have been using the official Strapi tutorial on how to deploy strapi to heroku with postgres and after following all the instructions, my heroku app is showing an error. However when I check the build logs, there are no errors and they show the build successful message.
build logs
2020-08-17T15:48:19.744258+00:00 app[web.1]: npm ERR!
2020-08-17T15:48:19.744486+00:00 app[web.1]: npm ERR! Failed at the radio-darya-backend#0.1.0 start script.
2020-08-17T15:48:19.744753+00:00 app[web.1]: npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
2020-08-17T15:48:19.756754+00:00 app[web.1]:
2020-08-17T15:48:19.757071+00:00 app[web.1]: npm ERR! A complete log of this run can be found in:
2020-08-17T15:48:19.757252+00:00 app[web.1]: npm ERR! /app/.npm/_logs/2020-08-17T15_48_19_747Z-debug.log
2020-08-17T15:48:19.825573+00:00 heroku[web.1]: Process exited with status 1
2020-08-17T15:48:19.869487+00:00 heroku[web.1]: State changed from starting to crashed
2020-08-17T15:48:32.221633+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=radio-darya-backend.herokuapp.com request_id=1bceee5d-4452-4b2a-9638-d5f242b4337c fwd="213.162.246.193" dyno= connect= service= status=503 bytes= protocol=https
2020-08-17T15:48:32.751425+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/favicon.ico" host=radio-darya-backend.herokuapp.com request_id=95d4de1a-5f17-49e3-bed2-b459bce9259e fwd="213.162.246.193" dyno= connect= service= status=503 bytes= protocol=https
package.json dependencies
"devDependencies": {},
"dependencies": {
"knex": "<0.20.0",
"pg": "^8.3.0",
"sqlite3": "latest",
"strapi": "3.1.4",
"strapi-admin": "3.1.4",
"strapi-connector-bookshelf": "3.1.4",
"strapi-plugin-content-manager": "3.1.4",
"strapi-plugin-content-type-builder": "3.1.4",
"strapi-plugin-email": "3.1.4",
"strapi-plugin-upload": "3.1.4",
"strapi-plugin-users-permissions": "3.1.4",
"strapi-utils": "3.1.4"
},
config database.js
module.exports = ({ env }) => ({
defaultConnection: 'default',
connections: {
default: {
connector: 'bookshelf',
settings: {
"client":"postgres",
"host":"${process.env.DATABASE_HOST}",
"port": "${process.env.DATABASE_PORT}",
"database": "${process.env.DATABASE_NAME}",
"username": "${process.env.DATABASE_USERNAME}",
"password": "${process.env.DATABASE_PASSWORD}",
"ssl": { "rejectUnauthorized": false }
},
options: {
},
},
},
});
her
This is working for me for v3.x strapi.
// Path: ./config/env/production/database.js
const parse = require('pg-connection-string').parse;
const config = parse(process.env.DATABASE_URL);
module.exports = ({ env }) => ({
defaultConnection: 'default',
connections: {
default: {
connector: 'bookshelf',
settings: {
client: 'postgres',
host: config.host,
port: config.port,
database: config.database,
username: config.user,
password: config.password,
},
options: {
ssl: false,
},
},
},
});
We also need to set the NODE_ENV variable on Heroku to production to ensure this new database configuration file is used.
heroku config:set NODE_ENV=production
see https://strapi.io/documentation/v3.x/deployment/heroku.html
Note here is v3.x instead of beta version. Google "strapi heroku postgres" for the moment still give legacy beta version.
You didn’t give the error message you’re having?
I did deploy my strapi to heroku a few weeks ago and there was no problems.
You are sure you followed all the steps from strapi documentation?
Only thing I could think to go wrong is database connection.
First you have to install postgress addon in Heroku, then get the config-info and last add environment variables in Heroku (settings/config vars) and also modify strapi config-files to get the database information from environment variables.
Strapi documentation: https://strapi.io/documentation/3.0.0-beta.x/deployment/heroku.html.
EDIT:
Strapi documentation isn’t correct at the moment, database.json file and location has been changed. See:
strapi database.js / multiple database configs
https://www.youtube.com/watch?v=xNE0TrI5OKk
ive just completed that tutorial a day ago... and i also had problems for a complete beginner to strapi and postgres and heroku... heres my experience.
follow along here to get postgres and the database setup:
https://tute.io/install-configure-strapi-postgresql
then complete your setup with the missing pieces from here:
https://strapi.io/documentation/v3.x/deployment/heroku.html
basically:
have postgres installed locally on your system, create the db and user and grant permissions.
install strapi without the quickstart flag and use the details that was used above.
use the heroku cli to set the configs derived from the database_url configvar.
commit and push and all should be well.
EDIT:
In Heroku under the app in question:
click setting and scroll down to Environment variable and click Reveal Config Vars
Ensure that you have a DATABASE_URL config var
Ensure that your postgres specific config vars match up to the DATABASE_URL
eg. DATABASE_URL = postgres://ebitxebvixeeqd:dc59b16dedb3a1eef84d4999sb4baf#ec2-50-37-231-192.compute-2.amazonaws.com: 5432/d516fp1u21ph7b
It's read like so: postgres:// USERNAME : PASSWORD # HOST : PORT : DATABASE_NAME
Also in your ./config/server.js file make sure your host is 0.0.0.0
module.exports = ({ env }) => ({
host: env('HOST', '0.0.0.0'),
port: env.int('PORT', 1337),
admin: {
auth: {
secret: env('ADMIN_JWT_SECRET', '***********************'),
},
},
});
Also change your database.js config to be:
settings: {
client: 'postgres',
host: env('DATABASE_HOST', '127.0.0.1'),
port: env.int('DATABASE_PORT', 5432),
database: env('DATABASE_NAME', 'strapi'),
username: env('DATABASE_USERNAME', 'postgres'),
password: env('DATABASE_PASSWORD', ''),
ssl: env.bool('DATABASE_SSL', false),
},
options: {}
Theres no much to go on from your question or the logs, but the above is basically some common issues ive experienced.
copy this code directly in the config/database.js
module.exports = ({ env }) => ({
defaultConnection: 'default',
connections: {
default: {
connector: 'bookshelf',
settings: {
client:'postgres',
host:`${process.env.DATABASE_HOST}`,
port: `${process.env.DATABASE_PORT}`,
database: `${process.env.DATABASE_NAME}`,
username: `${process.env.DATABASE_USERNAME}`,
password: `${process.env.DATABASE_PASSWORD}`,
ssl: { "rejectUnauthorized": false }
},
options: {
},
},
},
});

Heroku Application Error - code=H10 desc="App crashed"

I am trying to deploy a dash app on heroku.
Everything works fine (also locally) except from when I run heroku open I get this Application Error:
heroku logs --tail then gives:
2019-11-22T22:55:41.598239+00:00 heroku[web.1]: State changed from up to crashed
2019-11-22T22:55:41.587218+00:00 heroku[web.1]: Process exited with status 3
2019-11-22T22:55:42.000000+00:00 app[api]: Build succeeded
2019-11-22T22:55:59.485489+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=romania2019.herokuapp.com request_id=d48fde4a-bd7e-47e2-9a21-bdf589ef281a fwd="152.115.83.242" dyno= connect= service= status=503 bytes= protocol=https
2019-11-22T22:55:59.851491+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/favicon.ico" host=romania2019.herokuapp.com request_id=b6bbb5e8-8667-42f2-b083-b7bf26b5ae14 fwd="152.115.83.242" dyno= connect= service= status=503 bytes= protocol=https
Any suggestions where I should look next?
Here is my app.py I am trying to deploy:
import datetime
import time
import dash
import dash_core_components as dcc
import dash_html_components as html
import plotly
from dash.dependencies import Input, Output
import numpy as np
from selenium import webdriver
url = 'https://prezenta.bec.ro/prezidentiale24112019/abroad-precincts'
profile = webdriver.FirefoxProfile()
profile.set_preference("browser.preferences.instantApply",True)
profile.set_preference("browser.helperApps.neverAsk.saveToDisk",
"text/plain, application/octet-stream, application/binary, text/csv, application/csv, application/excel, text/comma-separated-values, text/xml, application/xml")
profile.set_preference("browser.helperApps.alwaysAsk.force",False)
profile.set_preference("browser.download.manager.showWhenStarting",False)
profile.set_preference("browser.download.folderList",1)
options = webdriver.FirefoxOptions()
options.headless = True
# create a new Firefox session
driver = webdriver.Firefox(firefox_profile=profile, options=options)
driver.implicitly_wait(30)
driver.get(url)
# Find button and click on it
ELEMENT = driver.find_element_by_xpath('//*[#id="root"]/main/section/div[1]/div[1]/div[5]/div/div[2]/div/h2')
def update(ELEMENT):
# time.sleep(2.9)
return int(ELEMENT.text.replace('.', ''))
external_stylesheets = ['https://codepen.io/chriddyp/pen/bWLwgP.css']
app = dash.Dash(__name__, external_stylesheets=external_stylesheets)
server = app.server
app.layout = html.Div(
html.Div([
html.H4('ROMANIA - Presidential Elections Live Feed'),
html.Div(id='live-update-text'),
dcc.Graph(id='live-update-graph'),
dcc.Interval(
id='interval-component',
interval=60*1000, # in milliseconds
n_intervals=0
)
])
)
time.sleep(5)
#app.callback(Output('live-update-text', 'children'),
[Input('interval-component', 'n_intervals')])
def update_metrics(n):
no = update(ELEMENT)
style = {'padding': '5px', 'fontSize': '16px'}
return [
html.Span('Voters: {}'.format(no), style=style),
html.Span('Voters/Min: {}'.format(np.diff(data['number'])[-5:].mean()), style=style)
]
data = {'time': [], 'number': []}
# Multiple components can update everytime interval gets fired.
#app.callback(Output('live-update-graph', 'figure'),
[Input('interval-component', 'n_intervals')])
def update_graph_live(n):
# Collect some data
data['time'].append(datetime.datetime.now()) # + datetime.timedelta(seconds=3))
try:
no = update(ELEMENT)
data['number'].append(no)
except:
data['number'].append(no)
# Create the graph with subplots
fig = plotly.subplots.make_subplots(rows=2, cols=1, vertical_spacing=0.1)
fig['layout']['margin'] = {
'l': 30, 'r': 10, 'b': 30, 't': 10
}
fig['layout']['legend'] = {'x': 0, 'y': 1, 'xanchor': 'left'}
fig.append_trace({
'x': data['time'],
'y': data['number'],
'name': 'Voters',
'type': 'scatter'
}, 1, 1)
fig.append_trace({
'x': data['time'],
'y': np.diff(data['number']),
'text': data['time'],
'name': 'Voters / Min',
'type': 'scatter'
}, 2, 1)
fig.layout.update(height=800, )
return fig
if __name__ == '__main__':
app.run_server(debug=True)
It basically reads the number of voters and displays the number of votes per minute.
And the deployment instructions I've been following are given here: https://dash.plot.ly/deployment.
open your git bash and post
log --tails it will show your the error type H10 is app crasing due to some error missing dependencies or your procfiles is not well set so try reading the error types and look in your code. if your app is working fine in your local host try
web: gunicorn index:server --preload --timeout60.

typeProviders with private GKE clusters

I am new to google cloud manager (GCM) and am writing some code in order to practice. I have read some interesting articles that detail how I can use deploymentmanager.v2beta.typeprovider in order to extend GCM and use it to configure Kubernetes objects themselves as additional deployment. This is a very appealing behavior for extension and seems to open up great opportunities to extend declarative automation of any API which is cool.
I am attempting to create a private node/public endpoint GKE cluster that is managed by custom typeProvider resources which correspond to GKE api calls. It seems that public node/public endpoint GKE cluster is the only way to support GCM custom typeProviders and this seems wrong considering private node/public endpoint GKE configuration possible.
It seems it would be weird to not have deploymentmanager.v2beta.typeprovider support a private node/public endpoint GKE configuration.
as an aside...
I feel a private node/private endpoint/Cloud Endpoint to expose it to the GCM typeProvider public API endpoint requirement should also be a valid architecture but I have yet to test.
Using the following code
def GenerateConfig(context):
# Some constant type vars that are not really
resources = []
outputs = []
gcp_type_provider = 'deploymentmanager.v2beta.typeProvider'
extension_prefix = 'k8s'
api_version = 'v1'
kube_initial_auth = {
'username': 'luis',
'password': 'letmeinporfavors',
"clientCertificateConfig": {
'issueClientCertificate': True
}
}
# EXTEND API TO CONTROL KUBERNETES BEGIN
kubernetes_exposed_apis = [
{
'name': '{}-{}-api-v1-type'.format(
context.env['deployment'],
extension_prefix
),
'endpoint': 'api/v1'
},
{
'name': '{}-{}-apps-v1-type'.format(
context.env['deployment'],
extension_prefix
),
'endpoint': 'apis/apps/v1'
},
{
'name': '{}-{}-rbac-v1-type'.format(
context.env['deployment'],
extension_prefix
),
'endpoint': 'apis/rbac.authorization.k8s.io/v1'
},
{
'name': '{}-{}-v1beta1-extensions-type'.format(
context.env['deployment'],
extension_prefix
),
'endpoint': 'apis/extensions/v1beta1'
}
]
for exposed_api in kubernetes_exposed_apis:
descriptor_url = 'https://{}/swaggerapi/{}'.format(
'$(ref.{}-k8s-cluster.endpoint)'.format(
context.env['deployment']
),
exposed_api['endpoint']
)
resources.append(
{
'name': exposed_api['name'],
'type': gcp_type_provider,
'properties': {
'options': {
'validationOptions': {
'schemaValidation': 'IGNORE_WITH_WARNINGS'
},
'inputMappings': [
{
'fieldName': 'name',
'location': 'PATH',
'methodMatch': '^(GET|DELETE|PUT)$',
'value': '$.ifNull($.resource.properties.metadata.name, $.resource.name)'
},
{
'fieldName': 'metadata.name',
'location': 'BODY',
'methodMatch': '^(PUT|POST)$',
'value': '$.ifNull($.resource.properties.metadata.name, $.resource.name)'
},
{
'fieldName': 'Authorization',
'location': 'HEADER',
'value': '$.concat("Bearer ", $.googleOauth2AccessToken())'
}
],
},
'descriptorUrl': descriptor_url
},
}
)
# EXTEND API TO CONTROL KUBERNETES END
# NETWORK DEFINITION BEGIN
resources.append(
{
'name': "{}-network".format(context.env['deployment']),
'type': "compute.{}.network".format(api_version),
'properties': {
'description': "{} network".format(context.env['deployment']),
'autoCreateSubnetworks': False,
'routingConfig': {
'routingMode': 'REGIONAL'
}
},
}
)
resources.append(
{
'name': "{}-subnetwork".format(context.env['deployment']),
'type': "compute.{}.subnetwork".format(api_version),
'properties': {
'description': "{} subnetwork".format(
context.env['deployment']
),
'network': "$(ref.{}-network.selfLink)".format(
context.env['deployment']
),
'ipCidrRange': '10.64.1.0/24',
'region': 'us-east1',
'privateIpGoogleAccess': True,
'enableFlowLogs': False,
}
}
)
# NETWORK DEFINITION END
# EKS DEFINITION BEGIN
resources.append(
{
'name': "{}-k8s-cluster".format(context.env['deployment']),
'type': "container.{}.cluster".format(api_version),
'properties': {
'zone': 'us-east1-b',
'cluster': {
'description': "{} kubernetes cluster".format(
context.env['deployment']
),
'privateClusterConfig': {
'enablePrivateNodes': False,
'masterIpv4CidrBlock': '10.0.0.0/28'
},
'ipAllocationPolicy': {
'useIpAliases': True
},
'nodePools': [
{
'name': "{}-cluster-pool".format(
context.env['deployment']
),
'initialNodeCount': 1,
'config': {
'machineType': 'n1-standard-1',
'oauthScopes': [
'https://www.googleapis.com/auth/compute',
'https://www.googleapis.com/auth/devstorage.read_only',
'https://www.googleapis.com/auth/logging.write',
'https://www.googleapis.com/auth/monitoring'
],
},
'management': {
'autoUpgrade': False,
'autoRepair': True
}
}],
'masterAuth': kube_initial_auth,
'loggingService': 'logging.googleapis.com',
'monitoringService': 'monitoring.googleapis.com',
'network': "$(ref.{}-network.selfLink)".format(
context.env['deployment']
),
'clusterIpv4Cidr': '10.0.0.0/14',
'subnetwork': "$(ref.{}-subnetwork.selfLink)".format(
context.env['deployment']
),
'enableKubernetesAlpha': False,
'resourceLabels': {
'purpose': 'expiramentation'
},
'networkPolicy': {
'provider': 'CALICO',
'enabled': True
},
'initialClusterVersion': 'latest',
'enableTpu': False,
}
}
}
)
outputs.append(
{
'name': '{}-cluster-endpoint'.format(
context.env['deployment']
),
'value': '$(ref.{}-k8s-cluster.endpoint)'.format(
context.env['deployment']
),
}
)
# EKS DEFINITION END
# bring it all together
template = {
'resources': resources,
'outputs': outputs
}
# give it to google
return template
if __name__ == '__main__':
GenerateConfig({})
Also, I will note the subsequent hello world template which uses the created typeProviders above.
def current_config():
'''
get the current configuration
'''
return {
'name': 'atrium'
}
def GenerateConfig(context):
resources = []
conf = current_config()
resources.append(
{
'name': '{}-svc'.format(conf['name']),
'type': "{}/{}-k8s-api-v1-type:/api/v1/namespaces/{}/pods".format(
context.env['project'],
conf['name'],
'{namespace}'
),
'properties': {
'namespace': 'default',
'apiVersion': 'v1',
'kind': 'Pod',
'metadata': {
'name': 'hello-world',
},
'spec': {
'restartPolicy': 'Never',
'containers': [
{
'name': 'hello',
'image': 'ubuntu:14.04',
'command': ['/bin/echo', 'hello', 'world'],
}
]
}
}
}
)
template = {
'resources': resources,
}
return template
if __name__ == '__main__':
GenerateConfig({})
If I leave enablePrivateNodes as False
'privateClusterConfig': {
'enablePrivateNodes': False,
'masterIpv4CidrBlock': '10.0.0.0/28'
}
I get this as a response
~/code/github/gcp/expiramentation/atrium_gcp_infra 24s
❯ bash atrium/recycle.sh
Waiting for delete [operation-1562105370163-58cb9ffb0b7b8-7479dd98-275c6b14]...done.
Delete operation operation-1562105370163-58cb9ffb0b7b8-7479dd98-275c6b14 completed successfully.
Waiting for delete [operation-1562105393399-58cba01134528-be47dc30-755cb106]...done.
Delete operation operation-1562105393399-58cba01134528-be47dc30-755cb106 completed successfully.
The fingerprint of the deployment is IiWcrdbZA5MedNlJLIicOg==
Waiting for create [operation-1562105786056-58cba187abee2-5d761e87-b446baca]...done.
Create operation operation-1562105786056-58cba187abee2-5d761e87-b446baca completed successfully.
NAME TYPE STATE ERRORS INTENT
atrium-k8s-api-v1-type deploymentmanager.v2beta.typeProvider COMPLETED []
atrium-k8s-apps-v1-type deploymentmanager.v2beta.typeProvider COMPLETED []
atrium-k8s-cluster container.v1.cluster COMPLETED []
atrium-k8s-rbac-v1-type deploymentmanager.v2beta.typeProvider COMPLETED []
atrium-k8s-v1beta1-extensions-type deploymentmanager.v2beta.typeProvider COMPLETED []
atrium-network compute.v1.network COMPLETED []
atrium-subnetwork compute.v1.subnetwork COMPLETED []
The fingerprint of the deployment is QJ2NS5EhjemyQJThUWYNHA==
Waiting for create [operation-1562106179055-58cba2fe76fe7-957ef7a6-f55257bb]...done.
Create operation operation-1562106179055-58cba2fe76fe7-957ef7a6-f55257bb completed successfully.
NAME TYPE STATE ERRORS INTENT
atrium-svc atrium-244423/atrium-k8s-api-v1-type:/api/v1/namespaces/{namespace}/pods COMPLETED []
~/code/github/gcp/expiramentation/atrium_gcp_infra 13m 48s
this is a good response and my custom typeProvider resource creates correctly using the API's of my freshly created cluster.
If I make this cluster have private nodes however... with
'privateClusterConfig': {
'enablePrivateNodes': True,
'masterIpv4CidrBlock': '10.0.0.0/28'
},
I fail with
~/code/github/gcp/expiramentation/atrium_gcp_infra 56s
❯ bash atrium/recycle.sh
Waiting for delete [operation-1562106572016-58cba47538c93-d34c17fc-8b863765]...done.
Delete operation operation-1562106572016-58cba47538c93-d34c17fc-8b863765 completed successfully.
Waiting for delete [operation-1562106592237-58cba4888184f-a5bc3135-4e662eed]...done.
Delete operation operation-1562106592237-58cba4888184f-a5bc3135-4e662eed completed successfully.
The fingerprint of the deployment is dk5nh_u5ZFFvYO-pCXnFBg==
Waiting for create [operation-1562106901442-58cba5af62f25-8b0e380f-3687aebd]...done.
Create operation operation-1562106901442-58cba5af62f25-8b0e380f-3687aebd completed successfully.
NAME TYPE STATE ERRORS INTENT
atrium-k8s-api-v1-type deploymentmanager.v2beta.typeProvider COMPLETED []
atrium-k8s-apps-v1-type deploymentmanager.v2beta.typeProvider COMPLETED []
atrium-k8s-cluster container.v1.cluster COMPLETED []
atrium-k8s-rbac-v1-type deploymentmanager.v2beta.typeProvider COMPLETED []
atrium-k8s-v1beta1-extensions-type deploymentmanager.v2beta.typeProvider COMPLETED []
atrium-network compute.v1.network COMPLETED []
atrium-subnetwork compute.v1.subnetwork COMPLETED []
The fingerprint of the deployment is 4RnscwpcYTtS614VXqtjRg==
Waiting for create [operation-1562107350345-58cba75b7e680-f548a69f-1a85f105]...failed.
ERROR: (gcloud.deployment-manager.deployments.create) Error in Operation [operation-1562107350345-58cba75b7e680-f548a69f-1a85f105]: errors:
- code: ERROR_PROCESSING_REQUEST
message: 'Error fetching URL https://10.0.0.2:443/api/v1/namespaces/default/pods,
reason: ERROR_EXCLUDED_IP'
The 10.0.0.2 seems to be the private endpoint of my cluster. I am having a difficult time tracking down where I can override the https://10.0.0.2:443/api/v1/namespaces/default/pods url host such that it would attempt to contact the publicEndpoint rather than the privateEndpoint.
If this call were to go out to the public endpoint it would create successfully I believe. As an interesting aside the typeProvider declarations in the descriptorUrl do attempt to hit the publicEndpoint of the cluster and are successful in doing so. However, despite this indication the creation of the actual api resources such as the hello world example attempt to interface with the private endpoint.
I feel this behavior should be overridable somewhere but I am failing to find this clue.
I have tried both a working public node configuration and a non-working private node configuration
So I ran into the same problem writing [this] (https://github.com/Aahzymandius/k8s-workshops/tree/master/8-live-debugging). While trying to debug this I ran into 2 issues
The public endpoont is used initially to fetch the swagger definition of the various k8s apis and resources, however, in that response is included the private endpoint for the API server which causes the k8s type to try to use that ip.
While further trying debug that error plus a new one I was hitting, I discovered that gke 1.14.x or later does not support insecure calls to the k8s api which caused the k8s type to fail even with fully public clusters.
Since the functionality seems to fail with newer versions of GKE, I just stopped trying to debug it. Though I'd recommend reporting an issue on the github repo for it. This functionality would be great to keep going

Resources