How to structure container logs in Vertex AI? - google-ai-platform

Question

I have a model in Vertex AI, from the logs it seems that Vertex AI has ingested the log into message field within jsonPayload field, but i would like to structure the jsonPayload field such that every key in message will be a field within jsonPayload, i.e: flatten/extract message

Answer 1

The logs in Stackdriver follow a defined LogEntry schema. Cloud Logging uses structured logs where log entries use the jsonPayload field to add structures to their payload.
For Vertex AI, the parameters are passed inside the message field which we see in the logs. These structures of the logs are predefined. However if you want to extract the fields that are present inside the message block you can refer to the below mentioned workarounds:
1. Create a sink :
You can export your logs to a Cloud Storage bucket, Bigquery,Pub/Sub etc.
If you use Bigquery as the sink, then in Bigquery you can use the JSON functions to extract the required data.
2. Download the logs and write your custom code :
You can download the log files and then write your custom logic to extract data as per your requirements.
You can refer to the client library (python) to write the custom logic and python JSON functions.

Answer 2

Using the gcloud logging client to write structure logs into a Vertex AI endpoint:
(Make sure you have a service account with premissions to write logs into gcloud, and, for clean logs make sure you don't stream any other logs into stderr or stdout)
import json
import logging
from logging import Handler, LogRecord
import google.cloud.logging_v2 as logging_v2
from google.api_core.client_options import ClientOptions
from google.oauth2 import service_account
data_to_write_to_endpoint = {key1: value1, ...}
#Json key for a Service account permitted to write logs into the gcp
# project where your endpoint is
credentials = service_account.Credentials.from_service_account_info(
json.loads(SERVICE_ACOUNT_KEY_JSON)
)
client = logging_v2.client.Client(
credentials=credentials, client_options=ClientOptions(api_endpoint="logging.googleapis.com",),
)
# This represent your Vertex AI endpoint
resource = logging_v2.Resource(
type="aiplatform.googleapis.com/Endpoint",
labels={"endpoint_id": YOUR_ENDPOINT_ID, "location": ENDPOINT_REGION},
)
logger = client.logger("LOGGER NAME")
logger.log_struct(
info=data_to_write_to_endpoint,
severity=severity,
resource=resource,
)