I have a Kubernetes pod using a readiness probe, and tied with the service this ensures that I don't receive traffic until I'm ready.
I'm using Spring Actuator as the health endpoint for this readiness probe.
But i'd like to trigger some actions whenever the pod is deemed ready by the kubelet.
What would be the simplest way to do this?
Perhaps implement your own HealthCheck. When you find that everything is ok for the first time, run your code.
I use a static variable firstHealthCheckOK is checked. Your logic should run only once.
I am assuming you are running Spring-boot 2.x and are calling a readiness probe on http://localhost:8080/actuator/health
The health() method below is called when Kubernetes calls http://localhost:8080/actuator/health
import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.HealthIndicator;
import org.springframework.stereotype.Component;
#Component
public class HealthCheck implements HealthIndicator {
static boolean firstHealthCheckOK = false;
#Override
public Health health() {
int errorCode = check(); // perform health check
if (errorCode != 0) {
return Health.down()
.withDetail("Error Code", errorCode).build();
}
if (firstHealthCheckOK == false){
firstHealthCheckOK = true;
doStartUpLogic();
}
return Health.up().build();
}
private int check() {
//some logic
return 0;
}
private void doStartUpLogic() {
//some startup logic
}
}
You can use a post-install hook provided by helm charts (in case you are using helm to deploy your application). This will perform necessary actions/script/jobs after the pod becomes up and running.
As part of the pod lifecycle events, you may want to attach additional handlers such as podStart and build your custom logic to manipulate the event happening as need be.
Alternatively, you can also run your code to read REST response from
GET /api/v1/namespaces/{namespace}/pods/{name}/log
build any downstream logic to get pod status
Note, in controlled environments, it's good to not base any of your conditional logic on pods (individual pods) rather rely on deployments. The REST endpoint which you should rather focus on is
GET /apis/apps/v1beta2/namespaces/{namespace}/deployments/{name}/status
Related
I am testing a service which heavily relies on project reactor.
For many tests I am mocking the return value of the component responsible for API calls.
The tests are split over multiple files.
When I run the tests of one file, they are green, but when I execute all of the test files at once, some tests fail, with the error message indicating that the mocking did not succeed (Either the injected component returned null, or the implementation of the actual component is invoked).
In the logs, there is no information about the mocking failing.
A code example:
interface API {
Flux<Bird> getBirds();
}
#Component
class BirdWatcher {
API api;
BirdWatcher(API api) {
this.api = api;
}
Flux<Bird> getUncommonBirds() {
return api.getBirds() // Although this is mocked in the test, in some runs it returns `null` or calls the implementation of the actual component
.filter(Bird::isUncommon);
}
}
#SpringBootTest
class BirdWatcherTests {
#Autowired
BirdWatcher birdWatcher;
#MockBean
API api;
#Test
void findsUncommonBirds() {
// Assemble
Bird birdCommon = new Bird("Sparrow", "common");
Bird birdUncommon = new Bird("Parrot", "uncommon");
Mockito.when(api.getBirds()).thenReturn(Flux.just(birdCommon, birdUncommon));
// Act
Flux<Bird> uncommonBirds = birdWatcher.getUncommonBirds();
// Assert
assertThat(uncommonBirds.collectList().block().size(), equalTo(1));
}
}
For me the issue seems like a race condition, but I don't know where and how this might happen, and how I can check and fix this.
I am using spring-boot-test:2.7.8, pulling in org.mockito:mockito-core:4.5.1 org.mockito:mockito-junit-jupiter:4.5.1, and org.junit.jupiter:junit-jupiter:5.8.2, with gradle 7.8.
For reactor, spring-boot-starter-webflux:2.7.8, depending on reactor:2.7.8.
I am using Masstransit with RabbitMQ. As part of some deployment procedure, At some point in time I need my service to disconnect and stop receiving any messages.
Assuming that I won't need the bus until the next restart of the service, will it be Ok to use bus.StopAsync()?
Is there a way to get list of end points and then remove them from listining ?
You should StopAsync the bus, and then when ready, call StartAsync to bring it back up (or start it at the next service restart).
To stop receiving messages without stopping the buss I needed a solution that will avoid the consume message pipeline from consuming any type of message. I tried with observers but unsuccessfully. My solution came up with custom consuming message filter.
The filter part looks like this
public class ComsumersBlockingFilter<T> :
IFilter<ConsumeContext<T>>
where T : class
{
public void Probe(ProbeContext context)
{
var scope = context.CreateFilterScope("messageFilter");
}
public async Task Send(ConsumeContext<T> context, IPipe<ConsumeContext<T>> next)
{
// Check if the service is degraded (true for this demo)
var isServiceDegraded = true;
if (isServiceDegraded)
{
//Suspend the message for 5 seconds
await Task.Delay(TimeSpan.FromMilliseconds(5000), context.CancellationToken);
if (!context.CancellationToken.IsCancellationRequested)
{
//republish the message
await context.Publish(context.Message);
Console.WriteLine($"Message {context.MessageId} has been republished");
}
// NotifyConsumed to avoid skipped message
await context.NotifyConsumed(TimeSpan.Zero, "messageFilter");
}
else
{
//Next filter in the pipe is called
await next.Send(context);
}
}
}
The main idea is to delay with cancellation token and the republish the message. After that call contect.NotifyConsumed to avoid the next pipeline filters and return normally.
Is there a way to hook into a Lambda's shutdown? I am opening a database connection and want to keep it open, but I want to make sure it gets closed when the Lambda is terminated.
You are probably interested in an event that is thrown when the Lambda instance is being killed and not when a single invocation ends, right? You have one option for both though, but I doubt that they'll help you..
You can either use the context method getRemainingTimeInMillis() (links to Node.js but similar in other programming languages) to find out when the current invocation of your Lambda function times out. This might be helpful to cleanup things or use the time of your Lambda function to the full extent. I don't recommend to cleanup your database connections at the end of each invocation because then you won't reuse them for future invocations which slows down your Lambda function. But if you're okay with that, then go for it. Remember that this only works as long as your function is running. As soon as you have returned a response, you can't perform any cleanup operations because your Lambda function will get into a 'sleep mode'. You need to do this before you return something.
Alternatively, you can make use of the Extensions API. It offers a shutdown phase and triggers an extension with a Shutdown event. However, since an extension sits besides your function (and not within your function code), I'm not sure if you have a chance to clean up any database connections with this approach... See also Lambda Execution Environment for additional information.
Assuming you have a pooled connection for a warm lambda, you may register a shutdown hook to close the DB connection or release any other resources, you only have 500 ms to perform this task.
class EnvironmentConfig {
private static volatile boolean shutdownRegistered;
private static volatile HikariDataSource ds;
private void registerShudownHook() {
if (!shutdownRegistered) {
synchronized (lock) {
if (!shutdownRegistered) {
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
if (ds != null) {
ds.close();
}
}));
EnvironmentConfig.shutdownRegistered = true;
}
}
}
}
public DataSource dataSource() throws PropertyVetoException {
HikariDataSource _ds = EnvironmentConfig.ds;
if (_ds == null) {
synchronized (lock) {
_ds = EnvironmentConfig.ds;
if (_ds == null) {
_ds = new HikariDataSource();
// TODO: set connection props
EnvironmentConfig.ds = _ds;
registerShudownHook();
}
}
}
return _ds;
}
}
You can reference the datasource anywhere to get a singleton copy which will create the instance and register the shutdown hook.
Your shutdown hook could do other tasks, provided it does them quickly, or you can register more than one hook, just don't go nuts with how many threads you're registering.
No, you can't hook into the shutdown of a Lambda Execution context.
Lambda handles that on it's own an decides if and when to re-use or destroy execution contexts.
You'll probably have to rely on the connections to time out on their own.
We have an AWS lambda with .net core 3.1, we use dependency injection to add some services, one of those services is a ConsoleLogger, we inject the logger like this:
private void ConfigureServices(IServiceCollection services)
{
this.Configuration = new ConfigurationBuilder().AddEnvironmentVariables().Build();
services.AddOptions();
services.AddLogging(builder =>
{
builder.AddConsole((x) =>
{
x.DisableColors = true;
x.Format = Microsoft.Extensions.Logging.Console.ConsoleLoggerFormat.Systemd;
});
});
// more services
}
Then in the function we use the logger like this:
[LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]
public async Task Handle(ILambdaContext lambdaContext)
{
var logger = this.ServiceProvider.GetService<ILogger<MyClass>>();
string startTime = DateTime.UtcNow.ToString("yyyy-MM-dd HH:mm:ss.fff", CultureInfo.InvariantCulture);
logger.LogInformation($"Start Time stamp:{startTime}|AwsRequestId:{lambdaContext.AwsRequestId}");
// more work
logger.LogInformation("processing x");
// more work
string endTime = DateTime.UtcNow.ToString("yyyy-MM-dd HH:mm:ss.fff", CultureInfo.InvariantCulture);
logger.LogInformation($"End Time stamp:{endTime}|AwsRequestId:{lambdaContext.AwsRequestId}");
}
The problem is that in cloudwatch the logs are out of order
Even the report of the cost is before my entry.
Is there a way to avoid this?
Thanks
ConsoleLogger buffers messages in an internal queue, so they're probably getting delayed there, and it's nothing to do with CloudWatch. Amazon's own CloudWatch logging library does the same thing, and they note in their own documentation that it can be a problem for Lambdas: https://github.com/aws/aws-logging-dotnet/#aws-lambda
Their recommended solution is to use Amazon.Lambda.Logging.AspNetCore which doesn't do any buffering.
No, I don't believe you can do this with CloudWatch. CloudWatch guarantees delivery, not timely delivery. You could set up a Dynamo or ElasticSearch database and write your log messages to the database with a timestamp. On retrieval you can sort by the timestamp. This also gives you more control over filtering the messages than is possible with CloudWatch.
After migration to Spring Boot 2.3.0 I need to create a custom implementation of StatusAggregator.
However, I don't know exactly how to do it properly after when the HealthAggregator is deprecated. I've got several external systems and I check their availability and in the previous version I had method Health aggregate(Map<String, Health> healths) but now it's Status getAggregateStatus(Set<Status> statuses) from interface StatusAggregator where as an argument it has Set of Statuses. I need to recognize from with system Status object comes, so during creation Health, I use the approach where I create Status with description, something like this:
Health.status(new Status((Status.DOWN.getCode(), "TEST_SYSTEM")).build())
Then I want to recognize it in method Status getAggregateStatus(Set<Status> statuses) by this description however when I go to Status.class and check equals it not include description field.
public boolean equals(Object obj) {
if (obj == this) {
return true;
} else {
return obj instanceof Status ? ObjectUtils.nullSafeEquals(this.code, ((Status)obj).code) : false;
}
}
so Set<Status> will return only Statuses with different code.
I was also looking to do something similar and did not find a way to identify the component sending the status. I defined custom health indicators for each API and checked the dependencies for external APIs within those individual health indicators. For custom health indicator, implement the health method in the HealthIndicator and annotate with 'component' to register in the application context as explained in the documentation.