List custom resources from caching client with custom fieldSelector - go

Question

I'm using the Operator SDK to build a custom Kubernetes operator. I have created a custom resource definition and a controller using the respective Operator SDK commands:
operator-sdk add api --api-version example.com/v1alpha1 --kind=Example
operator-sdk add controller --api-version example.com/v1alpha1 --kind=Example
Within the main reconciliation loop (for the example above, the auto-generated ReconcileExample.Reconcile method) I have some custom business logic that requires me to query the Kubernetes API for other objects of the same kind that have a certain field value. It's occurred to me that I might be able to use the default API client (that is provided by the controller) with a custom field selector:
func (r *ReconcileExample) Reconcile(request reconcile.Request) (reconcile.Result, error) {
ctx := context.TODO()
listOptions := client.ListOptions{
FieldSelector: fields.SelectorFromSet(fields.Set{"spec.someField": "someValue"}),
Namespace: request.Namespace,
}
otherExamples := v1alpha1.ExampleList{}
if err := r.client.List(ctx, &listOptions, &otherExamples); err != nil {
return reconcile.Result{}, err
}
// do stuff...
return reconcile.Result{}, nil
}
When I run the operator and create a new Example resource, the operator fails with the following error message:
{"level":"info","ts":1563388786.825384,"logger":"controller_example","msg":"Reconciling Example","Request.Namespace":"default","Request.Name":"example-test"}
{"level":"error","ts":1563388786.8255732,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"example-controller","request":"default/example-test","error":"Index with name field:spec.someField does not exist","stacktrace":"..."}
The most important part being
Index with name field:spec.someField does not exist
I've already searched the Operator SDK's documentation on the default API client and learned a bit about the inner workings of the client, but no detailed explanation on this error or how to fix it.
What does this error message mean, and how can I create this missing index to efficiently list objects by this field value?

Answer 1

The default API client that is provided by the controller is a split client -- it serves Get and List requests from a locally-held cache and forwards other methods like Create and Update directly to the Kubernetes API server. This is also explained in the respective documentation:
The SDK will generate code to create a Manager, which holds a Cache and a Client to be used in CRUD operations and communicate with the API server. By default a Controller's Reconciler will be populated with the Manager's Client which is a split-client. [...] A split client reads (Get and List) from the Cache and writes (Create, Update, Delete) to the API server. Reading from the Cache significantly reduces request load on the API server; as long as the Cache is updated by the API server, read operations are eventually consistent.
To query values from the cache using a custom field selector, the cache needs to have a search index for this field. This indexer can be defined right after the cache has been set up.
To register a custom indexer, add the following code into the bootstrapping logic of the operator (in the auto-generated code, this is done directly in main). This needs to be done after the controller manager has been instantiated (manager.New) and also after the custom API types have been added to the runtime.Scheme:
package main
import (
k8sruntime "k8s.io/apimachinery/pkg/runtime"
"example.com/example-operator/pkg/apis/example/v1alpha1"
// ...
)
function main() {
// ...
cache := mgr.GetCache()
indexFunc := func(obj k8sruntime.Object) []string {
return []string{obj.(*v1alpha1.Example).Spec.SomeField}
}
if err := cache.IndexField(&v1alpha1.Example{}, "spec.someField", indexFunc); err != nil {
panic(err)
}
// ...
}
When a respective indexer function is defined, field selectors on spec.someField will work from the local cache as expected.