故障排查
controller-manager 一直在BackOff状态：
$ oc -n kube-service-catalog get pods
...
controller-manager-x5dr8 0/1 CrashLoopBackOff 21 1h
查看日志：
$ oc -n kube-service-catalog logs -f --tail=30 controller-manager-x5dr8
I1008 06:24:21.655748 1 feature_gate.go:194] feature gates: map[OriginatingIdentity:true]
I1008 06:24:21.655957 1 feature_gate.go:194] feature gates: map[OriginatingIdentity:true AsyncBindingOperations:true]
I1008 06:24:21.655983 1 feature_gate.go:194] feature gates: map[OriginatingIdentity:true AsyncBindingOperations:true NamespacedServiceBroker:true]
I1008 06:24:21.656012 1 hyperkube.go:192] Service Catalog version v3.11.0-0.1.35+8d4f895-2;Upstream:v0.1.35 (built 2019-01-08T23:12:26Z)
I1008 06:24:21.659263 1 leaderelection.go:185] attempting to acquire leader lease kube-service-catalog/service-catalog-controller-manager...
I1008 06:24:21.677905 1 leaderelection.go:194] successfully acquired lease kube-service-catalog/service-catalog-controller-manager
I1008 06:24:21.678992 1 event.go:221] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-service-catalog", Name:"service-catalog-controller-manager", UID:"f4993f8c-93f0-11e9-9c59-00163e0a2de7", APIVersion:"v1", ResourceVersion:"189814138", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' controller-manager-x5dr8-external-service-catalog-controller became leader
F1008 06:24:21.726721 1 controller_manager.go:237] error running controllers: failed to get api versions from server: failed to get supported resources from server: unable to retrieve the complete list of server APIs: servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request
这个报错有点奇怪的，去对应代码看了看，就是APIServer那边返回的，我直接用cURL看了一下：
$ TOKEN=$(oc whoami -t)
$ curl -X GET -H "Authorization: Bearer ${TOKEN}" 'https://1.2.3.4:8443/apis/servicecatalog.k8s.io/v1beta1'
Error: 'x509: certificate has expired or is not yet valid'
原来是catalog服务的证书到期了。
解决方法
使用OpenShift-Ansible来重新部署更新证书。
复制一份playbooks/redeploy-certificates.yml，把playbooks/redeploy-certificates.yml中其他项目的playbook注释了，只保留init和catalog的。
$ cd openshift-ansible
$ cp -a playbooks/redeploy-certificates.yml playbooks/redeploy-certificates-catalog.yml
$ cat playbooks/redeploy-certificates-catalog.yml
---
- import_playbook: init/main.yml
- import_playbook: openshift-service-catalog/private/redeploy-certificates.yml
when: openshift_enable_service_catalog | default(true) | bool
重新部署：
$ ansible-playbook -i </path/to/inventory/file> playbooks/redeploy-certificates-catalog.yml
部署完后确认：
curl -I -X GET -H "Authorization: Bearer ${TOKEN}" 'https://1.2.3.4:8443/apis/servicecatalog.k8s.io/v1beta1'
HTTP/1.1 200 OK
Cache-Control: no-store
Content-Type: application/json
Date: Fri, 08 Oct 2021 06:37:39 GMT
Transfer-Encoding: chunked