TLS Certificate Debugging in Kubernetes

Common Kubernetes TLS errors

Inspect TLS secrets

List all TLS secrets in a namespace:

kubectl get secrets -n your-namespace --field-selector type=kubernetes.io/tls

Extract and inspect the certificate from a secret:

# Extract the certificate
kubectl get secret your-tls-secret -n your-namespace \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > cert.pem

# Inspect it
openssl x509 -in cert.pem -text -noout | grep -E "Subject|Issuer|Not Before|Not After|DNS"

Check expiry directly:

kubectl get secret your-tls-secret -n your-namespace \
  -o jsonpath='{.data.tls\.crt}' | base64 -d \
  | openssl x509 -noout -dates

Debugging cert-manager

Check the status of a Certificate resource:

kubectl describe certificate your-cert -n your-namespace

Check the CertificateRequest that cert-manager created:

kubectl get certificaterequest -n your-namespace
kubectl describe certificaterequest your-cert-xxxxx -n your-namespace

Check the Order (for ACME issuers):

kubectl get orders -n your-namespace
kubectl describe order your-cert-xxxxx -n your-namespace

Check cert-manager controller logs for errors:

kubectl logs -n cert-manager deploy/cert-manager | grep -E "ERROR|error|failed" | tail -20

Verify your ClusterIssuer is ready:

kubectl get clusterissuer
kubectl describe clusterissuer letsencrypt-prod

Ingress TLS configuration

A correctly configured Ingress with TLS:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  tls:
  - hosts:
    - example.com
    secretName: example-com-tls    # cert-manager creates this
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-service
            port:
              number: 80

Verify the ingress is using the right certificate:

kubectl describe ingress my-ingress -n your-namespace | grep -A5 TLS

Test what certificate the ingress is actually serving:

openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates

Trust issues between pods

When a pod gets x509: certificate signed by unknown authority connecting to another internal service, you need to inject the CA cert into the pod's trust store. The cleanest way is via a ConfigMap:

kubectl create configmap ca-bundle \
  --from-file=ca.crt=internal-ca.pem \
  -n your-namespace

# Mount it in your deployment
volumes:
- name: ca-bundle
  configMap:
    name: ca-bundle
volumeMounts:
- name: ca-bundle
  mountPath: /etc/ssl/certs/internal-ca.crt
  subPath: ca.crt

For Java pods, set the trust store via environment variables:

env:
- name: JAVA_TOOL_OPTIONS
  value: "-Djavax.net.ssl.trustStore=/etc/ssl/certs/truststore.jks
          -Djavax.net.ssl.trustStorePassword=changeit"

Quick reference

# List all TLS secrets
kubectl get secrets -A --field-selector type=kubernetes.io/tls

# Check cert expiry from secret
kubectl get secret my-tls -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates

# Force cert-manager to renew a certificate
kubectl annotate certificate my-cert cert-manager.io/issueTemporary="true" --overwrite

# Delete and recreate a stuck certificate
kubectl delete certificate my-cert -n my-namespace
# cert-manager will recreate it

# Check what cert an endpoint is serving
openssl s_client -connect my-service.namespace.svc.cluster.local:443 -showcerts