Route with cert-manager annotations is not created #38

gaohoward · 2023-09-05T14:59:25Z

Hi, I'm trying to install a route of edge termination using cert-manager annotations in openshift cluster (crc), this is part of our operator's reconcile loop. The following annotations are added to the route:

      "annotations": {
        "cert-manager.io/common-name": "arkamq.io",
        "cert-manager.io/issuer-kind": "Issuer",
        "cert-manager.io/issuer-name": "selfsigned-issuer-ingress"
      }

The dump of the full Route spec:

  "route": {
    "kind": "Route",
    "apiVersion": "v1",
    "metadata": {
      "name": "artemis-broker-wconsj-0-svc-rte",
      "namespace": "default",
      "creationTimestamp": null,
      "labels": {
        "ActiveMQArtemis": "artemis-broker",
        "application": "artemis-broker-app",
        "statefulset.kubernetes.io/pod-name": "artemis-broker-ss-0"
      },
      "annotations": {
        "cert-manager.io/common-name": "arkamq.io",
        "cert-manager.io/issuer-kind": "Issuer",
        "cert-manager.io/issuer-name": "selfsigned-issuer-ingress"
      }
    },
    "spec": {
      "host": "artemis-broker-wconsj-0-svc-rte-default.apps-crc.testing",
      "to": {
        "kind": "Service",
        "name": "artemis-broker-wconsj-0-svc",
        "weight": null
      },
      "port": {
        "targetPort": "wconsj-0"
      },
      "tls": {
        "termination": "edge",
        "insecureEdgeTerminationPolicy": "Redirect"
      },
      "wildcardPolicy": "None"
    },
    "status": {
      "ingress": null
    }
  }

However the route is not created. I dumped the openshift-route pod's log and I see something is going on but no obvious error messages. I'll attach the log. My environment is

CRC version: 2.25.0+97705a
OpenShift version: 4.13.6
Podman version: 4.4.4

cert-manager is installed from operatorHub from default channel
openshit-route is installed using the following command

oc apply -f https://github.com/cert-manager/openshift-routes/releases/latest/download/cert-manager-openshift-routes.yaml

I'll attach also the output of oc events to provide more information.
Please help. Thanks!
oc_events.log
openshift-routes-pod.log

The text was updated successfully, but these errors were encountered:

ctrought · 2023-09-13T22:59:57Z

It looks like there is an issue with the route definition? In the logs the route is visible, but in the pasted configuration apiVersion: v1 for kind Route is not valid, it should be apiVersion: route.openshift.io/v1. This might just be a copy paste issue though?

Can you also check to see whether a CertificateRequest CR was generated? I don't see the event for it in your attachment but there is a log message the CR is not ready. If it was cceated, can you check to make sure the issuer is in a Ready state in the default namespace and perhaps check your cert-manager controller logs (assuming the certificate request was created).

$ oc get issuers -n default
$ oc get certificaterequests -n default
$ oc describe certificaterequest <cr>
$ oc get events -n cert-manager
$ oc get events -n default
$ oc logs cert-manager-controller -n cert-manager

"CR not ready"

I0905 13:27:30.681290       1 app.go:90] cert-manager-openshift-routes/controller-manager "msg"="Event(v1.ObjectReference{Kind:\"Route\", Namespace:\"default\", Name:\"artemis-broker-wconsj-0-svc-rte\", UID:\"345b4999-f3db-4dc4-bc4c-2c842fb9f4f1\", APIVersion:\"route.openshift.io/v1\", ResourceVersion:\"140254\", FieldPath:\"\"}): type: 'Normal' reason: 'Issuing' Issuing cert as no certificate exists"  
I0905 13:27:30.690862       1 sync.go:99] cert-manager-openshift-routes/route/sync "msg"="cr is not ready yet" "resourceVersion"="140254" "route"={"Namespace":"default","Name":"artemis-broker-wconsj-0-svc-rte"}

I was able to successfully generate a certificate using your provided Route spec, aside from the apiVersion that needed a correction.

gaohoward · 2023-09-14T02:55:36Z

Thank you. I'll try.

gaohoward · 2023-09-14T16:02:04Z

@ctrought Hi,

I tried to fix the route's apiVersion (it's in our operator code) and re-deployed our resources.
I can see the apiVersion is correct now (route.openshift.io/v1).

But the route is still not able to be deployed. The route is created/deployed by our operator based on
our deployed CR. I observed during the process the route is first being created and then it's gone.
(I'm using oc get routes command)

Here is the output of the command I tried:

first I tried to get route:(i have everything in default namespace)

$ oc get route
NAME                              HOST/PORT                                                  PATH   SERVICES                      PORT       TERMINATION     WILDCARD
artemis-broker-wconsj-0-svc-rte   artemis-broker-wconsj-0-svc-rte-default.apps-crc.testing          artemis-broker-wconsj-0-svc   wconsj-0   edge/Redirect   None

A few seconds later the route is gone:

$ oc get route
No resources found in default namespace.

get issuer

$ oc get issuers -n default
NAME                        READY   AGE
selfsigned-issuer-ingress   True    2m19s

get csr

$ oc get certificaterequests -n default
No resources found in default namespace.

get cert-manager events

$ oc get events -n cert-manager (see the attached oc_get_events_n_cert_manager.output.log)

get default events

$ oc get events -n default (see attached oc_get_events_n_default.output.log )

get cert-manager log

$ oc logs cert-manager-7cc7959fbb-w6fzk -n cert-manager (see attached oc_logs_cert_manager.log)

get openshift-routes log

$ oc logs cert-manager-openshift-routes-5bfbc6fb88-75zvd -n cert-manager (see attached cert-manager-openshift-routes-pod.log)

In oc_get_events_n_default.output I see some error like

7m20s       Warning   InternalReconcileError                       route/artemis-broker-wconsj-0-svc-rte                      error while reconciling: Operation cannot be fulfilled on routes.route.openshift.io "artemis-broker-wconsj-0-svc-rte": the object has been modified; please apply your changes to the latest version and try again

Not sure if that's relevant. But I do see the CSR get approved in the same file:

m31s       Normal    cert-manager.io                              certificaterequest/artemis-broker-wconsj-0-svc-rte-q4r49   Certificate request has been approved by cert-manager.io

I'm not sure what could be the issue. Please help.

Thanks
cert-manager-openshift-routes-pod.log
oc_get_events_n_cert_manager.output.log
oc_get_events_n_default.output.log
oc_logs_cert_manager.log

ctml91 · 2023-09-14T16:18:14Z

I observed during the process the route is first being created and then it's gone

That sounds like something else is causing it. Cert-manager routes won't delete your route. Maybe whatever you're using to deploy the route is removing the route?

Have you tried deploying the route directly instead of using the operator you mentioned? You can check the kube api-server audit logs to see what's deleting the route. Since you're using OpenShift, you can use the cluster logging operator to send audit logs to elasticsearch or loki to query the audit logs if you don't already. Or you can use the CLI to view them https://docs.openshift.com/container-platform/4.13/security/audit-log-view.html

gaohoward · 2023-09-19T12:54:21Z

Thanks. Yes turns out it's out operator's bug that causes the Route resource being deleted.

Now with the bug fixed the route seems working (I can access the service via the route). However it looks like the route is not using the certificate from openshift-route. There is no cert resource generated (in default namespace). By examining the certificate it seems that the route is using the default certificate.

The route spec does show some fields being generated by openshift-route, but it's not complete.

For example if I deploy this route:

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  annotations:
    cert-manager.io/common-name: arkamq.io
    cert-manager.io/issuer-kind: Issuer
    cert-manager.io/issuer-name: selfsigned-issuer-ingress
  labels:
    ActiveMQArtemis: artemis-broker
    application: artemis-broker-app
    statefulset.kubernetes.io/pod-name: artemis-broker-ss-0
  name: my-route-artemis-broker
  namespace: default
spec:
  host: my-route-artemis-broker-default.apps-crc.testing
  port:
    targetPort: wconsj-0
  tls:
    insecureEdgeTerminationPolicy: Redirect
    termination: edge
  to:
    kind: Service
    name: artemis-broker-wconsj-0-svc
    weight: 100

and then use oc get route to get the created route, there are some new annotations
injected (cert-manager.io/certificate-revision and cert-manager.io/next-private-key)
But no certs in the route spec. See below for the full contents:

(20)[20:51:26]@route$ oc get route my-route-artemis-broker -o yaml
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  annotations:
    cert-manager.io/certificate-revision: "0"
    cert-manager.io/common-name: arkamq.io
    cert-manager.io/issuer-kind: Issuer
    cert-manager.io/issuer-name: selfsigned-issuer-ingress
    cert-manager.io/next-private-key: |
      -----BEGIN RSA PRIVATE KEY-----
      MIIEowIBAAKCAQEAudmUOn8c6KwCKB+ZI4tKBcUB1iJI3Baeh7gOpTcZmOmYJ3cn
      Zq+H1lQ5xcdc9c4tkEEcLscNOlWZiCTpMmKZFXbPJxr8K4g7s/KasTBY8XtnkPTL
      VQedaMiRy1aIJKYxE88ImUh/L/bY3iQIkUA4RedeaaTEE/CTFe+8+pOmTyEH/Ta4
      atNAffGbnhwHKOxjuMW3n0ePvHP0twSFxxRA+n6wHHzREftcDmyo3rr1MGfMwwh0
      7jAphdZ3jFrz+9glr6PUwdiK14bAYiQRDr99PabLtRqtADpXBuKmEr3RIoJ0R10Q
      YC+LFJ8l0jsp4yXdOkPAMTN8c9iLkesbm10IfwIDAQABAoIBAHbXBazeKWsyynJD
      ulX7NIs57bGy3Ko3oImijWLiN/EKtivOSpij95V66DJrCCt5+rK6n73dtLyvjex7
      xc5dNAyAm3gjDkr740lvAbH5+3X29kO/9OFbbDFbx7ahj4734nwLMDFsJUBTdVue
      6QeQbCtvBnVBUcx6UmN9mHcWwQrpOIs/5SpI49TNixN641k6JxE9YG060Ih1kEMg
      x0rE6R3Spb09/aFcFvJLgloJD4GC3IZEkZoqFvBc4myg7t3JuuA8HXfE34mcHLff
      NaAIjlVrO14e83NOMddIvrLqMXybJC3DUST+Cl9ZwMCwiWSteTsjkTv+b3aDnWrB
      /vATodkCgYEA9B8mWahrrFHe3XVh10qmbjlcGil4FU9T8dy4M9VKPEq8OOUWjvEe
      dWt7lX97VHy6jCjInJObLsbzZxiacea6E6t78V2QxvcCAR9uDjN4NDzKB/xIs9we
      U7rargeohUZNd8fncYEKO5TCzNN97k4+uzzC8i0pvI7ja1CSMVn+5gUCgYEAwuSU
      TgIQTutBxmGf4jmjlhH+6lK1QTdM1TW9bvtyI2i/JrvuRU8T0RzrMfFvU+lER3Su
      D20M9goFDgL51GsKKfmbJOWO22lAzz3ASHbl6+M7POY8pzb/+QZ2AAX+yHfOO2sz
      I7RxwN6gD0Jz/PcrVOJ9zZWw/hgX8LVeWo1k17MCgYAb9Z1c029RkyblUlZcZA5f
      vW4RCQLZ3fYNU2w9c4ZHFYAtO/+za2ZDqIXE/47OhGKRPvKLDFTPrLWf7y9xMC3L
      pu5RHJhCUXQHUOQ3+CkiQnwZIgnpn6JRkKsPF41zhjPFLj4Rv+e+HzGYZ/iaXJfO
      tgzV09pd4DenyNiVkmshSQKBgF3Yv+lIaR33fvbOL7U52CH9jGi9ZvZ+cIsCk7OL
      dQzD9bfrKvNI/tvlNwQXpGodm3llICPheSpI7OIwYKj+2igYP7Lmr4HMDx88Nwkt
      581M4EuU+oPyohQCHnQobuWNe0t1dWyuCVC8Ywq2DzAyvTTk03tZp57Tb0PDsimc
      iICtAoGBAOOZeVUpkHY/oDsMyenTfJWDxKTtncnNlV0tG5QII9J/AAdRX7gNGIrq
      cp1ubde4LZVpvB5HXf5LF+12ZZDbFCbDTK6Kx8HoDLRcv/1vs/A/heXdnxl1e4eh
      vRtYoO8fDx3tVC4QSdWoJr2HjNXp5GO28rviJusafeNfhAaEirwl
      -----END RSA PRIVATE KEY-----
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"route.openshift.io/v1","kind":"Route","metadata":{"annotations":{"cert-manager.io/common-name":"arkamq.io","cert-manager.io/issuer-kind":"Issuer","cert-manager.io/issuer-name":"selfsigned-issuer-ingress"},"labels":{"ActiveMQArtemis":"artemis-broker","application":"artemis-broker-app","statefulset.kubernetes.io/pod-name":"artemis-broker-ss-0"},"name":"my-route-artemis-broker","namespace":"default"},"spec":{"host":"my-route-artemis-broker-default.apps-crc.testing","port":{"targetPort":"wconsj-0"},"tls":{"insecureEdgeTerminationPolicy":"Redirect","termination":"edge"},"to":{"kind":"Service","name":"artemis-broker-wconsj-0-svc","weight":100}}}
  creationTimestamp: "2023-09-19T09:20:20Z"
  labels:
    ActiveMQArtemis: artemis-broker
    application: artemis-broker-app
    statefulset.kubernetes.io/pod-name: artemis-broker-ss-0
  name: my-route-artemis-broker
  namespace: default
  resourceVersion: "1060469"
  uid: 1d98e663-18b5-4ce8-bf1c-b8521068370f
spec:
  host: my-route-artemis-broker-default.apps-crc.testing
  port:
    targetPort: wconsj-0
  tls:
    insecureEdgeTerminationPolicy: Redirect
    termination: edge
  to:
    kind: Service
    name: artemis-broker-wconsj-0-svc
    weight: 100
  wildcardPolicy: None
status:
  ingress:
  - conditions:
    - lastTransitionTime: "2023-09-19T09:20:20Z"
      status: "True"
      type: Admitted
    host: my-route-artemis-broker-default.apps-crc.testing
    routerCanonicalHostname: router-default.apps-crc.testing
    routerName: default
    wildcardPolicy: None

I have no idea what's the problem. Please help. If you need more info please let me know.
Thanks

gaohoward · 2023-09-19T12:56:18Z

@ctml91 and I check the audit log and there seems to be no specific info regarding the route. Everything is pretty general.

ctrought · 2023-09-19T14:46:56Z

Can you gather the logs again from the controller, etc.?

$ oc get issuers -n default
$ oc get certificaterequests -n default
$ oc describe certificaterequest <cr>
$ oc get events -n default
$ oc get events -n cert-manager
$ oc logs <cert-manager-route-controller> -n cert-manager
$ oc logs <cert-manager-controller> -n cert-manager

gaohoward · 2023-09-19T15:39:25Z

Hi @ctrought
Here are the information I gathered.

$ oc get issuers -n default

(20)[23:22:08]@route$  oc get issuers -n default
NAME                        READY   AGE
selfsigned-issuer-ingress   True    11h

$ oc get certificaterequests -n default

(20)[23:22:17]@route$ oc get certificaterequests -n default
NAME                                    APPROVED   DENIED   READY   ISSUER                      REQUESTOR                                                          AGE
artemis-broker-wconsj-0-svc-rte-s5jd4   True                False   selfsigned-issuer-ingress   system:serviceaccount:cert-manager:cert-manager-openshift-routes   11h
my-route-artemis-broker-njg94           True                False   selfsigned-issuer-ingress   system:serviceaccount:cert-manager:cert-manager-openshift-routes   6h2m

$ oc describe certificaterequest

(20)[23:23:22]@route$ oc describe certificaterequests my-route-artemis-broker-njg94
Name:         my-route-artemis-broker-njg94
Namespace:    default
Labels:       <none>
Annotations:  cert-manager.io/certificate-revision: 1
API Version:  cert-manager.io/v1
Kind:         CertificateRequest
Metadata:
  Creation Timestamp:  2023-09-19T09:20:20Z
  Generate Name:       my-route-artemis-broker-
  Generation:          1
  Managed Fields:
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:cert-manager.io/certificate-revision:
        f:generateName:
        f:ownerReferences:
          .:
          k:{"uid":"1d98e663-18b5-4ce8-bf1c-b8521068370f"}:
      f:spec:
        .:
        f:duration:
        f:issuerRef:
          .:
          f:kind:
          f:name:
        f:request:
        f:usages:
    Manager:      Go-http-client
    Operation:    Update
    Time:         2023-09-19T09:20:20Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
          .:
          k:{"type":"Approved"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
    Manager:      cert-manager-certificaterequests-approver
    Operation:    Update
    Subresource:  status
    Time:         2023-09-19T09:20:20Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
          k:{"type":"Ready"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
        f:failureTime:
    Manager:      cert-manager-certificaterequests-issuer-selfsigned
    Operation:    Update
    Subresource:  status
    Time:         2023-09-19T09:20:20Z
  Owner References:
    API Version:           route.openshift.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Route
    Name:                  my-route-artemis-broker
    UID:                   1d98e663-18b5-4ce8-bf1c-b8521068370f
  Resource Version:        1060483
  UID:                     13169b94-65cc-4c79-a80f-6b09449c996a
Spec:
  Duration:  2160h0m0s
  Extra:
    authentication.kubernetes.io/pod-name:
      cert-manager-openshift-routes-5bfbc6fb88-75zvd
    authentication.kubernetes.io/pod-uid:
      7f3e6354-93c1-4911-9bda-be692822a411
  Groups:
    system:serviceaccounts
    system:serviceaccounts:cert-manager
    system:authenticated
  Issuer Ref:
    Kind:   Issuer
    Name:   selfsigned-issuer-ingress
  Request:  LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ3B6Q0NBWThDQVFBd0ZERVNNQkFHQTFVRUF4TUpZWEpyWVcxeExtbHZNSUlCSWpBTkJna3Foa2lHOXcwQgpBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF1ZG1VT244YzZLd0NLQitaSTR0S0JjVUIxaUpJM0JhZWg3Z09wVGNaCm1PbVlKM2NuWnErSDFsUTV4Y2RjOWM0dGtFRWNMc2NOT2xXWmlDVHBNbUtaRlhiUEp4cjhLNGc3cy9LYXNUQlkKOFh0bmtQVExWUWVkYU1pUnkxYUlKS1l4RTg4SW1VaC9ML2JZM2lRSWtVQTRSZWRlYWFURUUvQ1RGZSs4K3BPbQpUeUVIL1RhNGF0TkFmZkdibmh3SEtPeGp1TVczbjBlUHZIUDB0d1NGeHhSQStuNndISHpSRWZ0Y0RteW8zcnIxCk1HZk13d2gwN2pBcGhkWjNqRnJ6KzlnbHI2UFV3ZGlLMTRiQVlpUVJEcjk5UGFiTHRScXRBRHBYQnVLbUVyM1IKSW9KMFIxMFFZQytMRko4bDBqc3A0eVhkT2tQQU1UTjhjOWlMa2VzYm0xMElmd0lEQVFBQm9FNHdUQVlKS29aSQpodmNOQVFrT01UOHdQVEE3QmdOVkhSRUVOREF5Z2pCdGVTMXliM1YwWlMxaGNuUmxiV2x6TFdKeWIydGxjaTFrClpXWmhkV3gwTG1Gd2NITXRZM0pqTG5SbGMzUnBibWN3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUY1WWxGN3AKck1Pc0RyNGd3SWw4eVJkY1N2bGQ3VzFzaGxqMTVXRkwwd2JMWXRwSG5FRE5Lb1ZrS0xXRmNlRlQyU25wb2JjMgp6UnFySUFZU3oxeThZejBGTHJLQ1EwVG9tbXFMQXNsc0RPZUt6ZGhXOWJnYkFOcW5IUmowajUwb2I2WjJ1dCtyCkJ6TENSWHVpY3pDaUFPUjNHVVFkL2J6VGtFU2ZyUFoybStjZlF3THZnUEUrc2gzWFBaSHpwNVlyRDhlenpLY24KVHpBREJHcXRRL1puc1liWkMxQUVrMEJBS1JuOEppb05ybXAxRTVFandtclB5eldIOTlSdW9vV0VTS2tsRExOVQpHVHRyT3lxb3Q1enUwSmlqYU5YeWswby9BZGxHaDF0QStuckNuZDhSLzBJT1NvUlcrMU9Ca0tOelFZTUp4bmNoClYxREFJZFNKTE1aY2hjRT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUgUkVRVUVTVC0tLS0tCg==
  UID:      0d32d56f-00fe-4f7c-affe-f1d01a5c6598
  Usages:
    server auth
    digital signature
    key encipherment
  Username:  system:serviceaccount:cert-manager:cert-manager-openshift-routes
Status:
  Conditions:
    Last Transition Time:  2023-09-19T09:20:20Z
    Message:               Certificate request has been approved by cert-manager.io
    Reason:                cert-manager.io
    Status:                True
    Type:                  Approved
    Last Transition Time:  2023-09-19T09:20:20Z
    Message:               Annotation "cert-manager.io/private-key-secret-name" missing or reference empty: secret name missing
    Reason:                Failed
    Status:                False
    Type:                  Ready
  Failure Time:            2023-09-19T09:20:20Z
Events:                    <none>

$ oc get events -n default

(20)[23:28:34]@route$ oc get events -n default
No resources found in default namespace.

$ oc get events -n cert-manager

(20)[23:29:09]@route$ oc get events -n cert-manager
No resources found in cert-manager namespace.

$ oc logs -n cert-manager

oc logs cert-manager-openshift-routes-5bfbc6fb88-75zvd -n cert-manager > cert-manager-openshift-routes.log

$ oc logs -n cert-manager

oc logs cert-manager-7cc7959fbb-w6fzk -n cert-manager > cert-manager.log

cert-manager.log
cert-manager-openshift-routes.log

gaohoward · 2023-09-27T09:22:39Z

Annotation "cert-manager.io/private-key-secret-name" missing or reference empty: secret name missing
I'm not sure what this annotation is for?

gaohoward · 2023-09-27T15:05:40Z

@ctrought any idea what could be the problem? Thanks!

maelvls · 2023-12-15T15:03:09Z

I just realized that openshift-routes is not compatible with cert-manager's SelfSigned issuer at the moment. The SelfSigned issuer requires the private key to be stored in a Secret resource and requires the special annotation cert-manager.io/private-key-secret-name to be present on the CertificateRequest.

Usually, cert-manager creates this annotation when it creates the CertificateRequest. But since openshift-routes is the one that creates the CertificateRequest and openshift-routes doesn't know that the issuer is the SelfSigned issuer, it doesn't add the annotation and doesn't create the Secret.

I think we need to mention this limitation in the README! 😅

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Route with cert-manager annotations is not created #38

Route with cert-manager annotations is not created #38

gaohoward commented Sep 5, 2023

ctrought commented Sep 13, 2023

gaohoward commented Sep 14, 2023

gaohoward commented Sep 14, 2023 •

edited

Loading

ctml91 commented Sep 14, 2023

gaohoward commented Sep 19, 2023

gaohoward commented Sep 19, 2023

ctrought commented Sep 19, 2023

gaohoward commented Sep 19, 2023 •

edited

Loading

gaohoward commented Sep 27, 2023

gaohoward commented Sep 27, 2023

maelvls commented Dec 15, 2023

Route with cert-manager annotations is not created #38

Route with cert-manager annotations is not created #38

Comments

gaohoward commented Sep 5, 2023

ctrought commented Sep 13, 2023

gaohoward commented Sep 14, 2023

gaohoward commented Sep 14, 2023 • edited Loading

ctml91 commented Sep 14, 2023

gaohoward commented Sep 19, 2023

gaohoward commented Sep 19, 2023

ctrought commented Sep 19, 2023

gaohoward commented Sep 19, 2023 • edited Loading

gaohoward commented Sep 27, 2023

gaohoward commented Sep 27, 2023

maelvls commented Dec 15, 2023

gaohoward commented Sep 14, 2023 •

edited

Loading

gaohoward commented Sep 19, 2023 •

edited

Loading