Provision Storage for Image Registry

OCP comes with a in-built container image registry. The advantages of these container image registry are the following:

From Red Hat - The registry is typically used as a publication target for images built on the cluster, as well as being a source of images for workloads running on the cluster. When a new image is pushed to the registry, the cluster is notified of the new image and other components can react to and consume the updated image.

• Integrates with OCP clusters authentication and authorization system
• Provides a image source local to the cluster for all running workloads
• Integrates with OCP clusters CICD workflows - for example when a new container images is uploaded to this registry, the cluster can update the image of all the running running contianers
• Registry can be easily scaled up or down
• Image data is stored in a cloud storage

OCP Image Registry Scope

Internal image registry should not be used as enterprise-wide image registry. For this use-case Quay is part of OpenShift OPP subscription. For disconnected scenarios there is a limited Quay-Version available: “mirror-registry”

Single OCP cluster is the typical scope of the OCP inbuilt registry.

There are two ways of provisioning Nutanix HCI based storage to OpenShift Image Registry:

• Using S3 compliant storage (recommended by Nutanix and RedHat)
• Presenting a CSI Volumes PVC (RWO)

We will look at both in this section. Nutanix CSI Volumes PVC is optional. Use one or the other for OCP Image registry.

Image Registry Storage using Nutanix S3 Bucket

In this section we will provision Nutanix Objects based S3 storage to serve as a storage for all OpenShift image registry containers.

Create a DNS Entry for Nutanix Objects Store

In this section we will add nutanix objects store's DNS records for lookup by OCP Image registry.

1. Logon to the AutoAD windows VM

   • Username: administrator
   • Password: your HPOC password

2. We will add the following entries to DNS server

   Create DNS Entry If Not Present

   Use your HPOC cluster Object's public IP only.

   Create A record entry only if it is not present in DNS server.

   The IP addresses in the following commands are used as an example. You should use IP address details that belong to your HPOC cluster's Object's public IP. For information on locating your cluster IP see Getting Started Networking section.

   Sample A Record Entry

   10.38.18.221  ntnx-objects.ntnxlab.local

3. Open PowerShell as Administrator and create the two A records

   Add the API A record for Objects IP

   Add-DnsServerResourceRecordA -Name ntnx-objects -IPv4Address 10.38.18.221 -ZoneName ntnxlab.local -ZoneScope ntnxlab.local

4. Test name resolution for added entries

   nslookup ntnx-objects.ntnxlab.local
   Server: dc.ntnxlab.local
   Address: 10.38.18.203   # << This is the DNS server IP address
   
   Name: ntnx-objects.ntnxlab.local
   Address: 10.38.18.221   # << This is the ntnx-objects public IP address

Installing SSL Certificate on Objects

We will need to install SSL certificates on the pre-provisioned nutanix-objects store to be able to use it as a OCP registry storage and to avoid other security threats.

Sharing a Nutanix Cluster?

Only one person needs to do this section for generate and install SSL certificates for Objects Store.

If you are sharing a HPOC for multiple users, then you need to do this section only once. Decide with other participants sharing your cluster before proceeding.

If you are the certificate admin in the cluster you are sharing with other users, create and install the follwing SSL certificates on Objects store:

ntnx-objects.ntnxlab.local.pem

ntnx-objects.ntnxlab.local-key.pem

rootCA.crt

1. Logon to your UserXX-LinuxToolsVM using VSCode client or on the browser

2. Create public and private key certificates for ntnx-objects.ntnxlab.local

   mkcert ntnx-objects.ntnxlab.local"
   Command output

   mkcert ntnx-objects.ntnxlab.local
   #
   Created a new certificate valid for the following names 📜
    - "ntnx-objects.ntnxlab.local"
   
   The certificate is at "./ntnx-objects.ntnxlab.local.pem" and the key at "./ntnx-objects.ntnxlab.local-key.pem" ✅
   
   It will expire on 30 August 2027

3. cat out the contents of certificate files and copy them to your Mac/PC workstation in separate files

   cat ntnx-objects.ntnxlab.local.pem
   cat ntnx-objects.ntnxlab.local-key.pem

4. Create these files on Mac/PC

   • On a Mac, use vi in Terminal or VSCode (if you already have it installed)
   • On Windows PC, use Notepad or VSCode

5. Go to Prism Central > Services > Objects

6. Select the ntnxlab object store and choose Manage FQDNs & SSL Certificates

7. Click on Replace SSL Certificate

8. Upload the following files:

   • Private key - ntnx-objects.ntnxlab.local-key.pem
   • Public Certificate - ntnx-objects.ntnxlab.local.pem
   • CA Certificate/Chain - rootCA.pem (This was created in the previous section during IPI pre-requisites preparation)

9. Under New FQDN, add ntnx-objects.ntnxlab.local as an additional FQDN

10. Click on Save

Generating Access Keys for S3 Bucket

1. Go to Prism Central > Objects

2. On the top menu, click on Access Keys

3. Click on + Add people

4. Select Add people not in a directory service

5. Enter a email (ocpuserXX@nutanix.com) and name (ocpuserXX)

   

6. Click on Next

7. Click on Generate Keys

8. Once generated, click on Download Keys

9. Once downloaded, click on Close

10. Open the downloaded file to verify contents

    Username: ocpuserXX@ntnxlab.com
    Access Key: t1FzWhILj_CxU4NGNcWzZ0hW-xxxxxxx
    Secret Key: KbGdgYMoaYx1DnIJ6gTgu-xxxxxxxxxx
    Display Name: ocpuserXX
    Tag: buckets-access-key-xxxxxxxxxxxxxxxxxxxx

11. Store the access key and secret key in a safe place for later

Create Buckets Storage for OCP Image Registry

We will create a bucket for backup destination

1. On the top menu, click on Object Stores

2. Click on ntnxlab (or other objects store) objects store, this will open objects store management page in a separate browser tab

3. Click on Create Bucket

4. Enter ocpuserXX-registry (E.g. ocpuser01-registry) as the bucket name

5. Click on Create

6. In the list of buckets, click on the ocpuserXX-registry (E.g. ocpuser01-registry) bucket

7. Click on User Access menu and Edit User Access

   

8. In the Share Bucket ocpuserXX-registry window, type in your email that you configured in User Access section

9. Give Full Access permissions

10. Click on Save

Create Kubernetes Resources to use Objects Store

We will need to create kubernetes resources to use the Objects store as the OCP registry store

1. Change to the directory where rootCA.crt file is present. (if not already there)

2. Create a connection to your OCP cluster (if not already done so)

   export KUBECONFIG=/home/ubuntu/ocpuserXX/auth/kubeconfig

   List the nodes in the cluster to make sure the connection is working

   oc get nodes
   Output

   # oc get nodes
   NAME                     STATUS   ROLES    AGE    VERSION
   xyz-7q676-master-0       Ready    master   115m   v1.24.0+b62823b
   xyz-7q676-master-1       Ready    master   116m   v1.24.0+b62823b
   xyz-7q676-master-2       Ready    master   115m   v1.24.0+b62823b
   xyz-7q676-worker-4lxjt   Ready    worker   104m   v1.24.0+b62823b
   xyz-7q676-worker-fg7b5   Ready    worker   104m   v1.24.0+b62823b
   xyz-7q676-worker-kbsfw   Ready    worker   104m   v1.24.0+b62823b

3. Create a config map

   oc create configmap object-ca --from-file=ca-bundle.crt=rootCA.pem -n openshift-config 
   Output

   configmap/object-ca created

4. Confirm the config map with the rootCA.crt contents for proxy/cluster resource

   oc -n openshift-config get cm user-ca-bundle -oyaml

   This should match with the contents oc rootCA.crt file

5. Create a secret with the bucket access and secret key you generated in the previous section

   Template command

   oc create secret generic image-registry-private-configuration-user \
     --from-literal=REGISTRY_STORAGE_S3_ACCESSKEY=<your-access-key> \
     --from-literal=REGISTRY_STORAGE_S3_SECRETKEY=<your-secret-key> \
     --namespace openshift-image-registry

   Sample command

   oc create secret generic image-registry-private-configuration-user \
     --from-literal=REGISTRY_STORAGE_S3_ACCESSKEY=ofYqh4g2ImLNuXm5JjqDrRtiDLKU8YAr \
     --from-literal=REGISTRY_STORAGE_S3_SECRETKEY=jluOBDuENC7jeci7JiSH4tsB4uCHX0ST \
     --namespace openshift-image-registry
   Output

   secret/image-registry-private-configuration-user created

6. Update the image registry configuration to use the newly create nutanix objects S3 bucket

   Template command

   oc patch configs.imageregistry.operator.openshift.io/cluster \
     --type='json' \
     --patch='[
   {"op": "remove", "path": "/spec/storage" },
   {"op": "add", "path": "/spec/storage", "value":
   {"s3":
   {"bucket": "ocpuserXX-registry",                        ### <<< REMEMBER TO USE YOUR BUCKET NAME
   "regionEndpoint": "https://ntnx-objects.ntnxlab.local",
   "encrypt": false, 
   "region": "us-east-1"}}}]'

   Sample command

   oc patch configs.imageregistry.operator.openshift.io/cluster \
     --type='json' \
     --patch='[
   {"op": "remove", "path": "/spec/storage" },
   {"op": "add", "path": "/spec/storage", "value":
   {"s3":
   {"bucket": "ocpuser01-registry",                       ### <<< REMEMBER TO USE YOUR BUCKET NAME
   "regionEndpoint": "https://ntnx-objects.ntnxlab.local",
   "encrypt": false, 
   "region": "us-east-1"}}}]'
   Output

   config.imageregistry.operator.openshift.io/cluster patched

7. Enable the image registry (by default image registry is disabled)

   oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"managementState":"Managed"}}'
   Output

   config.imageregistry.operator.openshift.io/cluster patched

8. You can use the config description to check if the image registry successfully connected to Nutanix Objects store's bucket ocpuserXX-registry

   oc get config.imageregistry.operator.openshift.io/cluster -oyaml
   Output

   kind: Config
   
   spec:
     httpSecret: xxxxxxxxxx
     logLevel: Normal
     ## Snipped for brevity
       s3:
         bucket: ocpuserXX-registry                            ## << your Nutanix bucket for storing container images
         region: us-east-1
         regionEndpoint: https://ntnx-objects.ntnxlab.local  ## << your Nutanix Object's URL
         trustedCA:
           name: ""
         virtualHostedStyle: false
     unsupportedConfigOverrides: null
   ## Snipped for brevity
   status:
     - lastTransitionTime: "2022-10-04T01:56:40Z"
       reason: S3 Bucket Exists                             
       status: "True"
       type: StorageExists                                  ## << your Nutanix bucket connection is successful

Verify Image Contents in S3 Bucket

We will install a simple application to verify if the local OCP image registry is able to store container images in the S3 bucket. Verification of our setup as we go helps us setup our workload on OCP cluster without running into issues.

1. Create a new project (namespace) in OCP cluster

   oc new-project my-shared-storage

2. Create an app called new-app inside this project/namespace

   oc new-app openshift/php https://github.com/christianh814/openshift-php-upload-demo --name=file-uploader

3. Wait for a few seconds to check the logs of the application

   sleep 30s
   oc logs -f file-uploader-1-build
   Output

   ## Output snipped for brevity
   STEP 9/9: CMD /usr/libexec/s2i/run
   COMMIT temp.builder.openshift.io/my-shared-storage/file-uploader-1:00fde781
   Getting image source signatures
   Copying blob sha256:b38cb92596778e2c18c2bde15f229772fe794af39345dd456c3bf6702cc11eef
   Copying config sha256:9ef2b09224b2a0b312cc0c5a1b5c96afadb9e7f1c36f990ad1c47c50ac3ea82a
   Writing manifest to image destination
   Storing signatures
   --> 9ef2b09224b
   Successfully tagged temp.builder.openshift.io/my-shared-storage/file-uploader-1:00fde781
   9ef2b09224b2a0b312cc0c5a1b5c96afadb9e7f1c36f990ad1c47c50ac3ea82a
   Pushing image image-registry.openshift-image-registry.svc:5000/my-shared-storage/file-uploader:latest ...
   Getting image source signatures
   Copying blob sha256:b9cf13b728c6800670647c11df5701edf60214352ff4c3d721bf0277cf20350d
   Copying config sha256:9ef2b09224b2a0b312cc0c5a1b5c96afadb9e7f1c36f990ad1c47c50ac3ea82a
   Writing manifest to image destination
   Storing signatures
   Successfully pushed image-registry.openshift-image-registry.svc:5000/my-shared-storage/file-uploader@sha256:6a073686b1538fe6d2cc657f0116060d93721caef05c3bbee267e67b29b9fa79
   Push successful

   To verify the pods are running

   oc get pods
   Output

   # oc get po
   NAME                                READY   STATUS      RESTARTS   AGE
   file-uploader-1-build               0/1     Completed   0          32m
   file-uploader-5d56584787-bzkbs      1/1     Running     0          31m

4. To verify if the images are present in the S3 bucket, Logon to your Prism Central and check the contents using Object's browser.

5. From Prism Central > Services > Objects

6. Click on ntnx-object object store (this will open in a new tab)

7. Select your ocpuserXX-registry

8. Click on Launch Objects Browser (this will open in a new tab)

9. Here you will be able to see the contents of the uploaded container image (you will have to drill down the directory structure to see this as shown below)