Fixes ocp-power-automation#220

Fixes ocp-power-automation#220 Wildcard move Fixes ocp-power-automation#220 Wildcard move Fixes ocp-power-automation#220 Updated doc Fixes ocp-power-automation#220 Updated doc Fixes ocp-power-automation#220 Updated Docu Fixes ocp-power-automation#220 Updated Docu Update var.tfvars-doc.md Fixes ocp-power-automation#220 Update var.tfvars-doc.md Updated documentation, that selecting supported branches are also supported Enable external DNS/LB support Signed-off-by: CS Zhang <[email protected]> Update the doc Signed-off-by: CS Zhang <[email protected]> Update the helpernode_tag to latest level Signed-off-by: CS Zhang <[email protected]> Add RHCOS kernel options before installation Signed-off-by: Aishwarya Kamat <[email protected]> Allow OCP network customization before installation. (ocp-power-automation#224) add clusterNetwork_CIDR, serviceNetwork, hostprefix vars Added cs-zhang as approver To set mtu on private network Signed-off-by: Aishwarya Kamat <[email protected]> remove mkumatag from reviewer list Not actively involved, hence removing my entry from the reviewers to avoid getting assigned automatically for the review force centos stream to use ansible 2.9 like rhel8 Accessing cluster using non-root user Signed-off-by: Aishwarya Kamat <[email protected]> bastion fqdn with clusterID as subdmain To remove the scp error with Terraform v1.1.x Signed-off-by: Aishwarya Kamat <[email protected]> To Update the Terraform Version Signed-off-by: Aishwarya Kamat <[email protected]> FIPS enablement Signed-off-by: Aishwarya Kamat <[email protected]> Merging the code ocp-power-automation#220 Merging the code ocp-power-automation#220
torwen1 · Apr 6, 2022 · 09f362e · 09f362e
1 parent ab3dbff
commit 09f362e
Show file tree

Hide file tree

Showing 21 changed files with 356 additions and 67 deletions.
diff --git a/OWNERS b/OWNERS
@@ -1,10 +1,10 @@
 reviewers:
-  - mkumatag
   - Prajyot-Parab
   - sudeeshjohn
   - yussufsh
   - bpradipt
   - cs-zhang
 approvers:
   - bpradipt
+  - cs-zhang
   - yussufsh
diff --git a/docs/automation_host_prereqs.md b/docs/automation_host_prereqs.md
@@ -22,7 +22,7 @@ Install the following packages on the automation host. Select the appropriate in
 **Terraform >= 0.13.0**: Please refer to the [link](https://learn.hashicorp.com/terraform/getting-started/install.html) for instructions on installing Terraform. For validating the version run `terraform version` command after install.
 
 Install Terraform and providers for Power environment:
-1. Download the Terraform binary version 0.13.5 from https://www.power-devops.com/terraform and install it to /usr/local/bin.
+1. Download and install the Terraform binary (>= 0.13.0) for Linux/ppc64le from https://www.power-devops.com/terraform.
 2. Download the required Terraform providers for Power into your TF project directory:
 ```
 $ cd <path_to_TF_project>

diff --git a/docs/var.tfvars-doc.md b/docs/var.tfvars-doc.md
@@ -83,10 +83,11 @@ worker                      = {instance_type    = "<worker-compute-template>", i
 ```
 These set of variables specify the username and the SSH key to be used for accessing the bastion node.
 ```
-rhel_username               = "root"
+rhel_username               = "root"  #Set it to an appropriate username for non-root user access
 public_key_file             = "data/id_rsa.pub"
 private_key_file            = "data/id_rsa"
 ```
+rhel_username is set to root. rhel_username can be set to an appropriate username having superuser privileges with no password prompt.
 Please note that only OpenSSH formatted keys are supported. Refer to the following links for instructions on creating SSH key based on your platform.
 - Windows 10 - https://phoenixnap.com/kb/generate-ssh-key-windows-10
 - Mac OSX - https://www.techrepublic.com/article/how-to-generate-ssh-keys-on-macos-mojave/
@@ -139,10 +140,24 @@ If `cluster_if_prefix` is not set, the `cluster_id` will be used only without pr
 A random value will be used for `cluster_id` if not set.
 The total length of `cluster_id_prefix`.`cluster_id` should not exceed 14 characters.
 
+### FIPS Variable for OpenShift deployment
+
+These variables will be used for deploying OCP in FIPS mode.
+Change the values as per your requirement.
+```
+fips_compliant      = false
+```
+
 ### Misc Customizations
 
 These variables provides miscellaneous customizations. For common usage scenarios these are not required and should be left unchanged.
 
+The following variables are used to define the IP address for the preconfigured external DNS and the Load-balancer 
+```
+lb_ipaddr                       = ""
+ext_dns                         = ""
+```
+
 The following variable is used to set the network adapter type for the VMs. By default the VMs will use SEA. If SRIOV is required then uncomment the variable
 ```
 network_type                = "SRIOV"
@@ -179,14 +194,49 @@ This variable can be used for trying out custom OpenShift install image for deve
 release_image_override     = ""
 ```
 
-These variables specify the ansible playbooks that are used for OpenShift install and post-install customizations.
+These variables specify the ansible playbooks that are used for OpenShift install and post-install customizations. If the URL ends with a file name extension .zip, then it is assumed that it points to a HTTP/HTTPS server and curl/unzip will be used to extract the package. URLs without ending with .zip are recognized as GitHub repositories and git clone && git checkout are used.
+`Only .zip is supported file format on web servers. The all files must be placed in folders starting with ocp4-playbooks, or ocp4-helpernode! It is allowed to extend the directory name with additional informations: e.g. ocp4-helpernode-<master/version number)`
+Valid options: Requires a URL pointing to the packages/GitHub project.
 ```
+helpernode_repo            = "https://<HTTP SERVER>/ocp4-ansible-modules/ocp4-helpernode-master.zip"
+OR
 helpernode_repo            = "https://github.com/RedHatOfficial/ocp4-helpernode"
 helpernode_tag             = "5eab3db53976bb16be582f2edc2de02f7510050d"
+
+install_playbook_repo      = "https://<HTTP SERVER>/ocp4-ansible-modules/ocp4-playbooks-master.zip"
+OR
 install_playbook_repo      = "https://github.com/ocp-power-automation/ocp4-playbooks"
 install_playbook_tag       = "02a598faa332aa2c3d53e8edd0e840440ff74bd5"
 ```
 
+If you want to provide the ansible playbooks by your local HTTP server, follow these steps:
+```
+Use your web browser and visit https://github.com/RedHatOfficial/ocp4-helpernode
+On the main page, stay on the master repository page, or select any supported branch and click on the green "Code" button with a download symbol in front of it
+Click on "Download ZIP"
+Upload the file to your local HTTP server and place it in the appropriate directory
+
+Use your web browser and visit https://github.com/ocp-power-automation/ocp4-playbooks
+On the main page, stay on the master repository page, or select any supported branch and click on the green "Code" button with a download symbol in front of it
+Click on "Download ZIP"
+Upload the file to your local HTTP server and place it in the appropriate directory, like the example below
+
+ls -la /var/www/html/repos/
+total 13452
+-rw-r--r--. 1 root root 13624204 Jul  8 13:43 ocp4-helpernode.zip
+-rw-r--r--. 1 root root   145165 Jul  8 13:44 ocp4-playbooks.zip
+```
+
+This variable can be used to define a different source for the helm package, like a local web server. By default, the help package will be downloaded from the official internet source.
+```
+helm_repo                  = "https://<HTTP SERVER>/python-modules/helm-latest-linux-ppc64le.tar.gz"
+```
+
+This variable specify the MTU value for the private network interface on RHEL and RHCOS nodes. The CNI network will have <private_network_mtu> - 50 for OpenshiftSDN and <private_network_mtu> - 100 for OVNKubernetes network provider.
+```
+private_network_mtu         = 1450
+```
+
 These variables can be used when debugging ansible playbooks
 ```
 installer_log_level         = "info"
@@ -198,6 +248,16 @@ This variable specifies the external DNS servers to forward DNS queries that can
 dns_forwarders              = "1.1.1.1; 9.9.9.9"
 ```
 
+List of [day-1 kernel arguments](https://docs.openshift.com/container-platform/4.8/installing/install_config/installing-customizing.html#installation-special-config-kargs_installing-customizing) for the cluster nodes.
+To add kernel arguments to master or worker nodes, using MachineConfig object and inject that object into the set of manifest files used by Ignition during cluster setup.
+```
+rhcos_pre_kernel_options        = []
+```
+- Example 1
+  ```
+  rhcos_pre_kernel_options   = ["rd.multipath=default","root=/dev/disk/by-label/dm-mpath-root"]
+  ```
+
 List of [kernel arguments](https://docs.openshift.com/container-platform/4.4/nodes/nodes/nodes-nodes-working.html#nodes-nodes-kernel-arguments_nodes-nodes-working) for the cluster nodes.
 Note that this will be applied after the cluster is installed and all the nodes are in `Ready` status.
 ```
@@ -255,4 +315,7 @@ This variable is used to set the default Container Network Interface (CNI) netwo
 
 ```
 cni_network_provider       = "OpenshiftSDN"
-```
+cluster_network_cidr        = "10.128.0.0/14"
+cluster_network_hostprefix  = "23"
+service_network             = "172.30.0.0/16"
+```
diff --git a/modules/1_bastion/bastion.tf b/modules/1_bastion/bastion.tf
@@ -93,18 +93,18 @@ resource "null_resource" "bastion_init" {
     }
     provisioner "file" {
         content = var.private_key
-        destination = "$HOME/.ssh/id_rsa"
+        destination = ".ssh/id_rsa"
     }
     provisioner "file" {
         content = var.public_key
-        destination = "$HOME/.ssh/id_rsa.pub"
+        destination = ".ssh/id_rsa.pub"
     }
     provisioner "remote-exec" {
         inline = [
-            "sudo chmod 600 $HOME/.ssh/id_rsa*",
+            "sudo chmod 600 .ssh/id_rsa*",
             "sudo sed -i.bak -e 's/^ - set_hostname/# - set_hostname/' -e 's/^ - update_hostname/# - update_hostname/' /etc/cloud/cloud.cfg",
-            "sudo hostnamectl set-hostname --static ${lower(var.cluster_id)}-bastion-${count.index}.${var.cluster_domain}",
-            "echo 'HOSTNAME=${lower(var.cluster_id)}-bastion-${count.index}.${var.cluster_domain}' | sudo tee -a /etc/sysconfig/network > /dev/null",
+            "sudo hostnamectl set-hostname --static ${lower(var.cluster_id)}-bastion-${count.index}.${lower(var.cluster_id)}.${var.cluster_domain}",
+            "echo 'HOSTNAME=${lower(var.cluster_id)}-bastion-${count.index}.${lower(var.cluster_id)}.${var.cluster_domain}' | sudo tee -a /etc/sysconfig/network > /dev/null",
             "sudo hostname -F /etc/hostname",
             "echo 'vm.max_map_count = 262144' | sudo tee --append /etc/sysctl.conf > /dev/null",
         ]
@@ -265,12 +265,12 @@ resource "null_resource" "bastion_packages" {
     provisioner "remote-exec" {
         inline = [
             "#sudo yum update -y --skip-broken",
-            "sudo yum install -y wget jq git net-tools vim python3 tar"
+            "sudo yum install -y wget jq git net-tools vim python3 tar curl unzip"
         ]
     }
     provisioner "remote-exec" {
         inline = [
-           "sudo yum install -y ansible"
+           "sudo yum install -y ansible-2.9.*"
         ]
     }
     provisioner "remote-exec" {
@@ -327,11 +327,11 @@ resource "null_resource" "setup_nfs_disk" {
     }
     provisioner "remote-exec" {
         inline = [
-            "rm -rf mkdir ${local.storage_path}; mkdir -p ${local.storage_path}; chmod -R 755 ${local.storage_path}",
+            "sudo rm -rf mkdir ${local.storage_path}; sudo mkdir -p ${local.storage_path}; sudo chmod -R 755 ${local.storage_path}",
             "sudo chmod +x /tmp/create_disk_link.sh",
             # Fix for copying file from Windows OS having CR
-            "sed -i 's/\r//g' /tmp/create_disk_link.sh",
-            "/tmp/create_disk_link.sh",
+            "sudo sed -i 's/\r//g' /tmp/create_disk_link.sh",
+            "sudo /tmp/create_disk_link.sh",
             "sudo mkfs.ext4 -F /dev/${local.disk_config.disk_name}",
             "echo '/dev/${local.disk_config.disk_name} ${local.storage_path} ext4 defaults 0 0' | sudo tee -a /etc/fstab > /dev/null",
             "sudo mount ${local.storage_path}",

diff --git a/modules/1_bastion/versions.tf b/modules/1_bastion/versions.tf
@@ -33,5 +33,5 @@ terraform {
       version = "~> 2.3"
     }
   }
-  required_version = "~> 0.13.0"
+  required_version = ">= 0.13.0"
 }
diff --git a/modules/2_network/versions.tf b/modules/2_network/versions.tf
@@ -25,5 +25,5 @@ terraform {
       version = "~> 1.32"
     }
   }
-  required_version = "~> 0.13.0"
+  required_version = ">= 0.13.0"
 }
diff --git a/modules/3_helpernode/helpernode.tf b/modules/3_helpernode/helpernode.tf
@@ -38,6 +38,8 @@ locals {
         bastion_master_ip   = var.bastion_ip[0]
         bastion_backup_ip   = length(var.bastion_ip) > 1 ? slice(var.bastion_ip, 1, length(var.bastion_ip)) : []
         forwarders      = var.dns_forwarders
+        lb_ipaddr       = var.lb_ipaddr
+        ext_dns         = var.ext_dns
         gateway_ip      = var.gateway_ip
         netmask         = cidrnetmask(var.cidr)
         broadcast       = cidrhost(var.cidr,-1)
@@ -67,20 +69,21 @@ locals {
         ]
 
         local_registry           = local.local_registry
+        helm_repo                = var.helm_repo
         client_tarball           = var.openshift_client_tarball
         install_tarball          = var.openshift_install_tarball
     }
     helpernode_inventory = {
-        bastion_ip  = var.bastion_ip
+        rhel_username = var.rhel_username
+        bastion_ip    = var.bastion_ip
     }
 }
 
-resource "null_resource" "config" {
-
+resource "null_resource" "prep_helpernode_tools_git" {
     triggers = {
         bootstrap_count = var.bootstrap_port_ip == "" ? 0 : 1
-        worker_count    = length(var.worker_port_ips)
     }
+    count = length(regexall("\\.zip$", var.helpernode_repo)) == 0 ? 1 : 0
 
     connection {
         type        = "ssh"
@@ -101,23 +104,75 @@ resource "null_resource" "config" {
             "cd ocp4-helpernode && git checkout ${var.helpernode_tag}"
         ]
     }
+}
+
+resource "null_resource" "prep_helpernode_tools_curl" {
+    triggers = {
+        bootstrap_count = var.bootstrap_port_ip == "" ? 0 : 1
+    }
+    count = length(regexall("\\.zip$", var.helpernode_repo)) > 0 ? 1 : 0
+
+    connection {
+        type        = "ssh"
+        user        = var.rhel_username
+        host        = var.bastion_ip[0]
+        private_key = var.private_key
+        agent       = var.ssh_agent
+        timeout     = "${var.connection_timeout}m"
+        bastion_host = var.jump_host
+    }
+
+    provisioner "remote-exec" {
+        inline = [
+            "mkdir -p .openshift",
+            "rm -rf ocp4-helpernode",
+            "rm -rf ocp4-extract-helper",
+            "mkdir -p ocp4-extract-helper", 
+            "echo 'Downloading ocp4-helpernode...'",
+            "curl -o ocp4-extract-helper/ocp4-helpernode.zip ${var.helpernode_repo}",
+            "echo 'Extracting ocp4-helpernode...'",
+            "cd ocp4-extract-helper && unzip ocp4-helpernode.zip",
+            "cd .. && rm -rf ocp4-extract-helper/ocp4-helpernode.zip",
+            "mv ocp4-extract-helper/ocp4-helpernode* ocp4-helpernode",
+            "rm -rf ocp4-extract-helper"
+        ]
+    }
+}
+
+resource "null_resource" "config" {
+    depends_on = [null_resource.prep_helpernode_tools_git, null_resource.prep_helpernode_tools_curl]
+    triggers = {
+        bootstrap_count = var.bootstrap_port_ip == "" ? 0 : 1
+        worker_count    = length(var.worker_port_ips)
+    }
+
+    connection {
+        type        = "ssh"
+        user        = var.rhel_username
+        host        = var.bastion_ip[0]
+        private_key = var.private_key
+        agent       = var.ssh_agent
+        timeout     = "${var.connection_timeout}m"
+        bastion_host = var.jump_host
+    }
+
     provisioner "file" {
         content     = templatefile("${path.module}/templates/helpernode_inventory", local.helpernode_inventory)
-        destination = "$HOME/ocp4-helpernode/inventory"
+        destination = "ocp4-helpernode/inventory"
     }
     provisioner "file" {
         content     = var.pull_secret
-        destination = "$HOME/.openshift/pull-secret"
+        destination = ".openshift/pull-secret"
     }
     provisioner "file" {
         content     = templatefile("${path.module}/templates/helpernode_vars.yaml", local.helpernode_vars)
-        destination = "$HOME/ocp4-helpernode/helpernode_vars.yaml"
+        destination = "ocp4-helpernode/helpernode_vars.yaml"
     }
     provisioner "remote-exec" {
         inline = [
             "sed -i \"/^helper:.*/a \\ \\ networkifacename: $(ip r | grep \"${var.cidr} dev\" | awk '{print $3}')\" ocp4-helpernode/helpernode_vars.yaml",
             "echo 'Running ocp4-helpernode playbook...'",
-            "cd ocp4-helpernode && ansible-playbook  -i inventory -e @helpernode_vars.yaml tasks/main.yml ${var.ansible_extra_options}"
+            "cd ocp4-helpernode && ansible-playbook  -i inventory -e @helpernode_vars.yaml tasks/main.yml ${var.ansible_extra_options} --become"
         ]
     }
 }
diff --git a/modules/3_helpernode/templates/helpernode_inventory b/modules/3_helpernode/templates/helpernode_inventory
@@ -1,4 +1,4 @@
 [vmhost]
 %{ for ip in bastion_ip ~}
-${ip} ansible_connection=ssh ansible_user=root
+${ip} ansible_connection=ssh ansible_user=${rhel_username}
 %{ endfor ~}
diff --git a/modules/3_helpernode/templates/helpernode_vars.yaml b/modules/3_helpernode/templates/helpernode_vars.yaml
@@ -21,10 +21,16 @@ dns:
   domain: "${cluster_domain}"
   clusterid: "${cluster_id}"
   forwarder1: "${forwarders}"
+%{ if lb_ipaddr != "" }
+  lb_ipaddr: "${lb_ipaddr}"
+%{ endif }
 dhcp:
   router: "${gateway_ip}"
   bcast: "${broadcast}"
   netmask: "${netmask}"
+%{ if ext_dns != "" }
+  dns: "${ext_dns}"
+%{ endif }
   ipid: "${ipid}"
   netmaskid: "${netmask}"
   poolstart: "${pool.start}"
@@ -84,4 +90,4 @@ ocp_initramfs: "file:///dev/null"
 ocp_install_kernel: "file:///dev/null"
 
 # This is required for latest helpernode. TODO: Remove when https://github.com/RedHatOfficial/ocp4-helpernode/pull/140 is merged
-helm_source: "https://get.helm.sh/helm-v3.4.0-linux-ppc64le.tar.gz"
+helm_source: "${helm_repo}"
diff --git a/modules/3_helpernode/variables.tf b/modules/3_helpernode/variables.tf
@@ -29,6 +29,8 @@ variable "dns_forwarders" {
     default   = "8.8.8.8; 9.9.9.9"
 }
 
+variable "lb_ipaddr" {}
+variable "ext_dns" {}
 variable "gateway_ip" {}
 variable "cidr" {}
 variable "allocation_pools" {}
@@ -58,6 +60,7 @@ variable "ocp_release_tag" {}
 
 variable "helpernode_repo" {}
 variable "helpernode_tag" {}
+variable "helm_repo" {}
 
 variable "ansible_extra_options" {}
 

diff --git a/modules/3_helpernode/versions.tf b/modules/3_helpernode/versions.tf
@@ -25,5 +25,5 @@ terraform {
       version = "~> 2.1"
     }
   }
-  required_version = "~> 0.13.0"
+  required_version = ">= 0.13.0"
 }
diff --git a/modules/4_nodes/versions.tf b/modules/4_nodes/versions.tf
@@ -33,5 +33,5 @@ terraform {
       version = "~> 2.3"
     }
   }
-  required_version = "~> 0.13.0"
+  required_version = ">= 0.13.0"
 }