//// Purpose ------- Capture information about how to use the deployed and related systems. Recommended to primarily be workflow diagrams, possible with labeled and annotated steps. Examples -------- * Continues Integration and Continues Deployment/Delivery processes * DeclarativeOps / GitOps processes * Update/Upgrade processes * Self service processes, - EX: create a CR for a custom operator to create a new namespace and attach it to the correct team RBAC. //// [id="processes_{context}"] = Processes == Cluster Lifecycle === Installation New clusters are installed using the ROSA command line interface and bootstrap scripts. === Upgrades The HCP team performs two minor version (e.g. 4.14.x => 4.16.y) upgrades per year with weekly patches. Weekly pathes all on he same day to ensure all stages are on the same patch version. == Secure Software Development Lifecycle Developers and the HCP team operate on the basis of a shared responsibility model where the platform and certain SSDLC enhancing services are provided by the HCP team but the ultimate responsibility for operating applications lies with the developers themselves. The core tools used are GitHub Enterprise for version control and CI/CD pipelines, Nexus for artifact management, and ACS for vulnerability scanning. In addition, in house tools such as lifecycle and the chargeback tool are used. === Ordering a Namespace To order a namespace, users need a LeanIX number. LeanIX is an asset inventory tracking system akin to a configuration management database [CMDB]. The LeanIX number is entered into a ServiceNow ticket along with the target stage (non-prod or prod), and a name. Naming of namespaces follows a `--` convention. image::order-ns.png[Process of ordering a namespace visualised. Adapted from workshop notes.] The namespcae is created according to a project template. Upon creation, RBAC role bindings are created and an in-house operator creates a GitOps repository on GitHub. The namespace creation itself is not yet gitopsed. To begin using the GitOps repository, developers need to copy paste a secret value from OpenShift to GitHub. === Increasing Resource Quotas Up to a certain limit, developers are able to increase their resource quota in a self-service fashion. === Releasing and deploying Software Container images are built using GitHub actions. The runner for the pipelines are running on EKS and are provided by the Dev Tools team. Images are pushed to OpenShift ImageStreams and for production releases to Nexus. .Process of releasing, and deploying software to a cluster visualised. Adapted from workshop notes. image::release.png[] === Ordering managed services To order managed services such as an RDS database, users create a ServiceNow ticket. The resources are then manually provisioned using terraform by the on-call engineer. Credentials are delivered to users via encrypted email. === Vulnerability Management The HCP team's vulnerability management approach focuses on identifying and mitigating Common Vulnerabilities and Exposures [CVEs] at runtime. ACS and Gatekeeper are leveraged for policy enforcement, along with RBAC admission controls to limit access and reduce the attack surface. ACS is configured with a default policy set, augmented by a few custom policies tailored to specific operational needs. However, the setup currently lacks enforced security quality gates, meaning that identified vulnerabilities don’t halt deployments but are instead flagged for remediation. File integrity and compliance operators have been decommissioned, as Node level security and compliance is deemed a part of the ROSA managed service offering. === Debugging For troubleshooting, developers use pod logs on the OpenShift console as well as the external logging, and tracing tools provided. Support is managed through ServiceNow tickets, where engineers on call respond to issues during business hours. === End-User Training The HCP team has created training courses on the internal IT academy learning platform that focus on {cust} specific processes and technologies. The courses are adapted to different levels of know-how and range from read + click, to screen casts, and homework project-based trainings. === Conculsions Overall, the processes owned by the HCP team seem to be designed to make the developer experience as frictionless as possible. Based on the imformation shared, the time required for developers to go from zero to production seems significantly shorter than the average when compared to other Red Hat customers. The targeted, on-demand training offering is especially impressive and far exceeds what average customers offer their end-users. There is some potential for leaveraging templating and automation to provision managed resource which are currently manually created using terraform. Additionally, parts of the vulnerability management process could be shifted left to ensure security risks are identified earlier in the SSDLC.