//// Purpose ------- Information about how deployed systems can be observed. Examples -------- * Logging * Monitoring //// [id="observability_{context}"] = Observability == Infrastreucture Splunk Open Telemetry External monitoring team Alerting from prometheus/alertmanager to splunk on-call, notificationto email/teams chat. == User Workloads Pod logs forwarded to Splunk Enterprise. Currently using fluentd but migrating to vector. Internal application called "Lifecycle" consumes different data sources for vivibility on image versions... All application logs can be accessed by all developers == Audit Logs Requested through Red Hat support as per regular ROSA procedure. An additional internal audit tool intercepts the API for the production environment only and logs calls. == Cost management and chargeback Helvetia has developed a sophisticated in-house tool for cost management and chargeback that allocates costs based primarily on relative customer usage divided by actual costs. This tool uses a dedicated data pipeline that pulls and preprocesses data from a PostgreSQL database, providing ready-to-use, accurate cost allocation. High-level costs, such as those related to reserved instances and AWS enterprise savings plans, are managed by a company-wide AWS cloud solutions team. {cust}’s custom-built chargeback tool stands out for its advanced data architecture, offering tailored insights into cost distribution and optimization opportunities across the organization.