Duration: 52 minutes

Affected Components: OpenProject

Apr 16, 11:50:00 GMT+0
Identified - Due to <https://puzzle-status.ch/cmnd6w3ys00pur8lz76iv7y4u> we will update Openproject.

Apr 16, 10:58:14 GMT+0
Completed - Maintenance has completed successfully..

Duration: 1 hour and 2 minutes

Affected Components: OpenShift 4 Cluster LPG1 cloudscale.ch, OpenShift 4 Cluster cloudscale.ch, PuzzleTime

Apr 7, 14:59:03 GMT+0
Identified - We will only test PTime this time..

Apr 13, 16:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 13, 17:02:28 GMT+0
Completed - Maintenance has completed successfully. Disaster recovery tests for PTime CH and DE were successful..

Apr 13, 16:00:00 GMT+0
Identified - We are conducting disaster recovery tests for our test applications: Harbor, Rocketchat, PTime and PTime-DE. During these tests, the applications will not be available. This test occurs quarterly..

Duration: 42 minutes

Affected Components: Jenkins, SonarQube

Apr 7, 16:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 7, 16:41:37 GMT+0
Completed - Maintenance has completed successfully..

Apr 7, 16:00:00 GMT+0
Identified - As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and periods of degraded performance..

Duration: 2 hours and 47 minutes

Affected Components: CodiMD

Apr 1, 05:57:09 GMT+0
Investigating - ### **Summary** At approximately 08:00 AM, our monitoring systems detected a significant drop in **Markdown Pressure** within the CodiMD cluster. It appears the application layer has been compromised by a localized high-pressure system, causing "Variable-Width Character Drifting." ### **Current Observations** * **Packet Loss:** Several `.md` files have been reported hovering near the ceiling of the server rack. * **Infrastructure Impact:** The "Servicegedanken" module is currently unresponsive. We suspect it may be looking for a broom or a windbreaker. * **User Impact:** Users may experience a total lack of productivity, though this is currently indistinguishable from a standard Wednesday morning. ### **Root Cause Analysis (In Progress)** Initial logs suggest that someone forgot to close the `target="_blank"` windows in the frontend code, allowing a draft to enter the database. Our DevOps team is currently attempting to deploy **Digital Paperweights** to secure the remaining notes. ### **Next Steps** 1. Consulting with the local Meteorological Office for a patch. 2. Waiting for the "Servicegedanken" to manifest in physical form. 3. Monitoring the Beaufort scale for any increase in syntax errors. ---.

Apr 1, 08:44:34 GMT+0
Resolved - We are pleased to announce that the **Variable-Width Character Drifting** incident has been fully resolved. All `.md` files have been successfully coaxed down from the ceiling of the server rack using a combination of **Digital Paperweights** and a strongly-worded comment in the codebase. \## Root Cause (Final) After extensive consultation with the **Meteorological Office**, it was confirmed that a rogue `target="_blank"` window left open over the long weekend had created a **sustained indoor draft of approximately 4.2 on the Beaufort Scale**. This caused a localized low-pressure zone in the CodiMD cluster, which — as any senior infrastructure engineer will tell you — is a classic **Spring Syntax Migration Event**. \## Resolution Steps Taken \- All open `target="_blank"` windows have been closed. A company-wide policy on **Responsible Window Management** is being drafted (in CodiMD, naturally). \- The **Servicegedanken** module has returned from the supply closet with a broom and is fully operational. \- Digital Paperweights have been deployed cluster-wide and are performing admirably. \- Markdown Pressure has been restored to **nominal seasonal levels**. \## User Impact Productivity has returned to its pre-incident baseline. This remains **statistically indistinguishable from a standard Wednesday**, but we consider that a success. \## Lessons Learned We have learned nothing. However, we \*have\* updated our runbook to include a section on **Meteorological Incident Response**, which we feel reflects our commitment to continuous improvement. \--- \*We apologize for any inconvenience caused. As always, if you experience any further atmospheric anomalies, please open a ticket — or a window, but not `target="_blank"`.\* 🌬️ **Stay grounded. Keep your Markdown pressure stable.**.

Duration: 35 minutes

Affected Components: PuzzleTime

Mar 30, 12:00:00 GMT+0
Identified - This release includes several new features, improvements, and bug fixes. Check out the changelog after the rollout: PuzzleTime > click on version number in the upper-left corner next to the PuzzleTime-logo..

Mar 30, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 30, 12:35:13 GMT+0
Completed - PuzzleTime v2.16 was successfully deployed!.

Duration: 9 hours and 6 minutes

Affected Components: Network Office-Bern

Mar 27, 12:45:11 GMT+0
Monitoring - We implemented a fix and are seeing stable access using IPv6 from multiple client nets. We'll continue monitoring the situation..

Mar 27, 07:00:00 GMT+0
Investigating - We are currently investigating an Issue with IPv6 deployment in our BE3 Office network. We are observing intermittent issues regarding IPv6 reachability of internal services as well as public services. In case you are affected you can locally disable IPv6 in your network stack..

Mar 27, 14:48:13 GMT+0
Identified - We are noticing more degraded connections again. .

Mar 27, 16:05:30 GMT+0
Resolved - We have reached a stable setup again and we are seeing connections working normally again..

Affected Components: OpenProject

Mar 26, 19:00:00 GMT+0
Identified - We have observed a **significant drop in performance** in the Backlogs module after updating to OpenProject 17.2.2. The following issues are particularly noticeable: 1. **Slow loading times:** * Initial loading of backlogs takes **considerably longer** than in previous versions. * Opening individual features/stories within the backlog is **delayed** (2–3 seconds per interaction). 2. **Undesired GUI behavior:** * Certain interactions (e.g., drag-and-drop sorting) exhibit **unexpected delays or visual glitches**, even though the backlog content itself remains unaffected. **Context:** * The issues emerged after the **17.2.0 update**, which introduced the **Primer Design System and Turbo/StimulusJS** for dynamic updates in the Backlogs module. * The problems are **reproducible** with large backlogs and persist in 17.2.2. **Workaround:** * Disabling _"Show versions folded"_ in user settings **mitigates the drag-and-drop issue**, but is not a sustainable solution due to usability concerns. We are currently investigating potential solutions and will keep you updated on our progress..

Apr 10, 06:24:57 GMT+0
Monitoring - OpenProject is actively working on a bug fix for the issue we’ve been experiencing. Progress has been made, and we can expect a patch in the near future. If you are experiencing performance issues with OpenProject due to large backlogs, we recommend using "Arbeitspakete" (Work Packages) view and disabling the option **"Show versions folded"** to mitigate the issue..

Duration: 3 hours and 19 minutes

Affected Components: CodiMD, PuzzleSkills, Artifactory, Webmail, Vault, Zammad, Jitsi, Wiki, Loki, GitLab, Odoo, PuzzleTime, OpenProject, Jenkins, LDAP, Netbox, Puzzle News Site, Rocket.Chat, CardDAV, Puzzle SSO (Keycloak), Youtrack, Network Office-Bern, Network Office-Zürich, Whiteboard, Taiga, SonarQube, File storage & sharing, Grafana, Mail, PostgreSQL, Argo CD

Mar 26, 17:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 26, 20:19:26 GMT+0
Completed - Maintenance has completed successfully..

Mar 26, 17:00:00 GMT+0
Identified - As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/ or periods of degraded performance..

Duration: 3 hours and 45 minutes

Affected Components: CodiMD, PuzzleSkills, Odoo, , PuzzleTime, Netbox, Rocket.Chat, OKR, Puzzle SSO (Keycloak), Youtrack, Harbor, OpenShift 4 Cluster cloudscale.ch, Taiga, , Jenkins, SonarQube, Grafana, Puzzle Docs, Argo CD, , Collaboration → Project Management → Dev Tools →

Mar 23, 17:00:00 GMT+0
Identified - As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and periods of degraded performance..

Mar 23, 17:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 23, 20:45:03 GMT+0
Completed - Maintenance has completed successfully..

Duration: 4 minutes

Affected Components: Wiki

Mar 18, 06:00:00 GMT+0
Identified - We will be performing an update to our Foswiki instance outside the regular maintenance window. The update will begin at 07:00, during which Foswiki will be unavailable. Expected downtime is between 15 and 45 minutes. Thank you for your understanding..

Mar 18, 05:56:07 GMT+0
Completed - The scheduled update has been canceled and will be rescheduled for another time..

Duration: 20 minutes

Affected Components: Jenkins, SonarQube

Mar 3, 17:00:00 GMT+0
Identified - As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and periods of degraded performance..

Mar 3, 17:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 3, 17:20:30 GMT+0
Completed - Maintenance has completed successfully..

Duration: 2 hours and 52 minutes

Affected Components: CodiMD, Webmail, OpenProject, PuzzleSkills, Artifactory, Jenkins, SonarQube, Odoo, Zammad, Jitsi, GitLab, Wiki, PuzzleTime, Loki, LDAP, Netbox, Puzzle News Site, Rocket.Chat, CardDAV, Vault, Puzzle SSO (Keycloak), Youtrack, Network Office-Bern, Network Office-Zürich, Taiga, Whiteboard, File storage & sharing, Grafana, Mail, PostgreSQL, Argo CD

Feb 26, 17:00:01 GMT+0
Identified - Maintenance is now in progress.

Feb 26, 19:52:12 GMT+0
Completed - Maintenance has completed successfully..

Feb 26, 17:00:00 GMT+0
Identified - As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/ or periods of degraded performance..

Duration: 2 hours and 22 minutes

Affected Components: CodiMD, PuzzleSkills, Odoo, , OpenShift 4 Cluster LPG1 cloudscale.ch, PuzzleTime, SonarQube, Netbox, Rocket.Chat, OKR, Puzzle SSO (Keycloak), Youtrack, OpenShift 4 Cluster cloudscale.ch, Taiga, , Harbor, Jenkins, Grafana, Puzzle Docs, Argo CD, , Collaboration → Project Management → Dev Tools →

Feb 23, 17:00:00 GMT+0
Identified - As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and periods of degraded performance..

Feb 23, 17:00:01 GMT+0
Identified - Maintenance is now in progress.

Feb 23, 19:21:55 GMT+0
Completed - Maintenance has completed successfully..

Duration: 1 hour and 15 minutes

Affected Components: Rocket.Chat

Feb 20, 11:45:03 GMT+0
Identified - Due to rash actions on my side I accidentally deleted RocketChat Avatar Pictures and some File Uploads from our S3 Bucket. What happened: The Intention was to delete a connection profile in my S3 Client (to prevent things exactly like this issue here happening) but accidentally started deleting the Bucket (or part of it) instead of the connection because I haven't read the button correctly. Current State: \- All avatars are already restored. \- We are currently investigating some missing uploads, a very small percentage of upload files were also deleted and we are in the process to restore them from our backup instance..

Feb 20, 12:57:59 GMT+0
Monitoring - We restored the missing uploaded files that were also deleted. Our last fallback data sync was around 05:40 in the morning. There is a time window between 05:40 and around 10:00 (I don't have the exact time anymore) where newly uploaded files in your chats are gone. Also if you changed your avatar in this time frame it might be reverted to the old one. So if you are missing a file in a chat, you have to resend it to your peer. My apologies, have a nice rest of the day and enjoy your weekend.

Feb 20, 13:00:30 GMT+0
Resolved - This incident has been resolved..

Duration: 48 minutes

Affected Components: OpenProject

Feb 12, 17:00:00 GMT+0
Identified - We are planning a major database upgrade for our OpenProject instance..

Feb 12, 17:00:01 GMT+0
Identified - Maintenance is now in progress.

Feb 12, 17:48:26 GMT+0
Completed - The database upgrade was completed successfully and openproject is available again to all users..

Duration: 2 hours and 22 minutes

Affected Components: Jenkins, SonarQube

Feb 3, 17:00:00 GMT+0
Identified - As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and periods of degraded performance..

Feb 3, 17:00:01 GMT+0
Identified - Maintenance is now in progress.

Feb 3, 19:21:57 GMT+0
Completed - Maintenance has completed successfully..

Duration: 57 minutes

Affected Components: Puzzle SSO (Keycloak)

Jan 30, 07:13:56 GMT+0
Investigating - There are currently problems (certificate Issues, externalSecret, etc.) with the Openshift Cluster in RMA and therefore Apps might not work correctly. We are investigating the issue.

Jan 30, 08:10:58 GMT+0
Resolved - This incident has been resolved. A missing DNS entry led to the issue. We suspect a change of logic with the DNS provider as the cause..

Duration: 4 hours and 45 minutes

Affected Components: CodiMD, Webmail, Puzzle SSO (Keycloak), PuzzleSkills, SonarQube, Odoo, Taiga, Jenkins, Zammad, Jitsi, Artifactory, GitLab, Wiki, PuzzleTime, Loki, LDAP, Netbox, Puzzle News Site, Rocket.Chat, CardDAV, OpenProject, Vault, Network Office-Bern, Network Office-Zürich, Youtrack, Whiteboard, File storage & sharing, Grafana, Mail, PostgreSQL, Argo CD

Jan 29, 17:00:01 GMT+0
Identified - Maintenance is now in progress.

Jan 29, 21:44:51 GMT+0
Completed - Maintenance has completed successfully..

Jan 29, 17:00:00 GMT+0
Identified - As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/ or periods of degraded performance..

Duration: 18 minutes

Affected Components: Odoo

Jan 29, 11:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time..

Jan 29, 11:00:01 GMT+0
Identified - Maintenance is now in progress.

Jan 29, 11:17:45 GMT+0
Completed - Maintenance has completed successfully..

Duration: 39 minutes

Affected Components: Odoo

Jan 27, 05:57:15 GMT+0
Investigating - We are currently investigating this incident..

Jan 27, 06:06:09 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Jan 27, 06:36:23 GMT+0
Resolved - This incident has been resolved..

Duration: 4 hours

Affected Components: CodiMD, Taiga, Jenkins, , Odoo, PuzzleTime, Netbox, Rocket.Chat, OKR, Youtrack, , Puzzle SSO (Keycloak), PuzzleSkills, SonarQube, Harbor, OpenShift 4 Cluster cloudscale.ch, Grafana, Puzzle Docs, Argo CD, , Collaboration → Project Management → Dev Tools →

Jan 26, 17:00:00 GMT+0
Identified - As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and periods of degraded performance..

Jan 26, 17:00:01 GMT+0
Identified - Maintenance is now in progress.

Jan 26, 21:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 24 minutes

Jan 16, 09:51:51 GMT+0
Investigating - We are currently investigating this incident..

Jan 16, 10:16:00 GMT+0
Resolved - This incident has been resolved..

Duration: 3 hours

Affected Components: OpenShift 4 Cluster LPG1 cloudscale.ch, PuzzleTime, Rocket.Chat, Harbor, OpenShift 4 Cluster cloudscale.ch

Jan 12, 17:00:00 GMT+0
Identified - We are conducting disaster recovery tests for our test applications: Harbor, Rocketchat, PTime and PTime-DE. During these tests, the applications will not be available. This test occurs quarterly. This is the first test run, we will only test Harbor..

Jan 12, 17:00:01 GMT+0
Identified - Maintenance is now in progress.

Jan 12, 20:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 15 minutes

Affected Components: Grafana

Jan 9, 11:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. No user impact expected..

Jan 9, 11:00:01 GMT+0
Identified - Maintenance is now in progress.

Jan 9, 11:15:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 37 minutes

Affected Components: Jenkins, SonarQube

Jan 6, 17:00:00 GMT+0
Identified - As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and periods of degraded performance..

Jan 6, 17:00:01 GMT+0
Identified - Maintenance is now in progress.

Jan 6, 17:37:04 GMT+0
Completed - Maintenance has completed successfully..

Duration: 2 hours and 10 minutes

Affected Components: OpenShift 4 Cluster cloudscale.ch

Dec 23, 12:46:02 GMT+0
Resolved - This incident has been resolved..

Dec 23, 10:35:38 GMT+0
Investigating - We are currently investigating this incident..

Dec 23, 10:52:35 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Duration: 28 minutes

Affected Components: , OpenShift 4 Cluster LPG1 cloudscale.ch, OpenShift 4 Cluster cloudscale.ch, Container Platforms →

Dec 23, 09:30:10 GMT+0
Investigating - We are currently experiencing issues with one of the openshift nodes. This may cause short outages of applications running on the mentioned node. We are sorry for the inconvenience and will keep you updated..

Dec 23, 09:34:16 GMT+0
Identified - We are continuing to work on a fix for this incident..

Dec 23, 09:58:00 GMT+0
Resolved - This incident has been resolved..

Duration: 2 hours and 35 minutes

Affected Components: , CodiMD, Taiga, Harbor, Puzzle SSO (Keycloak), , Odoo, PuzzleSkills, SonarQube, OpenShift 4 Cluster cloudscale.ch, Netbox, Jenkins, PuzzleTime, Rocket.Chat, OKR, Youtrack, Grafana, Puzzle Docs, Argo CD, , Project Management → Collaboration → Dev Tools →

Dec 22, 17:00:00 GMT+0
Identified - As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and periods of degraded performance..

Dec 22, 17:00:01 GMT+0
Identified - Maintenance is now in progress.

Dec 22, 19:35:28 GMT+0
Completed - Maintenance has completed successfully..

Duration: 8 days, 18 hours and 47 minutes

Affected Components: CodiMD, Network Office-Bern, Taiga, Puzzle SSO (Keycloak), Odoo, PuzzleSkills, SonarQube, Network Office-Zürich, Zammad, Webmail, Jitsi, Artifactory, GitLab, Wiki, Loki, LDAP, , Netbox, Jenkins, Puzzle News Site, PuzzleTime, Rocket.Chat, CardDAV, OpenProject, Vault, Youtrack, Whiteboard, File storage & sharing, Grafana, Mail, PostgreSQL, Argo CD, Nextcloud →

Dec 18, 17:00:00 GMT+0
Identified - As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/ or periods of degraded performance..

Dec 18, 17:00:01 GMT+0
Identified - Maintenance is now in progress.

Dec 18, 20:30:46 GMT+0
Completed - Maintenance has completed successfully..

Duration: 1 minute

Affected Components: PostgreSQL

Dec 3, 17:00:00 GMT+0
Identified - We will be increasing the connection limit on [postgres04.cloud.puzzle.ch](http://postgres04.cloud.puzzle.ch), which hosts databases for various applications. This change requires a restart of the PostgreSQL service. During the restart, PostgreSQL will be temporarily unavailable, resulting in a short service interruption of only a few seconds. We appreciate your understanding..

Dec 3, 17:00:01 GMT+0
Identified - Maintenance is now in progress.

Dec 3, 17:01:22 GMT+0
Completed - Maintenance has completed successfully..

Duration: 2 hours and 24 minutes

Affected Components: SonarQube, Jenkins

Dec 2, 17:00:00 GMT+0
Identified - As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and periods of degraded performance..

Dec 2, 17:00:01 GMT+0
Identified - Maintenance is now in progress.

Dec 2, 19:24:05 GMT+0
Completed - Maintenance has completed successfully..

Duration: 5 hours and 41 minutes

Affected Components: Artifactory

Nov 28, 15:47:39 GMT+0
Resolved - This incident has been resolved..

Nov 28, 10:07:00 GMT+0
Investigating - After our weekly maintenance window from yesterday, it's currently not possible to login to the WebUI of <https://artifactory.puzzle.ch> We are currently investigating this incident..

Duration: 5 hours and 26 minutes

Affected Components: Netbox, CodiMD, PuzzleSkills, Network Office-Bern, Taiga, SonarQube, Jenkins, Zammad, Webmail, Jitsi, Artifactory, GitLab, Wiki, Puzzle SSO (Keycloak), Loki, LDAP, Puzzle Website, Puzzle News Site, , PuzzleTime, Rocket.Chat, CardDAV, OpenProject, Network Office-Zürich, Odoo, Vault, Youtrack, Whiteboard, File storage & sharing, Grafana, Mail, PostgreSQL, Argo CD, Nextcloud →

Nov 27, 17:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 27, 22:26:25 GMT+0
Completed - Maintenance has completed successfully..

Nov 27, 17:00:00 GMT+0
Identified - As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/ or periods of degraded performance..

Duration: 2 hours and 30 minutes

Affected Components: , Netbox, CodiMD, Taiga, Harbor, SonarQube, Jenkins, OpenShift 4 Cluster LPG1 cloudscale.ch, PuzzleSkills, Puzzle SSO (Keycloak), OpenShift 4 Cluster cloudscale.ch, PuzzleTime, Rocket.Chat, OKR, , Odoo, Youtrack, Grafana, Puzzle Docs, Argo CD, , Project Management → Collaboration → Dev Tools →

Nov 24, 17:00:00 GMT+0
Identified - As usual, we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and periods of degraded performance..

Nov 24, 17:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 24, 19:30:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 1 hour

Affected Components: PuzzleTime

Nov 18, 13:00:00 GMT+0
Identified - We are planning to release hotfixes for multiple issues..

Nov 18, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 18, 14:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 1 hour and 2 minutes

Affected Components: PuzzleTime

Nov 18, 08:30:00 GMT+0
Identified - The Puzzletime production instance is running on the Openshift node 14, which has issues. This leads to various 404 errors..

Nov 18, 09:32:00 GMT+0
Resolved - The Puzzletime production instance was moved off of the faulty node and is now fully functional again. Therefore, this incident is now resolved..

Duration: 1 hour and 40 minutes

Affected Components: OpenShift 4 Cluster cloudscale.ch

Nov 18, 07:49:37 GMT+0
Investigating - Our OpenShift Node [ocp4worker14.cloud.puzzle.ch](http://ocp4worker14.cloud.puzzle.ch) is currently out of space and therefore is not really usable..

Nov 18, 07:51:28 GMT+0
Investigating - The node is cordoned and no new Pods will be scheduled on the Node. We are currently trying to reschedule Pods running on the Node to mitigate issues with running pods on the Node..

Nov 18, 08:53:31 GMT+0
Monitoring - By rescheduling some of the pods and remote the unused container images, the node has again space on the disk and therefore the remaining pods should be fine again. None the less, we started mitigation of the problem in general, our Nodes needs bigger root disks to serve all the pods with its container images..

Nov 18, 09:29:30 GMT+0
Resolved - This incident has been resolved..

Duration: 4 days, 9 hours and 14 minutes

Affected Components: Whiteboard

Nov 13, 04:06:00 GMT+0
Identified - After a security patch, Nextcloud whiteboards cannot be saved. We discourage the use of the application while we implement a fix..

Nov 17, 11:48:53 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Nov 17, 13:20:24 GMT+0
Resolved - This incident has been resolved..

Duration: 2 hours and 22 minutes

Affected Components: Google Meet Hardware

Aug 18, 06:34:36 GMT+0
Identified - We are investigating possible fixes. The Hardware is unavailable in the meantime..

Aug 18, 08:56:44 GMT+0
Resolved - This incident has been resolved. the power cord has been plugged in again..

Duration: 2 hours and 51 minutes

Affected Components: OpenShift 4 Cluster cloudscale.ch, Jenkins, GitLab, SonarQube

Jul 22, 14:28:08 GMT+0
Identified - All CICD applications have accidentally been deleted, we are working on restoring the affected applications..

Jul 22, 15:01:30 GMT+0
Monitoring - Most services are up an running again, for Jenkins, Sonarqube and Deptrack we need to restore the backup..

Jul 22, 17:19:17 GMT+0
Resolved - This incident has been resolved. All services have been restored..

Duration: 2 hours

Affected Components: Network Office-Bern

Jul 17, 07:19:10 GMT+0
Investigating - IPv6 connectivity at Belpstrasse is currently degraded. We are actively investigating the issue. More updates will follow as we learn more..

Jul 17, 09:19:33 GMT+0
Resolved - This incident has been resolved..

Duration: 2 hours and 35 minutes

Affected Components: Webmail, Loki, Puzzle Website, Argo CD, CodiMD, Harbor, Puzzle Docs, Netbox, Taiga, CardDAV, Youtrack, Jenkins, GitLab, Odoo, Wiki, Puzzle SSO (Keycloak), Jitsi, Artifactory, Zammad, OpenProject, Puzzle News Site, , Rocket.Chat, PuzzleSkills, , PuzzleTime, Mail, Vault, LDAP, Grafana, SonarQube, OKR, PostgreSQL, Puzzle Services → Nextcloud →

Jul 11, 13:43:51 GMT+0
Investigating - We are currently experiencing unexpected outages of different cloud services. We are investigating this incident..

Jul 11, 16:17:33 GMT+0
Monitoring - We have encountered an abnormal number of connections to our LDAP Server resulting in many services not being able to create new LDAP connections. We have implemented a fix and are currently monitoring the result. .

Jul 11, 16:18:25 GMT+0
Resolved - This incident has been resolved..

Duration: 3 hours and 17 minutes

Affected Components: LDAP

Jul 1, 15:46:19 GMT+0
Identified - The LDAP Puzzle OU has been deleted by accident and we are currently recovering it. Until the data is recovered, logging on to puzzle services will not be possible. We're sorry for the inconvenciences caused by this..

Jul 1, 19:03:15 GMT+0
Resolved - This incident has been resolved..

Duration: 2 hours and 35 minutes

Affected Components: Network Office-Bern

Jun 2, 06:30:00 GMT+0
Identified - We are continuing to work on a fix for this incident..

Jun 2, 07:39:37 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Jun 2, 09:05:05 GMT+0
Resolved - This incident has been resolved..

Duration: 18 hours and 20 minutes

Affected Components: Google Meet Hardware

May 19, 15:15:36 GMT+0
Identified - After an unsuccessful Hardware upgrade, the Google meet Hardware in /git is not able to connect to Google. As a result, Google Meet (and the built-in Screen sharing) is unavailable in /git for the time being. Google meet service in the other meeting rooms is unaffected and fully available..

May 20, 09:35:57 GMT+0
Resolved - This incident has been resolved. Google Meet hardware is functional in all meeting rooms again..

Duration: 1 hour and 30 minutes

Affected Components: OpenShift 4 Cluster cloudscale.ch

Apr 28, 14:28:17 GMT+0
Investigating - We are currently investigating this incident..

Apr 28, 15:08:21 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Apr 28, 15:58:33 GMT+0
Resolved - This incident has been resolved..

Affected Components: OpenShift 4 Cluster cloudscale.ch

Mar 18, 20:20:56 GMT+0
Resolved - This incident has been resolved..

Mar 18, 19:50:18 GMT+0
Resolved - This incident has been resolved..

Duration: 2 hours and 1 minute

Jan 28, 08:03:13 GMT+0
Investigating - Currently you are not able to login into our Quay Container Registry using Puzzle SSO. LDAP still works. We are investigating the issue..

Jan 28, 10:01:31 GMT+0
Identified - A solution to fix this is being rolled out soon. Unfortunately a short interruption of quay is necessary to do the fix..

Jan 28, 10:04:01 GMT+0
Resolved - I config change did its magic. Login via SSO is possible again.

Duration: 4 days, 17 hours and 10 minutes

Affected Components: CardDAV

Jan 24, 15:33:38 GMT+0
Identified - We are currently not investigating this incident. We put everything we have into the Tonne and build it up from scratch again. Stay tuned!.

Jan 29, 08:43:56 GMT+0
Resolved - This incident has been resolved..

Duration: 2 hours and 26 minutes

Affected Components: Network Office-Zürich

Jan 23, 14:01:53 GMT+0
Identified - Update from Init 7 regarding the Issue: <https://as13030.net/status/?ticket=42258104>.

Jan 23, 16:28:14 GMT+0
Resolved - This incident has been resolved / See: <https://as13030.net/status/?ticket=42258104>.