Notice history

Operational

Jan 2021

Feb 2021

Mar 2021

Jan 2021

Feb 2021

Mar 2021

Operational

Jan 2021

Feb 2021

Mar 2021

Jan 2021

Feb 2021

Mar 2021

Jan 2021

Feb 2021

Mar 2021

Jan 2021

Feb 2021

Mar 2021

Jan 2021

Feb 2021

Mar 2021

Jan 2021

Feb 2021

Mar 2021

Jan 2021

Feb 2021

Mar 2021

Jan 2021

Feb 2021

Mar 2021

Jan 2021

Feb 2021

Mar 2021

Jan 2021

Feb 2021

Mar 2021

Jan 2021

Feb 2021

Mar 2021

Jan 2021

Feb 2021

Mar 2021

Jan 2021

Feb 2021

Mar 2021

Jan 2021

Feb 2021

Mar 2021

Jan 2021

Feb 2021

Mar 2021

Jan 2021

Feb 2021

Mar 2021

Jan 2021

Feb 2021

Mar 2021

Operational

Jan 2021

Feb 2021

Mar 2021

Jan 2021

Feb 2021

Mar 2021

Jan 2021

Feb 2021

Mar 2021

Jan 2021

Feb 2021

Mar 2021

Operational

Jan 2021

Feb 2021

Mar 2021

Jan 2021

Feb 2021

Mar 2021

Mar 2021

Resolved
31. March 2021 at 11:15
Resolved
31. March 2021 at 11:15
We just resolved the issue!
Monitoring
31. March 2021 at 10:32
Monitoring
31. March 2021 at 10:32
We implemented a fix and currently monitoring the result.
Identified
31. March 2021 at 10:20
Identified
31. March 2021 at 10:20
We are continuing to work on a fix for this incident. In the meantime use https://meet-staging.puzzle.ch/

Resolved
30. March 2021 at 8:28
Resolved
30. March 2021 at 8:28
We just resolved the issue!
Monitoring
30. March 2021 at 7:25
Monitoring
30. March 2021 at 7:25
We implemented a fix and currently monitoring the result.

All Rancher VM's are rebooted and are stable again.
Identified
30. March 2021 at 7:16
Identified
30. March 2021 at 7:16
We are continuing to work on a fix for this incident. One of the Rancher vm's is/was unreachable. I will reboot and cleanup the Rancher vm's in sequence. Rancher will continue to have degraded performance, as during reboot, the floating IPs of rancher.puzzle.ch will switch and also the Rancher Pods will be rescheduled a few times.
Investigating
30. March 2021 at 7:3
Investigating
30. March 2021 at 7:3
Our Rancher Management Server seems unstable and from time to time unreachable. I'm starting investigating the issue.

This has no effect on the downstream Kubernetes Cluster and the application running on it. Only access via Rancher can be degraded or not available.

RocketChat Security fixes

Completed
29. March 2021 at 15:46
Completed
29. March 2021 at 15:46
Maintenance has completed successfully.
In progress
29. March 2021 at 15:45
In progress
29. March 2021 at 15:45
Maintenance is now in progress.
Planned
29. March 2021 at 15:45
Planned
29. March 2021 at 15:45
There are some high severity security issues that requires an RocketChat upgrade. No downtime of RocketChat is expected, you might only see a short reconnect in your RocketChat client.

Nextcloud migration

Completed
28. March 2021 at 19:57
Completed
28. March 2021 at 19:57
The Nextcloud migration is completed successfully and you can access Nextcloud as before on https://files.puzzle.ch Clients will automatically trigger a resync upon initial connection. You can find the latest documentation on Nextcloud usage at: https://docs.puzzle.ch/guides/latest/nextcloud-user-guide/index.html In case you have questions or notice an anomaly please contact helpdesk.
Planned
26. March 2021 at 23:21
Planned
26. March 2021 at 23:21
The migration of our Nextcloud server has begun and the server is currently in maintenance mode. In case you urgently need read-only access to a file please contact our helpdesk. We will provide further updates on the progress of the migration over the weekend.

Resolved
26. March 2021 at 7:54
Resolved
26. March 2021 at 7:54
All connections issues are resolved. We'll monitor the situation closely for the rest of the day and post updates if necessary.
Update
26. March 2021 at 7:11
Update
26. March 2021 at 7:11
More connectivity issues have been identified at the office. We are continuing to work on a fix for this incident.
Investigating
26. March 2021 at 6:18
Investigating
26. March 2021 at 6:18
The VPN for be3 isn't working at the moment. This issue could be related to the maintenance window last night. We are currently investigating this incident and someone is on their way to the office. Services only availably via VPN are not reachable at this point in time.

Feb 2021

Jenkins Prometheus Plugin downgrade

Completed
26. February 2021 at 11:6
Completed
26. February 2021 at 11:6
Maintenance has completed successfully.
Planned
26. February 2021 at 11:0
Planned
26. February 2021 at 11:0
We will downgrade the Prometheus Plugin. Jenkins will be restarted during this time.

/sys Wartungsfenster

Completed
25. February 2021 at 20:58
Completed
25. February 2021 at 20:58
All maintenances have been completed successfully and OpenProject has been upgraded to the newest major release. In case you notice any irregularities please contact our helpdesk at help (at) puzzle (dot) ch
In progress
25. February 2021 at 17:8
In progress
25. February 2021 at 17:8
Maintenance is now in progress.
Planned
25. February 2021 at 17:0
Planned
25. February 2021 at 17:0
As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/ or periods of degraded performance. Additionally we will update OpenProject to major version 11.

Resolved
24. February 2021 at 14:13
Resolved
24. February 2021 at 14:13
Stable again. Closing incident
Monitoring
24. February 2021 at 13:33
Monitoring
24. February 2021 at 13:33
The master node remains stable, we continue to monitor. All applications running on the k8s cluster are up and running
Identified
24. February 2021 at 13:19
Identified
24. February 2021 at 13:19
One of the master nodes was not ready anymore. This resulted in short outage of the Kubernetes control plane. This then (for yet unknwon reason) also affected some of the applicactions. I'm currently investigating what exacly happened. I rebooted the vm and it seems stable again.
Investigating
24. February 2021 at 13:8
Investigating
24. February 2021 at 13:8
Our Kubernetes Cluster has currently trouble which seems to affect some application. So far we see outages on SSO and Codi, possible more applications.

investigation started

Networkchange on Cloudscale VMs with multiple Network Interfaces

Completed
23. February 2021 at 18:7
Completed
23. February 2021 at 18:7
Maintenance has completed successfully.
Planned
23. February 2021 at 17:0
Planned
23. February 2021 at 17:0
We are planning for a scheduled maintenance during that time to add some more routes for Servers with multiple network interfaces.

Resolved
23. February 2021 at 15:8
Resolved
23. February 2021 at 15:8
Still stable. Closing this incident.
Monitoring
23. February 2021 at 14:39
Monitoring
23. February 2021 at 14:39
Rancher Management Server seems stable again. I rebooted all nodes and I'm monitoring the situation now.
Investigating
23. February 2021 at 14:18
Investigating
23. February 2021 at 14:18
We see some restarts of Rancher Podss which leads to an unstable Rancher Management Server and API Access to Downstream Cluster.

Downstream Cluster (Puzzle Kubernetes Production Cluster) is not affected and all applications running on it are working

Jan 2021

Monthly Maintenance Window /sys

Completed
28. January 2021 at 20:34
Completed
28. January 2021 at 20:34
Maintenance has completed successfully.
Planned
28. January 2021 at 17:0
Planned
28. January 2021 at 17:0
Regular monthly maintenance: 28.01.2021 18:00 - 23:00

Cloudscale Staging Network Changes

Completed
28. January 2021 at 20:34
Completed
28. January 2021 at 20:34
Maintenance has completed successfully. Thanks to cloudscale.ch for the support!
Planned
28. January 2021 at 13:0
Planned
28. January 2021 at 13:0
In preparation for a major networking change in the production environment, we will conduct a maintenance window to test all the necessary steps in the staging environment. These changes are necessary to enable the new OCP4 platform on cloudscale.ch. There is no impact on production servers. Announcement for the production change following.

Resolved
28. January 2021 at 18:29
Resolved
28. January 2021 at 18:29
The Incident was caused by a bug of OpenShift version 4.6.9
Identified
28. January 2021 at 11:6
Identified
28. January 2021 at 11:6
There is currently a issue with the "cert-utils-operator" to provide the certificate to the target route: "openshift.aws.puzzle.ch" which leads to a invalid certificate. Other applications on the platform should not be affected by this issue.

We are continuing to work on a fix for this incident.

Resolved
28. January 2021 at 7:47
Resolved
28. January 2021 at 7:47
Internet connectivity is stable again. cloudscale.ch will provide a detailed report about the incident. https://cloudscale-status.net/incident/163
Identified
27. January 2021 at 16:23
Identified
27. January 2021 at 16:23
cloudscale.ch is experiencing restricted internet connectivity. Access to applications can be limited. For further information please visit: https://cloudscale-status.net/incident/163

Taiga Update

Completed
22. January 2021 at 17:0
Completed
22. January 2021 at 17:0
Maintenance has completed successfully. Taiga is happily running on v5.5.9 now.
In progress
22. January 2021 at 16:30
In progress
22. January 2021 at 16:30
Maintenance is now in progress.
Planned
22. January 2021 at 16:15
Planned
22. January 2021 at 16:15
We are planning for a scheduled maintenance during that time. Taiga will be updated to the latest release.

Jan 2021 to Mar 2021

Puzzle ITC - Notice history

All systems operational

Notice history

Mar 2021

Feb 2021

Jan 2021