Notice history

Operational

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Operational

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Operational

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Operational

Sep 2021

Oct 2021

Nov 2021

Sep 2021

Oct 2021

Nov 2021

Nov 2021

Update IP Failover Setup on Kubernetes Cluster

Completed
30. November 2021 at 16:41
Completed
30. November 2021 at 16:41
Maintenance has completed successfully.
Planned
30. November 2021 at 16:30
Planned
30. November 2021 at 16:30
Our IP failover setup (based on keepalived) need a new API token to update the IP assignment at our cloud provider cloudscale. During redeployment of the keepalived Pods, our public ingress IPv4 and IPv6 addresses (e.g. behind chat.puzzle.ch) will shortly be not available and therefore applications running on the Kubernetes cluster will have a short outage. I will update them in sequence to keep the impact minimal
In progress
30. November 2021 at 16:29
In progress
30. November 2021 at 16:29
Maintenance is now in progress.

Migration Artifactory Centos > Rocky

Completed
30. November 2021 at 16:58
Completed
30. November 2021 at 16:58
Maintenance has completed successfully.
In progress
30. November 2021 at 16:31
In progress
30. November 2021 at 16:31
Maintenance is now in progress.
Planned
30. November 2021 at 16:30
Planned
30. November 2021 at 16:30
We will migrate Artifactory from Centos to Rocky, during this time Artifactory will be offline.

/sys Wartungsfenster

Completed
25. November 2021 at 21:26
Completed
25. November 2021 at 21:26
Maintenance has completed successfully.
In progress
25. November 2021 at 17:0
In progress
25. November 2021 at 17:0
Maintenance is now in progress.
Planned
25. November 2021 at 17:0
Planned
25. November 2021 at 17:0
As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/ or periods of degraded performance.

MariaDB01 Rocky Migration (affects Puzzle Website & HAST Services)

Completed
18. November 2021 at 17:0
Completed
18. November 2021 at 17:0
We will migrate one of our mariadb instances to RockyLinux which will affect the puzzle website (http://www.puzzle.ch) and some HAST Services. There will be a scheduled downtime for the Website & the HAST Services during the migration of the database server.

Please be aware that you can't use the website (https://www.puzzle.ch) or the HAST Services on Thursday starting from 18:00 until the end of this maintenance.

Resolved
16. November 2021 at 7:12
Resolved
16. November 2021 at 7:12
We just resolved the issue!
Monitoring
16. November 2021 at 6:31
Monitoring
16. November 2021 at 6:31
We restored connectivity and are monitoring the situation. The issue was most likely related to a planned maintenance of our ISP: https://as13030.net/status/?ticket=4246996.
Investigating
16. November 2021 at 6:24
Investigating
16. November 2021 at 6:24
We are experiencing some issues with our VPN connections to our cloud providers (cloudscale.ch and AWS). As a result, internal connections are not possible at the moment.

Oct 2021

Resolved
22. October 2021 at 10:59
Resolved
22. October 2021 at 10:59
Traffic path to region Bern was changed. Stable since 2021-10-22 9:50 CEST.
Identified
22. October 2021 at 6:18
Identified
22. October 2021 at 6:18
Init7 is currently experiencing interruptions on multiple Fiber7 POPs. Init7 status info: https://as13030.net/status/?ticket=OPS-160

/sys Wartungsfenster

Completed
21. October 2021 at 21:58
Completed
21. October 2021 at 21:58
We have completed our monthly maintenance. If you encounter any problems or errors please contact our helpdesk.
In progress
21. October 2021 at 16:1
In progress
21. October 2021 at 16:1
Maintenance is now in progress.
Planned
21. October 2021 at 16:0
Planned
21. October 2021 at 16:0
We will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/ or periods of degraded performance.

ATTENTION: this maintenance window is NOT occurring on the 4th Thursday of the month but on the 3rd. This is due to a company wide event the following week. Please adjust blackout periods and monitoring systems to this change of dates. We apologize for the inconvenience caused by this.

PuzzleTime (CH+DE) OpenShift 4 Migration

Completed
22. October 2021 at 8:35
Completed
22. October 2021 at 8:35
Bei PTime DE gab es nach der Umstellung noch Zertifikats- und Caching-Probleme. Diese sind nun behoben und der Umzug von PTime DE somit abgeschlossen.
Update
21. October 2021 at 15:3
In progress
21. October 2021 at 15:3
Der Umzug der DE Umgebung ist nach der DNS Umstellung abgeschlossen.
Update
21. October 2021 at 14:41
In progress
21. October 2021 at 14:41
Die DNS Umstellung der CH Umgebung ist durch und https://time.puzzle.ch ist wieder online.
Update
21. October 2021 at 14:35
In progress
21. October 2021 at 14:35
Der Umzug der CH Umgebung ist nach der DNS Umstellung abgeschlossen.

Wir beginnen nun mit dem Umzug der DE Umgebung. Diese wird bis zum Ende des Umzugs nicht erreichbar sein.
In progress
21. October 2021 at 14:10
In progress
21. October 2021 at 14:10
Wir starten mit dem Umzug der PTime CH Umgebung. Diese wird bis nach der Umstellung nicht erreichbar sein.
Planned
21. October 2021 at 14:0
Planned
21. October 2021 at 14:0
Nach der gestrigen erfolgreichen Migration auf S3 Storage werden wir PuzzleTime auf OpenShift 4 migrieren.

Resolved
21. October 2021 at 11:30
Resolved
21. October 2021 at 11:30
Taiga is back online Again, apologies for the inconvenience
Identified
21. October 2021 at 11:27
Identified
21. October 2021 at 11:27
One component of Taiga is currently not working and therefore Taiga is not up and running.
We are currently investigating and fixing this incident . Apologies for the inconvenience

Kubernetes & Rancher Node Maintenance

Completed
20. October 2021 at 17:42
Completed
20. October 2021 at 17:42
Maintenance has completed successfully.
Planned
20. October 2021 at 16:0
Planned
20. October 2021 at 16:0
All Rancher and Kubernetes nodes will get an OS update. We are planning for a scheduled maintenance during that time. All applications running on the Kubernetes cluster will be rescheduled multiple times during the maintenance window, therefore we expect some outages of the applications.
In progress
20. October 2021 at 15:51
In progress
20. October 2021 at 15:51
Maintenance is now in progress.

Sep 2021

OpenShift 4 Cluster Update

Completed
30. September 2021 at 19:30
Completed
30. September 2021 at 19:30
Maintenance has completed successfully.
In progress
30. September 2021 at 15:0
In progress
30. September 2021 at 15:0
Maintenance is now in progress.
Planned
30. September 2021 at 15:0
Planned
30. September 2021 at 15:0
We are planning for a scheduled maintenance during that time. The OpenShift 4 clusters on AWS and cloudscale are going to be updated to the latest version.

Resolved
15. October 2021 at 12:59
Resolved
15. October 2021 at 12:59
We just resolved the issue!
Monitoring
30. September 2021 at 23:39
Monitoring
30. September 2021 at 23:39
We implemented a fix and currently monitoring the result.
Identified
30. September 2021 at 23:14
Identified
30. September 2021 at 23:14
This is fixed, a post mortem is available in #devruby-ops
Investigating
30. September 2021 at 14:27
Investigating
30. September 2021 at 14:27
We are currently investigating this incident. Seems to be an issue regarding expired mTLS certificates.

OpenShift 4 Cluster cloudscale.ch Master Reboot

Completed
30. September 2021 at 11:47
Completed
30. September 2021 at 11:47
Maintenance has completed successfully.
In progress
30. September 2021 at 11:32
In progress
30. September 2021 at 11:32
Maintenance is now in progress.
Planned
30. September 2021 at 11:30
Planned
30. September 2021 at 11:30
The OpenShift 4 Masters will be drained and rebooted sequentially. No Service interruption is expected.

Resolved
30. September 2021 at 11:18
Resolved
30. September 2021 at 11:18
CodiMD pod was not responsive and hat to be restarted.

Mailserver maintenance

Completed
29. September 2021 at 16:37
Completed
29. September 2021 at 16:37
Maintenance has completed successfully.
Planned
29. September 2021 at 15:30
Planned
29. September 2021 at 15:30
We will implement SRS (Sender Rewriting Scheme) to minimize the effects of email forwards to gmail and other providers. During the maintenance you might experience some delays when receiving emails.
In progress
29. September 2021 at 15:29
In progress
29. September 2021 at 15:29
Maintenance is now in progress.

Sep 2021 to Nov 2021

Puzzle ITC - Notice history

All systems operational

Notice history

Nov 2021

Oct 2021

Sep 2021