Notice history

Operational

Jul 2021

Aug 2021

Sep 2021

Jul 2021

Aug 2021

Sep 2021

Operational

Jul 2021

Aug 2021

Sep 2021

Jul 2021

Aug 2021

Sep 2021

Jul 2021

Aug 2021

Sep 2021

Jul 2021

Aug 2021

Sep 2021

Jul 2021

Aug 2021

Sep 2021

Jul 2021

Aug 2021

Sep 2021

Jul 2021

Aug 2021

Sep 2021

Jul 2021

Aug 2021

Sep 2021

Jul 2021

Aug 2021

Sep 2021

Jul 2021

Aug 2021

Sep 2021

Jul 2021

Aug 2021

Sep 2021

Jul 2021

Aug 2021

Sep 2021

Jul 2021

Aug 2021

Sep 2021

Jul 2021

Aug 2021

Sep 2021

Jul 2021

Aug 2021

Sep 2021

Jul 2021

Aug 2021

Sep 2021

Jul 2021

Aug 2021

Sep 2021

Operational

Jul 2021

Aug 2021

Sep 2021

Jul 2021

Aug 2021

Sep 2021

Jul 2021

Aug 2021

Sep 2021

Jul 2021

Aug 2021

Sep 2021

Operational

Jul 2021

Aug 2021

Sep 2021

Jul 2021

Aug 2021

Sep 2021

Sep 2021

OpenShift 4 Cluster Update

Completed
30. September 2021 at 19:30
Completed
30. September 2021 at 19:30
Maintenance has completed successfully.
In progress
30. September 2021 at 15:0
In progress
30. September 2021 at 15:0
Maintenance is now in progress.
Planned
30. September 2021 at 15:0
Planned
30. September 2021 at 15:0
We are planning for a scheduled maintenance during that time. The OpenShift 4 clusters on AWS and cloudscale are going to be updated to the latest version.

Resolved
15. October 2021 at 12:59
Resolved
15. October 2021 at 12:59
We just resolved the issue!
Monitoring
30. September 2021 at 23:39
Monitoring
30. September 2021 at 23:39
We implemented a fix and currently monitoring the result.
Identified
30. September 2021 at 23:14
Identified
30. September 2021 at 23:14
This is fixed, a post mortem is available in #devruby-ops
Investigating
30. September 2021 at 14:27
Investigating
30. September 2021 at 14:27
We are currently investigating this incident. Seems to be an issue regarding expired mTLS certificates.

OpenShift 4 Cluster cloudscale.ch Master Reboot

Completed
30. September 2021 at 11:47
Completed
30. September 2021 at 11:47
Maintenance has completed successfully.
In progress
30. September 2021 at 11:32
In progress
30. September 2021 at 11:32
Maintenance is now in progress.
Planned
30. September 2021 at 11:30
Planned
30. September 2021 at 11:30
The OpenShift 4 Masters will be drained and rebooted sequentially. No Service interruption is expected.

Resolved
30. September 2021 at 11:18
Resolved
30. September 2021 at 11:18
CodiMD pod was not responsive and hat to be restarted.

Mailserver maintenance

Completed
29. September 2021 at 16:37
Completed
29. September 2021 at 16:37
Maintenance has completed successfully.
Planned
29. September 2021 at 15:30
Planned
29. September 2021 at 15:30
We will implement SRS (Sender Rewriting Scheme) to minimize the effects of email forwards to gmail and other providers. During the maintenance you might experience some delays when receiving emails.
In progress
29. September 2021 at 15:29
In progress
29. September 2021 at 15:29
Maintenance is now in progress.

Aug 2021

/sys Wartungsfenster

Completed
26. August 2021 at 19:52
Completed
26. August 2021 at 19:52
Maintenance has completed successfully.
Planned
26. August 2021 at 16:0
Planned
26. August 2021 at 16:0
As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/ or periods of degraded performance.
In progress
26. August 2021 at 15:1
In progress
26. August 2021 at 15:1
Maintenance is now in progress.

Resolved
20. August 2021 at 9:53
Resolved
20. August 2021 at 9:53
Worker node k8s-worker03 is in a working state again.

OpenShift 4 Update

Completed
17. August 2021 at 18:58
Completed
17. August 2021 at 18:58
Maintenance has completed successfully.
In progress
17. August 2021 at 16:1
In progress
17. August 2021 at 16:1
Maintenance is now in progress.
Planned
17. August 2021 at 16:0
Planned
17. August 2021 at 16:0
We are planning for a scheduled maintenance during that time. Applications running as single pod instances are going to encounter a short downtime while they redeploy.

Resolved
23. August 2021 at 12:40
Resolved
23. August 2021 at 12:40
New cables, new happiness… fix worked
Monitoring
17. August 2021 at 11:15
Monitoring
17. August 2021 at 11:15
We implemented a possible fix for our issues in the GNU meeting room. Please continue to report all issues in all our meeting rooms to help@puzzle.ch
Identified
9. August 2021 at 20:16
Identified
9. August 2021 at 20:16
The last couple days have unfortunately not shown significant improvements regarding the stability of the Google Meet conferencing setup in the rooms "BASH", "GNU" and "TUX". Preliminary findings from Google support point to non-optimal or defunct USB cabling between the Logitech TAP and the Chromebox. We will pursue an immediate replacement of the affected cables and will keep this issue updated with new findings or updates about improved performance.
Monitoring
4. August 2021 at 15:35
Monitoring
4. August 2021 at 15:35
While our case with Google is still ongoing we did manually reboot the affected rooms and have working conferencing setups again. We will keep monitoring the situation closely and advise everyone to please create a ticket if you encounter an unresponsive Google Meet Setup. Please do not reboot or unplug Meet hardware on your own as this may hinder troubleshooting.
Investigating
4. August 2021 at 7:8
Investigating
4. August 2021 at 7:8
Multiple meeting rooms at Belpstrasse are experiencing problems with the Google Meet HW setups. A Ticket has been opened at Google and we'll post an update as soon as the problem could be solved.

Until then please don't unplug or power off any of the hardware. We recommend using your laptop and headphones for meetings if you experience Google Meet hardware issues in a meeting room.

Jul 2021

Update CSI drivers OpenShift CloudScale Production

Update
30. July 2021 at 11:35
Completed
30. July 2021 at 11:35
Maintenance has completed successfully.
Completed
30. July 2021 at 11:23
Completed
30. July 2021 at 11:23
We are planning to update the CSI drivers on OpenShift CloudScale Production. We do not expect any impact or downtime during this time. During maintenance, requesting a new persistent volume may take longer.

Sentry Migration

Completed
29. July 2021 at 16:9
Completed
29. July 2021 at 16:9
Maintenance has completed successfully.
In progress
29. July 2021 at 15:31
In progress
29. July 2021 at 15:31
Maintenance is now in progress.
Planned
29. July 2021 at 15:30
Planned
29. July 2021 at 15:30
We will be migrating Sentry from Openshift 3 to 4 tonight. During the migration Sentry will be temporarily unavailable.

Resolved
30. July 2021 at 6:31
Resolved
30. July 2021 at 6:31
We just resolved the issue!
Investigating
29. July 2021 at 12:0
Investigating
29. July 2021 at 12:0
We are currently investigating this incident.

/sys Wartungsfenster

Completed
22. July 2021 at 20:2
Completed
22. July 2021 at 20:2
Maintenance has completed successfully.
Planned
22. July 2021 at 16:0
Planned
22. July 2021 at 16:0
As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/ or periods of degraded performance.
In progress
22. July 2021 at 15:54
In progress
22. July 2021 at 15:54
Maintenance is now in progress.

OpenShift 4 maintenance

Completed
7. July 2021 at 17:7
Completed
7. July 2021 at 17:7
Maintenance has completed successfully.
In progress
7. July 2021 at 16:11
In progress
7. July 2021 at 16:11
Maintenance is now in progress.
Planned
7. July 2021 at 16:0
Planned
7. July 2021 at 16:0
We are planning for a scheduled maintenance during that time. The following production-labelled pods are going to encounter a short downtime due to rescheduling:
- Pod "cm-acme-http-solver-bq6vm" in namespace "pitc-lightning-prod"
- Pod "pfeedback-waf-1-gwfg9" in namespace "pitc-pfeedback-prod"
- Pod "rails-5df8d96f8b-cjbj4" in namespace "pitc-skills-prod"

Jul 2021 to Sep 2021

Puzzle ITC - Notice history

All systems operational

Notice history

Sep 2021

Aug 2021

Jul 2021