Puzzle ITC - Notice history

Experiencing minor outage

Operational

OpenShift 4 Cluster cloudscale.ch - Operational

100.0% uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

OpenShift 4 Cluster LPG1 cloudscale.ch - Operational

100.0% uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021
Operational

PuzzleTime - Operational

100.0% uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Puzzle SSO (Keycloak) - Operational

100.0% uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

PuzzleSkills - Operational

100.0% uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Harbor - Operational

100.0% uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Grafana - Operational

100.0% uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Puzzle Backstage - Operational

100.0% uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Zammad - Operational

100.0% uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

CardDAV - Operational

100.0% uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Argo CD - Operational

100.0% uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Vault - Operational

100.0% uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Loki - Operational

100.0% uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

LDAP - Operational

100.0% uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Puzzle Docs - Operational

100.0% uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Odoo - Operational

100.0% uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Netbox - Operational

100.0% uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

PostgreSQL - Operational

100.0% uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Package Mirror - Operational

100.0% uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021
Partial outage

Network Office-Bern - Partial outage

99.91% uptime
Aug 2021 · 100.0%Sep · 99.92%Oct · 99.81%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Network Office-Zürich - Partial outage

Network Office-Solothurn - Operational

100.0% uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Google Meet Hardware - Operational

93.60% uptime
Aug 2021 · 81.39%Sep · 100.0%Oct · 99.81%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Telephone Setup - Operational

92.31% uptime
Aug 2021 · 100.0%Sep · 89.47%Oct · 87.61%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021
Operational

cloudscale.ch - Operational

99.99% uptime
Aug 2021 · 100.0%Sep · 99.97%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Notice history

View current status

Oct 2021

Network connectivity outage due to Init7 Issues.
ResolvedPartial outage4 hours 40 minutes
  • ResolvedUTC

    Traffic path to region Bern was changed. Stable since 2021-10-22 9:50 CEST.

  • IdentifiedUTC

    Init7 is currently experiencing interruptions on multiple Fiber7 POPs. Init7 status info: https://as13030.net/status/?ticket=OPS-160

OCT21
/sys Wartungsfenster
CompletedMaintenance5 hours 58 minutes
  • Completed21. October 2021 at 21:58UTC

    We have completed our monthly maintenance. If you encounter any problems or errors please contact our helpdesk.

  • In progress21. October 2021 at 16:1UTC

    Maintenance is now in progress.

  • Planned21. October 2021 at 16:0UTC

    We will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/ or periods of degraded performance.

    ATTENTION: this maintenance window is NOT occurring on the 4th Thursday of the month but on the 3rd. This is due to a company wide event the following week. Please adjust blackout periods and monitoring systems to this change of dates. We apologize for the inconvenience caused by this.

OCT21
PuzzleTime (CH+DE) OpenShift 4 Migration
CompletedMaintenance18 hours 35 minutes
  • Completed22. October 2021 at 8:35UTC

    Bei PTime DE gab es nach der Umstellung noch Zertifikats- und Caching-Probleme. Diese sind nun behoben und der Umzug von PTime DE somit abgeschlossen.

  • Update21. October 2021 at 15:3UTC

    Der Umzug der DE Umgebung ist nach der DNS Umstellung abgeschlossen.

  • Update21. October 2021 at 14:41UTC

    Die DNS Umstellung der CH Umgebung ist durch und https://time.puzzle.ch ist wieder online.

  • Update21. October 2021 at 14:35UTC

    Der Umzug der CH Umgebung ist nach der DNS Umstellung abgeschlossen.

    Wir beginnen nun mit dem Umzug der DE Umgebung. Diese wird bis zum Ende des Umzugs nicht erreichbar sein.

  • In progress21. October 2021 at 14:10UTC

    Wir starten mit dem Umzug der PTime CH Umgebung. Diese wird bis nach der Umstellung nicht erreichbar sein.

  • Planned21. October 2021 at 14:0UTC

    Nach der gestrigen erfolgreichen Migration auf S3 Storage werden wir PuzzleTime auf OpenShift 4 migrieren.

Taiga not available
ResolvedMajor outage2 minutes
  • ResolvedUTC

    Taiga is back online Again, apologies for the inconvenience

  • IdentifiedUTC

    One component of Taiga is currently not working and therefore Taiga is not up and running.
    We are currently investigating and fixing this incident . Apologies for the inconvenience

OCT20
Kubernetes & Rancher Node Maintenance
CompletedMaintenance1 hour 42 minutes
  • Completed20. October 2021 at 17:42UTC

    Maintenance has completed successfully.

  • Planned20. October 2021 at 16:0UTC

    All Rancher and Kubernetes nodes will get an OS update. We are planning for a scheduled maintenance during that time. All applications running on the Kubernetes cluster will be rescheduled multiple times during the maintenance window, therefore we expect some outages of the applications.

  • In progress20. October 2021 at 15:51UTC

    Maintenance is now in progress.

Sep 2021

SEP30
OpenShift 4 Cluster Update
CompletedMaintenance4 hours 30 minutes
  • Completed30. September 2021 at 19:30UTC

    Maintenance has completed successfully.

  • In progress30. September 2021 at 15:0UTC

    Maintenance is now in progress.

  • Planned30. September 2021 at 15:0UTC

    We are planning for a scheduled maintenance during that time. The OpenShift 4 clusters on AWS and cloudscale are going to be updated to the latest version.

Cryptopus down
ResolvedMinor outage358 hours 32 minutes
  • ResolvedUTC

    We just resolved the issue!

  • MonitoringUTC

    We implemented a fix and currently monitoring the result.

  • IdentifiedUTC

    This is fixed, a post mortem is available in #devruby-ops

  • InvestigatingUTC

    We are currently investigating this incident. Seems to be an issue regarding expired mTLS certificates.

SEP30
OpenShift 4 Cluster cloudscale.ch Master Reboot
CompletedMaintenance17 minutes
  • Completed30. September 2021 at 11:47UTC

    Maintenance has completed successfully.

  • In progress30. September 2021 at 11:32UTC

    Maintenance is now in progress.

  • Planned30. September 2021 at 11:30UTC

    The OpenShift 4 Masters will be drained and rebooted sequentially. No Service interruption is expected.

CodiMD not available
Resolved
  • ResolvedUTC

    CodiMD pod was not responsive and hat to be restarted.

SEP30
WeKan Update
CompletedMaintenance1 hour 27 minutes
  • Completed30. September 2021 at 9:27UTC

    Maintenance has completed successfully. We have successfully updated WeKan to version 5.51 and MongoDB to 4.4.9.

  • In progress30. September 2021 at 8:33UTC

    Maintenance is now in progress.

  • Planned30. September 2021 at 8:0UTC

    We are planning for a scheduled maintenance during that time and will update WeKan to the latest version. The application will not be available during that time.

Aug 2021

AUG26
/sys Wartungsfenster
CompletedMaintenance3 hours 52 minutes
  • Completed26. August 2021 at 19:52UTC

    Maintenance has completed successfully.

  • Planned26. August 2021 at 16:0UTC

    As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/ or periods of degraded performance.

  • In progress26. August 2021 at 15:1UTC

    Maintenance is now in progress.

k8s-worker03 outage
Resolved
  • ResolvedUTC

    Worker node k8s-worker03 is in a working state again.

AUG17
OpenShift 4 Update
CompletedMaintenance2 hours 58 minutes
  • Completed17. August 2021 at 18:58UTC

    Maintenance has completed successfully.

  • In progress17. August 2021 at 16:1UTC

    Maintenance is now in progress.

  • Planned17. August 2021 at 16:0UTC

    We are planning for a scheduled maintenance during that time. Applications running as single pod instances are going to encounter a short downtime while they redeploy.

AUG4
Update CSI drivers Rancher Kubernetes Production
CompletedMaintenance47 minutes
  • Completed4. August 2021 at 15:47UTC

    Maintenance has completed successfully.

  • In progress4. August 2021 at 15:15UTC

    Maintenance is now in progress.

  • Planned4. August 2021 at 15:0UTC

    We are planning to update the CSI drivers on Rancher Kubernetes Production. We do not expect any impact or downtime during this time. During maintenance, requesting a new persistent volume may take longer. check https://puzzle-status.ch for more details

Google Meet Hardware at Belpstasse
ResolvedMinor outage461 hours 32 minutes
  • ResolvedUTC

    New cables, new happiness… fix worked

  • MonitoringUTC

    We implemented a possible fix for our issues in the GNU meeting room. Please continue to report all issues in all our meeting rooms to help@puzzle.ch

  • IdentifiedUTC

    The last couple days have unfortunately not shown significant improvements regarding the stability of the Google Meet conferencing setup in the rooms "BASH", "GNU" and "TUX". Preliminary findings from Google support point to non-optimal or defunct USB cabling between the Logitech TAP and the Chromebox. We will pursue an immediate replacement of the affected cables and will keep this issue updated with new findings or updates about improved performance.

  • MonitoringUTC

    While our case with Google is still ongoing we did manually reboot the affected rooms and have working conferencing setups again. We will keep monitoring the situation closely and advise everyone to please create a ticket if you encounter an unresponsive Google Meet Setup. Please do not reboot or unplug Meet hardware on your own as this may hinder troubleshooting.

  • InvestigatingUTC

    Multiple meeting rooms at Belpstrasse are experiencing problems with the Google Meet HW setups. A Ticket has been opened at Google and we'll post an update as soon as the problem could be solved.

    Until then please don't unplug or power off any of the hardware. We recommend using your laptop and headphones for meetings if you experience Google Meet hardware issues in a meeting room.

Previous

Aug 2021 to Oct 2021

Next