Puzzle ITC - Notice history

All systems operational

Operational

OpenShift 4 Cluster cloudscale.ch - Operational

100% - uptime
Sep 2021 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2021
Oct 2021
Nov 2021

OpenShift 4 Cluster LPG1 cloudscale.ch - Operational

100% - uptime
Sep 2021 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2021
Oct 2021
Nov 2021
Operational

Puzzle Website - Operational

100% - uptime
Sep 2021 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2021
Oct 2021
Nov 2021

PuzzleTime - Operational

100% - uptime
Sep 2021 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2021
Oct 2021
Nov 2021

Puzzle SSO (Keycloak) - Operational

100% - uptime
Sep 2021 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2021
Oct 2021
Nov 2021

PuzzleSkills - Operational

100% - uptime
Sep 2021 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2021
Oct 2021
Nov 2021

Harbor - Operational

100% - uptime
Sep 2021 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2021
Oct 2021
Nov 2021

Grafana - Operational

100% - uptime
Sep 2021 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2021
Oct 2021
Nov 2021

Puzzle News Site - Operational

100% - uptime
Sep 2021 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2021
Oct 2021
Nov 2021

Zammad - Operational

100% - uptime
Sep 2021 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2021
Oct 2021
Nov 2021

CardDAV - Operational

100% - uptime
Sep 2021 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2021
Oct 2021
Nov 2021

Argo CD - Operational

100% - uptime
Sep 2021 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2021
Oct 2021
Nov 2021

Vault - Operational

100% - uptime
Sep 2021 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2021
Oct 2021
Nov 2021

Loki - Operational

100% - uptime
Sep 2021 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2021
Oct 2021
Nov 2021

LDAP - Operational

100% - uptime
Sep 2021 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2021
Oct 2021
Nov 2021

Puzzle Docs - Operational

100% - uptime
Sep 2021 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2021
Oct 2021
Nov 2021

Odoo - Operational

100% - uptime
Sep 2021 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2021
Oct 2021
Nov 2021

Netbox - Operational

100% - uptime
Sep 2021 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2021
Oct 2021
Nov 2021

PostgreSQL - Operational

100% - uptime
Sep 2021 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2021
Oct 2021
Nov 2021
Operational

Network Office-Bern - Operational

100% - uptime
Sep 2021 · 99.92%Oct · 99.81%Nov · 100.0%

Previous page

Next page

Sep 2021
Oct 2021
Nov 2021

Network Office-Zürich - Operational

100% - uptime
Sep 2021 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2021
Oct 2021
Nov 2021

Google Meet Hardware - Operational

100% - uptime
Sep 2021 · 100.0%Oct · 99.81%Nov · 100.0%

Previous page

Next page

Sep 2021
Oct 2021
Nov 2021

Telephone Setup - Operational

92% - uptime
Sep 2021 · 89.47%Oct · 87.61%Nov · 100.0%

Previous page

Next page

Sep 2021
Oct 2021
Nov 2021
Operational

cloudscale.ch - Operational

100% - uptime
Sep 2021 · 99.97%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2021
Oct 2021
Nov 2021

Amazon AWS - Operational

100% - uptime
Sep 2021 · 99.97%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2021
Oct 2021
Nov 2021

Notice history

Nov 2021

Migration Artifactory Centos > Rocky
  • Completed
    30. November 2021 at 16:58
    Completed
    30. November 2021 at 16:58

    Maintenance has completed successfully.

  • In progress
    30. November 2021 at 16:31
    In progress
    30. November 2021 at 16:31

    Maintenance is now in progress.

  • Planned
    30. November 2021 at 16:30
    Planned
    30. November 2021 at 16:30

    We will migrate Artifactory from Centos to Rocky, during this time Artifactory will be offline.

Update IP Failover Setup on Kubernetes Cluster
  • Completed
    30. November 2021 at 16:41
    Completed
    30. November 2021 at 16:41

    Maintenance has completed successfully.

  • Planned
    30. November 2021 at 16:30
    Planned
    30. November 2021 at 16:30

    Our IP failover setup (based on keepalived) need a new API token to update the IP assignment at our cloud provider cloudscale. During redeployment of the keepalived Pods, our public ingress IPv4 and IPv6 addresses (e.g. behind chat.puzzle.ch) will shortly be not available and therefore applications running on the Kubernetes cluster will have a short outage. I will update them in sequence to keep the impact minimal

  • In progress
    30. November 2021 at 16:29
    In progress
    30. November 2021 at 16:29

    Maintenance is now in progress.

/sys Wartungsfenster
  • Completed
    25. November 2021 at 21:26
    Completed
    25. November 2021 at 21:26

    Maintenance has completed successfully.

  • In progress
    25. November 2021 at 17:0
    In progress
    25. November 2021 at 17:0

    Maintenance is now in progress.

  • Planned
    25. November 2021 at 17:0
    Planned
    25. November 2021 at 17:0

    As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/ or periods of degraded performance.

MariaDB01 Rocky Migration (affects Puzzle Website & HAST Services)
  • Completed
    18. November 2021 at 17:0
    Completed
    18. November 2021 at 17:0

    We will migrate one of our mariadb instances to RockyLinux which will affect the puzzle website (http://www.puzzle.ch) and some HAST Services. There will be a scheduled downtime for the Website & the HAST Services during the migration of the database server.

    Please be aware that you can't use the website (https://www.puzzle.ch) or the HAST Services on Thursday starting from 18:00 until the end of this maintenance.

VPN connectivity issues
  • Resolved
    Resolved

    We just resolved the issue!

  • Monitoring
    Monitoring

    We restored connectivity and are monitoring the situation. The issue was most likely related to a planned maintenance of our ISP: https://as13030.net/status/?ticket=4246996.

  • Investigating
    Investigating

    We are experiencing some issues with our VPN connections to our cloud providers (cloudscale.ch and AWS). As a result, internal connections are not possible at the moment.

Oct 2021

Network connectivity outage due to Init7 Issues.
  • Resolved
    Resolved

    Traffic path to region Bern was changed. Stable since 2021-10-22 9:50 CEST.

  • Identified
    Identified

    Init7 is currently experiencing interruptions on multiple Fiber7 POPs. Init7 status info: https://as13030.net/status/?ticket=OPS-160

/sys Wartungsfenster
  • Completed
    21. October 2021 at 21:58
    Completed
    21. October 2021 at 21:58

    We have completed our monthly maintenance. If you encounter any problems or errors please contact our helpdesk.

  • In progress
    21. October 2021 at 16:1
    In progress
    21. October 2021 at 16:1

    Maintenance is now in progress.

  • Planned
    21. October 2021 at 16:0
    Planned
    21. October 2021 at 16:0

    We will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/ or periods of degraded performance.

    ATTENTION: this maintenance window is NOT occurring on the 4th Thursday of the month but on the 3rd. This is due to a company wide event the following week. Please adjust blackout periods and monitoring systems to this change of dates. We apologize for the inconvenience caused by this.

PuzzleTime (CH+DE) OpenShift 4 Migration
  • Completed
    22. October 2021 at 8:35
    Completed
    22. October 2021 at 8:35

    Bei PTime DE gab es nach der Umstellung noch Zertifikats- und Caching-Probleme. Diese sind nun behoben und der Umzug von PTime DE somit abgeschlossen.

  • Update
    21. October 2021 at 15:3
    In progress
    21. October 2021 at 15:3

    Der Umzug der DE Umgebung ist nach der DNS Umstellung abgeschlossen.

  • Update
    21. October 2021 at 14:41
    In progress
    21. October 2021 at 14:41

    Die DNS Umstellung der CH Umgebung ist durch und https://time.puzzle.ch ist wieder online.

  • Update
    21. October 2021 at 14:35
    In progress
    21. October 2021 at 14:35

    Der Umzug der CH Umgebung ist nach der DNS Umstellung abgeschlossen.

    Wir beginnen nun mit dem Umzug der DE Umgebung. Diese wird bis zum Ende des Umzugs nicht erreichbar sein.

  • In progress
    21. October 2021 at 14:10
    In progress
    21. October 2021 at 14:10

    Wir starten mit dem Umzug der PTime CH Umgebung. Diese wird bis nach der Umstellung nicht erreichbar sein.

  • Planned
    21. October 2021 at 14:0
    Planned
    21. October 2021 at 14:0

    Nach der gestrigen erfolgreichen Migration auf S3 Storage werden wir PuzzleTime auf OpenShift 4 migrieren.

Taiga not available
  • Resolved
    Resolved

    Taiga is back online Again, apologies for the inconvenience

  • Identified
    Identified

    One component of Taiga is currently not working and therefore Taiga is not up and running.
    We are currently investigating and fixing this incident . Apologies for the inconvenience

Kubernetes & Rancher Node Maintenance
  • Completed
    20. October 2021 at 17:42
    Completed
    20. October 2021 at 17:42

    Maintenance has completed successfully.

  • Planned
    20. October 2021 at 16:0
    Planned
    20. October 2021 at 16:0

    All Rancher and Kubernetes nodes will get an OS update. We are planning for a scheduled maintenance during that time. All applications running on the Kubernetes cluster will be rescheduled multiple times during the maintenance window, therefore we expect some outages of the applications.

  • In progress
    20. October 2021 at 15:51
    In progress
    20. October 2021 at 15:51

    Maintenance is now in progress.

Sep 2021

OpenShift 4 Cluster Update
  • Completed
    30. September 2021 at 19:30
    Completed
    30. September 2021 at 19:30

    Maintenance has completed successfully.

  • In progress
    30. September 2021 at 15:0
    In progress
    30. September 2021 at 15:0

    Maintenance is now in progress.

  • Planned
    30. September 2021 at 15:0
    Planned
    30. September 2021 at 15:0

    We are planning for a scheduled maintenance during that time. The OpenShift 4 clusters on AWS and cloudscale are going to be updated to the latest version.

Cryptopus down
  • Resolved
    Resolved

    We just resolved the issue!

  • Monitoring
    Monitoring

    We implemented a fix and currently monitoring the result.

  • Identified
    Identified

    This is fixed, a post mortem is available in #devruby-ops

  • Investigating
    Investigating

    We are currently investigating this incident. Seems to be an issue regarding expired mTLS certificates.

OpenShift 4 Cluster cloudscale.ch Master Reboot
  • Completed
    30. September 2021 at 11:47
    Completed
    30. September 2021 at 11:47

    Maintenance has completed successfully.

  • In progress
    30. September 2021 at 11:32
    In progress
    30. September 2021 at 11:32

    Maintenance is now in progress.

  • Planned
    30. September 2021 at 11:30
    Planned
    30. September 2021 at 11:30

    The OpenShift 4 Masters will be drained and rebooted sequentially. No Service interruption is expected.

CodiMD not available
  • Resolved
    Resolved

    CodiMD pod was not responsive and hat to be restarted.

Mailserver maintenance
  • Completed
    29. September 2021 at 16:37
    Completed
    29. September 2021 at 16:37

    Maintenance has completed successfully.

  • Planned
    29. September 2021 at 15:30
    Planned
    29. September 2021 at 15:30

    We will implement SRS (Sender Rewriting Scheme) to minimize the effects of email forwards to gmail and other providers. During the maintenance you might experience some delays when receiving emails.

  • In progress
    29. September 2021 at 15:29
    In progress
    29. September 2021 at 15:29

    Maintenance is now in progress.

Sep 2021 to Nov 2021

Next