Puzzle ITC - Notice history

All systems operational

Operational

OpenShift 4 Cluster cloudscale.ch - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

OpenShift 4 Cluster LPG1 cloudscale.ch - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021
Operational

Puzzle Website (puzzle.ch) - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Mail - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Rocket.Chat - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Taiga - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 99.99%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

PuzzleTime - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Wiki - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Puzzle SSO (Keycloak) - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

CodiMD - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

OpenProject - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Nextcloud - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

PuzzleSkills - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

GitLab - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Jenkins - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Quay (registry.puzzle.ch ) - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Harbor (harbor.puzzle.ch) - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Grafana - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Puzzle News Site (news.puzzle.ch) - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Cryptopus - Operational

95% - uptime
Aug 2021 · 100.0%Sep · 99.60%Oct · 85.93%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Artifactory - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

SonarQube - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Puzzle Meet - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Zammad (help.puzzle.ch) - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Sentry - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

CardDAV - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Webmail - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Argo CD - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Youtrack - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

OKR - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Vault - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Loki - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

LDAP - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Puzzle Docs (docs.puzzle.ch) - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021
Operational

Network Office-Bern - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 99.92%Oct · 99.81%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Network Office-Zürich - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Google Meet Hardware - Operational

94% - uptime
Aug 2021 · 81.39%Sep · 100.0%Oct · 99.81%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Telephone Setup - Operational

92% - uptime
Aug 2021 · 100.0%Sep · 89.47%Oct · 87.61%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021
Operational

cloudscale.ch - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 99.97%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Amazon AWS - Operational

100% - uptime
Aug 2021 · 100.0%Sep · 99.97%Oct · 100.0%

Previous page

Next page

Aug 2021
Sep 2021
Oct 2021

Notice history

Oct 2021

Network connectivity outage due to Init7 Issues.
  • Resolved
    Resolved

    Traffic path to region Bern was changed. Stable since 2021-10-22 9:50 CEST.

  • Identified
    Identified

    Init7 is currently experiencing interruptions on multiple Fiber7 POPs. Init7 status info: https://as13030.net/status/?ticket=OPS-160

/sys Wartungsfenster
  • Completed
    21. October 2021 at 21:58
    Completed
    21. October 2021 at 21:58

    We have completed our monthly maintenance. If you encounter any problems or errors please contact our helpdesk.

  • In progress
    21. October 2021 at 16:1
    In progress
    21. October 2021 at 16:1

    Maintenance is now in progress.

  • Planned
    21. October 2021 at 16:0
    Planned
    21. October 2021 at 16:0

    We will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/ or periods of degraded performance.

    ATTENTION: this maintenance window is NOT occurring on the 4th Thursday of the month but on the 3rd. This is due to a company wide event the following week. Please adjust blackout periods and monitoring systems to this change of dates. We apologize for the inconvenience caused by this.

PuzzleTime (CH+DE) OpenShift 4 Migration
  • Completed
    22. October 2021 at 8:35
    Completed
    22. October 2021 at 8:35

    Bei PTime DE gab es nach der Umstellung noch Zertifikats- und Caching-Probleme. Diese sind nun behoben und der Umzug von PTime DE somit abgeschlossen.

  • Update
    21. October 2021 at 15:3
    In progress
    21. October 2021 at 15:3

    Der Umzug der DE Umgebung ist nach der DNS Umstellung abgeschlossen.

  • Update
    21. October 2021 at 14:41
    In progress
    21. October 2021 at 14:41

    Die DNS Umstellung der CH Umgebung ist durch und https://time.puzzle.ch ist wieder online.

  • Update
    21. October 2021 at 14:35
    In progress
    21. October 2021 at 14:35

    Der Umzug der CH Umgebung ist nach der DNS Umstellung abgeschlossen.

    Wir beginnen nun mit dem Umzug der DE Umgebung. Diese wird bis zum Ende des Umzugs nicht erreichbar sein.

  • In progress
    21. October 2021 at 14:10
    In progress
    21. October 2021 at 14:10

    Wir starten mit dem Umzug der PTime CH Umgebung. Diese wird bis nach der Umstellung nicht erreichbar sein.

  • Planned
    21. October 2021 at 14:0
    Planned
    21. October 2021 at 14:0

    Nach der gestrigen erfolgreichen Migration auf S3 Storage werden wir PuzzleTime auf OpenShift 4 migrieren.

Taiga not available
  • Resolved
    Resolved

    Taiga is back online Again, apologies for the inconvenience

  • Identified
    Identified

    One component of Taiga is currently not working and therefore Taiga is not up and running.
    We are currently investigating and fixing this incident . Apologies for the inconvenience

Kubernetes & Rancher Node Maintenance
  • Completed
    20. October 2021 at 17:42
    Completed
    20. October 2021 at 17:42

    Maintenance has completed successfully.

  • Planned
    20. October 2021 at 16:0
    Planned
    20. October 2021 at 16:0

    All Rancher and Kubernetes nodes will get an OS update. We are planning for a scheduled maintenance during that time. All applications running on the Kubernetes cluster will be rescheduled multiple times during the maintenance window, therefore we expect some outages of the applications.

  • In progress
    20. October 2021 at 15:51
    In progress
    20. October 2021 at 15:51

    Maintenance is now in progress.

Sep 2021

OpenShift 4 Cluster Update
  • Completed
    30. September 2021 at 19:30
    Completed
    30. September 2021 at 19:30

    Maintenance has completed successfully.

  • In progress
    30. September 2021 at 15:0
    In progress
    30. September 2021 at 15:0

    Maintenance is now in progress.

  • Planned
    30. September 2021 at 15:0
    Planned
    30. September 2021 at 15:0

    We are planning for a scheduled maintenance during that time. The OpenShift 4 clusters on AWS and cloudscale are going to be updated to the latest version.

Cryptopus down
  • Resolved
    Resolved

    We just resolved the issue!

  • Monitoring
    Monitoring

    We implemented a fix and currently monitoring the result.

  • Identified
    Identified

    This is fixed, a post mortem is available in #devruby-ops

  • Investigating
    Investigating

    We are currently investigating this incident. Seems to be an issue regarding expired mTLS certificates.

OpenShift 4 Cluster cloudscale.ch Master Reboot
  • Completed
    30. September 2021 at 11:47
    Completed
    30. September 2021 at 11:47

    Maintenance has completed successfully.

  • In progress
    30. September 2021 at 11:32
    In progress
    30. September 2021 at 11:32

    Maintenance is now in progress.

  • Planned
    30. September 2021 at 11:30
    Planned
    30. September 2021 at 11:30

    The OpenShift 4 Masters will be drained and rebooted sequentially. No Service interruption is expected.

CodiMD not available
  • Resolved
    Resolved

    CodiMD pod was not responsive and hat to be restarted.

Mailserver maintenance
  • Completed
    29. September 2021 at 16:37
    Completed
    29. September 2021 at 16:37

    Maintenance has completed successfully.

  • Planned
    29. September 2021 at 15:30
    Planned
    29. September 2021 at 15:30

    We will implement SRS (Sender Rewriting Scheme) to minimize the effects of email forwards to gmail and other providers. During the maintenance you might experience some delays when receiving emails.

  • In progress
    29. September 2021 at 15:29
    In progress
    29. September 2021 at 15:29

    Maintenance is now in progress.

Aug 2021

/sys Wartungsfenster
  • Completed
    26. August 2021 at 19:52
    Completed
    26. August 2021 at 19:52

    Maintenance has completed successfully.

  • Planned
    26. August 2021 at 16:0
    Planned
    26. August 2021 at 16:0

    As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/ or periods of degraded performance.

  • In progress
    26. August 2021 at 15:1
    In progress
    26. August 2021 at 15:1

    Maintenance is now in progress.

k8s-worker03 outage
  • Resolved
    Resolved

    Worker node k8s-worker03 is in a working state again.

OpenShift 4 Update
  • Completed
    17. August 2021 at 18:58
    Completed
    17. August 2021 at 18:58

    Maintenance has completed successfully.

  • In progress
    17. August 2021 at 16:1
    In progress
    17. August 2021 at 16:1

    Maintenance is now in progress.

  • Planned
    17. August 2021 at 16:0
    Planned
    17. August 2021 at 16:0

    We are planning for a scheduled maintenance during that time. Applications running as single pod instances are going to encounter a short downtime while they redeploy.

Google Meet Hardware at Belpstasse
  • Resolved
    Resolved

    New cables, new happiness… fix worked

  • Monitoring
    Monitoring

    We implemented a possible fix for our issues in the GNU meeting room. Please continue to report all issues in all our meeting rooms to help@puzzle.ch

  • Identified
    Identified

    The last couple days have unfortunately not shown significant improvements regarding the stability of the Google Meet conferencing setup in the rooms "BASH", "GNU" and "TUX". Preliminary findings from Google support point to non-optimal or defunct USB cabling between the Logitech TAP and the Chromebox. We will pursue an immediate replacement of the affected cables and will keep this issue updated with new findings or updates about improved performance.

  • Monitoring
    Monitoring

    While our case with Google is still ongoing we did manually reboot the affected rooms and have working conferencing setups again. We will keep monitoring the situation closely and advise everyone to please create a ticket if you encounter an unresponsive Google Meet Setup. Please do not reboot or unplug Meet hardware on your own as this may hinder troubleshooting.

  • Investigating
    Investigating

    Multiple meeting rooms at Belpstrasse are experiencing problems with the Google Meet HW setups. A Ticket has been opened at Google and we'll post an update as soon as the problem could be solved.

    Until then please don't unplug or power off any of the hardware. We recommend using your laptop and headphones for meetings if you experience Google Meet hardware issues in a meeting room.

Aug 2021 to Oct 2021

Next