Puzzle ITC - Notice history

Experiencing minor outage

Operational

OpenShift 4 Cluster cloudscale.ch - Operational

100.0% uptime
Feb 2021 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2021
Mar 2021
Apr 2021

OpenShift 4 Cluster LPG1 cloudscale.ch - Operational

100.0% uptime
Feb 2021 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2021
Mar 2021
Apr 2021
Operational

PuzzleTime - Operational

100.0% uptime
Feb 2021 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2021
Mar 2021
Apr 2021

Puzzle SSO (Keycloak) - Operational

99.22% uptime
Feb 2021 · 97.58%Mar · 100.0%Apr · 99.97%

Previous page

Next page

Feb 2021
Mar 2021
Apr 2021

PuzzleSkills - Operational

100.0% uptime
Feb 2021 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2021
Mar 2021
Apr 2021

Harbor - Operational

100.0% uptime
Feb 2021 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2021
Mar 2021
Apr 2021

Grafana - Operational

100.0% uptime
Feb 2021 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2021
Mar 2021
Apr 2021

Puzzle Backstage - Operational

100.0% uptime
Feb 2021 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2021
Mar 2021
Apr 2021

Zammad - Operational

100.0% uptime
Feb 2021 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2021
Mar 2021
Apr 2021

CardDAV - Operational

100.0% uptime
Feb 2021 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2021
Mar 2021
Apr 2021

Argo CD - Operational

100.0% uptime
Feb 2021 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2021
Mar 2021
Apr 2021

Vault - Operational

100.0% uptime
Feb 2021 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2021
Mar 2021
Apr 2021

Loki - Operational

100.0% uptime
Feb 2021 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2021
Mar 2021
Apr 2021

LDAP - Operational

100.0% uptime
Feb 2021 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2021
Mar 2021
Apr 2021

Puzzle Docs - Operational

100.0% uptime
Feb 2021 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2021
Mar 2021
Apr 2021

Odoo - Operational

100.0% uptime
Feb 2021 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2021
Mar 2021
Apr 2021

Netbox - Operational

100.0% uptime
Feb 2021 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2021
Mar 2021
Apr 2021

PostgreSQL - Operational

100.0% uptime
Feb 2021 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2021
Mar 2021
Apr 2021

Package Mirror - Operational

100.0% uptime
Feb 2021 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2021
Mar 2021
Apr 2021
Partial outage

Network Office-Bern - Partial outage

97.92% uptime
Feb 2021 · 98.64%Mar · 95.32%Apr · 100.0%

Previous page

Next page

Feb 2021
Mar 2021
Apr 2021

Network Office-Zürich - Partial outage

Network Office-Solothurn - Operational

100.0% uptime
Feb 2021 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2021
Mar 2021
Apr 2021

Google Meet Hardware - Operational

100.0% uptime
Feb 2021 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2021
Mar 2021
Apr 2021

Telephone Setup - Operational

100.0% uptime
Feb 2021 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2021
Mar 2021
Apr 2021
Operational

cloudscale.ch - Operational

100.0% uptime
Feb 2021 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2021
Mar 2021
Apr 2021

Notice history

View current status

Apr 2021

APR30
Unhappy MachineConfig Operator...
ResolvedMaintenance192 hours 54 minutes
  • ResolvedUTC

    We just resolved the issue! We hit a bug in OpenShift 4.6.

  • IdentifiedUTC

    The MachineConfig Operator on OCP4 AWS Prod is not happy, although he should be - it's Friday after all.

APR29
OpenShift 4 Cluster Update
CompletedMaintenance5 hours 22 minutes
  • Completed29. April 2021 at 20:22UTC

    Maintenance has completed successfully.

  • In progress29. April 2021 at 15:2UTC

    Maintenance is now in progress.

  • Update29. April 2021 at 15:0UTC

    We are planning to update OpenShift 4.6 to the latest stable version.

  • Planned19. April 2021 at 13:32UTC

    Wrong title, there is no GlusterFS on the OCP4 cluster. Update will be done in one take.

APR29
OpenShift 3.11 Cluster Update - Infra + Compute Nodes
CompletedMaintenance5 hours 21 minutes
  • Completed29. April 2021 at 20:21UTC

    Maintenance has completed successfully.

  • Update29. April 2021 at 15:14UTC

    We will update the compute nodes in parallel, meaning most of all applications will not be available for at least 30min.

  • In progress29. April 2021 at 15:3UTC

    Maintenance is now in progress.

  • Planned29. April 2021 at 15:0UTC

    We are planning to update OpenShift 3.11 to the latest stable version.

APR28
OpenShift 3.11 Cluster Update - Control Plane + GlusterFS
CompletedMaintenance5 hours 58 minutes
  • Completed28. April 2021 at 20:58UTC

    Maintenance has completed successfully.

  • In progress28. April 2021 at 15:4UTC

    Maintenance is now in progress.

  • Planned28. April 2021 at 15:0UTC

    We are planning to update OpenShift 3.11 to the latest stable version.

APR26
Nextcloud Server Downtime
CompletedMaintenance3 minutes
  • Completed26. April 2021 at 16:3UTC

    The maintenance has completed successfully and Nextcloud is available again.

  • In progress26. April 2021 at 16:0UTC

    This maintenance is now in progress and we will scale the server momentarily.

  • Planned26. April 2021 at 16:0UTC

    The nextcloud upgrade to version 20 and a combined downsizing of it's resources has led to perceived slowness of the server. Therefore we are scaling the server to a bigger flavour. An downtime of less than 10 minutes is expected, beginning today at 18:00.

Mar 2021

Jitsi is non-operational
ResolvedMajor outage54 minutes
  • ResolvedUTC

    We just resolved the issue!

  • MonitoringUTC

    We implemented a fix and currently monitoring the result.

  • IdentifiedUTC

    We are continuing to work on a fix for this incident. In the meantime use https://meet-staging.puzzle.ch/

Rancher Management Server unstable
ResolvedDegraded performance1 hour 25 minutes
  • ResolvedUTC

    We just resolved the issue!

  • MonitoringUTC

    We implemented a fix and currently monitoring the result.

    All Rancher VM's are rebooted and are stable again.

  • IdentifiedUTC

    We are continuing to work on a fix for this incident. One of the Rancher vm's is/was unreachable. I will reboot and cleanup the Rancher vm's in sequence. Rancher will continue to have degraded performance, as during reboot, the floating IPs of rancher.puzzle.ch will switch and also the Rancher Pods will be rescheduled a few times.

  • InvestigatingUTC

    Our Rancher Management Server seems unstable and from time to time unreachable. I'm starting investigating the issue.

    This has no effect on the downstream Kubernetes Cluster and the application running on it. Only access via Rancher can be degraded or not available.

MAR29
RocketChat Security fixes
CompletedMaintenance1 minute
  • Completed29. March 2021 at 15:46UTC

    Maintenance has completed successfully.

  • In progress29. March 2021 at 15:45UTC

    Maintenance is now in progress.

  • Planned29. March 2021 at 15:45UTC

    There are some high severity security issues that requires an RocketChat upgrade. No downtime of RocketChat is expected, you might only see a short reconnect in your RocketChat client.

MAR26
Nextcloud migration
CompletedMaintenance44 hours 35 minutes
  • Completed28. March 2021 at 19:57UTC

    The Nextcloud migration is completed successfully and you can access Nextcloud as before on https://files.puzzle.ch Clients will automatically trigger a resync upon initial connection. You can find the latest documentation on Nextcloud usage at: https://docs.puzzle.ch/guides/latest/nextcloud-user-guide/index.html In case you have questions or notice an anomaly please contact helpdesk.

  • Planned26. March 2021 at 23:21UTC

    The migration of our Nextcloud server has begun and the server is currently in maintenance mode. In case you urgently need read-only access to a file please contact our helpdesk. We will provide further updates on the progress of the migration over the weekend.

BE3 vpn not working
ResolvedMajor outage1 hour 35 minutes
  • ResolvedUTC

    All connections issues are resolved. We'll monitor the situation closely for the rest of the day and post updates if necessary.

  • UpdateUTC

    More connectivity issues have been identified at the office. We are continuing to work on a fix for this incident.

  • InvestigatingUTC

    The VPN for be3 isn't working at the moment. This issue could be related to the maintenance window last night. We are currently investigating this incident and someone is on their way to the office. Services only availably via VPN are not reachable at this point in time.

Feb 2021

FEB26
Jenkins Prometheus Plugin downgrade
CompletedMaintenance6 minutes
  • Completed26. February 2021 at 11:6UTC

    Maintenance has completed successfully.

  • Planned26. February 2021 at 11:0UTC

    We will downgrade the Prometheus Plugin. Jenkins will be restarted during this time.

FEB25
/sys Wartungsfenster
CompletedMaintenance3 hours 58 minutes
  • Completed25. February 2021 at 20:58UTC

    All maintenances have been completed successfully and OpenProject has been upgraded to the newest major release. In case you notice any irregularities please contact our helpdesk at help (at) puzzle (dot) ch

  • In progress25. February 2021 at 17:8UTC

    Maintenance is now in progress.

  • Planned25. February 2021 at 17:0UTC

    As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/ or periods of degraded performance. Additionally we will update OpenProject to major version 11.

Interruption of Kubernetes Cluster
ResolvedMajor outage1 hour 4 minutes
  • ResolvedUTC

    Stable again. Closing incident

  • MonitoringUTC

    The master node remains stable, we continue to monitor. All applications running on the k8s cluster are up and running

  • IdentifiedUTC

    One of the master nodes was not ready anymore. This resulted in short outage of the Kubernetes control plane. This then (for yet unknwon reason) also affected some of the applicactions. I'm currently investigating what exacly happened. I rebooted the vm and it seems stable again.

  • InvestigatingUTC

    Our Kubernetes Cluster has currently trouble which seems to affect some application. So far we see outages on SSO and Codi, possible more applications.

    investigation started

FEB23
Networkchange on Cloudscale VMs with multiple Network Interfaces
CompletedMaintenance1 hour 7 minutes
  • Completed23. February 2021 at 18:7UTC

    Maintenance has completed successfully.

  • Planned23. February 2021 at 17:0UTC

    We are planning for a scheduled maintenance during that time to add some more routes for Servers with multiple network interfaces.

Rancher Management Server unstable
ResolvedPartial outage50 minutes
  • ResolvedUTC

    Still stable. Closing this incident.

  • MonitoringUTC

    Rancher Management Server seems stable again. I rebooted all nodes and I'm monitoring the situation now.

  • InvestigatingUTC

    We see some restarts of Rancher Podss which leads to an unstable Rancher Management Server and API Access to Downstream Cluster.

    Downstream Cluster (Puzzle Kubernetes Production Cluster) is not affected and all applications running on it are working

Previous

Feb 2021 to Apr 2021

Next