Puzzle ITC - Notice history

All systems operational

Operational

OpenShift 4 Cluster cloudscale.ch - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

OpenShift 4 Cluster LPG1 cloudscale.ch - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021
Operational

Puzzle Website (puzzle.ch) - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Mail - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Rocket.Chat - Operational

100% - uptime
Mar 2021 · 99.98%Apr · 100.0%May · 99.77%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Taiga - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

PuzzleTime - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Wiki - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Puzzle SSO (Keycloak) - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 99.97%May · 99.77%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

CodiMD - Operational

100% - uptime
Mar 2021 · 99.98%Apr · 99.97%May · 99.77%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

OpenProject - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Nextcloud - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

PuzzleSkills - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

GitLab - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 99.17%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Jenkins - Operational

98% - uptime
Mar 2021 · 99.82%Apr · 99.17%May · 95.38%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Quay (registry.puzzle.ch ) - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 99.78%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Harbor (harbor.puzzle.ch) - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Grafana - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Puzzle News Site (news.puzzle.ch) - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Cryptopus - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Artifactory - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

SonarQube - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Puzzle Meet - Operational

100% - uptime
Mar 2021 · 99.87%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Zammad (help.puzzle.ch) - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Sentry - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

CardDAV - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Webmail - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Argo CD - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Youtrack - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

OKR - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Vault - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Loki - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

LDAP - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Puzzle Docs (docs.puzzle.ch) - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Odoo - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021
Operational

Network Office-Bern - Operational

98% - uptime
Mar 2021 · 95.32%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Network Office-Zürich - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Google Meet Hardware - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Telephone Setup - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021
Operational

cloudscale.ch - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Amazon AWS - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Notice history

May 2021

/sys Wartungsfenster
  • Completed
    27. May 2021 at 20:19
    Completed
    27. May 2021 at 20:19

    Maintenance has completed successfully.

  • In progress
    27. May 2021 at 16:24
    In progress
    27. May 2021 at 16:24

    Maintenance is now in progress.

  • Planned
    27. May 2021 at 16:0
    Planned
    27. May 2021 at 16:0

    As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/ or periods of degraded performance.

Jenkins freestyle job configuration
  • Resolved
    Resolved

    We just resolved the issue!

  • Monitoring
    Monitoring

    We implemented a fix and currently monitoring the result.

  • Identified
    Identified

    We found the Plugin which causes the Issue. The Hudson Locks and Latches plugin is no longer compatible with the newest Jenkins Version.

    We will delete the plugin and restart Jenkins today at 12:30.

  • Investigating
    Investigating

    Saving freestyle job configuration changes does not work.

    We are currently investigating this incident.

Partial outage of PROD Rancher cluster worker 08
  • Resolved
    Resolved

    We just resolved the issue!

  • Identified
    Identified

    Cordoned, drained & rebooted node. The "flapping" behavior seems to be caused by the "broken" runC version we are currently running on our Rancher nodes: https://github.com/kubernetes/kubernetes/issues/45419#issuecomment-819574343

    [root@k8s-worker08 ~]# runc --version runc version 1.0.0-rc93 commit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec spec: 1.0.2-dev go: go1.13.15 libseccomp: 2.3.1

Jenkins Gitlab Plugin Update
  • Completed
    6. May 2021 at 16:7
    Completed
    6. May 2021 at 16:7

    Maintenance has completed successfully.

  • Planned
    6. May 2021 at 16:0
    Planned
    6. May 2021 at 16:0

    We are planning for a scheduled maintenance during that time. Jenkins will be restartet during this time.

  • In progress
    6. May 2021 at 15:59
    In progress
    6. May 2021 at 15:59

    Maintenance is now in progress.

Kubernetes Worker down
  • Resolved
    Resolved

    I checked the resource request/limits and optimized them a bit. Furthermore there is a new worker node running. I'm closing this incident all services are up and running.

  • Monitoring
    Monitoring

    We implemented a fix and currently monitoring the result.

  • Identified
    Identified

    The node is up and running again. But we need to identify what caused the problem, as earlier there was already a node outage. Current guess is, not enough memory, nodes are too crowed and resource management is probably not the best. We are adding a new node (maybe temporary), to mitigate the problem

  • Investigating
    Investigating

    Anonther Kubernetes Worker is unfortunatly down

Apr 2021

Unhappy MachineConfig Operator...
  • Resolved
    Resolved

    We just resolved the issue! We hit a bug in OpenShift 4.6.

  • Identified
    Identified

    The MachineConfig Operator on OCP4 AWS Prod is not happy, although he should be - it's Friday after all.

Nextcloud Server Downtime
  • Completed
    26. April 2021 at 16:3
    Completed
    26. April 2021 at 16:3

    The maintenance has completed successfully and Nextcloud is available again.

  • In progress
    26. April 2021 at 16:0
    In progress
    26. April 2021 at 16:0

    This maintenance is now in progress and we will scale the server momentarily.

  • Planned
    26. April 2021 at 16:0
    Planned
    26. April 2021 at 16:0

    The nextcloud upgrade to version 20 and a combined downsizing of it's resources has led to perceived slowness of the server. Therefore we are scaling the server to a bigger flavour. An downtime of less than 10 minutes is expected, beginning today at 18:00.

Jenkins OpenShift Slaves fail
  • Resolved
    Resolved

    We just resolved the issue!

  • Identified
    Identified

    openshiftv3slave-xxx are not starting because of GitLab auth problems. We are continuing to work on a fix for this incident.

/sys Wartungsfenster
  • Completed
    22. April 2021 at 20:14
    Completed
    22. April 2021 at 20:14

    Our maintenance has completed successfully.

    Please remember the Gitlab authentication change that happened tonight: https://info.puzzle.ch/posts/2021-03-05-gitlab-authentifizierung-nur-noch-via-sso/

    As usual, if you encounter any unexpected behaviour please contact us via our helpdesk.

  • Planned
    22. April 2021 at 16:0
    Planned
    22. April 2021 at 16:0

    As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/ or periods of degraded performance.

  • In progress
    22. April 2021 at 15:57
    In progress
    22. April 2021 at 15:57

    Our monthly maintenance is now in progress.

Jenkins OWASP Dependency-Check not working
  • Resolved
    Resolved

    NVD data is accessible again.

  • Investigating
    Investigating

    We are currently investigating this incident. The NVD data feeds are broken -> no data to check against.

Mar 2021

Jitsi is non-operational
  • Resolved
    Resolved

    We just resolved the issue!

  • Monitoring
    Monitoring

    We implemented a fix and currently monitoring the result.

  • Identified
    Identified

    We are continuing to work on a fix for this incident. In the meantime use https://meet-staging.puzzle.ch/

Rancher Management Server unstable
  • Resolved
    Resolved

    We just resolved the issue!

  • Monitoring
    Monitoring

    We implemented a fix and currently monitoring the result.

    All Rancher VM's are rebooted and are stable again.

  • Identified
    Identified

    We are continuing to work on a fix for this incident. One of the Rancher vm's is/was unreachable. I will reboot and cleanup the Rancher vm's in sequence. Rancher will continue to have degraded performance, as during reboot, the floating IPs of rancher.puzzle.ch will switch and also the Rancher Pods will be rescheduled a few times.

  • Investigating
    Investigating

    Our Rancher Management Server seems unstable and from time to time unreachable. I'm starting investigating the issue.

    This has no effect on the downstream Kubernetes Cluster and the application running on it. Only access via Rancher can be degraded or not available.

RocketChat Security fixes
  • Completed
    29. March 2021 at 15:46
    Completed
    29. March 2021 at 15:46

    Maintenance has completed successfully.

  • In progress
    29. March 2021 at 15:45
    In progress
    29. March 2021 at 15:45

    Maintenance is now in progress.

  • Planned
    29. March 2021 at 15:45
    Planned
    29. March 2021 at 15:45

    There are some high severity security issues that requires an RocketChat upgrade. No downtime of RocketChat is expected, you might only see a short reconnect in your RocketChat client.

Nextcloud migration
  • Completed
    28. March 2021 at 19:57
    Completed
    28. March 2021 at 19:57

    The Nextcloud migration is completed successfully and you can access Nextcloud as before on https://files.puzzle.ch Clients will automatically trigger a resync upon initial connection. You can find the latest documentation on Nextcloud usage at: https://docs.puzzle.ch/guides/latest/nextcloud-user-guide/index.html In case you have questions or notice an anomaly please contact helpdesk.

  • Planned
    26. March 2021 at 23:21
    Planned
    26. March 2021 at 23:21

    The migration of our Nextcloud server has begun and the server is currently in maintenance mode. In case you urgently need read-only access to a file please contact our helpdesk. We will provide further updates on the progress of the migration over the weekend.

BE3 vpn not working
  • Resolved
    Resolved

    All connections issues are resolved. We'll monitor the situation closely for the rest of the day and post updates if necessary.

  • Update
    Update

    More connectivity issues have been identified at the office. We are continuing to work on a fix for this incident.

  • Investigating
    Investigating

    The VPN for be3 isn't working at the moment. This issue could be related to the maintenance window last night. We are currently investigating this incident and someone is on their way to the office. Services only availably via VPN are not reachable at this point in time.

Mar 2021 to May 2021

Next