Puzzle ITC - Notice history

All systems operational

Operational

OpenShift 4 Cluster cloudscale.ch - Operational

100% - uptime
Apr 2021 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2021
May 2021
Jun 2021

OpenShift 4 Cluster LPG1 cloudscale.ch - Operational

100% - uptime
Apr 2021 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2021
May 2021
Jun 2021
Operational

Puzzle Website - Operational

100% - uptime
Apr 2021 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2021
May 2021
Jun 2021

PuzzleTime - Operational

100% - uptime
Apr 2021 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2021
May 2021
Jun 2021

Puzzle SSO (Keycloak) - Operational

100% - uptime
Apr 2021 · 99.97%May · 99.77%Jun · 100.0%

Previous page

Next page

Apr 2021
May 2021
Jun 2021

PuzzleSkills - Operational

100% - uptime
Apr 2021 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2021
May 2021
Jun 2021

Harbor - Operational

100% - uptime
Apr 2021 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2021
May 2021
Jun 2021

Grafana - Operational

100% - uptime
Apr 2021 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2021
May 2021
Jun 2021

Puzzle News Site - Operational

100% - uptime
Apr 2021 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2021
May 2021
Jun 2021

Zammad - Operational

100% - uptime
Apr 2021 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2021
May 2021
Jun 2021

CardDAV - Operational

100% - uptime
Apr 2021 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2021
May 2021
Jun 2021

Argo CD - Operational

100% - uptime
Apr 2021 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2021
May 2021
Jun 2021

Vault - Operational

100% - uptime
Apr 2021 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2021
May 2021
Jun 2021

Loki - Operational

100% - uptime
Apr 2021 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2021
May 2021
Jun 2021

LDAP - Operational

100% - uptime
Apr 2021 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2021
May 2021
Jun 2021

Puzzle Docs - Operational

100% - uptime
Apr 2021 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2021
May 2021
Jun 2021

Odoo - Operational

100% - uptime
Apr 2021 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2021
May 2021
Jun 2021

Netbox - Operational

100% - uptime
Apr 2021 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2021
May 2021
Jun 2021

PostgreSQL - Operational

100% - uptime
Apr 2021 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2021
May 2021
Jun 2021
Operational

Network Office-Bern - Operational

100% - uptime
Apr 2021 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2021
May 2021
Jun 2021

Network Office-Zürich - Operational

100% - uptime
Apr 2021 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2021
May 2021
Jun 2021

Google Meet Hardware - Operational

100% - uptime
Apr 2021 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2021
May 2021
Jun 2021

Telephone Setup - Operational

100% - uptime
Apr 2021 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2021
May 2021
Jun 2021
Operational

cloudscale.ch - Operational

100% - uptime
Apr 2021 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2021
May 2021
Jun 2021

Amazon AWS - Operational

100% - uptime
Apr 2021 · 100.0%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2021
May 2021
Jun 2021

Notice history

Jun 2021

Rancher & Kubernetes Maintenance
  • Completed
    1. July 2021 at 18:15
    Completed
    1. July 2021 at 18:15

    Maintenance has completed successfully.

    Everything is up and running again

  • In progress
    1. July 2021 at 15:52
    In progress
    1. July 2021 at 15:52

    Maintenance is now in progress.

  • Planned
    30. June 2021 at 16:0
    Planned
    30. June 2021 at 16:0

    We are planning for a scheduled maintenance during that time for:

    • OS patching
    • Ingress controller upgrade
    • worker02 node replacement/rebuild
/sys Wartungsfenster
  • Completed
    25. June 2021 at 15:40
    Completed
    25. June 2021 at 15:40

    We completed this Maintenance yesterday at 23:30 but somehow forgot to update this status, sorry for that. Have a nice Weekend

  • In progress
    24. June 2021 at 16:0
    In progress
    24. June 2021 at 16:0

    Maintenance is now in progress.

  • Planned
    24. June 2021 at 16:0
    Planned
    24. June 2021 at 16:0

    As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/ or periods of degraded performance.

Major Gitlab Outage
  • Resolved
    Resolved

    This has been resolved last week during our maintenance window

  • Monitoring
    Monitoring

    We implemented a fix and currently monitoring the result. Another smaller fix is in the making, and we will inform you as soon as we know more.

  • Investigating
    Investigating

    We are currently investigating this incident.

OpenShift 4 Updates
  • Completed
    7. June 2021 at 19:31
    Completed
    7. June 2021 at 19:31

    Maintenance has completed successfully.

  • In progress
    7. June 2021 at 16:1
    In progress
    7. June 2021 at 16:1

    Maintenance is now in progress.

  • Planned
    7. June 2021 at 16:0
    Planned
    7. June 2021 at 16:0

    We are planning for a scheduled maintenance during that time. The OpenShift 4 clusters on AWS and cloudscale are going to be updated to the latest version.

Emergency RocketChat DB Maintenance
  • Completed
    1. June 2021 at 12:42
    Completed
    1. June 2021 at 12:42

    Maintenance has completed successfully. The DB has now more resources available.

  • In progress
    1. June 2021 at 12:29
    In progress
    1. June 2021 at 12:29

    Maintenance is now in progress.

  • Planned
    1. June 2021 at 12:26
    Planned
    1. June 2021 at 12:26

    Because of the Puzzle Workshop we have an unusual high number of users in the chat, which leads to CPU throttling and degraded performance. We will upgrade the deployment to improve performance. RocketChat will be affected for ~10min.

May 2021

/sys Wartungsfenster
  • Completed
    27. May 2021 at 20:19
    Completed
    27. May 2021 at 20:19

    Maintenance has completed successfully.

  • In progress
    27. May 2021 at 16:24
    In progress
    27. May 2021 at 16:24

    Maintenance is now in progress.

  • Planned
    27. May 2021 at 16:0
    Planned
    27. May 2021 at 16:0

    As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/ or periods of degraded performance.

Jenkins freestyle job configuration
  • Resolved
    Resolved

    We just resolved the issue!

  • Monitoring
    Monitoring

    We implemented a fix and currently monitoring the result.

  • Identified
    Identified

    We found the Plugin which causes the Issue. The Hudson Locks and Latches plugin is no longer compatible with the newest Jenkins Version.

    We will delete the plugin and restart Jenkins today at 12:30.

  • Investigating
    Investigating

    Saving freestyle job configuration changes does not work.

    We are currently investigating this incident.

Partial outage of PROD Rancher cluster worker 08
  • Resolved
    Resolved

    We just resolved the issue!

  • Identified
    Identified

    Cordoned, drained & rebooted node. The "flapping" behavior seems to be caused by the "broken" runC version we are currently running on our Rancher nodes: https://github.com/kubernetes/kubernetes/issues/45419#issuecomment-819574343

    [root@k8s-worker08 ~]# runc --version runc version 1.0.0-rc93 commit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec spec: 1.0.2-dev go: go1.13.15 libseccomp: 2.3.1

Jenkins Gitlab Plugin Update
  • Completed
    6. May 2021 at 16:7
    Completed
    6. May 2021 at 16:7

    Maintenance has completed successfully.

  • Planned
    6. May 2021 at 16:0
    Planned
    6. May 2021 at 16:0

    We are planning for a scheduled maintenance during that time. Jenkins will be restartet during this time.

  • In progress
    6. May 2021 at 15:59
    In progress
    6. May 2021 at 15:59

    Maintenance is now in progress.

Kubernetes Worker down
  • Resolved
    Resolved

    I checked the resource request/limits and optimized them a bit. Furthermore there is a new worker node running. I'm closing this incident all services are up and running.

  • Monitoring
    Monitoring

    We implemented a fix and currently monitoring the result.

  • Identified
    Identified

    The node is up and running again. But we need to identify what caused the problem, as earlier there was already a node outage. Current guess is, not enough memory, nodes are too crowed and resource management is probably not the best. We are adding a new node (maybe temporary), to mitigate the problem

  • Investigating
    Investigating

    Anonther Kubernetes Worker is unfortunatly down

Apr 2021

Unhappy MachineConfig Operator...
  • Resolved
    Resolved

    We just resolved the issue! We hit a bug in OpenShift 4.6.

  • Identified
    Identified

    The MachineConfig Operator on OCP4 AWS Prod is not happy, although he should be - it's Friday after all.

Nextcloud Server Downtime
  • Completed
    26. April 2021 at 16:3
    Completed
    26. April 2021 at 16:3

    The maintenance has completed successfully and Nextcloud is available again.

  • In progress
    26. April 2021 at 16:0
    In progress
    26. April 2021 at 16:0

    This maintenance is now in progress and we will scale the server momentarily.

  • Planned
    26. April 2021 at 16:0
    Planned
    26. April 2021 at 16:0

    The nextcloud upgrade to version 20 and a combined downsizing of it's resources has led to perceived slowness of the server. Therefore we are scaling the server to a bigger flavour. An downtime of less than 10 minutes is expected, beginning today at 18:00.

Jenkins OpenShift Slaves fail
  • Resolved
    Resolved

    We just resolved the issue!

  • Identified
    Identified

    openshiftv3slave-xxx are not starting because of GitLab auth problems. We are continuing to work on a fix for this incident.

/sys Wartungsfenster
  • Completed
    22. April 2021 at 20:14
    Completed
    22. April 2021 at 20:14

    Our maintenance has completed successfully.

    Please remember the Gitlab authentication change that happened tonight: https://info.puzzle.ch/posts/2021-03-05-gitlab-authentifizierung-nur-noch-via-sso/

    As usual, if you encounter any unexpected behaviour please contact us via our helpdesk.

  • Planned
    22. April 2021 at 16:0
    Planned
    22. April 2021 at 16:0

    As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/ or periods of degraded performance.

  • In progress
    22. April 2021 at 15:57
    In progress
    22. April 2021 at 15:57

    Our monthly maintenance is now in progress.

Jenkins OWASP Dependency-Check not working
  • Resolved
    Resolved

    NVD data is accessible again.

  • Investigating
    Investigating

    We are currently investigating this incident. The NVD data feeds are broken -> no data to check against.

Apr 2021 to Jun 2021

Next