Puzzle ITC - Notice history

All systems operational

Container Platforms
Operational

OpenShift 4 Cluster cloudscale.ch - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

OpenShift 4 Appuio cloudscale.ch - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

OpenShift 4 Cluster LPG1 cloudscale.ch - Operational

100% - uptime
May 2024 · 100.0%Jun · 99.78%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024
Puzzle Services
Operational

Puzzle Website (puzzle.ch) - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Mail - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Rocket.Chat - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Taiga - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

PuzzleTime - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Wiki - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Puzzle SSO (Keycloak) - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

CodiMD - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

OpenProject - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Nextcloud - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

PuzzleSkills - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

GitLab - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Jenkins - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Quay (registry.puzzle.ch ) - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Grafana - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Puzzle News Site (news.puzzle.ch) - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Cryptopus - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Artifactory - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

SonarQube - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Puzzle Meet - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Zammad (help.puzzle.ch) - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Sentry - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

CardDAV - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Webmail - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Argo CD - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Youtrack - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

OKR - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Vault - Operational

99% - uptime
May 2024 · 99.39%Jun · 99.07%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Loki - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024
Office
Operational

Network Office-Bern - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Google Meet Hardware - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Telephone Setup - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024
Cloud and Datacenter
Operational

cloudscale.ch - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Amazon AWS - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Notice history

Jul 2024

/sys Wartungsfenster
  • Completed
    25. July 2024 at 21:4
    Completed
    25. July 2024 at 21:4

    Our maintenance has concluded and services are fully usable again.

  • In progress
    25. July 2024 at 16:0
    In progress
    25. July 2024 at 16:0
    Maintenance is now in progress
  • Planned
    25. July 2024 at 16:0
    Planned
    25. July 2024 at 16:0

    As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/ or periods of degraded performance.

Quay registry currently not available
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating
    We are currently investigating this incident.
OpenShift 4 Monthly Maintenance
  • Completed
    22. July 2024 at 18:33
    Completed
    22. July 2024 at 18:33
    Maintenance has completed successfully.
  • In progress
    22. July 2024 at 16:0
    In progress
    22. July 2024 at 16:0
    Maintenance is now in progress
  • Planned
    22. July 2024 at 16:0
    Planned
    22. July 2024 at 16:0

    As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/or periods of degraded performance.

Upgrade - PuzzleSkills
  • Completed
    18. July 2024 at 8:26
    Completed
    18. July 2024 at 8:26
    Maintenance has completed successfully.
  • In progress
    18. July 2024 at 6:30
    In progress
    18. July 2024 at 6:30
    Maintenance is now in progress
  • Planned
    18. July 2024 at 6:30
    Planned
    18. July 2024 at 6:30
    We are planning for a scheduled maintenance during that time.
Security Update for Gitlab
  • Completed
    10. July 2024 at 20:58
    Completed
    10. July 2024 at 20:58

    The upgrade is completed and Gitlab is available normally to all users again.

  • In progress
    10. July 2024 at 20:45
    In progress
    10. July 2024 at 20:45
    Maintenance is now in progress
  • Planned
    10. July 2024 at 20:45
    Planned
    10. July 2024 at 20:45

    Due to a critical vulnerability that surfaced in the Gitlab server software we will be doing an emergency patch. Gitlab will be unavailable for around 10 minutes.

    We apologize for the short notice as well as for the disruption coming up.

Jun 2024

/sys Wartungsfenster
  • Completed
    27. June 2024 at 21:18
    Completed
    27. June 2024 at 21:18
    Maintenance has completed successfully.
  • In progress
    27. June 2024 at 16:0
    In progress
    27. June 2024 at 16:0
    Maintenance is now in progress
  • Planned
    27. June 2024 at 16:0
    Planned
    27. June 2024 at 16:0

    As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/ or periods of degraded performance.

Migration MX Servers
  • Completed
    27. June 2024 at 16:0
    Completed
    27. June 2024 at 16:0
    Maintenance has completed successfully
  • In progress
    27. June 2024 at 15:0
    In progress
    27. June 2024 at 15:0
    Maintenance is now in progress
  • Planned
    27. June 2024 at 15:0
    Planned
    27. June 2024 at 15:0

    We will be migrating our inbound mail infrastructure (mx servers) today, starting at 5pm. During this time, there might be slight delays of a couple of minutes for inbound messages. Thank you for your understanding.

OpenShift 4 Monthly Maintenance
  • Completed
    24. June 2024 at 16:18
    Completed
    24. June 2024 at 16:18

    The upgrade to 4.15.17 has completed successfully. All services are operational.

  • In progress
    24. June 2024 at 15:0
    In progress
    24. June 2024 at 15:0
    Maintenance is now in progress
  • Planned
    24. June 2024 at 15:0
    Planned
    24. June 2024 at 15:0

    As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/or periods of degraded performance.

OpenShift LPG/Staging Cluster Network Outage
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    Update

    Still unclear, what the root cause is. We are analyzing possible workarounds.

  • Investigating
    Investigating

    We are currently investigating this incident. During the cluster upgrade of the LPG/Staging cluster, the network pods are crashlooping.

Outage of files.puzzle.ch
  • Resolved
    Resolved

    We are happy with the results of the mitigation and observed a stable operation over the last hours.

  • Monitoring
    Update

    We took corrective actions and we are monitoring the results. files.puzzle.ch is available again.

  • Monitoring
    Monitoring

    We took corrective actions and we are monitoring the results. files.puzzle.ch is available again.

May 2024

Partial outage of Vault
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    Vault was successfully restored to a known good db state of Thursday 30.05.2024 20:11.

    Secrets that were added between 20:11 and the next day around 08:30 (after which the instance was offline) are unfortunately lost.

    We apologize for the inconvenience.

    We are continuing to monitor the performance of the restored cluster.

  • Identified
    Identified

    We were not able to successfully restore a current version of the main vault instance.

    Please continue to use the read-only instance provided for the duration of the weekend.

    Please accept our sincere apologies for the outage.

  • Investigating
    Investigating

    We are currently experiencing an outage with our main vault cluster.

    A read-only copy is available at vault.aws.puzzle.ch

    Write operations are currently not possible. Any entries made on the read-only instance will be overwritten shortly.

GitLab runners
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Identified
    Identified

    Ein Teil der GitLab runner sind im Moment nicht erreichbar. Dies äussert sich mit dieser Fehlermeldung:
    This job is stuck because the project doesn't have any runners online assigned to it.

    Das betrifft alle builds ohne Tags oder Builds mit diesen Tags: openshift-runner, openshift4-runner, medium,small,dagger

OpenShift 4 Monthly Maintenance
  • Completed
    27. May 2024 at 17:3
    Completed
    27. May 2024 at 17:3
    Maintenance has completed successfully.
  • In progress
    27. May 2024 at 16:0
    In progress
    27. May 2024 at 16:0
    Maintenance is now in progress
  • Planned
    27. May 2024 at 16:0
    Planned
    27. May 2024 at 16:0

    As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/or periods of degraded performance.

/sys Wartungsfenster
  • Update
    27. May 2024 at 9:3
    Completed
    27. May 2024 at 9:3
    Maintenance has completed successfully.
  • Completed
    23. May 2024 at 20:13
    Completed
    23. May 2024 at 20:13
    Maintenance has completed successfully.
  • In progress
    23. May 2024 at 16:0
    In progress
    23. May 2024 at 16:0
    Maintenance is now in progress
  • Planned
    23. May 2024 at 16:0
    Planned
    23. May 2024 at 16:0

    As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/ or periods of degraded performance.

Jenkins ArgoCD migration
  • Completed
    7. May 2024 at 18:22
    Completed
    7. May 2024 at 18:22

    Die Migration wurde wegen technischen Problemen abgebrochen.

  • In progress
    7. May 2024 at 15:30
    In progress
    7. May 2024 at 15:30
    Maintenance is now in progress
  • Planned
    7. May 2024 at 15:30
    Planned
    7. May 2024 at 15:30

    Wir werden unsere Jenkins Instanz während dieser Zeit vom alten auf das neue ArgoCD Setup migrieren. Jenkins wird während diesem Wartungsfenster nicht erreichbar sein.

May 2024 to Jul 2024

Next