Puzzle ITC - Partial outage of Vault – Incident details

All systems operational

Partial outage of Vault

Resolved
Operational
Started 3 months agoLasted 3 days

Affected

Puzzle Services

Partial outage from 8:53 AM to 10:13 PM, Operational from 10:13 PM to 7:18 AM

Vault

Partial outage from 8:53 AM to 10:13 PM, Operational from 10:13 PM to 7:18 AM

Updates
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    Vault was successfully restored to a known good db state of Thursday 30.05.2024 20:11.

    Secrets that were added between 20:11 and the next day around 08:30 (after which the instance was offline) are unfortunately lost.

    We apologize for the inconvenience.

    We are continuing to monitor the performance of the restored cluster.

  • Identified
    Identified

    We were not able to successfully restore a current version of the main vault instance.

    Please continue to use the read-only instance provided for the duration of the weekend.

    Please accept our sincere apologies for the outage.

  • Investigating
    Investigating

    We are currently experiencing an outage with our main vault cluster.

    A read-only copy is available at vault.aws.puzzle.ch

    Write operations are currently not possible. Any entries made on the read-only instance will be overwritten shortly.