Puzzle ITC - New Certificates that do not Match the deplyed key, Services won't start anymore – Incident details

Experiencing partial outage

New Certificates that do not Match the deplyed key, Services won't start anymore

Monitoring
Major outage
Started about 8 hours ago

Affected

Puzzle Services

Major outage from 4:20 AM to 12:00 AM

Collaboration

Major outage from 4:20 AM to 12:00 AM

Mail

Major outage from 4:20 AM to 12:00 AM

Jitsi

Major outage from 4:20 AM to 12:00 AM

Webmail

Major outage from 4:20 AM to 12:00 AM

Updates
  • Monitoring
    UTC
    Monitoring

    Most services have the new certificates and are happily running again. We are sorry for any inconvenience this incident caused, and we wish you a pleasant weekend.

  • Identified
    UTC
    Identified

    somehow the key of the new certificates was never uploaded with the certs which caused a missmatch. we are currently regenerating certificates and keys to have matching pairs and will run them out shortly

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.