Puzzle ITC - Rocket Chat Missing Avatar and Channel Pictures – Incident details

All systems operational

Rocket Chat Missing Avatar and Channel Pictures

Resolved
Operational
Started 8 months agoLasted 1 hour 15 minutes

Affected

Puzzle Services

Operational from 11:45 AM to 1:00 PM

Collaboration

Operational from 11:45 AM to 1:00 PM

Rocket.Chat

Operational from 11:45 AM to 1:00 PM

Updates
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    We restored the missing uploaded files that were also deleted.
    Our last fallback data sync was around 05:40 in the morning. There is a time window between 05:40 and around 10:00 (I don't have the exact time anymore) where newly uploaded files in your chats are gone. Also if you changed your avatar in this time frame it might be reverted to the old one. So if you are missing a file in a chat, you have to resend it to your peer.

    My apologies, have a nice rest of the day and enjoy your weekend

  • Identified
    UTC
    Identified

    Due to rash actions on my side I accidentally deleted RocketChat Avatar Pictures and some File Uploads from our S3 Bucket.

    What happened:
    The Intention was to delete a connection profile in my S3 Client (to prevent things exactly like this issue here happening) but accidentally started deleting the Bucket (or part of it) instead of the connection because I haven't read the button correctly.

    Current State:
    - All avatars are already restored.
    - We are currently investigating some missing uploads, a very small percentage of upload files were also deleted and we are in the process to restore them from our backup instance.