Malformed JSON files

Incident Report for Panther Labs

Postmortem

We’d like to share a follow-up on the data ingestion issue that occurred on July 25, 2025 between 1:30 and 6:30pm ET.

The root cause was an update to field discovery functionality that was incorrectly deployed before end-to-end testing was complete. This resulted in certain JSON files being malformed, meaning they could not be properly ingested into Panther.

Not all Panther customers were affected. The impact was limited to data:

Ingested through an S3 bucket
Parsed by a schema that has field discovery enabled
- This includes custom schemas, as well as Panther-managed GitHub schemas. Field discovery was enabled for managed GitHub schemas for most Panther customers.

No data was lost, and we are currently working to reprocess affected files. We will follow-up directly with affected customers when reprocessing is complete, which we expect to be tomorrow, July 29.

We apologize for the inconvenience caused by this issue and thank you for your patience throughout remediation.

Posted Jul 28, 2025 - 07:45 PDT

Resolved

We have deployed a fix for the issue that was causing JSON log files to be written in a malformed format, and have confirmed that everything is operating as expected.

The issue was active from approximately 1:30 PM ET until the fix was fully deployed to all customers around 6:30 PM ET. During this time, malformed log files were not ingested, which may have temporarily impacted downstream detections or processing.

We believe the affected data can be safely reingested and are currently evaluating the best approach. We will follow up with an additional update once the reingestion plan is finalized.

Thank you for your patience as we work through this.

Posted Jul 25, 2025 - 15:40 PDT

Identified

We have identified the root cause of this malformed JSON log issue and are deploying a fix across customer accounts now.

We will follow up as soon as we confirm that the issue has been resolved.

Posted Jul 25, 2025 - 14:02 PDT

Investigating

We are currently investigating an issue where some JSON log files are being written in a malformed state. This may impact log ingestion and downstream detections and searches that rely on these files.

We will update this incident as our investigation and remediation process continues.

Posted Jul 25, 2025 - 13:07 PDT

This incident affected: Data Ingestion into Panther (Log Processing).