We're pleased to announce that the incident has been successfully resolved. All members have continued to have successful and on time ingests. Regular monitoring will continue to ensure ingest stability. Please refer to our status page for updates on any future incidents.
Posted May 15, 2024 - 14:55 UTC
Monitoring
A fix was deployed yesterday afternoon to address ingestion failures and extended running processes. As a result, all member ingestions last night were successful without the need for manual intervention. Currently, all member context stores are updated with fresh data, and all UDP marts will refresh at their regularly scheduled times. We will continue to monitor the situation to prevent any additional occurrences.
Posted May 14, 2024 - 15:44 UTC
Investigating
Over the weekend, we experienced another increase in ingestion failures and extended running processes, which resulted in outdated ingestions for multiple members.
We are actively working to address this issue and will provide an update as soon as possible.
Posted May 13, 2024 - 15:46 UTC
Update
The occurrence rate of this issue has significantly reduced, now only affecting a few tables for a small number of members. While all context store tables have been updated with new data, the downstream BigQuery marts (context_store_* and mart_* datasets) may still display data that's over 24 hours old for these affected tables.
Posted May 09, 2024 - 17:41 UTC
Monitoring
A fix has been implemented and we will be monitoring the results over the next ingest.
Posted May 08, 2024 - 18:43 UTC
Identified
The source of the issue has been identified as a resource constraints on the processes loading data into the context store tables. We are working on implementing a fix.
Posted May 08, 2024 - 18:33 UTC
Investigating
We are currently investigating an issue causing a few various tables to have significantly longer ingestion times into the context store. We have been able to get most tables to update where they should be no more than 24 hours behind. Some of the impacted tables are, but not limited to: