We have continued to see successful ingests of CD2 data for all members since 3/31/2024.
Instructure has informed us that they have identified the source of the system errors and plan to implement changes to prevent such issues in the future.
Our support team will provide any future updates related to this incident.
Posted Apr 18, 2024 - 21:38 UTC
Update
We have continued to see successful ingests of CD2 data for all members since 3/31/2024.
This morning (4/9/2024) Unizin partnered with Instructure to conduct a load test aimed at replicating encountered errors. During this exercise, our engineers and Instructure were able to gather valuable information to help identify a root cause. Instructure is currently investigating the issue, and we will share any updates as soon as they become available.
We are happy to report that only an error rate of ~0.2% was seen from CD2, with all Unizin systems able to gracefully recover and succeed on pulling full CD2 dumps for each of our member institutions without manual intervention. As a result, we will resume our CD2 validation efforts in our SIT environments for members who have not yet migrated to CD2.
Posted Apr 09, 2024 - 20:35 UTC
Monitoring
We have continued to see successful ingests of CD2 data for all members since 3/31/2024.
We will continue to monitor our CD2 ingest processes as we prepare to enable more ingests to production and SIT. Collaboration with Instructure to resolve these issues is ongoing.
Posted Apr 03, 2024 - 17:38 UTC
Update
All production UDP raw CD2 BigQuery and GCS marts are now updated with 3/31/2024 data for each tenant. Tonight (4/1/2024), all tenants that have migrated to CD2 in production will have their Context Store updated with CD2 data from at least 3/31/2024.
Currently, we have suspended all CD2 data imports into SIT BigQuery and GCS marts in order to alleviate some pressure on the CD2 API. We will also be suspending new CD2 Context Store imports for tenants currently undergoing CD2 validation until this issue can be resolved.
We will continue to monitor the situation around the issues we are seeing with the CD2 API. More updates will be provided as we know more and implement any fixes.
Posted Apr 01, 2024 - 20:52 UTC
Update
Update: Unizin is continuing to work with Instructure to identify and remediate errors coming from the CD2 API. While some UDP tenants successfully updated early on the morning of March 30th, the inability to successfully pull complete CD2 data for all Unizin members is still under investigation.
Posted Mar 30, 2024 - 17:39 UTC
Identified
Unizin is currently receiving errors when attempting to access various files via the CD2 API resulting in the inability to pull complete CD2 data dumps. This may result in stale UDP context data or raw CD2 marts. We have identified where the errors are being returned and are working with Instructure support to resolve as quickly as possible.
Posted Mar 29, 2024 - 02:09 UTC
This incident affected: UDP - Batch ETL and UDP - Marts.