GCP Support has provided clarification on the root cause of this incident with Storage Transfer Service:
> Our product engineer mentioned that at the time you reported the case, an internal service associated with the transfer jobs was not not responding correctly, leading them to make an internal change to increase the number of workers in the Storage Transfer Service tool to mitigate the issue. This hopefully will prevent the same issue in the future.
Subsequently, Unizin has observed successful completions of STS jobs within expected durations since July 11th. We appreciate your patience as we collaborated with GCP Support to determine the underlying cause of this incident.
Posted Jul 24, 2024 - 19:12 UTC
Update
Over the weekend, GCP's Storage Transfer Service maintained its standard performance, ensuring successful ingestion of Canvas Data for all members. We are currently awaiting feedback from GCP Support regarding the issues we encountered. They have escalated the matter to their Cloud Storage Product Engineering team, and we will promptly share any insights they provide with you.
Posted Jul 15, 2024 - 15:09 UTC
Update
We have observed improved performance from GCP's Storage Transfer Service during tonight's Canvas Data ingestion for all members. Each member's UDP context store and marts will be updated with fresh Canvas Data. We are actively monitoring the situation and will promptly notify you of any recurring performance issues. Currently, we are awaiting an update from Google Support regarding the root cause and anticipated resolution of the issue.
Posted Jul 11, 2024 - 06:16 UTC
Update
We have began to observe the performance issue with GCP's Storage Transfer Service occur for every member whose scheduled Canvas Data ingestion has triggered so far this evening. We have notified GCP support of this and are awaiting their internal Storage support team's investigation. Currently it appears that this will affect the ingestion of new Canvas Data into all every members' UDP tonight. We will update as soon as we have have more information.
Posted Jul 09, 2024 - 22:15 UTC
Investigating
We are currently encountering significant performance issues with a Google Cloud Platform managed service responsible for data ingestion from Canvas into our cloud environments. While we engage with Google Support to address these delays, we are actively exploring alternative solutions. As a result, exports from Canvas used for batch ingestion, BigQuery marts, and GCS marts may experience delays for some institutions. Presently, the impact on institutions has been minimal, though this situation remains fluid.
Posted Jul 09, 2024 - 20:14 UTC
This incident affected: UDP - Batch ETL and UDP - Marts.