The WisTransPortal System - TEST

2 0 2 0

• May 13, there will be a low-risk maintenance window from 9pm-midnight, to perform an update to the Crash Resolve System. ~ COMPLETED
• May 10th UW Network Services is planning to perform maintenance on the data center hardware firewall from 5am-6:30am on Thursday. There is no anticipated disruption to the network during this maintenance.

  See Outage #10590 for a complete list of affected services

  This work will introduce a patch to fix an intermittent bug that has been causing disruptions on UW firewalls recently. ~ COMPLETED

• April 29 There will be a TIA maintenance on Wednesday, between 8:00 PM and 11:00 PM. TIA will not be available for <15 min during the maintenance window. ~ COMPLETED
• April 26 WisTransPortal server maintenance window scheduled from 6am to 3pm ~ COMPLETED
• March 22 WisTransPortal server maintenance window scheduled from 6am to 10am ~ COMPLETED
• March 9 TOPS planned an emergency maintenance on the WisTransPortal Web server to fix an network issue. The WisTransPortal was unavailable for 6 minutes total between 11:32pm and 11:56pm.
• March 4 A low-risk maintenance window, from 9pm-midnight, to perform an update to the Crash Resolve System. This maintenance window will implement several updates and improvements to the Resolve System “notice to report” letters along with several other minor JIRA fixes. ~ COMPLETED
• March 2 There was a DoIT network issue that took down access to the WisTransportal and all its applications. The outage started at 11:40 am and was resolved at 12:53 pm. Campus Outage Link
• February 23 WisTransPortal server maintenance window is scheduled from 6am to 3pm. ~ COMPLETED
• February 5 there will be a Teleste Receiver Station Update beginning at 2am and ending at 6 am. This work is to alleviate an issue with the Receiver Stations keep crashing (issue #22). During the update, one of the LINK server will not be available, which means that the few of the video tours will not be available in LINK for short period of time. ~ COMPLETED
• February 5 A low-risk maintenance window is scheduled from 9pm-midnight to perform an update to the Crash Resolve System. ~ CANCELED
• January 26 WisTransPortal server maintenance window scheduled from 6AM to 3PM. ~ COMPLETED
• January 14 There will be a Teleste Receiver Station Update beginning at 2am and ending at 6am. During the update, the LINK server will not be available, which means that some video tours will not be available in LINK for short period of time. ~ COMPLETED
• January 12 DoIT Network Services is planning to perform maintenance 12am-8am on the data center hardware firewall that will have a significant impact on a number of DoIT-hosted services. The scope of impact for this work includes many systems, including the WisTransPortal. Some systems may only experience disruption for a few minutes, but others may be impacted for an hour or more while system and application admins bring them back online, if necessary. DoIT list of affected services ~ COMPLETED

2 0 1 9

• December 22 WisTransPortal server maintenance window scheduled from 6AM to 9AM. ~ COMPLETED
• December 8 WisTransPortal server maintenance window scheduled from 6AM to 9AM. ~ COMPLETED
December 4 DANE InterCAD server move had some issues so DANE InterCAD was down from 1:00pm to 1:45pm
• December 4 DANE InterCAD server move from 12:00pm - 12:15pm ~ COMPLETED
• November 28: @ 3:03AM the virtual environment storage, 1 volume filled max reserved space and caused some systems to go into standby mode. This affected some automated services to include Crash Resolve and Winter Roads 511. The storage was corrected around 9:40AM. Systems recovered at 11:04am and all is running well at this time.
• WisTransPortal server maintenance window scheduled for Sunday, November 24 ~ COMPLETED
• November 6 11:45 AM - UW Network firewall started experiencing issues starting @ 11:18 PM on Nov 5. This event slowed connection to the WisTransPortal and all applications Campus Outage. UW DoIT Firewall failed to secondary and things came back up. This caused some stale sessions and prolonged outages being experienced until about 9am. Systems were good and then the UW DoIT firewall switched to primary and caused WisTransPortal to respond slowly and LINK video to fail starting aroun 9:45am. We are still working to clear LINK video to smooth operation.
• Nov 7 Update - I continue to have to reset LINK about every 30 minutes. I try to keep up but you may have to wait a bit and then refresh until we get this settled with the DoIT Network team.
• Nov 8 update - Still dealing with streaming video issues every 30 mintues. No new break throughs.
• Nov 8 3:30 pm update - actively working on this issue with Palo Alto techs and the UW DoIT Network team.
• Nov 8 5pm update - Issue resolved.
• October 29-30 A Teleste update occured from 2am to 6am. ~ COMPLETED
• WisTransPortal server maintenance window scheduled for Sunday, October 27 ~ COMPLETED
• WisTransPortal server maintenance window scheduled for Sunday, September 29 ~ COMPLETED
• WisTransPortal server maintenance window scheduled for Sunday, August 25 ~ COMPLETED
• 8/22/2019 - The Crash Data Analysis Tools locked up this morning around 9:50am causing the WisTransPortal to be unresponsive. WisTransPortal was back at 9:55am but Crash Data Tools is still unresponsive. It appears to be an issue with the GIS /WISLR map at this time.

  Systems were back at 10:14 AM

• WisTransPortal server maintenance window scheduled for Sunday, July 21 ~ COMPLETED
• On July 9, the WisTransPortal became unresponsive between 8:38 AM and 8:55 AM. UW Campus was experiencing a network issue.
• WisDOT will upgrade the Teleste Video System on the scheduled time below:
  • Early Monday morning, July 8th from 2:00 am to 2:00 pm
  • Early Tuesday morning, July 9th from 2:00 am to 2:00 pm

  During the upgrade, there will be intermittent LINK video outages throughout the process

• On June 27, the WisTransPortal became unresponsive between 4:08 PM and 4:11 PM. We are looking into the cause.
• On June 6, 2019, the WisTransPortal became unresponsive starting at 10:04 AM. Services were back by 10:14 AM. We are investigating the issue.
• On Tuesday, May 28, 5:15 AM, TOPS Lab was notified that half the video was down. One of the servers did stop over the weekend again. This is a re-occuring issue and we are still working on it with Wowza Media support. Video was restored by 7:55 AM.
• On Wednesday, May 22, emergency maintenance on some WisTransPortal servers. Reboots will have all TMC apps offline briefly. Window is from 9-10PM., ~ COMPLETED
• Teleste Maintenance work will start late Sunday night, May 5th, 2019 and last until Friday afternoon, May 10th, 2019. There will be intermittent outages with LINK Video throughout the process but they will be limited due to the redundant nature of the system. NVRs will also be upgraded during this process and recordings will be temporarily interrupted for 30-50 cameras at a time as each NVR is upgraded. ~ CANCELED
• WisTransPortal server maintenance window scheduled for Sunday, April 28, ~ COMPLETED
• InterCAD maintenance window scheduled for Wednesday, April 24, from 12-12:30 pm. ~ COMPLETED
• 4/22/2019 6:24 AM - A number of the LINK video tours were reported down. One of the Media servers locked and needed restarting. Tours were restored by 7:32 AM.
• 4/16/2019 8:55 AM - The WisTransPortal became slow and unresponsive. Web site was back at 9:11 AM. We are looking into the issue.
• 4/15/2019 12:00 PM - 8 Video streams started failing due to a service error on one of Media servers. There were issues on the backend video systems as well during this time. Service and video was restored by 9:15 PM.
• 4/13/2019 11:00 PM - Storage maintenance resulted in some stalled services starting at 9:45 AM. We continue to work on the issues. TPIMS, WRS, and LCS updates are some of the updates not going out from TOPS.
• 4/14/2019 6:32 PM - all systems were back up.
• 4/13/2019 Storage maintenance from 6AM to 12PM. No systems or applications will be taken down. ~ COMPLETED
• 4/4/2019 11:14 AM - At 12:04 AM the Dane county InterCAD puller service failed due to DNS issues. The cause is being investigated. Puller was restored at 9:35am
• VSPOC Calendar update - Sunday March 24th @ 4:00pm. ~ COMPLETED
• WisTransPortal server maintenance window scheduled for Sunday, March 24, COMPLETED. - NO DOT Crash Resolve (CRU) systems updated or rebooted due to the upcoming election code freeze.
• TMP 2.0 roll-out on Wednesday, March 20, from 12p-1p. ~ COMPLETED
• WisTransPortal storage maintenance on March 17. Window is from 10am - 6pm. No production services will be affected. ~ CANCELED
• CRASH Resolve patch on March 17. Window is from 6am - 9am. Services will be unavailable for short restarts of Tomcat during this time. ~ CANCELED
• 3/13/09 - At 10:41am, TOPS was notified that V-SPOC was unresponsive. Services had locked up after a search. VSPOC app was restarted and services were back at 10:47am.
• 3/9/2019 8:20 PM - at 3:15 AM, one of the volumes on the WisTransPortal virtual storage reached capacity and production systems started being unresponsive at times. These servers include the scheduled services, Storm Reports, and Resolve. Issue was resolved at 9:20PM.
• 3/6/2019 - At 2:54 PM, the WisTransPortal was reported to be responding really slowly by the CRU. This appears to be due to an application they were testing with issues returning information. The application was restarted and systems were back to normal at 3:13pm.
• Sunday 3/3 at 8am - 9am is emergency maintenance on the main Web server. This will affect all applications when system is rebooting. Maintenance may be canceled due to weather. ~ Completed
• Friday 3/1 at 11:30am is emergency maintenance for the LINK video system. This is a performance update. ~ Completed
• on 2/28/2019, the WisTransPortal web server dropped network connectivity between 11:24:24am and 11:35:27am. The issue was caused by system load. Increased resources are being added to the virtual environment to compensate.
• The 9am - 3pm portion of Sunday, Februrary 24 maintenance, was canceled due to weather conditions.
• Wednesday 2/20 between 8pm-10pm: LCS update ~ Completed
• Wednesday 2/20 at noon: LINK update ~ Completed
• Monday 2/18 at noon: InterCAD update ~ Completed
• Monday 2/18 at 10am: Recreate RX station for Dane County and SW Tour ~ Completed
• On February 2nd, some WTP applications became unresponsive around 4:00am. This included TIA and WRS. The issue was with the database being misconfigured after early maintenance on the backups. Sytem was restored at 9:05am on 2/2.
• Today, January 29, starting at 7:10am, the WisTransPortal became slow and often unresponsive. Things were back at 8:48am. The WTP web server was locked up and needed to restart.
• WisTransPortal server maintenance window is scheduled for Sunday, January 27 - completed
• LINK video maintenance on January 3rd, 2019 from 12:30 - 1:30pm ~ COMPLETED

2 0 1 8

• WisTransPortal server maintenance window scheduled for Sunday, December 16, - completed
• On December 8th @ 8:45pm, DoIT Network team contacts TOPS about network issues at the University and we are one of the 3 or so other high traffic departments flagged. They want to shut down our traffic for 10-15 minutes to help identify the issue. UW Madison IT Outages
  • Saturday Dec 8 @ 9:30pm(ish) — DoIT team takes WisTransPortal traffic offline for 15 minutes. Steven P and I have communicated with DOT and TMC about the outage and have received the verbal okay.
  • Sunday Dec 9 @ 2:30pm — DoIT network team contacts me to see if we would be willing to move off to a private network. After discussions, DoIT decides this in undesirable and cancels the request. No issues on the WisTransPortal up through this time to my knowledge.
  • Sunday Dec 9 @ 7:15pm — LINK video has stopped. After looking into the issue, it appears that the same issue we had when first migrating to the firewall has returned. Sessions need to be cleared regularly to keep video going.
  • Monday Dec 10 @ 8:06am — The first notice, besides video (6:18am), saying that the WisTransPortal is unresponsive at times – Crash Resolve System (CRU).
  • Monday Dec 11 @ 11:04am — DoIT implements a fix that cuts down the network interface processing from 90%+ down to under 30%.
  • Monday Dec 12 @ 12:30pm — WisTransPortal web site and internal communication seems to have stabilized although some external traffic still has issues.
• On November 20th @ 12:00 — the DoIT Data Center team worked with TOPS staff and DOT staff in migrating the firewall. This affected all WisTransPortal networks and applications. The maintenance window extended to 5:30 pm with intermittent outages on applications.
  • 11/26 UPDATE. Working through dropped connections through the firewall to include the issues with LINK video, LCX XML, WRS XML, and InterCAD web services.
• WisTransPortal server maintenance window scheduled for Sunday, October 21, - completed.
  
• 10/19/2018 - At 11:03 AM, TOPS was notified that the DMV was having issues importing crash data. After futher investigating, we found that this started at 10:42am and lasted till 12:19. Other IPs were hitting TOPS servers except the mainframe source. It appears that the issue was between the DOT and UW-TOPS.
• 10/3/2018 - At 9:09 am, initial reports of the WisTransPortal being down were reported on Crash Resolve. The web site was responsive again by 9:20am. Issue is still being investigated.
  Update - 2:25pm / The issue was caused by the main WisTransPortal database being too busy. Project was adjusted so that more resources were left available.
• WisTransPortal server maintenance window scheduled for Sunday, September 23, - completed.
• 9/14/2015 - At 12:15 PM, the WisTransPortal Database was restarted to release connections that have been causing issues for a number of weeks. Most applications were unavailable for 8 minutes during the reboot to include TIA, VSPOC, LCS, and TMP.
• WisTransPortal server maintenance window scheduled for Sunday, August 19, - completed.
  The Wisconsin DOT has core router maintenance was from 2pm - 6pm.
• On August 15, @ 9:14am, TOPS lab was notified that the WisTransPortal was responding slowly or down. This happened again sometime between 10 and 10:30am. The cause of the slowness has not yet been identified but we continue to investigate.
  Update 8/16: Still looking into the cause of the slowness that was experienced. A ticket has been created with the DoIT Network team. So far, it appears to be a network load issue between clients and WisTransPortal.
• WisTransPortal server maintenance scheduled for Sunday, July 22, - completed.
• Please note that WisDOT will perform an emergency network work tonight (Thursday, July 12th) beginning at 7pm and ending at 10pm.
• Please note that WisDOT will continue to perform network maintenance (fiber splicing) at multiple locations in Madison area. This will possibly affect LINK Video.
  • Monday, June 18, 2018 beginning at 7pm and ending at 5am the following morning
  • Tuesday, June 19, 2018 beginning at 7pm and ending at 5am the following morning
• WisTransPortal server maintenance scheduled for Sunday, June 17 - Completed.
• WisDOT will continue to perform network maintenance (fiber splicing) at multiple locations in Madison area. This will possibly affect LINK Video. The work is scheduled during overnight hours on three consecutive nights next week after Memorial Day:
  • Tuesday, May 29th at 7:00 PM to Wednesday, May 30th at 5:00 AM
  • Wednesday, May 30th at 7:00 PM to Thursday, May 31st at 5:00 AM
  • Thursday, May 31st at 7:00 PM to Friday, June 1st at 5:00 AM
• AT 10:50 PM on 6/5/2018, TOPS was notified that TMC Operators were unable to connect to the WisTransPortal web site. The WisTransPortal virtual environment ran out of space in one of the configured storage pools and many VMs locked up. Some became corrupted so they could not be started again. Web server stopped at 10:22 PM on 6/5/2018. Web server was back online at 4:17 pm on 6/6/2018.
• WisTransPortal server maintenance scheduled for Sunday, May 20 - completed.
• 2018-05-09 - At 6:50 AM, TOPS was notified that TMC Operators were unable to connect to TIA. Subsequent reports verified that all WisTransPortal applications were down. Problem was determined to be caused by a UW Firewall acting irregular. Systems were restored by 10:00 AM
• WisTransPortal server maintenance scheduled for Sunday, April 22, - completed.
• 2018-04-11 - at 1:00 AM, the DANE interCAD data collector stopped. A required update was pushed out for the VPN client. The update was installed and DANE was working again at 10:00 AM.
• 2018-04-10 - at 10:30 AM, TOPS staff notice a slowing in performance on WisTransportal and then at 10:48, Web services were down. One of TOPS storage thresholds were maxed out and locked up a number of servers. By 11:08 systems were restored when the storage configuration was adjusted.
• 2018-04-05 - At 8:35 AM, TOPS determined that the InterCAD Reporting had stopped around 12:40 AM. Services were restarted and system was back up by 8:35 AM. Staff are investigating the issue.
• 2018-04-02 - At 11:25 AM, TOPS was notified that the WisTransPortal was not responding. It was quickly determined that VSPOC was locked and preventing further communication with the server. After a restart @ 11:35am, applications were available again.
• WisTransPortal server maintenance scheduled for Sunday, March 25, - completed.
• 2018-03-15 - At 9:37 AM, Dane County InterCAD XML pull quit working. The VPN client had disconnected again. Connection was restored at 12:00 PM.
• 2018-03-14 - At around 7:00 AM this morning, Dane County InterCAD XML pull quit working. The VPN client had failed due to a update request. The client failed to update. By 1:30PM, it started working again.
• 2018-03-12 - At 9:07 AM, TOPS was notified that the LCS map was unavailable. The application serving up ArcGIS for LCS had crashed. At 5:00 PM, it was running again.
• WisTransPortal server maintenance scheduled for Sunday, February 25 - Completed.
• 2018-02-19 - AT 4:00 PM it was noted that delayed and dropped connections to data was occurring more and more frequently from WisTransPortal applications. This was affecting most applications including TIA, VSPOC, and InterCAD. A reboot of the database server was done at 1:00 AM on 2018-02-20 to alleviate the issue. A long term solution is under development.
• WisTransPortal server maintenance scheduled for Sunday, January 21 - Completed.

2 0 1 7

• Thursday afternoon, October 26th, the WisTransPortal became unresponsive at 1:57PM. the VSPOC application had locked up resources. Services were restarted and the WisTransPortal was back by 2:00 PM
• WisTransPortal server maintenance window scheduled for Sunday, October 22 - partially completed. SAN virtual volume work did not allow for complete server patching in the alloted maintenance window.
• Thursday morning, September 14, the WisTransPortal wsa reported to be responding slowly. First report came in around 9:49am on the TMP system. Resolve was reported slow around 10:00am. After combing through the logs, going through all performance history, and testing applications, nothing turned up as cause on any of the applications or equipment that TOPS has control of. We continue to monitor. Everything was responding normally at 10:00am at the University. No other reports outside of the university since 10:21AM.
• WisTransPortal server maintenance window scheduled for Sunday, September 24 - Completed.
• Tuesday morning, September 5, the DOT Resolve system was reported to be down starting around 9:06 AM. Further tests showed that all of transportal.cee.wisc.edu was slow and/or unavailable. By 9:15 AM, Resolve was reported running again. TOPS staff are still looking into the cause
• Tuesday morning, August 29, no new crash data extracts were created. This was due to an expired software license at midnight of August 28. License was updated on production by 9:20 AM. UAT and TEST were done by 11:45 on 8/29.
• Monday, August 28. At 8:44AM, it was reported that the WisTransPortal became very slow and sometimes unresponsive. At 9:00AM the Apache service was restarted and things were immediately better. We are still looking into the cause.
• On Tuesday, August 15, the LINK video was scheduled to be down from 8:00 - 8:30PM. Some issues came up during the process and the video was down till 9:30PM - 1hour and 30 min. This was a major security upgrade to the backend media server.
• On Monday, July 31, many WisTransPortal maps were offline including WisLR Maps, ITS Inventory, and GLRTOC. We are working with some licensing issues and hope to have those back up in the afternoon of Tuesday, August 1.

  UPDATE - still working on the license issue (10:54AM 8/2)

  Maps were back up by 12:14PM on 8/2.

• On Friday, July 28, around 6:20 am, the WisTransPortal main database had an issue with the log file space. The system was cleared and applications were back up by 7:30am.
• WisTransPortal server maintenance scheduled for Sunday, July 30 - Completed.
• The LINK video camera feeds are down due to some persistent network issues. The problem has been on going all week. We expect to have it repaired the late evening of 6/29/2017.
• The WisTransPortal servers were experiencing delays and slowness around 10:13am this morning (6/21/2017). The delays were related to problems on the MV4000 Crash Data Resources. Systems were restored by 10:45am. Cause is being investigated.
• 5/1/2017 - > Bulk emails out of WisTransPortal are being delayed in the AM (TIAs). - UW Madison IT Outages
• WisTransPortal server maintenance scheduled for Sunday, April 30th - Completed.
• 4/26/2017 - > WisTransPortal applications were experienceing some slowness between 9:50 AM and about 10:30 AM. DOT Crash systems were not affected. Still looking into the cause.
• WisTransPortal server maintenance scheduled for Sunday, April 23rd - Completed.
• 4/21/2017 - > WisTransPortal applications were experienceing some slowness between 8:50 AM and about 9:30 AM. We are still uncertain of the cause. VSPOC had locked up some of the WisTransPortal applications. Resolve was still seeing some issues for a little time after VSPOC was cleared. Everything else was fine. An outage is planned for 4/30 to move equipment to new network services for the longterm solution.
• 4/18/2017 - > WisTransPortal applications are expierencing an outage starting around 7:56 AM this morning. Issue was temporarily resolved around 9:49 AM. It appears to have started up again around 10:45 AM and resolved around 11:18 AM. This ticket/issue closed at 3:30 PM. The problem is the result of one of the 2 primary network switches having extrememly high usage and other issues not yet resolved. This switch has been moved to failover status until new equipment is put in place - planned 4/30/2017.
• 4/18/2017 - > Bulk emails out of WisTransPortal are being delayed in the AM (TIAs). - UW Madison IT Outages
• 4/10/2017 - 10:50 AM > Bulk emails out of WisTransPortal are being delayed (TIAs). - UW Madison IT Outages
• DOT-CRASH web server maintenance scheduled for Sunday April 9 from 6-9 AM. C A N C E L E D - A small application update will occur at this time.
• WisTransPortal application maintenance scheduled for Wednesday, April 5, from 10 to Midnight, completed.
• Network issues started again around 10:40 AM on 3/30. Some web applications were unrepsonsive. Network communication restored itself by 5:42 PM on 3/30. Issue was due to heavy traffic on main shared UW Switch.
• Network issues started around 10:30 AM on 3/29. Web applications were unresponsive. Around 11:10 AM, things appear to be back to normal. Still looking into the issue.
• WisTransPortal server maintenance completed on Sunday, March 19th.
• WisTransPortal TEST server maintenance completed on Friday, March 17th.
• Some services that originate on the DoIT platform have been degraded on March 9, due to an issue with one of the network communications devices. The problem started at 1:40 PM. Resolved at 2:51 PM.
• The very brief outage today on WisTransportal was noticed around 1:40 PM on Tuesday, February 28. The problem is still being investigated but appears to be due to a UW network issue.
• VSPOC traffic data was not accessible most of the morning of Friday 2/17. It was back up by 2:20 PM. Problem is being investigated.
• WisTransPortal server maintenance window scheduled for Sunday, February 19th, was completed.
• WisTransPortal TEST server maintenance window on February 17th, starting @ 12:00 PM, was completed
• WisTransPortal server maintenance window was CANCELED for Sunday, December 18th, due to inclement weather. Maintenance scheduled for Sunday, January 22nd, was completed.
• WisTransPortal TEST server maintenance scheduled for Wednesday, January 18th, was completed. No production or DOT-CRASH servers will be affected
• VSPOC was down the morning of Jan 9. It may have been down as early as 6:22 PM on Jan 6, 2017. Services were restarted and VSPOC was back online at 9:43 AM on Jan 9.
• An update to the DOT-CRASH system is scheduled for today, January 6, starting at 8 PM. The window goes till 11 PM. The main WisTransPortal web server will also be restarted during this window.
• Many WisTransPortal applications went down at 10:03 AM on January 6th. The Oracle Listener service stopped on the main DB server. Server was restarted and services were restored by 10:50 AM.
• LINK video went out starting on January 2nd at 1:13 AM. The VLC encoders stopped working. Video was back up on January 3rd by 8:15 AM.

2 0 1 6

• VSPOC was unresponsive on Dec 7, 2016 starting around 10:50 AM. Application was restarted and available at 11:14 AM.
• WisTransPortal server maintenance window scheduled for Sunday, November 13th, completed.
• On October 27th around 9:57 AM, the WisTransPortal locked up. After service restarts, the WisTransportal locked up again at 10:05 AM. All applications were unavailable. Services were restarted and things were back up at 10:15 AM. We are not sure what caused the freeze but we are looking into it.
• The new LINK system upgrade was started on October 24th at 11:30 AM. Due to some Media server problems, it was not published till around 6:00 PM.
• WisTransPortal server maintenance window scheduled for Sunday, October 23rd, completed.
• The WisTransPortal was unstable 9/19/2016 starting around 2:25 PM. Applications were offline for up to 2 minutes at a time off and on for 20 minutes or so. This issue has been connected with the WISLR map usage on the MV4000 Crash Data Retrieval Facility. We apologize for the inconvenience. We are working on resolving the issue.
• WisTransPortal TEST server maintenance window is scheduled for Tuesday, October 18th, was completed on Sunday, October 16th.
• The Link Beta site was down for about 20 minutes on 9/12/2016 for switch over to private network. It was back up by 2:08 PM.
• WisTransPortal server maintenance scheduled for Sunday, September 25th, 2016 was completed.
• WisTransPortal TEST server maintenance scheduled for Tuesday, September 20th, 2016 was completed.
• The Link Beta site was down for about 20 minutes on 9/12/2016 for switch over to private network. It was back up by 2:08 PM.
• The WisTransPortal database was unstable 9/14/2016 starting around 1:50 PM. Applications started having problems shortly after 2:30 and then most applications became unavailable by 3:00. The server was restarted and by 3:12 PM, all applications were back. We apologize for the inconvenience and will continue to investigate the issue.
• Around 12:50 PM on Wednesday, August 31st, VSPOC application locked up all of WisTransPortal. Services was restored by 12:57 PM.
• WisTransPortal server maintenance scheduled for Saturday, August 27th, 2016 was completed.
• WisTransPortal TEST server maintenance scheduled for Tuesday, August 23rd, 2016 was completed on Thursday, August 25th.
• WisTransPortal had an unexpected outage on August 9th, starting around 6:20 AM. The database archival storage reached maximum size and locked up. There were some other issues related to the newest Oracle patch applied last maintenance window that made this a little tougher to track down. System was back up by 11:35 AM.
• A critical database maintenance window was set for Thursday, August 4th. The WisTransPortal service window started at 8:00 PM and ended at 10:05 PM. The outage affected all web based systems including TIA, InterCAD, LINK, and WisLCS.

• WisTransPortal server maintenance window scheduled for Saturday, July 23rd, was completed. A longer then expected outage for applications happened due to a Oracle 11g update.
• WisTransPortal TEST server maintenance window is scheduled for Tuesday, July 19th, was completed Friday, July 22nd.
• WisTransPortal server maintenance scheduled for Saturday, June 25th - Completed.
• WisTransPortal TEST server maintenance scheduled for Thursday, June 23rd - Completed
• WisTransPortal server maintenance scheduled for Sunday, May 22nd, 2016 - COMPLETED

  Web server was slow again coming up after a reboot around 7:45AM and wasn't functional till around 10:00AM. After applying some VMware firmware update to host servers, server is responding much faster. A manual defrag on the drive was also done on Sunday.

• TEST server maintenance window was completed on Friday, May 20th, 2016.
• The WisTransPortal had a outage this morning (5/19/2016) around 7:00AM. There was a temporarily lock on the database due to a large import process that was running. The issue was resolved around 8:10AM.
• WisTransPortal server maintenance scheduled for Sunday, May 1st, 2016 - COMPLETED

  Web server was slow coming up and wasn't functional till around 10:30AM. Researching the issue.

• TEST server maintenance window was completed on Thursday, April 28, 2016.
• WisTransPortal maintenance completed on Wednesday April 13th. starting @ 10:00 PM. This maintenance was necessary for adding additional storage capacity to the WisTransPortal database server. Maintenance was completed by 2:20AM on April 14th.
• WisTransPortal Web services were locked up on 3/17/2016 from 2:46 - 3:00 PM. Web services were said to be slow much of the afternoon and VSPOC locking up at 2:46 put it over. An application issue with VSPOC appears to be the issue. IT is looking into the resolution.
• VSPOC was down from 7:44 AM on 3/11/2016 till 10:24 AM on 3/11/2016. IT is looking into the issue.
• WisTransPortal server maintenance on Sunday, March 20, 2016 - COMPLETED
• TEST server maintenance window was skipped on Tuesday, March 15, 2016.
• After WisTransPortal maintenance on Saturday, February 20th, there were a few applications that experienced an extended outage due to the updates. VSPOC was down until Sunday, February 21st at 9:40 PM. InterCAD was not updating on public websites till after 10:00 AM on Sunday, February 21st. The scheduling service was down until around 8:30 AM on Sunday morning. This affected updates for TIA, WisTMP, and VSPOC. All applications are back online but further review will be done to help alleviate the problems with future updates.
• WisTransPortal server maintenance on Saturday, February 20, 2016 - COMPLETED
• maintenance window was completed on Friday, February 19, 2016. No production servers were affected
• WisTransPortal server maintenance on Saturday, January 23, 2016 - COMPLETED
<

2016
Scheduled WisTransPortal Server Maintenance Days	Completed	Notes
December 18	Skipped	PLANNED: Servers patched up to Microsoft December Updates; Symantec updates; VMware patches
November 13	Completed	Servers patched up to Microsoft November Updates; Symantec updates; New Remote Desktop Services server configured
October 23	Completed	Servers patched up to Microsoft October Updates; Symantec updates; VMware patches
September 25	Completed	Servers patched up to Microsoft September Updates; Symantec updates
August 27	Completed	Servers patched up to Microsoft August Updates; Dell firmware on select servers; Java Updates; Symantec updates
July 23	Completed	Servers patched up to Microsoft July Updates; Dell firmware on select servers; Java Updates; Symantec updates; Oracle 11g Update
June 25	Completed	Servers patched up to Microsoft June Updates; VMware Host patches; VM Client Tools upgraded
May 22	Completed	Servers patched up to Microsoft May Updates; VMware Host patches; Java Updates; Apache Security Patch
May 1	Completed	Servers patched up to Microsoft April Updates; VM Server Host Memory Upgrades
March 20	Completed	Servers patched up to Microsoft March Updates; Dell firmware on select hardware
February 20	Completed	Servers patched up to Microsoft February Updates; Apache Update; Tomcat Patch
January 23	Completed	Servers patched up to Microsoft January Updates; VMware Host patches; Java Update

2 0 1 5

• WisTransPortal server maintenance on Wednesday, December 19, 2015 - COMPLETED
• WisTransPortal server maintenance on Wednesday, November 18, 2015 - COMPLETED
• WisTransPortal server maintenance on Saturday, October 24, 2015 - COMPLETED
• WisTransPortal server maintenance on Saturday, September 19, 2015 - COMPLETED
• WisTransPortal server maintenance on Saturday, August 22, 2015 - COMPLETED @ 3:00 PM
• GIS services had an issue after the windows patches were installed so some mapping services were not functional until Sunday 23 around 7PM.
• On 8/3/2015 around 10:05 AM, WisTransPortal admin was notified that the MV4000 Crash Data Resources was not showing documents as being available. Network configurations on the server were shown as being incomplete and the matter was repaired by 10:39 AM. This is a permanent solution. The issue on the server may have affected intermittent outages reported on Friday 7/31.
• WisTransPortal server maintenance on Saturday, July 25, 2015 - COMPLETED
• Around 12:30 PM on July 9, WisTransPortal Applications failed to work. The applications included TIA, Community Maps, VSPOC and WRS. The problem was caused by some new rules set on the WisTransPortal domain for a new stack of servers being brought up. These rules conflicted with the current ones in place and blocked access. The new rules were removed. Outage was approximately 20 minutes.
• WisTransPortal server maintenance on Saturday, June 27, 2015 - COMPLETED
• WisTransPortal server maintenance on Saturday, May 16, 2015 - COMPLETED
• WisTransPortal server maintenance on Saturday, April 25, 2015 - COMPLETED on main production servers
• WisTransPortal server maintenance on Saturday, March 21, 2015 - COMPLETED
• On March 11 starting around 12:00 AM, the WISLR crash map and other GIS maps on the WisTransPortal were unavailable. This was due to a network change that inadvertanly affected permissions in these applications. Map services were restored around 1:00 PM on 3/11.
• WisTransPortal server maintenance on Saturday, February 21, 2015 - COMPLETED
• BITS/DET network maintenance on Sunday morning, February 8th, @ 5:00 AM, was unsuccessful. A roleback was performed at 3:30 PM. WisTransPortal tested fully back online by 6:00 PM
• WisTransPortal server maintenance on Saturday, January 24, 2015 - COMPLETED
• A short maintenance on the WisTransPortal main web server was completed on the evening of 1/6/2015. An upgrade of the Apache Web server with the latest patched version of 2.2.x.
• On Sunday January 4th starting at midnight UW Network Services performed network maintenance. All BAN ASAs were upgraded to a version to protect them from a known vulnerability. ASA5505s were rebooted at midnight.
• WisTransPortal server maintenance on Saturday, January 3, 2015 - SKIPPED DUE TO INCLEMENT WEATHER.

2015
Scheduled WisTransPortal Server Maintenance Days	Completed	Notes
December 19	Completed	Servers patched up to Microsoft December Updates; Dell firmware on select servers; Dell SAN firmware
November 18	Completed	Servers patched up to Microsoft November Updates
October 24	Completed	Servers patched up to Microsoft October Updates; VMware vCenter upgrade; Java update 65
September 19	Completed	Servers patched up to Microsoft September Updates; Apache Web patch
August 22	Completed	Servers patched up to Microsoft August Updates; VMware Client Tools Updates; Java Update; Symantec EP Version Update
July 25	Completed	Servers patched up to Microsoft July Updates; VMware Client Tools Updates; Oracle 11g Patch 39; Java Update on some PRD servers
June 27	Completed	Servers patched up to Microsoft June Updates; VMware Updates; Oracle 11g Patch 38
May 16	Completed	Servers patched up to Microsoft May Updates; Dell firmware on select servers; Virtual NIC fix on some virtual servers; NAS firmware; Hard Disk firmware update on VM SAN
April 25	Completed	Main production servers patched up to Microsoft April Updates
March 21	Completed	Select production servers patched up to Microsoft March Updates; Dell firmware on select servers; VMware patches and updates on Host servers; Virtual NIC fix on select virtual servers; Dell EqualLogic firmware (VM SAN and File SAN); VMWare Tools update on all VM servers
February 21	Completed	Servers patched up to Microsoft February Updates; Dell firmware on select servers; Java 7 Update 76; Dell EqualLogic firmware (NAS SAN)
January 24	Completed	Servers patched up to Microsoft January Updates; VMware patches and updates on Host servers (PVMH1 and PVMH2); Virtual NIC fix on virtual servers

2 0 1 4

• WisTransPortal server maintenance on Saturday, November 15, 2014 - COMPLETED.
• WisTransPortal server maintenance on Saturday, October 25, 2014 - COMPLETED.
• SUNDAY, OCTOBER 19 - Disabled SSL 3.0 support on the WisTransPortal web server to safeguard against the recent "POODLE" attack vulnerability.
• WisTransPortal maintenance for GIS servers only, Friday, September 19, 2014 - COMPLETED
• WisTransPortal server maintenance on Saturday, September 20, 2014 - COMPLETED.
• WisTransPortal maintenance for GIS servers only, Friday, August 22, 2014 - COMPLETED
• WisTransPortal server maintenance on Saturday, August 23, 2014 - COMPLETED.
• WisTransPortal server maintenance on Saturday, July 26, 2014 - COMPLETED.
• On June 18th at 7:55am, there was an interruption to electrical power to the UW-Madison Dayton Street data center, which houses many of the enterprise IT services on campus. The services that are housed in this data center were down. This included the WisTransPortal. Power was restored at 3:30 p.m. and campus systems started to be restored immediately after. Most of WisTrasnportal was up by 1:05 AM on June 19th and was fully back online at 10:00 AM on June 19th
• LINK Video hardware has moved from the STOC in Milwaukee to DoIT Datacenter at the UW Madison on Wednesday evening, May 21 starting at 8:00 PM. Since the move, there has been issues with some hardware that causes the video to stop occasionally. Technicians are working on the issue. [CLOSED 6/2/2014 11:00 AM]
• WisTransPortal server maintenance scheduled for Saturday, May 17, 2014 - Completed.
• WisTransPortal server maintenance scheduled for Saturday, April 19, 2014 - Completed.
• WisTransPortal server maintenance scheduled for Saturday, March 22nd, 2014 - Completed.
• At about 11 AM on March 13th, email alerts were no longer being sent out from the WisTransPortal including TIA. The issue was on CAE servers and was resolved by 6:00 PM on 3/13.
• At about 11 AM on March 13th, CAD incidents were not showing up on production. The issue was on the WisDOT servers. Problem was resolved by 6:30 PM on 3/13.
• After the planned maintenance window of February 22nd, the 511Projects data was unavailable. Actual outage for the 511Projects data was from 2/22 @ 9:00 AM till 10:35 AM on 2/23.
• WisTransPortal server maintenance scheduled for Saturday, February 22nd, 2014 - Completed.
• During the planned maintenance window of 12:00 PM - 2:00 PM on Wednesday, January 29th, the Southwest video was unavailable. Actual outage for the SW Link video was from 12:15 PM till 4PM on 1/29.
• WisTransPortal server maintenance scheduled for Saturday, January 18, 2014 - Completed.
• On January 5, 2014 12:00 AM - 8:00 AM, DoIT will be doing network maintenance that will affect most of UW Madison. Router/Switch Code upgrades have been cancelled so we don't anticipate any outages for the WisTransPortal but these types of maintenance may sometimes produce some unpredicted results so notification is being given here just in case. (DoIT Notifications - Maintenance Notification; Router/Switch Upgrade CANCELLED)




2014
Scheduled WisTransPortal Server Maintenance Days	Completed	Notes
December 20	Skipped	a few servers received Windows patches but most canceled due to Inclement weather
November 15	Completed	Servers patched up to Microsoft November Updates; Dell firmware on select servers; DB2 Oracle 11.2.0.4.0 update + 10/2014 patches; Java 7 Update 71
October 25	Completed	Servers patched up to Microsoft October Updates; Symantec ver 12.1.5
September 20	Completed	Servers patched up to Microsoft September Updates; VMWare Tools updated on all VMs; Firmware updates on TOPS SAN switches; Firmware update on DOT switches
August 23	Completed	Servers patched up to Microsoft August Updates; Java 7 Update 67; VMHost updates; SEP 12.1 RU4 MP1b
July 26	Completed	Servers patched up to Microsoft June and July Updates; Dell firmware on select servers
June 28	Skipped
May 17	Completed	Servers patched up to Microsoft May Updates; Java 7 update 55
April 19	Completed	Servers patched up to Microsoft April Updates; VMware upgrade; Symantec Upgrade; Dell firmware on select servers
March 22	Completed	Servers patched up to Microsoft March Updates; Oracle patch 11.2.0.3 Patch28; Java 7 Update 51
February 22	Completed	Servers patched up to Microsoft February Updates; Dell EqualLogic firmware; VMware Patches;
January 18	Completed	Servers patched up to Microsoft January Updates; Symantec (all Managed); Dell firmware on select servers;

2 0 1 3

• SW LINK Video was noted to be down on Dec 31 at 11:20 AM. Not certain how long they were out but SW LINK server had CPU errors starting on Dec 30 at 11:55 PM. Server was rebooted by DOT and the Encoders were running by 3:00 PM.
• TIA email notifications stopped going out on Dec 30, around 7:00 PM. It was noted that CAE server maintenance was taking place on Dec 30, between 8 and 10 PM. Email services were not going to work during this time. When services were brought back online, CAE inadvertently limited the access to the mail servers. It was repaired by CAE and services were back on Dec 31 at 10:30 AM.
• WisTransPortal server maintenance on Saturday, December 21, 2013 - Completed on main production servers only.

• ArcGIS Server for 511 Projects stopped working. Problem was resolved at 11:05 AM on December 2, 2013. Service may have been out since Saturday maintenance on November 30, 2013, but uncertain.

• WisTransPortal server maintenance on Saturday, November 30, 2013 - Completed

• The WisTransPortal Database server ran out of log file space on October 22, 2013 around 11:30 PM. This caused an outage for many web apps including TIA and was due to some uploads of new data. Issue was resolved by 3:45 AM on October 23, 2013.

• WisTransPortal server maintenance on Saturday, October 19, 2013 - Completed

• The College locked down their mail server on Monday, September 30, 2013 around 2:50 PM. This was done in order to restrict a recent occurrence of spam originating from the campus VPN. This had the affect of inadvertently blocking email from the WisTransPortal servers. In order to resolve the issue, the College hostmaster added the two (public and private) WisTransPortal subnets into their configuration. The outage was resolved at 7:15 PM.

• WisTransPortal server maintenance on Saturday, August 24, 2013 - Completed

• WisTransPortal server maintenance on Saturday, July 27, 2013 - Completed

• WisTransPortal applications became unresponsive around 1:30 PM on Tuesday, July 9 (TIA, LCS, Crash). Applications were back up by 1:49 PM.

• WisTransPortal server maintenance on Saturday, June 22, 2013 - Skipped

• WisTransPortal server maintenance on Saturday, May 25, 2013 from 7:00 AM - 11:00 AM. Completed.

• Around 3:00PM on Monday, April 29, the WisTransPortal database server lost communication with the Web Applications. Applications were down for about 30 Minutes. Technicians restored communication by 3:33 PM.

• Intermittent timeout and long delay issues have been identified on WisTransPortal Web applications as early as 12/17/2012 related to Java. A possible fix has been put in place and technicians watching results.(updated 4/24/2013). This issues is believed to be resolved (5/30/2013)

• WisTransPortal server maintenance on Saturday, April 20, 2013 from 7:00 AM - 11:00 AM.

• Issue on one of the WisTransPortal servers disabled some applications around 10:45 AM on 04/11/2013. Issue has been resolved with about a 10min outage affecting V-SPOC, WSR, InterCAD, and a few others.

• WisTransPortal server maintenance took place Saturday, March 23, 2013 from 7:00 AM - 11:00 AM.

• The WisTransPortal locked up on Tuesday, March 12, 2013 at around 12:30 PM. Tomcat was restarted on one of the servers to solve the problem. TIA, WISLCS, and Crash Data was affected. Outage was about 20 minutes.

• WisTransPortal maintenance took place Saturday February 23, from 7:00 AM - 12:15 AM. Time was extended due to some non-production servers having some cleanup done and then not coming up right away. WisTransPortal Applications were not affected after 11:00 AM.

• ** C A N C E L E D ** System maintenance window scheduled for Saturday January 19, from 7:00 AM - 10:30 AM.

• ** C A N C E L E D ** SW Link server is tentatively scheduled to be moved in January 2013 from Madison to Milwaukee. This will be a half day outage for SW LINK video.
  
  

  2013
  Scheduled WisTransPortal Server Maintenance Days	Completed	Notes
  December 21	Completed	Main production servers patched up to Microsoft December Updates; Symantec;
  November 30	Completed	Servers patched up to Microsoft November Updates; Symantec; Dell firmware on select servers;
  October 19	Completed	Servers patched up to Microsoft October Updates; Symantec;
  September 14	Completed	Servers patched up to Microsoft September Updates; Symantec; Dell firmware on select servers; Java 7 Update 40
  August 24	Completed	Servers patched up to Microsoft August Updates; Symantec; Dell firmware on select servers; EqualLogic Firmware and Software updated
  July 27	Completed	Servers patched up to Microsoft July Updates; Symantec; Dell firmware on select servers; DNS/DC replacement
  June 22	Skipped	WisTransPortal services needed for road closures due to heavy rains
  May 25	Completed	Servers patched up to Microsoft May Updates; Symantec
  April 20	Completed	Servers patched up to Microsoft April Updates; Java versions updated (No upgrades); Firmware on select servers; Symantec
  March 23	Completed	Public servers patched up to Microsoft March Updates; WEB1 gets Java 7; Symantec
  February 23	Completed	Servers patched up to Microsoft February Updates; All servers have minimum IE8; Java 7 testing; Symantec
  January 19	Skipped	WisTransPortal closure services needed concerning work being done around the Greenfield bridge; Snow days

2 0 1 2

• Storm Reports was down from 11:54 PM on 12/26 till 9:11 AM on 12/27. Issue has been resolved.

• WisTransPortal server maintenance was done on Saturday, December 22, 2012 from 7:00 AM - 11:00 AM. Services have been restored.

• Circuit rebuilds between TOPS Lab, Hillfarms, and STOC were completed around 11:00 AM on December 21, 2012. Service is completed. iNOC Ticket# 79154

• Scheduled ITSNET Fiber work was done on December 8, 2012. The outage was from 3:20 AM till 6:10 AM. The following applications were affected:
  InterCAD

  511 XML

• ** C A N C E L E D ** Scheduled ITSNET Fiber work to be done November 16, starting from 10:00 PM till November 17 ending at 6:00 AM. The following services will be affected:
  LINK Video

  InterCAD

  511 XML

• ** C A N C E L E D ** Next system maintenance window scheduled for Saturday November 24, from 7:00 AM - 10:30 AM.

• The WisTransPortal was having intermittent connection issues the morning of November 7, 2012 till 10:00 AM. Services have been restored. Fix was a WinSock patch.
• The WisTransPortal was having intermittent connection issues from around 7:00 PM, October 30, 2012 till 5:05 AM on October 31, 2012. Services have been restored but cause is still undetermined.
• The WisTransPortal was unavailable between 7:00 AM - 10:45 AM October 20, 2012 to install Windows Updates and Patches. Service has been restored.
• SW LINK video was reported down @ 3:51 PM on Thursday September 27, 2012. Services was restored at 9:25 PM.

• The WisTransPortal was unavailable between 7:00 AM - 10:45 AM September 23, 2012 to install Windows Updates and Patches. Service has been restored.

• The WisTransPortal was unavailable on Wednesday, August 15, between 7:00 PM and Midnight. Equipment was being relocated to the DoIT Data Center. Service was restored at 1:00 AM on August 16.

• The WisTransPortal was unavailable Monday, August 13, between 9:30 AM and 2:00 PM when network was switched over to new equipment. Service has been restored

• The WisTransPortal was unavailable between 7:00 AM - 11:30 AM July 28, 2012 to perform hardware maintenance and install Windows Updates and Patches. Service has been restored.

• The WisTransPortal was unavailable between 7:00 AM - 10:30 AM June 23, 2012 to perform hardware maintenance and install Windows Updates and Patches. Service has been restored.

  2 0 0 9

• The WisTransPortal was unavailable between 6:00 AM - 9:30 PM September 26, 2009 due to an unexpected issue with a priority upgrade to one of the webserver components. Service has been restored.

  2 0 0 7

• The WisTransPortal was unavailable between 8:00 AM - 4:00 PM June 24, 2007 to perform hardware maintenance on the webserver and apply an Oracle security patch. Service has been restored.

• The WisTransPortal was temporarily unavailable between 5:12 AM - 6:05 AM May 24, 2007 due to unexpected system issues after installing a Windows software patch. Service has been restored.

• The WisTransPortal was temporarily unavailable between 8:00 AM - 10:00 AM May 7, 2007 due to unexpected network issues. Service has been restored.

• The WisTransPortal Oracle database was offline between 11:00 PM 3/10/2007 and 1:30 AM 3/11/2007 to apply DST and security maintenance patches.

• There was a brief interruption in service between 9:00 AM - 10:00 AM Sunday, February 4, 2007 to upgrade the webserver software. The process completed successfully.

• The WisTransPortal was offline briefly between 3:00 - 3:30 PM Friday, January 25, 2007, while a temporary hardware device was installed to address the intermittent hardware failure reported on January 11, 2007.

• The WisTransPortal system lost network connectivity from approximately 1:30 AM - 5:00 AM Monday, January 18, 2007. This outage affected access to all internet applications including the LINK video sharing system. Service has now been restored. We apologize for any inconvenience.

• The WisTransPortal system lost network connectivity from approximately 6:45 PM - 8:45 PM Monday, January 15, 2007. This outage affected access to all internet applications including the LINK video sharing system. Service has now been restored. We apologize for any inconvenience.

• The WisTransPortal system lost network connectivity from approximately 5:30 PM - 9:30 PM Thursday, January 11, 2007. This outage affected access to all internet applications including the LINK video sharing system. Service has now been restored. We apologize for any inconvenience.

• The WisTransPortal system lost network connectivity from approximately 8:00 AM - 9:30 AM Thursday, January 11, 2007. This outage affected access to all internet applications including the LINK video sharing system. Service has now been restored. We apologize for any inconvenience.

• The WisTransPortal system lost network connectivity from 9:15 PM - 10:30 PM Tuesday, January 9, 2007. This outage affected access to all internet applications including the LINK video sharing system. Service has now been restored. We apologize for any inconvenience.

• The WisTransPortal system lost network connectivity from approximately 3:00 PM - 4:30 PM Thursday, January 4, 2007. This outage affected access to all internet applications including the LINK video sharing system. Service has now been restored. We apologize for any inconvenience.

2 0 0 6

• As of 10/30/2006, the WisTransPortal login form has been replaced with the pre-existing pop-up dialog. The login information (username and password) has not changed. This change was implemented in order to address several performance issues that were discovered during the initial testing phase.

• The WisTransPortal system was offline from 9:43 PM 10/16/2006 - 5:35 AM 10/17/2006. Service has since been restored. We apologize for any inconvenience.

• A maintenance window has been scheduled between the hours of 10:00 PM 10/7/2006 to 12:00 AM 10/8/2006 to upgrade the WisTransPortal webserver. All services, including WisDOT Traffic Video (LINK), the Crash Data export tool, and VSPOC, will be unavailable during that time.