tag:status.check21.com,2005:/historyCheck21.com Status - Incident History2024-03-29T05:22:54ZCheck21.comtag:status.check21.com,2005:Incident/202932072024-03-18T06:00:00Z2024-03-18T17:02:22ZAzure scheduled maintenance<p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>06:00</var> UTC</small><br><strong>Resolved</strong> - DB maintenance.<br />Temporary down time 1:53am EST for 30 seconds.</p>tag:status.check21.com,2005:Incident/198211072024-01-23T17:30:00Z2024-01-24T19:13:32ZTimeout Reported Across All Services<p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>17:30</var> UTC</small><br><strong>Resolved</strong> - All Check21/Payology services experienced multiple timeouts for 13 minutes between 12:48 PM and 1:13 PM EST on 1/23/24. During this period, the Check21 IT team investigated the errors and determined the cause was a SQL connection pool issue. The IT team solved the issue and is determining how the error can be prevented.</p>tag:status.check21.com,2005:Incident/198211452024-01-04T10:00:00Z2024-01-24T19:16:45ZImage Processing API Timeout<p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>10:00</var> UTC</small><br><strong>Resolved</strong> - On 1/4/24 between 4:50AM and 5:10AM, Image Processing services were down for scheduled maintenance.</p>tag:status.check21.com,2005:Incident/181694812023-08-27T09:00:06Z2023-08-27T09:00:06ZMaintenance of Check Verification Platform<p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>09:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>03:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>04:41</var> UTC</small><br><strong>Scheduled</strong> - The ATM/NCN and Early Warning check verification options, will be unavailable for a period of between 1 hour and 6 hours. During that time, submissions may or may not result in time-outs.</p>tag:status.check21.com,2005:Incident/180609402023-08-05T15:50:20Z2023-08-05T15:50:20ZServer Updates<p><small>Aug <var data-var='date'> 5</var>, <var data-var='time'>15:50</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug <var data-var='date'> 5</var>, <var data-var='time'>15:46</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'> 5</var>, <var data-var='time'>15:43</var> UTC</small><br><strong>Scheduled</strong> - The Check21.com team will be performing a maintenance update on the Image Processing server.</p>tag:status.check21.com,2005:Incident/178647422023-07-15T20:02:32Z2023-07-15T20:02:32ZUnscheduled maintenance<p><small>Jul <var data-var='date'>15</var>, <var data-var='time'>20:02</var> UTC</small><br><strong>Resolved</strong> - We investigated the outage and found out that Microsoft was performing an unscheduled maintenance that effected this component for a few seconds past 12:57am 7/15/2023. The component is fully operational.</p>tag:status.check21.com,2005:Incident/91252512022-01-20T03:54:16Z2022-05-23T05:43:41ZPartial Outage Under Investigation<p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>03:54</var> UTC</small><br><strong>Resolved</strong> - The Check21.com team has identified the source of the issue as a known intermittent Microsoft bug related to SQL server and SSL connection communication. Microsoft has directed the team to a possible permanent resolution which has been implemented but will take effect after midnight tonight. The team will monitor the systems to ensure functionality after the recommended change has been made.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>02:09</var> UTC</small><br><strong>Update</strong> - Check21.com has determined that the issue has no affected file generation or connection between the Check21.com Gateway and any financial institutions. We will continue to investigate the outage and provide updates accordingly.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>02:02</var> UTC</small><br><strong>Investigating</strong> - Check21.com is investigating a partial outage of 9 minutes. We will provide updates. The system is fully operational at this time.</p>tag:status.check21.com,2005:Incident/91061162022-01-17T21:45:02Z2022-01-17T21:45:35ZSignificant Performance Degradation<p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>21:45</var> UTC</small><br><strong>Resolved</strong> - At 4:19PM EST Check21.com began to experience significant performance degradation that led to wide-scale timeouts for all API endpoints and frontend portals. The IT team has identified communication with SQL as the cause. After investigation, the team decided to restart the communication layer. All components are now operational. The team will continue to investigate the cause and determine how it can be avoided in the future.</p>tag:status.check21.com,2005:Incident/89756722022-01-01T19:38:50Z2022-01-01T19:38:50ZOCR Engine Failing<p><small>Jan <var data-var='date'> 1</var>, <var data-var='time'>19:38</var> UTC</small><br><strong>Resolved</strong> - Per the vendor, the Primary OCR engine experienced a licensing issue which led to service errors. The vendor has issued a workaround until the incident can be resolved later in the week. All services are operational and a transition from the secondary engine to the primary engine has been completed. No further updates will be made available.</p><p><small>Jan <var data-var='date'> 1</var>, <var data-var='time'>01:14</var> UTC</small><br><strong>Update</strong> - The OCR Vendor is currently diagnosing the Primary service. All systems operational on secondary engine.</p><p><small>Jan <var data-var='date'> 1</var>, <var data-var='time'>01:05</var> UTC</small><br><strong>Update</strong> - The Check21.com team has rolled over to a secondary OCR engine for all endpoints. All endpoints are operational. Another update will be made available as soon as the primary service is back online.</p><p><small>Jan <var data-var='date'> 1</var>, <var data-var='time'>00:59</var> UTC</small><br><strong>Investigating</strong> - The Check21.com team is investigating a major outage of our OCR Engine. At this time, Mobile Capture and Check Scanning functions are not operational.</p>tag:status.check21.com,2005:Incident/87294242021-11-30T00:30:00Z2021-12-01T03:35:25ZX9 File Generation Misconfiguration<p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>00:30</var> UTC</small><br><strong>Resolved</strong> - A configuration file was uploaded prematurely to the Check21.com Payment Gateway production instance causing X9 files which were designated for delivery between 6:45PM and 7:00PM EST to be transmitted prior to 8:00PM EST. Files were generated and transmitted within that timeframe. Organizations whose bank cutoff times are set to 7:00PM EST may have been negatively affected - but it is unknown whether such issue impacted cash flow by one business day. Please check with your financial institution to confirm.</p>tag:status.check21.com,2005:Incident/82839392021-10-20T21:00:00Z2021-10-21T17:42:25ZICL-X9 File Generation Error<p><small>Oct <var data-var='date'>20</var>, <var data-var='time'>21:00</var> UTC</small><br><strong>Resolved</strong> - The Check21.com Payment Gateway experienced a Java-based error when performing ICL/X9 file generation functions. The Check21.com team was made aware of the issue at 4:50PM EST and resolved the issue at 5:15PM EST. The files were generated and transmitted at that time. Organizations which utilize Bank United or Frost Bank may have been affected - but it is unknown whether such issue impacted cash flow by one business day. Please check with your financial institution to confirm.</p>tag:status.check21.com,2005:Incident/77963952021-08-19T06:00:00Z2021-08-19T06:07:59ZIntermittent Timeouts on Image Processor APIs<p><small>Aug <var data-var='date'>19</var>, <var data-var='time'>06:00</var> UTC</small><br><strong>Resolved</strong> - The Check21.com is currently investigating intermittent timeouts on calls to the Mobile Check Capture API endpoint. We expect to have an update in the early morning hour.</p>tag:status.check21.com,2005:Incident/82336512021-07-31T09:30:00Z2021-10-15T04:26:19ZMicrosoft Azure Power Failure<p><small>Jul <var data-var='date'>31</var>, <var data-var='time'>09:30</var> UTC</small><br><strong>Resolved</strong> - On 7/31/2021, between 5:20AM and 5:59AM EST, a select number of Microsoft Azure Virtual Machines experienced sudden power failure. The power failure also caused redundancy points to fail. Since the incident, the Check21.com team has added additional monitoring capabilities and resolved point of failures which attributed to the lack of redundancy.</p>tag:status.check21.com,2005:Incident/70168252021-05-16T18:20:04Z2021-05-16T18:20:04ZPlatform-Wide Outage<p><small>May <var data-var='date'>16</var>, <var data-var='time'>18:20</var> UTC</small><br><strong>Resolved</strong> - At 1:55PM EST the Production environment server unexpectedly restarted to perform OS updates. The OS updates took a total of 12 minutes to perform causing all API's and portal access to timeout. The team is investigating why the OS was restarted when automatic restarts (for updates) is disabled.</p>tag:status.check21.com,2005:Incident/68531462021-04-27T16:52:25Z2021-04-27T16:52:25ZCheck Image Processing is Down<p><small>Apr <var data-var='date'>27</var>, <var data-var='time'>16:52</var> UTC</small><br><strong>Resolved</strong> - The root cause of the incident was a corruption in the licensing module file. The corrupt file was identified and replaced. All systems are operational now.</p><p><small>Apr <var data-var='date'>27</var>, <var data-var='time'>16:12</var> UTC</small><br><strong>Update</strong> - We continue to work with our vendor on resolving the issue. The vendor has identified the licensing key module as the cause of the error.</p><p><small>Apr <var data-var='date'>27</var>, <var data-var='time'>12:03</var> UTC</small><br><strong>Identified</strong> - We have identified an issue which is causing the check scanning and check capture functions to fail. We are on the line with the vendor in an attempt to fix the issue.</p>tag:status.check21.com,2005:Incident/66599462021-04-02T02:25:17Z2021-04-02T04:12:54ZAzure DNS issues<p><small>Apr <var data-var='date'> 2</var>, <var data-var='time'>02:25</var> UTC</small><br><strong>Resolved</strong> - Microsoft has resolved the incident by moving its services to a different DNS server. All systems are operational.</p><p><small>Apr <var data-var='date'> 2</var>, <var data-var='time'>01:41</var> UTC</small><br><strong>Update</strong> - DNS issue impacting multiple Microsoft services - Mitigated (Tracking ID GVY5-TZZ)<br /><br />Summary of Impact: Between approximately 21:30 UTC and 22:30 UTC on 01 Apr 2021, customers may have experienced intermittent issues accessing Microsoft services, including Azure, Dynamics, and Xbox Live.<br /><br />Preliminary Root Cause: We are continuing to investigate the underlying cause for the DNS outage but we have observed that Microsoft DNS servers saw a spike in DNS traffic.<br /><br />Mitigation: Microsoft engaged resilient DNS capabilities to absorb the spike in DNS traffic.<br /><br />Next Steps: We apologize for the impact caused by this outage. We are continuing to investigate to establish the full root cause.</p><p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>23:59</var> UTC</small><br><strong>Update</strong> - Warning DNS issue impacting multiple Microsoft services - Validating recovery<br /><br />SUMMARY OF IMPACT: Starting at approximately 21:30 UTC on 01 Apr 2021, customers may experience intermittent issues accessing Microsoft services, including Azure, Dynamics, and Xbox Live.<br /><br /><br />CURRENT STATUS: We have confirmed that the underlying DNS outage has been mitigated. Currently we are validating the recovery of the downstream services called out here on the status page. We will be providing a preliminary root cause and additional details within 30 minutes.<br /><br />This message was last updated at 23:54 UTC on 01 April 2021</p><p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>22:47</var> UTC</small><br><strong>Update</strong> - DNS issue impacting multiple Microsoft services - Recovery in progress<br /><br />SUMMARY OF IMPACT: Starting at approximately 21:30 UTC on 01 Apr 2021, customers may experience intermittent issues accessing Microsoft services, including Azure, Dynamics, and Xbox Live.<br /><br /><br />CURRENT STATUS: Microsoft rerouted traffic to our resilient DNS capabilities and are seeing improvement in service availability. We are continuing to investigate the cause of the DNS issue. The next update will be provided in 30 minutes or as events warrant.<br /><br />This message was last updated at 22:36 UTC on 01 April 2021</p><p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>22:16</var> UTC</small><br><strong>Investigating</strong> - Warning DNS issue impacting multiple Microsoft services - Investigating<br /><br />SUMMARY OF IMPACT: Starting at approximately 21:30 UTC on 01 Apr 2021, a subset of customers may experience intermittent issues accessing Microsoft services, including Azure, Dynamics, and Xbox Live.<br /><br />CURRENT STATUS: Engineering teams have been engaged and are currently investigating a potential underlying DNS issue. The next update will be provided in 60 minutes or as events warrant.<br /><br />This message was last updated at 22:12 UTC on 01 April 2021</p>tag:status.check21.com,2005:Incident/66182512021-03-26T17:10:00Z2021-03-26T20:10:54ZDatabase Timeout Affecting Global System Operations<p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>17:10</var> UTC</small><br><strong>Resolved</strong> - At 1:10PM EST today for a period of 6 minutes, database timeouts appeared. The timeouts appear to be caused by a index which required repair. The IT team repaired the Index and continue to monitor for additional incidents. At the moment, all solutions are operating well.</p>tag:status.check21.com,2005:Incident/66100692021-03-26T06:43:24Z2021-03-26T06:43:24ZService Outage Reported<p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>06:43</var> UTC</small><br><strong>Resolved</strong> - The incident has been resolved. All API's have been operational since the last status update. The IT team is still investigating the cause of the incident.</p><p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>19:17</var> UTC</small><br><strong>Monitoring</strong> - A service outage was reported between 2:36PM EST and 2:41PM EST. The service affected all Check21.com Payment Gateway API endpoints and its portal. The IT team has restarted the service and can confirm that all endpoints and services are operational. The team continues to investigate the outage occurred.</p>tag:status.check21.com,2005:Incident/60116872021-01-12T16:00:00Z2021-01-13T07:09:34ZCorrupt Index<p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>16:00</var> UTC</small><br><strong>Resolved</strong> - At 10:49AM EST, users within the management portal reported issues when attempting to view transaction details. The issue reported led the IT team to determine that the error was caused by a corrupt SQL Index. The index also led to intermittent API endpoint timeouts totaling 9 minutes. To resolve the issue, the team disabled portal access and, thereafter dropped and recreated all indexes. The system and health of the data is well.</p>tag:status.check21.com,2005:Incident/57560172020-12-08T19:00:00Z2020-12-08T21:56:50ZUnknown Issue<p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>19:00</var> UTC</small><br><strong>Resolved</strong> - Monitoring services reported an error which occurred on POST Transaction submission between 2:16PM EST and 2:17PM EST. Logs have been analyzed across all services which show no downtime or performance issues. Submissions into our database continued during the timeframe specified. Therefore, we believe the issue to be that of the monitoring service.</p>tag:status.check21.com,2005:Incident/56376812020-11-23T19:22:41Z2020-11-23T19:22:41ZNetwork Routing Incident<p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>19:22</var> UTC</small><br><strong>Resolved</strong> - There have been no subsequent errors detected since the incident was identified at 11:39am EST. The has resolved itself.</p><p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>17:03</var> UTC</small><br><strong>Monitoring</strong> - We're monitoring what may have been a network routing issue which caused between 10 and 35 seconds of transmission errors. It seems the errors were compartmentalized to Payment API's. No additional actions are required at this moment.</p>tag:status.check21.com,2005:Incident/55170852020-11-08T06:30:00Z2020-11-08T14:58:09ZImprovement in Disk Performance<p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>06:30</var> UTC</small><br><strong>Resolved</strong> - At 1:19AM EST and 1:23AM EST, for a period of between 10 and 30 seconds, Check21.com transitioned its hard disks to high performance drives. The impact was minimal and expected, causing no known client errors. All systems are operational and the maintenance is complete.</p>tag:status.check21.com,2005:Incident/54303602020-10-28T03:12:22Z2020-10-28T03:15:23ZNetwork Connection Error<p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>03:12</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>03:11</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor network traffic and have had over 35 minutes of uptime with no latency or connection errors.</p><p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>03:05</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>02:40</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating a network incident which resulted in intermittent connection errors. The incident started at 10:30PM EST and was resolved at 10:33PM EST. We believe the error was confined to the Customer POST/PUT's.</p>