tag:status.journeyapps.com,2005:/historyJourneyApps Status - Incident History2024-03-28T23:24:01-06:00JourneyAppstag:status.journeyapps.com,2005:Incident/198360542024-02-07T00:43:14-07:002024-02-07T00:43:14-07:00Additional Certificate Authority for HTTPS Services: Some API clients may be affected.<p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>00:43</var> MST</small><br><strong>Completed</strong> - The update is now complete.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>06:16</var> MST</small><br><strong>Update</strong> - This update is still in progress. Changes will only take affect on 7 or 8 February, with no downtime expected. We will provide updates as necessary. <br /><br />More details can be found here: https://whatsnew.journeyapps.com/announcements/upcoming-change-in-certificate-authorities-for-https-services-some-api-clients-may-be-affected</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>00:00</var> MST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>01:36</var> MST</small><br><strong>Scheduled</strong> - We will be introducing an additional Certificate Authority for our HTTPS services: ZeroSSL. Clients connecting via our backend API and which explicitly specify CA certificates instead of using system defaults should additionally include the certificate bundle from ZeroSSL by this date. More details can be found here: https://whatsnew.journeyapps.com/announcements/upcoming-change-in-certificate-authorities-for-https-services-some-api-clients-may-be-affected</p>tag:status.journeyapps.com,2005:Incident/194813932023-12-20T10:21:10-07:002023-12-20T10:21:10-07:00Intermittent errors from the Backend data-browser<p><small>Dec <var data-var='date'>20</var>, <var data-var='time'>10:21</var> MST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>16:06</var> MST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>16:05</var> MST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>14:32</var> MST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>09:57</var> MST</small><br><strong>Investigating</strong> - We are currently investigating reports of 504s and 502s from the Backend databrowser in the EU and US regions.</p>tag:status.journeyapps.com,2005:Incident/191973472023-11-22T15:10:30-07:002023-11-22T15:10:30-07:00CloudCode degraded performance<p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>15:10</var> MST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>13:16</var> MST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.journeyapps.com,2005:Incident/184324492023-10-17T07:28:07-06:002023-10-17T07:28:07-06:00Issue causing new Windows container builds to fail<p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>07:28</var> MDT</small><br><strong>Resolved</strong> - This incident has been resolved with the release of container version 23.10.1. Please use this version or greater for new Windows builds going forward.</p><p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>02:54</var> MDT</small><br><strong>Update</strong> - We are continuing to work on resolving this issue.<br /><br />The issue specifically affects signing new Windows builds. In the meantime, we can create unsigned builds if needed. Unsigned builds may present a warning dialog when a user installs the app and we recommend that the build is tested in your specific app distribution setup.</p><p><small>Sep <var data-var='date'> 8</var>, <var data-var='time'>02:37</var> MDT</small><br><strong>Identified</strong> - We have identified an issue that causes new Windows container builds to fail. In OXIDE, developers may see new Windows builds remain in a pending state for a while, and at a later stage fail. Existing Windows builds are unaffected by this issue - i.e. they can still be installed and used. Other platforms are also unaffected.<br /><br />We are actively working on a resolution.</p>tag:status.journeyapps.com,2005:Incident/185538322023-09-24T15:00:30-06:002023-09-24T15:00:30-06:00Scheduled maintenance - OXIDE<p><small>Sep <var data-var='date'>24</var>, <var data-var='time'>15:00</var> MDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'>24</var>, <var data-var='time'>14:02</var> MDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>03:00</var> MDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance affecting developers using OXIDE. <br /><br />During this time we expect that OXIDE will be disconnected for a few minutes, meaning that it will be in a read-only state. OXIDE's footer will show a "Disconnected" label and most updates to an app will be disabled.<br /><br />After this window, work in OXIDE can resume normally.</p>tag:status.journeyapps.com,2005:Incident/177519552023-07-03T06:55:27-06:002023-07-03T06:55:27-06:00Degraded Performance in EU/ZA regions<p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>06:55</var> MDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>06:25</var> MDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>06:22</var> MDT</small><br><strong>Monitoring</strong> - A degraded node caused degraded performance on the EU and ZA regions. We have rotated out the node, and performance should be back to normal. We are monitoring for any further issues.<br /><br />The issue was observed between 11:05 and 12:15 GMT. Symptoms were backend API requests taking longer than usual, or failing with a 503 or 504 error. CloudCode tasks and other API clients were affected by this.</p>tag:status.journeyapps.com,2005:Incident/175632762023-06-13T17:29:16-06:002023-06-13T17:29:16-06:00CloudCode US Partial Outage<p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>17:29</var> MDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>14:42</var> MDT</small><br><strong>Monitoring</strong> - AWS Lambda function invocation is starting to recover.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>13:30</var> MDT</small><br><strong>Identified</strong> - AWS Lambda function invocation is experiencing elevated error rates</p>tag:status.journeyapps.com,2005:Incident/174404432023-06-02T01:42:47-06:002023-06-02T01:42:47-06:00Intermittent failures with container builds on version 23.6.1<p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>01:42</var> MDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>23:58</var> MDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>02:39</var> MDT</small><br><strong>Investigating</strong> - We are aware of and are currently investigating intermittent failures when creating new container builds, particularly with container version 23.6.1. We recommend retrying the container build, as that fixed the issue in cases we've tested.</p>tag:status.journeyapps.com,2005:Incident/159423462023-04-04T08:15:32-06:002023-04-04T08:15:32-06:00CloudCode Logs and Audit Logs unavailable<p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>08:15</var> MDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>27</var>, <var data-var='time'>15:45</var> MST</small><br><strong>Monitoring</strong> - This incident has been resolved for all backend regions except AU. Deployments outside of AU are now fully operational.<br /><br />We are continuing to monitor the relevant subsystems.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>07:16</var> MST</small><br><strong>Investigating</strong> - We are investigating an issue where CloudCode Logs and Audit Logs are not being displayed in OXIDE and the Data Browser UI. CloudCode log search is also not working for files approximately 1 day or older.<br /><br />This incident is applicable to all Audit Logs and CloudCode Logs from backend regions.<br /><br />Cloudcode tasks themselves are running and the audit log entries are being saved normally.</p>tag:status.journeyapps.com,2005:Incident/126091092022-10-25T14:36:18-06:002022-10-25T14:36:18-06:00Intermittent issues with multiple JourneyApps cloud services<p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>14:36</var> MDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>03:03</var> MDT</small><br><strong>Monitoring</strong> - We are monitoring our services.</p><p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>03:02</var> MDT</small><br><strong>Update</strong> - We have identified the cause of the outage and applied the relevant fixes to ensure that all services are running smoothly. We will be monitoring our services in the coming hours.<br /><br />We apologize for the impact of the outage.</p><p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>02:32</var> MDT</small><br><strong>Update</strong> - We are continuing to investigate this issue. It relates to a part of our permissions systems, which is why the impact is broad across many of our cloud services.</p><p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>02:18</var> MDT</small><br><strong>Investigating</strong> - We are investigating intermittent issues with multiple JourneyApps cloud services. These include OXIDE, the Admin portal, and the regional backends. App users should be unaffected.<br /><br />We will provide an update within 30 minutes.</p>tag:status.journeyapps.com,2005:Incident/109114112022-08-25T02:38:25-06:002022-08-25T02:38:25-06:00Issues with building Typescript apps<p><small>Aug <var data-var='date'>25</var>, <var data-var='time'>02:38</var> MDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>25</var>, <var data-var='time'>02:38</var> MDT</small><br><strong>Update</strong> - The AWS outage is resolved, and TypeScript builds are working again.</p><p><small>Aug <var data-var='date'>25</var>, <var data-var='time'>01:24</var> MDT</small><br><strong>Identified</strong> - There is an AWS outage which is the root cause of the Typescript builds failing. More information on the ongoing AWS outage can be found here https://health.aws.amazon.com/health/status<br /><br />We are implementing a temporary workaround to deploy on a different region.</p><p><small>Aug <var data-var='date'>25</var>, <var data-var='time'>00:45</var> MDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.journeyapps.com,2005:Incident/104234772022-07-05T10:00:29-06:002022-07-05T10:00:29-06:00Scheduled Maintenance on the Backend System<p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>10:00</var> MDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>06:00</var> MDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>10:57</var> MDT</small><br><strong>Scheduled</strong> - During this time we will be rolling out new backend partitions and several internal updates to our backend system and services.<br /><br />While we don't expect any downtime during the maintenance window, you may experience some brief interruptions in the following affected services:<br />- App enrollments<br />- Creating new deployments<br />- Deploying to the backend or CloudCode<br />- CloudCode requests to the backend</p>tag:status.journeyapps.com,2005:Incident/104201182022-06-28T03:53:45-06:002022-06-28T06:38:36-06:00App sync issues<p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>03:53</var> MDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>03:39</var> MDT</small><br><strong>Update</strong> - ZA backend is fully recovered.<br />US backend may still have some minor performance issues.</p><p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>03:11</var> MDT</small><br><strong>Monitoring</strong> - An issue on the backend caused app sync to fail or continuously retry since 07:40 UTC.<br /><br />We have fixed the underlying issue. The systems are recovering, but may still have some performance issues until fully recovered.</p>tag:status.journeyapps.com,2005:Incident/103767412022-06-21T14:45:00-06:002022-06-22T02:10:52-06:00Authentication issue with PDF reports<p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>14:45</var> MDT</small><br><strong>Resolved</strong> - A change in tokens issued for CloudCode tasks caused authentication failures on the PDF service. We reverted the change, and everything is back to normal.<br /><br />The issue caused requests to generate PDFs to fail between 17:30 and 20:45 GMT.</p>tag:status.journeyapps.com,2005:Incident/100434272022-05-21T13:07:40-06:002022-05-21T13:07:40-06:00degraded performance<p><small>May <var data-var='date'>21</var>, <var data-var='time'>13:07</var> MDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>21</var>, <var data-var='time'>05:01</var> MDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>May <var data-var='date'>21</var>, <var data-var='time'>04:55</var> MDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>May <var data-var='date'>21</var>, <var data-var='time'>04:43</var> MDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.journeyapps.com,2005:Incident/98788862022-04-29T00:00:00-06:002022-04-29T05:50:39-06:00Password resets<p><small>Apr <var data-var='date'>29</var>, <var data-var='time'>00:00</var> MDT</small><br><strong>Resolved</strong> - Due to a bug introduced in a recent deploy, password resets performed from the JourneyApps Container would trigger a password reset for all users with the same company id, instead of only the user requesting the reset.<br /><br />We have fixed the issue as soon as we became aware of the issue, but by that time some users already received multiple password reset emails.<br /><br />If you have received a password reset email, you will be requested to change your password the next time you sign in.<br /><br />No credentials have been compromised during this incident.</p>tag:status.journeyapps.com,2005:Incident/98620982022-04-27T01:13:58-06:002022-04-27T01:13:58-06:00Backend performance<p><small>Apr <var data-var='date'>27</var>, <var data-var='time'>01:13</var> MDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>23:05</var> MDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>22:28</var> MDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.journeyapps.com,2005:Incident/97283112022-04-07T02:54:55-06:002022-04-07T02:54:56-06:00Backend Performance Issues<p><small>Apr <var data-var='date'> 7</var>, <var data-var='time'>02:54</var> MDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'> 7</var>, <var data-var='time'>02:30</var> MDT</small><br><strong>Monitoring</strong> - A fix has been implemented and performance should be back to normal. We are actively monitoring for any further issues.</p><p><small>Apr <var data-var='date'> 7</var>, <var data-var='time'>02:06</var> MDT</small><br><strong>Investigating</strong> - We have identified a major performance issue on all backend regions following a recent deploy, starting around 08:07 UTC.<br /><br />This would have affected the backend API, data browser, and OnlineDB calls from apps.</p>tag:status.journeyapps.com,2005:Incident/95135772022-03-11T20:09:29-07:002022-03-11T20:09:29-07:00Slow queueing of tasks<p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>20:09</var> MST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>18:25</var> MST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>14:12</var> MST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p><p><small>Mar <var data-var='date'>10</var>, <var data-var='time'>19:43</var> MST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>10</var>, <var data-var='time'>19:05</var> MST</small><br><strong>Investigating</strong> - We are currently investigating the issue.</p>tag:status.journeyapps.com,2005:Incident/93301812022-02-16T04:01:19-07:002022-02-21T05:33:03-07:00OXIDE search unavailable<p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>04:01</var> MST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>03:15</var> MST</small><br><strong>Monitoring</strong> - We have fixed the issue, and will continue monitoring for any performance issues.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>00:36</var> MST</small><br><strong>Identified</strong> - We have identified an issue with OXIDE being slow or unable to load. To mitigate the issue, we have temporarily disabled all search functionality in OXIDE.<br /><br />Everything in OXIDE should now load correctly, but search functionality will be unavailable.</p>tag:status.journeyapps.com,2005:Incident/89045752022-01-12T07:00:00-07:002022-02-16T00:33:35-07:00Disabling TLS 1.0 - US region<p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>07:00</var> MST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>21</var>, <var data-var='time'>09:18</var> MST</small><br><strong>Scheduled</strong> - We are permanently disabling TLS 1.0 and 1.1 support on our US regional cloud backends at 07:00 MST.<br /><br />Any clients not supporting TLS 1.2 or later must be upgraded as soon as possible.</p>tag:status.journeyapps.com,2005:Incident/90021902022-01-04T23:32:49-07:002022-01-04T23:32:49-07:00Slow CloudCode Queuing<p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>23:32</var> MST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>12:53</var> MST</small><br><strong>Monitoring</strong> - CloudCode queueing is happening at expected speeds. We are continuing to monitor for performance.</p><p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>12:51</var> MST</small><br><strong>Update</strong> - CloudCode queueing is happening at expected speeds. We are continuing to monitor for performance.</p><p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>12:06</var> MST</small><br><strong>Investigating</strong> - CloudCode tasks are queueing for a longer time than expected. CloudCode tasks are queued, but execute after a delay.<br /><br />We are actively investigating the issue.</p>tag:status.journeyapps.com,2005:Incident/89114532021-12-22T10:15:01-07:002021-12-22T10:15:01-07:00US Backend outage<p><small>Dec <var data-var='date'>22</var>, <var data-var='time'>10:15</var> MST</small><br><strong>Resolved</strong> - We can confirm that the incident is now resolved.</p><p><small>Dec <var data-var='date'>22</var>, <var data-var='time'>07:37</var> MST</small><br><strong>Monitoring</strong> - All systems have recovered as a result of the workarounds that we applied, in combination with the recovery of AWS. <br /><br />We are monitoring and will post updates here should any issues persist.<br /><br />We would like to apologize sincerely for the outage. We have already identified ways to make JourneyApps more resilient to upstream AWS outages such as these and will conduct the necessary root cause investigations to identify further improvements.</p><p><small>Dec <var data-var='date'>22</var>, <var data-var='time'>07:21</var> MST</small><br><strong>Update</strong> - We have applied our first workarounds to circumvent the affected AWS Availability Zones (where practically possible).<br /><br />That said, the performance of the US backend region is still severely affected. We are working on scaling up performance now.<br /><br />In addition, we are still not seeing any marked improvement from AWS despite their status updates.<br /><br />To summarize the current impact:<br />- Some API requests to US region deployments may be successful.<br />- OXIDE seems to be working for code edits, though deploying to US region backends doesn't seem to work as expected.<br />- CloudCode tasks in the US region seem to run, though due to the degraded performance these tasks may still time out frequently and/or run into other errors. Deploying changes to CloudCode tasks will also not work currently.<br /><br />We are downgrading the impact to "Partial outage" for now, and will update you as soon as performance of the backend is back to normal.</p><p><small>Dec <var data-var='date'>22</var>, <var data-var='time'>06:47</var> MST</small><br><strong>Update</strong> - AWS has communicated that power is being restored to the affected services, though we're not yet seeing the positive impact on our side. The outage continues.<br /><br />We are in the process of applying workarounds that should restore service to core backend functions for the US region.</p><p><small>Dec <var data-var='date'>22</var>, <var data-var='time'>06:13</var> MST</small><br><strong>Identified</strong> - This is the latest update from AWS:<br /><br />"5:01 AM PST We can confirm a loss of power within a single data center within a single Availability Zone (USE1-AZ4) in the US-EAST-1 Region. This is affecting availability and connectivity to EC2 instances that are part of the affected data center within the affected Availability Zone. We are also experiencing elevated RunInstance API error rates for launches within the affected Availability Zone. Connectivity and power to other data centers within the affected Availability Zone, or other Availability Zones within the US-EAST-1 Region are not affected by this issue, but we would recommend failing away from the affected Availability Zone (USE1-AZ4) if you are able to do so. We continue to work to address the issue and restore power within the affected data center."<br /><br />We have been attempting to fail away from the affected Availability Zone as recommended, but have not been successful due to errors from AWS. From our experience it seems like the outage at AWS is more extensive than being reported and is potentially affecting other Availability Zones too.<br /><br />We are currently working on possible workarounds to restore partial functionality and will keep you updated.</p><p><small>Dec <var data-var='date'>22</var>, <var data-var='time'>05:48</var> MST</small><br><strong>Update</strong> - From our initial investigations it seems like our infrastructure provider, Amazon Web Services, is having an outage which is affecting all our services hosted in the US region.<br /><br />This is their most current update:<br />"4:35 AM PST We are investigating increased EC2 launched failures and networking connectivity issues for some instances in a single Availability Zone (USE1-AZ4) in the US-EAST-1 Region. Other Availability Zones within the US-EAST-1 Region are not affected by this issue."<br /><br />This includes, as mentioned previously:<br /><br />- US-region deployments, which are currently down. This includes the Backend Data Browser.<br />- API clients (e.g. integration brokers, CloudCode tasks and apps) will see errors when making API calls.<br />- [New]: OXIDE is intermittently working for some customers, but some customers might see errors when trying to load apps in OXIDE. Deploying apps will possibly not work currently for US-based deployments.<br />- It is not currently clear whether CloudCode tasks in the US region are running at the moment - we are investigating.</p><p><small>Dec <var data-var='date'>22</var>, <var data-var='time'>05:31</var> MST</small><br><strong>Investigating</strong> - We are currently investigating an outage on the US backend region. Other regions are unaffected<br /><br />The outage will affect API calls to this region, whether that is from the app (syncing data or OnlineDB calls), CloudCode (DB calls), or other API clients. <br /><br />All US-region deployments are affected, including Testing, Staging, and Production deployments.<br /><br />The Backend Data browser is also affected, and currently does not load.<br /><br />This outage started at 12:10PM UTC (i.e. roughly 20 minutes ago).</p>tag:status.journeyapps.com,2005:Incident/88626732021-12-21T12:00:30-07:002021-12-21T12:00:30-07:00TLS 1.0 Deprecation - US region<p><small>Dec <var data-var='date'>21</var>, <var data-var='time'>12:00</var> MST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>21</var>, <var data-var='time'>08:01</var> MST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>00:52</var> MST</small><br><strong>Scheduled</strong> - We are disabling TLS 1.0 and 1.1 support on our US regional cloud backends for a period of 4 hours. Clients using TLS 1.2 or 1.3 will not be affected.<br /><br />Any clients not supporting TLS 1.2 or 1.3 must be upgraded as soon as possible. We will permanently disable TLS 1.0 and 1.1 support in January.</p>tag:status.journeyapps.com,2005:Incident/88537762021-12-16T00:02:48-07:002021-12-16T00:02:48-07:00Issues with Searching through App Code in OXIDE<p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>00:02</var> MST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>07:21</var> MST</small><br><strong>Monitoring</strong> - We have applied fixes and are now monitoring. <br /><br />Some intermittent issues may still occur in the next hour as the backing services to the code search features recover. We will mark this incident as "Resolved" once we have confirmed that everything is working 100% and fully recovered.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>11:00</var> MST</small><br><strong>Identified</strong> - We are aware of issues with the code search features (quick and advanced search) in OXIDE. <br /><br />Searches are sometimes returning outdated results, and some other searches are failing intermittently. <br /><br />These issues first occurred yesterday at approximately 20:00 (UTC) and have occurred intermittently since then.<br /><br />We have identified the underlying causes, and are actively working on the issues. <br /><br />We expect to have it resolved within the next few hours.</p>