tag:status.hydip.com,2005:/historyHyDip Status - Incident History2024-03-28T18:40:08ZHyDiptag:status.hydip.com,2005:Incident/190568482023-11-08T23:15:41Z2023-11-08T23:15:41ZFT2/X2 slow connectivity<p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>23:15</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>13:33</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>13:02</var> UTC</small><br><strong>Investigating</strong> - We are investigating an issue where FT2 and X2 devices are slow to connect.</p>tag:status.hydip.com,2005:Incident/189836452023-10-31T22:56:09Z2023-10-31T22:56:09ZDevice sync issue<p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>22:56</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>22:44</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>22:21</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>22:20</var> UTC</small><br><strong>Investigating</strong> - We are investigating a device syncronisation issue impacting HD1, FT1 and HD2 devices. Tank level updates are not reporting in for these device types.<br /><br />X2, FT2 devices are not impacted. Fuelcharge is not impacted. Offline tags are operating as normal while we are working on a fix.</p>tag:status.hydip.com,2005:Incident/177608552023-07-05T10:02:08Z2023-07-05T10:02:25ZDevice gateway is experiencing a higher than normal error rate.<p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>10:02</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'> 4</var>, <var data-var='time'>12:24</var> UTC</small><br><strong>Monitoring</strong> - A temporary fix has been put in place which has allowed us to clear the backlog. Service appears to be operating normally at this time.</p><p><small>Jul <var data-var='date'> 4</var>, <var data-var='time'>11:20</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating an issue where device gateway is responding unusually slowly to queries. This is causing some transactions to run slowly.<br /><br />We will provide a further update shortly.</p>tag:status.hydip.com,2005:Incident/174072882023-05-30T08:59:23Z2023-05-30T08:59:23ZFT1 device sync is incomplete<p><small>May <var data-var='date'>30</var>, <var data-var='time'>08:59</var> UTC</small><br><strong>Resolved</strong> - Closing this issue after a fix was deployed and stable for the past 24 hours.</p><p><small>May <var data-var='date'>29</var>, <var data-var='time'>08:22</var> UTC</small><br><strong>Investigating</strong> - We are investigating a synchronisation issue with FT1 devices. Transactions are processing, but credit limits are not consistently updating on devices. Offline tags is generally working, though some tags are reporting out of credit as they are unable to receive new credit.<br /><br />This does not impact FT2 or X2 devices which are operating normally.<br /><br />Engineers are working on the issue and will provide an update shortly.</p>tag:status.hydip.com,2005:Incident/173589182023-05-27T10:56:43Z2023-05-27T10:56:43ZDatabase maintenance underway<p><small>May <var data-var='date'>27</var>, <var data-var='time'>10:56</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>25</var>, <var data-var='time'>08:20</var> UTC</small><br><strong>Update</strong> - Engineers are still assessing the root cause of the performance issues.</p><p><small>May <var data-var='date'>25</var>, <var data-var='time'>01:15</var> UTC</small><br><strong>Identified</strong> - We are making adjustments to our database infrastructure which are expected to improve system responsiveness at peak time. <br /><br />As we are presently seeing an elevated error rate, this work is considered urgent and some brief glitches may occur during the course of today while we address the issue.</p>tag:status.hydip.com,2005:Incident/173375942023-05-23T12:51:54Z2023-05-23T12:51:54ZSporadic connectivity issues resulting in 503 and problems with real time transactions.<p><small>May <var data-var='date'>23</var>, <var data-var='time'>12:51</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>23</var>, <var data-var='time'>08:15</var> UTC</small><br><strong>Update</strong> - There appears to be a residual issue authorising remote transactions. Fuelcharge transactions get stuck at 'Unlocking pump'. We are investigating.</p><p><small>May <var data-var='date'>23</var>, <var data-var='time'>07:43</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are waiting for telemetry from sites to confirm full operation. System telemetry reports operational, the database fault indicators have not activated since the fix was applied.</p><p><small>May <var data-var='date'>23</var>, <var data-var='time'>06:12</var> UTC</small><br><strong>Identified</strong> - A database fault has been identified which is resulting in partial service degradation, particularly in relation to real time transactions. <br /><br />Offline transactions are not affected.<br />Tag transactions are minimally affected (reverting to offline use from time to time).<br />Fuelcharge and console/terminal transactions are heavily impacted.<br /><br />Engineers are working on the issue.</p>tag:status.hydip.com,2005:Incident/113235312022-09-30T08:51:25Z2022-09-30T08:51:25ZDevice gateway performance issue<p><small>Sep <var data-var='date'>30</var>, <var data-var='time'>08:51</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>30</var>, <var data-var='time'>08:28</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Sep <var data-var='date'>30</var>, <var data-var='time'>06:31</var> UTC</small><br><strong>Identified</strong> - We are currently investigating an issue with the device gateway. In most cases, this issue is causing degraded performance in reporting of HFT transactions. However in some cases, such as with real-time notifications, the impact is more significant. Engineers have narrowed the scope of the issue to a database congestion problem and are working on a resolution.</p>tag:status.hydip.com,2005:Incident/74681412021-07-10T17:00:00Z2021-07-10T21:23:29ZDegraded performance<p><small>Jul <var data-var='date'>10</var>, <var data-var='time'>17:00</var> UTC</small><br><strong>Resolved</strong> - A redis cluster node was automatically updated as part of an automated process to deploy critical security updates. Unfortunately the failover process was incomplete causing degraded performance for about 3-4 hours overnight. The situation has now been rectified.<br /><br />Most services including cross-network tagging will have seen some level of impact however offline tag buffers have mitigated most problems.</p>tag:status.hydip.com,2005:Incident/70526352021-05-21T03:45:52Z2021-05-21T03:45:52ZService outage<p><small>May <var data-var='date'>21</var>, <var data-var='time'>03:45</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>21</var>, <var data-var='time'>02:53</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>May <var data-var='date'>21</var>, <var data-var='time'>02:51</var> UTC</small><br><strong>Update</strong> - We have restarted the impacted database node and service appears to have been restored. Further analysis is underway.</p><p><small>May <var data-var='date'>21</var>, <var data-var='time'>02:50</var> UTC</small><br><strong>Investigating</strong> - We have encountered a database issue which is causing a high error rate at HyDip.com and other endpoints. <br /><br />Engineers are presently investigating.<br /><br />This should not impact offline tags.</p>tag:status.hydip.com,2005:Incident/60263092021-01-15T03:03:47Z2021-01-15T03:03:47ZDevice connection issue<p><small>Jan <var data-var='date'>15</var>, <var data-var='time'>03:03</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>14</var>, <var data-var='time'>22:01</var> UTC</small><br><strong>Update</strong> - Telemetry appears to have settled and we have direct confirmation that systems are back to normal.<br /><br />We will continue to actively monitor the system for any signs of recurrence.</p><p><small>Jan <var data-var='date'>14</var>, <var data-var='time'>22:01</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jan <var data-var='date'>14</var>, <var data-var='time'>21:58</var> UTC</small><br><strong>Monitoring</strong> - We identified a filesystem issue which was impacting device connection. <br /><br />The issue has been rectified and we are actively monitoring for any recurrence and are waiting for telemetry to settle.</p>tag:status.hydip.com,2005:Incident/51413242020-09-21T23:32:01Z2020-09-21T23:32:01ZConnectivity issues with X2/FT2<p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>23:32</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>12:30</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>10:13</var> UTC</small><br><strong>Investigating</strong> - Engineers are actively investigating an issue which is resulting in frequent timeouts on the communication channel between X2/FT2 devices and the HyDip Cloud. Other services are unaffected.</p>tag:status.hydip.com,2005:Incident/42656542020-06-03T10:33:43Z2020-06-03T10:33:43ZDatabase load issue<p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>10:33</var> UTC</small><br><strong>Resolved</strong> - The issue appears to have been resolved.</p><p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>10:27</var> UTC</small><br><strong>Update</strong> - A component of our database host was determined to be in a stuck state. This has been cleared, and migration is complete. We are expecting to restore service momentarily.</p><p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>09:40</var> UTC</small><br><strong>Update</strong> - A migration is taking significantly longer than expected in testing. We will temporarily pause service to release contention on the database.</p><p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>09:26</var> UTC</small><br><strong>Identified</strong> - We have identified a database issue which is causing problems when swiping tags. Engineers are working on it now.</p>tag:status.hydip.com,2005:Incident/35193502020-02-04T07:40:06Z2020-02-04T07:40:06ZDatabase performance issues<p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>07:40</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>07:27</var> UTC</small><br><strong>Monitoring</strong> - We have substantially increased database capacity and are continuing to monitor system load. Devices appear to be reporting in and telemetry is reporting normal status.</p><p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>06:45</var> UTC</small><br><strong>Investigating</strong> - We are presently investigating ongoing database performance issues causing generalised problems across the system. Engineers are working on the problem and an update will be provided shortly.</p>tag:status.hydip.com,2005:Incident/35192462020-02-04T06:28:03Z2020-02-04T06:28:03ZDevice sync issues on FT1, HD1 and HD2 models<p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>06:28</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>06:25</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>06:05</var> UTC</small><br><strong>Investigating</strong> - We are investigating a performance impairment that is causing sync issues with FT1, HD1 and HD2 device models.<br /><br />Engineers are presently working on the issue and an update will be provided when we have further information.</p>tag:status.hydip.com,2005:Incident/30051412019-09-17T09:10:41Z2019-09-17T09:10:41ZCharts not visible in mobile apps<p><small>Sep <var data-var='date'>17</var>, <var data-var='time'>09:10</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>17</var>, <var data-var='time'>02:03</var> UTC</small><br><strong>Investigating</strong> - We are aware of an issue affecting charts in our mobile apps. <br /><br />Sites are still recording dips, and the dips remain available through the web site and API. <br /><br />We have engineers working on the issue and expect an update around 6PM AEST.</p>tag:status.hydip.com,2005:Incident/21951112019-02-07T22:52:25Z2019-02-07T22:52:25ZHigh error rate causing 500 errors<p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>22:52</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>22:55</var> UTC</small><br><strong>Monitoring</strong> - We're presently monitoring the issue for any recurrence, but service has been restored.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>22:30</var> UTC</small><br><strong>Update</strong> - An update is being deployed to work around the issue. Expect a few more errors over the next 5-10 minutes while this resolves.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>22:23</var> UTC</small><br><strong>Identified</strong> - We have identified a fault and are in the process of implementing a workaround.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>22:16</var> UTC</small><br><strong>Investigating</strong> - We are investigating a database issue causing a high error rate related to the task database.</p>tag:status.hydip.com,2005:Incident/17984202018-07-03T14:30:32Z2018-07-03T14:30:32ZInfrastructure migration<p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>14:30</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>11:30</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>08:04</var> UTC</small><br><strong>Scheduled</strong> - We are undertaking a migration to new infrastructure. Services will be impacted during the course of the migration.</p>tag:status.hydip.com,2005:Incident/17577152018-06-06T02:38:03Z2018-06-06T02:38:03ZHigh error rate impacting HyDip system.<p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>02:38</var> UTC</small><br><strong>Resolved</strong> - Devices are rolling in and telemetry is responding.</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>00:27</var> UTC</small><br><strong>Update</strong> - A file system migration is in progress.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>23:15</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>23:08</var> UTC</small><br><strong>Investigating</strong> - There appears to be a service fault with our AWS database. We are attempting to work around the issue.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>22:29</var> UTC</small><br><strong>Identified</strong> - A server fault has been identified and the server is being removed from the pool.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>22:22</var> UTC</small><br><strong>Investigating</strong> - We are aware of the issue and are investigating the cause.</p>tag:status.hydip.com,2005:Incident/16741152018-04-05T11:22:58Z2018-04-05T11:22:58ZFremont datacenter offline<p><small>Apr <var data-var='date'> 5</var>, <var data-var='time'>11:22</var> UTC</small><br><strong>Resolved</strong> - Connectivity has been stable for the past few hours. Closing as resolved.</p><p><small>Apr <var data-var='date'> 5</var>, <var data-var='time'>08:50</var> UTC</small><br><strong>Monitoring</strong> - Connectivity has been restored. We are actively monitoring connectivity for recurrence.</p><p><small>Apr <var data-var='date'> 5</var>, <var data-var='time'>08:44</var> UTC</small><br><strong>Identified</strong> - Connectivity has dropped for all servers in our Fremont datacenter. We are working on the issue in conjunction with our upstream providers and will provide an update as soon as we have one.</p>tag:status.hydip.com,2005:Incident/12490132017-05-23T10:50:39Z2017-05-23T10:50:39ZSlow device syncing<p><small>May <var data-var='date'>23</var>, <var data-var='time'>10:50</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>22</var>, <var data-var='time'>23:02</var> UTC</small><br><strong>Monitoring</strong> - We have upgraded hardware to manage at higher load levels and are monitoring the issue at this time.</p><p><small>May <var data-var='date'>22</var>, <var data-var='time'>22:26</var> UTC</small><br><strong>Investigating</strong> - Device sync times are longer than usual and are resulting in degraded performance. We are working on this issue and will have more information soon.</p>tag:status.hydip.com,2005:Incident/4940022016-05-05T17:03:00Z2016-05-05T17:03:00ZUpstream networking issue<p><small>May <var data-var='date'> 5</var>, <var data-var='time'>17:03</var> UTC</small><br><strong>Resolved</strong> - Networking has been stable for the past half hour, so at this time we are going to close out the issue as resolved.</p><p><small>May <var data-var='date'> 5</var>, <var data-var='time'>16:42</var> UTC</small><br><strong>Monitoring</strong> - The upstream network appears to have stabilised and is currently working well. We are still monitoring the issue until our upstream provider has confirmed it is fixed.</p><p><small>May <var data-var='date'> 5</var>, <var data-var='time'>16:26</var> UTC</small><br><strong>Investigating</strong> - We are currently experiencing issues related to an upstream fault in our Newark datacenter. This is causing devices to fail to connect. We are working on this issue with our provider and will provide an update as soon as we have one.</p>