tag:hydrogenapi.statuspage.io,2005:/historyHydrogen Status - Incident History2024-02-29T03:40:16-05:00Hydrogentag:hydrogenapi.statuspage.io,2005:Incident/171357272023-05-03T16:36:26-04:002023-05-03T16:36:26-04:00Prod API Gateway Outage<p><small>May <var data-var='date'> 3</var>, <var data-var='time'>16:36</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'> 3</var>, <var data-var='time'>16:00</var> EDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:hydrogenapi.statuspage.io,2005:Incident/108683582022-08-21T12:14:24-04:002022-08-21T12:14:24-04:00White Label Apps Down<p><small>Aug <var data-var='date'>21</var>, <var data-var='time'>12:14</var> EDT</small><br><strong>Resolved</strong> - This outage has been resolved.</p><p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>12:41</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>12:40</var> EDT</small><br><strong>Investigating</strong> - We are investigating this outage. Your users will see the following error on all white label apps:<br /><br />"This site is inactive...<br />Seems that the site you are looking for is no longer active. We’re sorry for the inconvenience."</p>tag:hydrogenapi.statuspage.io,2005:Incident/106554602022-07-21T08:47:20-04:002022-07-21T08:47:20-04:00Email Verification & Password Reset Not Sending<p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>08:47</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>15:44</var> EDT</small><br><strong>Identified</strong> - After signing up for a new account, the Email Confirmation link is not being delivered correctly from our SMTP server. This is also affecting the Re-Send Confirmation and Reset Password.</p>tag:hydrogenapi.statuspage.io,2005:Incident/106247312022-07-18T09:18:24-04:002022-07-18T09:18:24-04:00OAuth 500 Prod<p><small>Jul <var data-var='date'>18</var>, <var data-var='time'>09:18</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>18</var>, <var data-var='time'>09:10</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:hydrogenapi.statuspage.io,2005:Incident/105569752022-07-12T21:18:48-04:002022-07-12T21:18:48-04:00Nucleus API Outage<p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>21:18</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>18:01</var> EDT</small><br><strong>Identified</strong> - The Nucleus API is experiencing an outage which is affecting callbacks and data storage. We are working to fix ASAP.</p>tag:hydrogenapi.statuspage.io,2005:Incident/105185612022-07-09T18:14:32-04:002022-07-09T18:14:32-04:00500 Error on Cards App Signup<p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>18:14</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>17:58</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating this issue affecting new signups on white label web apps. This is not affecting existing users that have accounts.</p>tag:hydrogenapi.statuspage.io,2005:Incident/104244802022-06-29T17:55:46-04:002022-06-29T17:55:46-04:00Web App 2FA Login Error<p><small>Jun <var data-var='date'>29</var>, <var data-var='time'>17:55</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>15:13</var> EDT</small><br><strong>Investigating</strong> - 2FA validation requests are returning a rate limiting error incorrectly on the white label app and not allowing 2FA users to log in.</p>tag:hydrogenapi.statuspage.io,2005:Incident/103411002022-06-17T19:26:34-04:002022-06-17T19:26:34-04:00Custom CNAME 503<p><small>Jun <var data-var='date'>17</var>, <var data-var='time'>19:26</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>17</var>, <var data-var='time'>14:16</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jun <var data-var='date'>17</var>, <var data-var='time'>13:12</var> EDT</small><br><strong>Investigating</strong> - Custom CNAME white label apps are not properly resolving and displaying a 503 Gateway error.</p>tag:hydrogenapi.statuspage.io,2005:Incident/96063252022-03-23T11:44:01-04:002022-03-23T11:44:01-04:00Card Phone Support Not Operational<p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>11:44</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>19:47</var> EDT</small><br><strong>Investigating</strong> - Cardholders are experiencing disconnection of calls to the card support phone number.</p>tag:hydrogenapi.statuspage.io,2005:Incident/92172192022-03-22T19:42:50-04:002022-03-22T19:42:50-04:00Physical Card Activation Failures<p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>19:42</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>11:25</var> EST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>14:39</var> EST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>23:39</var> EST</small><br><strong>Investigating</strong> - The Card Activation component is experiencing intermittent failures for some users in the white label apps. If your cardholders report an error, please send us the activation code and card last 4 digits, so we can activate it on our end.</p>tag:hydrogenapi.statuspage.io,2005:Incident/91376972022-01-22T18:47:30-05:002022-01-22T18:47:30-05:00White Label Cards CORS Errors<p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>18:47</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>14:13</var> EST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>13:30</var> EST</small><br><strong>Investigating</strong> - There are CORS errors on the white label cards apps that are causing the app to not display tenant configs and allow signups or logins.</p>tag:hydrogenapi.statuspage.io,2005:Incident/90923802022-01-16T09:03:41-05:002022-01-16T09:03:41-05:00504 Gateway Error<p><small>Jan <var data-var='date'>16</var>, <var data-var='time'>09:03</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved by our DevOps team and all Hydrogen platforms and portals are fully operational.</p><p><small>Jan <var data-var='date'>15</var>, <var data-var='time'>19:13</var> EST</small><br><strong>Identified</strong> - We have identified an error in AWS Kubernetes and are working to patch it.</p><p><small>Jan <var data-var='date'>15</var>, <var data-var='time'>17:12</var> EST</small><br><strong>Investigating</strong> - We are currently investigating a "504 Gateway Time-out" error that will affect app.hydrogenplatform.com and all cards white label apps.</p>tag:hydrogenapi.statuspage.io,2005:Incident/83333832022-01-15T17:14:25-05:002022-01-15T17:14:25-05:00Cardholder & Reserve Funding Failures<p><small>Jan <var data-var='date'>15</var>, <var data-var='time'>17:14</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>18:00</var> EST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Oct <var data-var='date'>29</var>, <var data-var='time'>13:30</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>16:28</var> EDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Oct <var data-var='date'>27</var>, <var data-var='time'>14:39</var> EDT</small><br><strong>Investigating</strong> - Our engineering team is currently investigating failures reported with the Card Funding Web Component, Card Funding on the white label cards apps, and Reserve Account funding on the account portal.</p>tag:hydrogenapi.statuspage.io,2005:Incident/88461292021-12-16T08:48:02-05:002021-12-16T08:48:02-05:00Card Admin Load Failure<p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>08:48</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>13:15</var> EST</small><br><strong>Identified</strong> - The Card Admin on the account portal is failing to load cards with reserve funds. We have patched the issue and will be deploying the fix ASAP.</p>tag:hydrogenapi.statuspage.io,2005:Incident/86661562021-11-22T17:19:33-05:002021-11-22T17:19:33-05:00OAuth 503<p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>17:19</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>16:49</var> EST</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>16:43</var> EST</small><br><strong>Identified</strong> - We have identified an issue with our authorization service. It is throwing a 503 "name resolution failed" error when trying to retrieve a token.</p>tag:hydrogenapi.statuspage.io,2005:Incident/86619232021-11-22T15:01:46-05:002021-11-22T15:01:46-05:00Card Issuance & Funding Failures<p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>15:01</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>11:25</var> EST</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>10:52</var> EST</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>10:52</var> EST</small><br><strong>Identified</strong> - Card issuance and funding is currently failing. We have identified the problem and are currently deploying a fix.</p>tag:hydrogenapi.statuspage.io,2005:Incident/81836892021-10-08T09:55:37-04:002021-10-08T09:55:37-04:00White Label Login Errors<p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>09:55</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>08:25</var> EDT</small><br><strong>Investigating</strong> - Login will always respond with "Incorrect Email or Password"</p>tag:hydrogenapi.statuspage.io,2005:Incident/79847552021-09-14T15:37:09-04:002021-09-14T15:37:09-04:00Card Issuance Failing For Approved Tenants<p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>15:37</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>13</var>, <var data-var='time'>13:58</var> EDT</small><br><strong>Identified</strong> - We have identified a bug that is causing virtual and physical card issuance to fail for approved tenants that are in pre-production testing. This bug is not affecting issuance of Hydrogen branded cards. We are working on a patch for this bug.</p>tag:hydrogenapi.statuspage.io,2005:Incident/79549302021-09-09T10:46:45-04:002021-09-09T10:46:45-04:00Internal Server Error on Debit Cards Web App<p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>10:46</var> EDT</small><br><strong>Resolved</strong> - This has been patched.</p><p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>08:51</var> EDT</small><br><strong>Identified</strong> - There is an internal server error on the login for the debit cards web app. Our DevOps team is investigating this error urgently. We will have a patch for it shortly.</p>tag:hydrogenapi.statuspage.io,2005:Incident/78749622021-08-31T12:24:30-04:002021-08-31T12:24:30-04:00Cards Web App Configurations Error<p><small>Aug <var data-var='date'>31</var>, <var data-var='time'>12:24</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>11:28</var> EDT</small><br><strong>Identified</strong> - We have identified an error that is causing all White Label Cards Web Apps to revert back to the default Hydrogen branded configurations. This is also limiting the ability for end cardholders to log into their accounts. We have identified the issue and are working on a hot fix for deployment as soon as possible.</p>tag:hydrogenapi.statuspage.io,2005:Incident/77930232021-08-18T17:28:14-04:002021-08-18T17:28:14-04:00Card Issuance KYC Failing<p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>17:28</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved, and all card issuance KYC should now be fully operational.</p><p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>14:50</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>14:48</var> EDT</small><br><strong>Investigating</strong> - We are currently fixing an issue with our card issuance component and KYC. You will be taken to a Pending or Failed state screen after submission, until this is resolved.</p>tag:hydrogenapi.statuspage.io,2005:Incident/76643372021-08-01T19:00:00-04:002021-08-02T10:40:19-04:00400 Error on Billing Page After Credit Card Validation<p><small>Aug <var data-var='date'> 1</var>, <var data-var='time'>19:00</var> EDT</small><br><strong>Resolved</strong> - There is a 400 error being received by customers after inputting credit card information on the Hydrogen Billing/Upgrade page.</p>tag:hydrogenapi.statuspage.io,2005:Incident/76344472021-07-26T16:00:00-04:002021-07-29T11:30:31-04:00Chase OAuth 2.0 Connection<p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>16:00</var> EDT</small><br><strong>Resolved</strong> - 404 error being received through the Chase OAuth 2.0 connection for the Hydrogen account aggregation and bank link service. While this is being corrected with Chase, clients will not be able to use the Chase connection. We apologize for the inconvenience.</p>tag:hydrogenapi.statuspage.io,2005:Incident/76346582021-07-19T11:52:00-04:002021-07-29T11:54:25-04:00Password Re-Set Error<p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>11:52</var> EDT</small><br><strong>Resolved</strong> - There is an error being reported by Hydrogen clients on the password re-set feature of the Hydrogen client portal.</p>