One of Australia’s largest electric vehicle charging networks, Chargefox, are pissed off at Australia’s largest telco, Telstra.
Update 2
Thankfully Chargefox confirmed late last night that the issue had been resolved and normal operation (and charges) had returned to their charging network.
Update
Yesterday we received the following details from Telstra, who emailed me directly to provide an update after they read this post.
A Telstra spokesman said the following:
A hardware issue is affecting a cloud service for a small number of Enterprise and Business customers.
It is a highly complex issue, and we’ve been working around the clock with our partner since it was identified to bring the service back online.
We are working with individual customers on migrating to alternative solutions, and some have successfully done so.
We know that the time it’s taking to restore service, and our inability to provide certainty on restoration timeframes has made it even more difficult for our customers.
The issue has had the full focus of our team, and our partner and we will continue to provide regular updates to our customers impacted directly until the issue is resolved.
We also learned that Chargefox was not actually a direct customer of Telstra, rather they relied on a 3rd party IoT provider who is the actual customer to Telstra in this instance. This places the responsibility on communicating with this intermediatory who were informed regularly.
While this may technically be the correct reporting chain, I think there is still an obligation on Telstra to communicate publicly once it is clear the outage impacts downstream customers operating critical services like a charging network. If this was an EFTPOS outage at Woolworths, that relied on an intermediatory, customers would still be out of luck and not be able to buy groceries.
The issue began on Monday (3/6/24) around 2PM when Chargefox reported an ongoing Telstra service disruption which affected connectivity to a number of charging stations. This made it difficult (in some cases impossible) for users to access Chargefox charging services.
By 3:50PM, less than 2 hours later, the cause of the issue was determined to be a faulty storage array in Telstra’s Clayton data centre. Telstra described the issue as a major incident and yet here we are, more than 2 days later and the issue has not yet been resolved.
Chargefox is the customer we know about, however the issue is relates to a Telstra Cloud service for a number of Enterprise and Business customers.
Just before 5PM last night, Telstra issued an update to Chargefox about their impacted service.
Our team continues to work at pace on restoration efforts alongside our partner. While we identified the cause of the issue quickly, a faulty storage array, restoring service is taking longer than expected as we have identified new issues along the way.
They go on to say that it’s not as easy as simply restoring the data (I assume it was backed up), that the restoration tasks are lengthy, and each task must be planned, staged, analysed and monitored. Based on this, it sounds like its a bad day to be a Telstra engineer.
Further in the response, Telstra says they have made good progress towards the primary fix and now also have a number of customers back up and running.. ok, but Chargefox isn’t, so does that mean they were lower on the priority list?
Having enough electric vehicle infrastructure available is challenging at the best of times, but having one of the larger networks offline is quite frankly, pretty unacceptable and the blame for this is fairly and squarely on the shoulders of Telstra.
Today at 10am, Chargefox got their latest update from Telstra and it’s not good and you can quickly see their patients is running out, sharing the following on their incident page.
Chargefox has strongly and repeatedly requested an estimated time of resolution be provided by Telstra, especially in the face of this prolonged and unacceptable outage.
Telstra has to this point refused to provide even an estimated timeline for the current outage to be fixed.
We have been informed that this failure is impacting the communications of over 30,000 devices across Australia, including critical service providers.
Telstra continues to refer to ‘working with our partner’ which I assume is a cloud service provider like Microsoft or Amazon. Unless Telstra was using some kind of hybrid cloud arrangement, it doesn’t make sense that a disk array in their infrastructure would impact the cloud service provided to customers like Chargefox, anyway, I’m sure the post mortem will be extensive on this one.
Perhaps the most significant line from Telstra is this.
We know the time it is taking to restore your service and our inability to give you certainty on a restoration timeframe is clearly unacceptable.
When you openly and now publicly admit to your service not being acceptable, that’s a pretty good indication some money is about to be exchanged between the two parties for breaching Service Level Agreements.
The next update was due at 12.30 AEST, but almost an hour later, that hasn’t been shared by Chargefox.
For EV owners, Chargefox took the extraordinary step of recommending you seek alternative charging options (i.e. use the competition). Where possible, Chargefox also put their chargers into a mode that allows you to charge for free (they don’t have comms to know who charged and how much was taken).
What You Can Do
- Check the Chargefox app for real-time updates on station availability.
- Plan your trips accordingly and consider alternative charging options if necessary.
- Stay tuned to our social media channels for ongoing updates.
Jason, it’s not a “cloud service provider like Microsoft or Amazon”. The issue is with a Dell storage array in Telstra’s own data centre, which is comprehensively borked.