Hard reset Rachio 3, Still "Boot Error One"

Hey @Storman! Would you be up for forwarding that email so I can do some research for you? I’ll DM you my direct email :slight_smile:

Same problem here - issue started after power outage. Opened ticked a week ago - so far no reply - very disappointing.
Can someone please get in touch with me directly!?

This does not help anyone with current problems, but Rachio support was fast and effective for me…probably helps that my power outage happened in late winter when they are slow and I was not in urgent need of watering. I agree that this problem has to be fixable and if “smart” controllers are going to work long term they have to have reliability that approaches my 20 year old hunter.

Having this same issue. Power outage 2 days after initial install. Now can’t get past initial quadrant boot fail.

Have filed ticket. Getting daily message from tech support, but it is just suggesting the next troubleshooting article that doesn’t address not getting past 1st light boot error.

Guess I will have to return to Amazon

1 Like

This happened to me on the day I installed it. After getting it all set up and working, I unplugged the controller to re-run the power cord. After that the controller is stuck in the first quadrant failure. Thankfully I didn’t throw out my old Hunter controller.

This has been a problem since October 2018? It seems crazy that a power cycle can so easily brick the controller.

We’ll see what support comes back with. I am dealing with support via email.

Having the same problem after power went out I have only had this controller 26 days. Is there a fix for this problem or what do I do.

@Rojo - I’d email support@rachio.com or call the support line. I’m thinking the unit will need to be RMA’ed.

I’m assuming that the unplug the Rachio transformer, count to 10 and plug it back in has been tried. And, the factory reset option. If so, and one is still getting Boot Error One then I think a new unit is in order. Rachio has a two year warranty, so there shouldn’t be an issue on getting a replacement.

1 Like

I ran to the same issue and as I’m outside US couldn’t RMA it. However I’m handy with electronics so I decided to plug it into bus pirate as described here: https://medium.com/tenable-techblog/reversing-the-rachio3-smart-sprinkler-controller-ae7fc06aab9

As the result I got the boot log and where it fails and why it keeps on rebooting:

1525468575584:DEBUG :https :unable to init ident
1525468575590:DEBUG :rachio_pair :https start fail
init fail: rachio_app_framework_event(RACHIO_EVENT_INIT_DONE):4
1525468575602:DEBUG :iro_main :app_boot fail
boot up failed!!
1525468575611:DEBUG :led_engine :enable animation 15:BOOT_FAIL center 0

The cause seems to be ‘rachio_pair :https start fail’. I haven’t started to figure out what causes this, but it seems like there’s something messed up in the EEPROM. I have a new unit so might try to copy the EEPROM contents from that one. If Rachio is listening and interested, let me know, I can get the memory dump from the problematic one.

Also in case somebody can guess what that error is, let me know.

Found one possible cause and it might be a bad SSL certificate in amazon cloud.

In the logs it refers to https://a3bmbcwe3hybwy.iot.us-west-2.amazonaws.com/ , which when opened shows an error in Chrome, NET::ERR_CERT_SYMANTEC_LEGACY. The certificate is:

Subject: *.iot.us-west-2.amazonaws.com
Issuer: Symantec Class 3 ECC 256 bit SSL CA - G2
Expires on: Oct 20, 2019

If the thing suddenly boots up after October 20th, then amazon has fixed the certificate problem. Just a hunch, but this might well be good news.

I also see the first report was October '18. The certificate in question was created on October 18th, 2018.

I have a Gen 2 that is different in some regards, so some of this may be incorrect.

I don’t believe that there is anything wrong with the cert. The device presumably has a Symantec CA cert in flash that it uses to verify the signature; it need not match anything trusted by consumer browsers.

I don’t know whether the “https start fail” is significant. If that refers to the embedded HTTPS server (which the Gen 2 doesn’t even have), it could be caused by there being no network interface up at the time, i.e. normal for a unit not yet paired.

The timestamps are from May 2018, somewhat surprising given that the device has a hardware clock with battery backup. Possibly, this is read further along in the boot process, or perhaps only gets read when NTP access fails.

That may indeed bring it to life, but you would likely have two units with the same MAC address and serial number. Some network wizardry could get them both on line, but if they look like the same device to the cloud, you couldn’t get them independently configured.

I’d be interested in seeing the complete boot log, perhaps put it on pastebin and post the link here.

Have you tried repeating the factory reset sequence?

Thank you Stewart, you shot down my initial theory and gave me another clue. The full log is here: https://pastebin.com/MduyXwEL

I tried factory reset multiple times and even plugged out the battery for the RTC, which is why you see the timestamps from 2018, I think the firmware was compiled back then. I bought the unit from Amazon a couple of months ago and the firmware version seems to be iro3-firmware-5-488, so units with firmware that has the problem are still being sold.

Rachio has since released new firmware so the problem might be corrected already, first thing I did was to update the new unit I got. I don’t have access to the new unit right now as it’s been installed but I’m keeping it offline as I don’t want a repeat incident.

You mentioned Rachio 3 has an embedded https server and after realizing that, I can now see from the log it’s setting up a wifi AP before trying to start that https server and that’s where it fails. More specifically, the cause of https initialization failure seems to be ‘https :unable to init ident’.

At this point the unit is not connected to internet so the amazon certificate is irrelevant.

I haven’t been programming embedded for a few years now, if some one who recognizes the https log messages can guess if it’s a library in use, perhaps they could shed some light into what that ‘ident’ it’s trying to initialize is.

I also had power failures before this happened, but when I found the unit offline it was still trying to connect to the wifi and wasn’t in the boot loop. I then reset the wifi, after which the unit rebooted and went into this boot loop.

AFAIK there’s no automatic update of the firmware, needs to be done manually? Otherwise my guess would have been bodged firmware upgrade due to power outage.

I’m unfamiliar with the chip being used and it looks like the firmware it’s encrypted anyway, so I likely would need an SDK with the encryption keys, which again likely means buying something fairly expensive and that’s outside my limit of my curiosity. I’m sort of doing this as a trip down memory lane as a former embedded systems engineer.

I’m thinking of getting the firmware from the URL that was in the pirate bus link I gave earlier and trying to patch it into an EEPROM image I first pull out of the malfuntioning unit, would need to know the memory offset to put it into, from the hex dump I can see the update firmware package does have a ‘RACHIO_OTA’ header which I probably could locate in the memory dump, unless that OTA upgrade is somehow processed before written to the EEPROM.

There’s also a place for a debug header on the PCB but I don’t know where to start with that one and would probably need the SDK and source code to get anything worth while done anyway. Thanks to the bus pirate I can access the EEPROM which for me seems to be the easiest way.

Or, if Rachio’s engineers are listening, is there an area of the EEPROM I could wipe or an address where a flag could be set to get it to boot?

For those that find all of this like it’s coming out of an alien planet: There’s something wrong with the software of the unit that can’t be fixed without reprogramming it and is causing it to reboot itself in a loop. Without the expertise, RMA seems to be the only solution, unfortunately.

To save you time I’d reach out to our customer success team and they can RMA for you.

:cheers:

Hi franz, I did and was informed I need to have US/CA address for the RMA, which I don’t have. The courier costs from/to Thailand would make it a bit silly too, not to mention battling with the customs department.

I don’t mind debugging, it’s a hobby and I already bought a new unit that’s working fine.

Would you be able to confirm the problem in question has been fixed in the new firmware releases?

Rachio Support has been awesome BUT I have now received my third Rachio 3 controller and it still gets the boot error 1. I want to love this thing but I cannot use it. At this point I think I want my money back.

Do you happen to remember if you upgraded the firmware before it got stuck in the boot error?

I did not run any firmware updates. This is pretty frustrating.

Thanks mkmetz, I understand the frustration. The reason I asked is franz didn’t yet confirm if the software bug in the firmware has been fixed in the upgrades, I recommend upgrading as soon as the unit is online.

I have in the meanwhile been able to get a full memory dump from the unit I have. I will proceed to take one from another working unit and copying it to the bricked one. I’ll keep this forum posted. However even that still won’t tell me if the bug has been fixed or not, but it might give me the ability to resurrect the bricked controllers using the pirate bus.

Ok, so dumping the EEPROM from the working unit and putting it in the malfunctioning one does not work, it won’t even log anything to UART. I suspect these units have a unique encryption key for the EEPROM contents and moving the contents from another unit does not work, the OS is encrypted in a unique way upon the initial factory line assembly.

For Stewart, the log from the running unit after factory reset is at: https://pastebin.com/JsfcLwGw . As can be seen, at ‘https started for 0’, this one booted up correctly and started blinking orange in the second led segment.

At least I now have a backup of the working unit in case the problem happens again. I’d still really like to have the confirmation the firmware problem where the https server used in pairing fails to start has been corrected in the firmware version I now have (iro3-firmware-hk-5-630).

In plain text for others, it’s RMA and when you do get a new unit, update the firmware the first thing you do.

Just want to add that after having the Gen 3 controller work flawlessly for the past year and a half, I also got the boot error one with no at-home fix working. I will say that Rachio support has been awesome and was able to work out the issue in under 12 hours (and I was sleeping 8 of those :smile: ) A new one is on its way and hopefully a new firmware fixes this issue because it makes no sense that a power failure would cause a piece of hardware to brick.

I do think there must be an issue with the previous firmware (or something else I don’t understand) because in the year and half of having the 3 we’ve had power failures before with no problems…so what changed??

1 Like

Update. New one arrived and it was installed and up & running in about 10 minutes. I’m thankful for the app feature to transfer the settings from the old controller to the new one. This one has a newer (I assume) firmware (650), then the old one (615).

Thanks again to Rachio support, specifically Vitor, for the expedient responses and resolution. I found out it wasn’t working Monday evening and had a new one delivered and installed by Thursday night.

1 Like