Ubiquiti EdgeRouter Lite ERLite-3 – Internal USB Storage Failure

Some Background:

I currently own 6 of these Ubiquiti Edgerouter Lite ERLITE-3 routers, I have been using them since late 2013 and was impressed by their sustained speed and never needing to be reset under heavy load (30+ users) that would bring most consumer grade routers to a standstill.. Unfortunately it also turned out that sometimes upon reboot they would not come back online, which led me to RMA two of them(Ubiquiti has a good 1 year return policy) and also put all of them on UPS backup power supplies.  Unfortunately this did not stop the failures from happening, it didn’t seem to matter if the router was hard reset from power failure or the reboot command from the GUI, upon reboot the device would not restart with nothing visible on the ETH0 interface. Pushing the hard reset switch on the device did nothing.

If you happen to have or purchase on Ebay for $15 a Cisco CAT5 – USB serial cable you can access the internal hardware serial port through a serial terminal locally connected to the device. – Keep in mind this is not necessary and you can use the guide further on to rewrite your USB drive without needing any serial cables.

Terminal Output Upon Boot:

Looking for valid bootloader image....
Jumping to start of image at address 0xbfc80000

U-Boot 1.1.1 (UBNT Build ID: 4493936-g009d77b) (Build time: Sep 20 2012 - 15:48:51)

BIST check passed.
UBNT_E100 r1:2, r2:12, serial #: DC9FDB29B637
Core clock: 500 MHz, DDR clock: 266 MHz (532 Mhz data rate)
DRAM: 512 MB
Clearing DRAM....... done
Flash: 4 MB
Net: octeth0, octeth1, octeth2

USB: (port 0) scanning bus for devices... 1 USB Devices found
scanning bus for storage devices...
Device 0: Vendor: Prod.: USB DISK 2.0 Rev: PMAP
Type: Removable Hard Disk
Capacity: 3700.6 MB = 3.6 GB (7579008 x 512)
 0
reading vmlinux.64

** Unable to read "vmlinux.64" from usb 0:1 **
argv[2]: coremask=0x3
argv[3]: root=/dev/sda2
argv[4]: rootdelay=15
argv[5]: rw
argv[6]: rootsqimg=squashfs.img
argv[7]: rootsqwdir=w
argv[8]: mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@3072k(eeprom)
## No elf image at address 0x09f00000
Octeon ubnt_e100#

As you can see the 4GB USB flash drive image is either not present or corrupted..

Solution:

If you have another working router with un-corrupted flash you can use this tool to directly image the whole drive and make an exact duplicate. Since I had a brand new replacement router handy fresh from the RMA process this is what I did.

You will need a new 4+ GB USB 2.0 Flash Drive:

The SanDisk Cruzer Fit 8GB USB 2.0 drive is NO LONGER recommended as Sandisk seems to have come out with a slightly different newer version (under the same part number!) that is no longer compatible with these routers.. It was reported in post 308 in this thread: http://community.ubnt.com/t5/EdgeMAX/EdgeMax-rescue-kit-now-you-can-reinstall-EdgeOS-from-scratch/m-p/1494957

I have not looked too far into finding a new replacement drive that always works, as all my routers are still functioning on the older Sandisk ones. My recommendation to anyone doing this fix is to try any USB 2.0 drives you already own that will fit in the case. Anything from 4GB – 16 (maybe 32gb?) should work. You wont gain the extra storage space unless you reformat the partitions, but the extra space may help the flash drive keep itself running longer as it should in theory be able to reallocate bad sectors to the un-formatted space – if its controller is smart enough.


Method 1 – Requires windows computer – Simple

Here is a fresh copy (default user pass) of the USB flash drive compressed in a Zip file. It is only 97MB but will be 4GB once you decompress it 😉 shows you how much of that space on the partition is empty.
EdgeOS 4GB Image.zip MD5 file checksum B57BFED8BC721D3E58F5D290DD12538A Verify it here

  1. Download drive image from above – extract the .bin file somewhere.
  2. Download Image USB utility http://www.osforensics.com/tools/write-usb-images.html
  3. On your computer plug in the blank Sandisk or other quality flash drive you want to copy the image onto.
  4. Select the flash drive you wish to write to in “Image USB”
  5. Select “Write to UFD”
  6. Select the .bin image you downloaded above.
  7. Click “Write to UFD”
  8. Wait for process to finish
  9. Plug back into your ERL and turn on! – all settings are default

Done!


Method 2 – Requires Linux command line knowledge – If not familiar use Method 1

Mike from below in the comments section has created this neat script to generate your own flash drive image from up to date firmware upgrades downloaded direct from Ubiquiti. Below is a quote from him:

“Major differences from other methods:

  1. you can use an up-to-date system image
  2. you can include a config backup, if you have one, in the generated .img so that your router boots back into your own network setup right away, rather than having to fuss with static IP addresses and cables
  3. it’s in a plain .img format so that non-Windows users can dd the image with already-installed system utilities;
  4. it fixes a few bugs in other similar utilities that were formatting the USB drive incorrectly, possibly causing problems with certain brands of USB drives.”

His script code with instructions: https://github.com/sowbug/mkeosimg

I have not personally tried this method yet – but it sounds promising next time something comes up with one of my routers. – According to Roberto and Shawn in the comments it works great!

I have since tried this method and it works great!! You will probably get an error like this “chown: changing ownership of “/mnt/boot/vmlinux.64”: Operation not permitted” and the same for “vmlinux.64.md5” which can be ignored – image works fine.

Here is a quote from Shawn “Thank You! Just rescued an early (plastic case) ER-lite using Mike’s script. Script did throw a couple errors: ‘chown: changing ownership of “/mnt/boot/vmlinux.64”: Operation not permitted’, and an same error for ‘vmlinux.64.md5’, but the resulting IMG works.”


There are newer versions of the firmware since I made the image so my first step after doing this is to download and install the latest firmware using the normal web interface procedure.

Only caveat is that I just performed this operation a few days ago so only time will tell if the Sandisk flash drives really work in the long run.

Update 2015-12-17: It has been 13 months now with no failures of the Sandisk 8GB drives.

Update 2016-9-27: It has been 23 months now with no failures of the Sandisk 8GB drives. – Note the same drives Sandisk ships today are not compatible with these routers. – Don’t ask me why.

Ubiquiti Community Forum member NVX is responsible for the idea behind this(thanks!) – he reports no failures since replacing the original drives with Sandisk ones. So lets hope for the best!

Lots of other useful information is in this thread:
http://community.ubnt.com/t5/EdgeMAX/EdgeMax-rescue-kit-now-you-can-reinstall-EdgeOS-from-scratch/m-p/1081065#M46148

Including information on how to manually reinstall the router images into a blank flash drive and increase the partition sizes if you want more than 4GB accessible to your router.

Router Reference: http://www.ubnt.com/edgemax/edgerouter-lite/

Defective Drive (4 have failed on me):

Drive in most recent RMA replacement router I have received (this drive may not fail prematurely):

54 thoughts on “Ubiquiti EdgeRouter Lite ERLite-3 – Internal USB Storage Failure

  1. Thanks, you saved my 3 days old erlite3 after i accidentally bricked it by trying to upgrade to v1.7.0rc1 :D.

  2. Smart solution. I had the exact same issue you had.

    It seems Ubiquiti has some issues with their USB mem-sticks.

    I really thought the router in general was too good to be sent to the trash, so I am very happy this worked out!

    Thank you for sharing this solution!

  3. Thank you very much for this solution… after severals attempts with other kind of solutions this was an elegant and awesome simple.

    this routers are great… and if the storage is a problem… just change the usb drive!! heheheh

  4. 2 here with dead/corrupted USB drive. I’ve had trouble finding brand name replacements that are small enough to fit in the chassis. mine dead outside the 1yr warranty.

    is there any filesystem/config tuning that needs to be done to stop these USB drives from being corrupted / worn out ?

    1. Hi, as to replacements, most flash drives can also be disassembled by removing the outer plastic case, this should make almost all of them small enough to fit on the PCB connector, as long as they aren’t super long. Makes them have better cooling too which may help with longevity.

      And to your second question on file system tuning, I don’t have enough expertise in that area to know if any improvements are possible, but from personal experience I think it is mostly the fault of Ubiquiti using inferior flash drives that don’t perform proper error correction or wear leveling.. Buy Sandisk is my recommendation.

  5. Success with the 32GB version of the recommended SanDisk Cruzer Fit model, but the 8GB version refuses to work. On the theory that the update to the latest 1.7 caused the hardware to be unable to boot older versions of the firmware, I tried mirroring both partitions and MBR of the 32GB drive to the 8GB, but that didn’t do anything.

    I have an RJ45-DB9 serial cable on order and will see if the console output tells us anything helpful. At this point it’s academic, because I have a working router once again — albeit one that wastes ~30GB of flash memory.

    1. Mike – your 30GB is not “wasted” – it’s now extra capacity for wear-leveling or using for bad block reallocation.

      Indeed, over provisioning is commonly used. In enterprise SANs one of the reason flash arrays are so epensive is they can have dramatic over-provisioning; as much as 60% of the flash memory you buy is typical not provisioned, but kept in reserve for wear leveling and error recovery.

      1. EricE, that’s interesting. It implies that the wear-leveling on the controller knows at least a bit about partitioning in order to use the extra space. Obviously it couldn’t reuse the 30GB if it were part of the partition, but it theoretically could use it if the 30GB were part of unpartitioned space. That means that reusing the advertised space for wear-leveling depends on knowing how to parse the GPT/MBR, which seems unlikely. But I don’t know the first thing about USB-drive controllers, so for all I know that could indeed be the case.

        1. Wear leveling needs no knowledge of the filesystem, all it does is dynamically shift writes to areas that are less used as the drive ages and is written to.
          Over provisioning (eg, either having a drive advertise itself as 4GB but really having 8GB of flash OR having an 8GB drive with only a 4GB partition) takes this a step further by providing more flash for the wear to be spread across.
          Something i don’t know however is how things are handled when the time comes that a piece of flash becomes too unreliable to use.
          With the smarter controllers and factory over provisioned drives i assume this is easily handled by having the faulty cells permanently made unavailable.
          In the case of a lesser smart drive that is advertised and actually is size X with no over provisioning i guess that is when the drive dies.

          A very late reply, came across this as my ERL flash has died (revived once, but now seems to be write locking itself again, not long to go i think) and i’ll be looking for a suitable replacement.

          In the last 3 years i’ve lost faith in Sandisk, will likely be looking for something else.

    2. An update now that my EdgeRouter has failed again and I’m going through all this again: I ended up writing a script (based on some other similar projects I’ve seen out there) to generate a .img based on any system image downloaded from the ubnt.com site: https://github.com/sowbug/mkeosimg.

      Major differences from other methods: (1) you can use an up-to-date system image; (2) you can include a config backup, if you have one, in the generated .img so that your router boots back into your own network setup right away, rather than having to fuss with static IP addresses and cables; (3) it’s in a plain .img format so that non-Windows users can dd the image with already-installed system utilities; (4) it fixes a few bugs in other similar utilities that were formatting the USB drive incorrectly, possibly causing problems with certain brands of USB drives.

      1. Hi Mike,
        I ended up almost managin to go that route (without prior Linux knowledge googling Ubuntu skills). I wanted to ask how I manage to get this last command done with Ubuntu. Somehow it works, but nothing appears on the stick.

        sudo dd if=ER-e100.v1.xxxxxxx.img of=/dev/path/to/usb/drive bs=1M`

  6. Thanks alot for this guide, and for the update the sandisk drives are still working. Merry christmas and a happy new year 😉

  7. It would be great if there was an ISO version of that image. I’m on a Mac and BIN is not supported. I just sent mine back to Amazon after it bricked. I would like to be prepared is the next one fails. Thanks for the post.

    1. I’m not too familiar with what options are available for writing images to flash drives on OSX, if you wanted to you could use the flash drive inside your brand new router to create the ISO image, you’d just open it up and take the USB drive out and create the image. There was no “warranty void if broken” seal on any of mine. If you do this I’d be happy to host the ISO file here for others to use. The other option is to use virtualbox with windows or borrow a windows PC, it only takes a few minutes to write the image.

  8. I, too had an ERlite fail in the same manner. My USB drive contained the exact same model #’s as the defective drive in the first picture.
    I used what I had at the time – an old 8Gb Verbatim. Had to chisel some of the case out of the way as it was a little large, but it works. Thanks for the image file!

  9. I notice the EdgeLite has more than the USB flash storage, it has a Windbond 25X05CL SPI flash chip and a MX29LV320ETTI-70G NAND flash chip. What are all these used for?

  10. Just tried to upgrade to 1.8 today, rebooted and it would not start. Found your page, took the ERlite apart and sure enough, the first partition was corrupt and would not work. I first tried you trick on a 8gb USB but it wouldn’t boot. I found another 4gb USB stick (had to trim it down), and presto – it worked! Do you think there is any chance that the USB sticks are failing because the unit is getting too hot? I have seen a number of folks put fans into the cases to cool things off. Thoughts? Is it worth adding a external cooling fan to the case to help keep it cool?

    1. Hi Marc, I did speculate that heat was causing the USB flash drives to fail when mine first started to die.. I even have a small brush-less blower directing air over the case of my main home router. However after replacing all my flash drives I haven’t had anymore failures, and only 2 of my 4 routers have fans on them. So my conclusion at the moment is that the problem is inferior flash drives.. I’ll update this if I ever have one of my Sandisk drives fail. I really only trust Sandisk flash now 😉 and would never buy a no-name USB flash. But at the same time a fan to keep things cool shouldn’t hurt so if you have some fans lying around go ahead, even a little 80mm 12V box fan blowing air over the case keeps the router at least half the temps.

      Thanks for the comment which allerted me to the V1.8 firmware 🙂 I am installing it now, some nice new features!

  11. Just a heads up, I’m seconding the comment saying this doesn’t work with the 8GB SanDisk drive referenced at the start of this article. I followed the instructions multiple times, trying multiple permutations, but router never comes up with this drive. The drive works fine and comes up with no errors when connected to the computer, but never fires up at all in the router.

    1. Did you try with other USB flash drives? If you only tried with the one I’m guessing it has something to do with how the drive was formatted and or the image uploaded to it. Of course its possible Ubiquiti made it harder to copy images but I haven’t encountered that yet.

  12. Holy shit batman – that was a seriously handy fix.
    I had one of the early usb sticks. Also died when trying to install 1.8.
    Mine failing on the upload of the 1.8FW so I suspect it was out of storage (mentioned on the forums somewhere).
    Might have corrupted it somehow.
    FYI – attempted to flash a USB stick with DD from OSX – no joy.
    Managed to use the win application listed above from within a virtual machine with no issues.
    Took a while but I got there.
    Used a little Apacer 16Gb stick. Super skinny – fit perfectly.
    So far so good.

    Thanks again 🙂

  13. Hello, thank you for the great post. Really glad to have my router back online. I appreciate you taking the time to provide this helpful post.

  14. Just had to do this again – 3rd time in 4 years. I had added a cooling fan – to no avail other than it looks cool! I also tried a different set of instructions. Just follow STEP 1 on this site:

    https://community.ubnt.com/t5/EdgeMAX/EdgeMax-rescue-kit-now-you-can-reinstall-EdgeOS-from-scratch/td-p/514857

    After STEP 1 – Reboot with console on, edit /config/config.boot to have your old IP address, then connect to WEB UI and restore config – one last reboot and done!

    1. I used to have similar problems (replacing the drive repeatedly), finally swapped out the little ubiquiti power brick for an old Linksys one and I haven’t had to touch my edgerouter since.

      1. hi
        i have device with 5 ports. Image is good, in Config tree just add eth 4 and eth5. Now working all 5 ports:)
        Thanks

        1. Interesting that works! You may have some other unexpected issues as it is the wrong firmware image for your device, but I am not sure. Can you use the web GUI to upload the correct firmware for your 5 port device?

          1. yes, i install update firmware to 1.9 for 5 ports edge. 3 and 5 ports router have the same size and name of update.

          2. Oh yeah you’re right they have the same firmware image! Must just be a configuration file change that stays on the router between updates. Glad its working!

  15. Just used Mike’s method to replace the corrupted USB on my Edgerouter Lite. Worked like a champ. I edited the script to take advantage of the full size of my 16Gb Kingston Datatraveler USB when creating the root partition. Router is running beautifully. Thanks for the tips.

  16. Thank You! Just rescued an early (plastic case) ER-lite using Mike’s script. Script did throw a couple errors: ‘chown: changing ownership of “/mnt/boot/vmlinux.64”: Operation not permitted’, and an same error for ‘vmlinux.64.md5’, but the resulting IMG works. Used a Verbatim Clip-it 4GB I’ve had laying around. Nice thing about the Clip-it is it fits w/o modification. Used v1.9 firmware and included my configuration (likely saved from v1.7 or 1.8). Thanks again!

  17. Thank you so much. Several months ago, my ERL became unresponsive and I was unable to login, even after a full factory reset.
    I had tried to manually load the system image using the ‘rescue kit’ but the router wouldn’t boot correctly, and gave waht appeared to be disk errors.
    After suffering for months with a generic ISP supplied router I was about to buy a new ERL but thought I’d give it one more go. Today I found your page and I have successfully used your drive image and used it to flash a new usb drive.
    LIFE
    SAVER.

    Well, maybe that’s a bit dramatic but thanks just the same.

  18. Big thanks to you and Mike for figuring this out and publishing it. Guys like you have been saving my bacon with stuff like this for years, but this one is special. It took me longer to scrounge around and find a thumb drive small enough to fit, than it did to copy the files and fire it up.
    Thanks again!

  19. Hi everyone. Tried to recover my router ERL from dd-wrt. But I have “counterfeit” in system. But see many recovered their routers with this image without problems. Also saw on ubnt forum this problem has some people, there is no clear answer from anyone what is this can be. Some answer that this can be hardware failure but not the usb drive. But on another forum read that there is a partition on flash drive with some specific information. Who knows something about this problem?

  20. I just got a new Edgerouter Lite. Thinking ahead of replacing a defective flash drive I made an image backup from a running Edgerouter Lite. It does dd over ssh as follows:
    ssh into the Edgerouter
    sudo su
    dd if=/dev/sda | ssh @ dd of=ubnt.dd
    Wait 5 to 10 minutes. After the image copy has finished you will find ubnt.dd on the remote host. This you can use to create a new fresh flash drive in case the old one is damaged.
    Although not tested with an Edgerouter I tried this method with Raspberry Pies I use at work.

  21. I see some characters are not posted. The dd command is: dd if=/dev/sda | ssh username@ipaddress_of_remote_host dd of=ubnt.dd

  22. Tested 8GB Philips pendrive model: FM08FD70B and works perfect !!

    Probado el pendrive de 8GB Philips modelo: FM08FD70B y funciona perfecto!!

  23. Hi there,

    First of all thanks for this tutorial, I was able to bring my ERL back to life.
    I am running into this weird issue, every time I powercycle the router I get this error:

    USB: (port 0) scanning bus for devices…
    USB device not responding, giving up (status=0)
    1 USB Devices found
    scanning bus for storage devices…
    No device found. Not initialized? 0
    ** Can’t read from device 0 **

    ** Unable to use usb 0:1 for fatload **
    argv[2]: coremask=0x3
    argv[3]: root=/dev/sda2
    argv[4]: rootdelay=15
    argv[5]: rw
    argv[6]: rootsqimg=squashfs.img
    argv[7]: rootsqwdir=w
    argv[8]: mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@3072k(eeprom)
    ## No elf image at address 0x09f00000
    Octeon ubnt_e100#

    What I have to do at this point is connect the console cable and type reboot, the ERL boots up properly after that, but I can see this getting very annoying down the road.
    I have tried two different flash drives so far (same results): Sandisk Ultra Fit 64GB and Kingston Datatraveler G3 32GB.

    Any ideas?
    Thanks a lot!

    1. I had some trouble getting a flash drive that functioned. The router would not boot automatically on power on, but it could be booted manually. Adding a boot delay helped.

      If you waiting for a bit (a few seconds) after receiving this error, can you manually continue the boot?

  24. Hi,

    Just wanted to say thank you for the article and thanks to Mike for the script. I managed to get my ER-Lite working again (after a whole year collecting dust). I created the image with Mike’s script, firmware ER-e100.v1.9.7+hotfix.4.5024004, and flashed it to Kingston DataTraveler SE9 16GB stick with Etcher app.

    The script did throw a couple of errors as mentioned before (“chown: changing ownership of “/mnt/boot/vmlinux.64”: Operation not permitted” and the same for “vmlinux.64.md5”), but the output image is working just fine.

  25. This worked perfectly for me using one of the older EdgeRouter 3 Lites (the plastic cover version with rounded off top-corners). I had saved it hoping it might not be complete dead, although I’ve since bought a couple of the new versions with a metal case which is entirely rectangular. I used an 8GB Sandisk Cruiser Glide USB key I had laying around, but had to do a tiny cut in the case to make it fit. So I now have a backup router just in case. Thank you!!

  26. Nearly 5 years later and this page is worth 10x its weight in gold. I deconstructed an old 8GB cruzer to make it fit. Downloaded the newest image from elsewhere. Used imageUSB and after 20 minutes the brick I bought on ebay (sold as functioning unit no less) is responding on eth0. I knew there were various methods for recovery on these units so I took a gamble and accepted a partial refund which was over 50% what I paid. After reading many different methods, this appealed to me out of ease (imo) and it did not disappoint! $35 (plus old-AF thumbdrive) for a working edgeouter lite. If I knew you, I’d buy you a beer.

    1. Hi Rob, If you ever make it to Hilo Hawaii I might take you up on the beer 😉 but seriously glad this is still helping people! The first router I replaced with the Sandisk drive is still working so thats good too..

  27. Hi chester Lowrey,can you make video guide by mkeosimg method,i don’t understand how to do it

  28. Just came across this method, and for a novice its sound interesting, but the image file is missing. Where can i found one?

Leave a Reply

Your email address will not be published.