Note: Christi normally writes the blog posts, however, since this is a boat technical post, Eric has authored this one.

Tuesday, February 25: We rolled out of bed at 0400 and did all the last minute things we always do before going to sea. Our first inkling that something could be wrong came at about 0530, when I turned on the main engine to warm it up. Before we get into the details, here is some important background information:

While underway, electricity to charge the batteries for Kosmos is made from an alternator on the main engine. It is an important system, and having electricity from the main engine allows us to run all the boats systems without running the generator. While underway, the boat uses quite a bit of electricity for air circulation, navigation/radar, and all the usual loads (refrigeration, freezer, lights, Internet etc.).

The main engine alternator is externally regulated by a product from ARCO called Zeus. External regulation is a general practice for interfacing with Lithium Iron Phosphate (LiFePo) batteries, which was part of the upgrades made to Kosmos after we got back from the Sea of Cortez. Zeus has an app and can connect via Bluetooth for configuration and monitoring. Mainly, it can reduce the load on the alternator ensuring it does not overheat. The Zeus box has several sensor wires for temperature, RPM, voltage, ignition, regulator control, etc.

During the Zeus box install it was discovered the voltage sensor would not work without doing some rewiring to the main battery shunt. This turned out to not be an issue, because interestingly, the voltage sensor did not stop the charging operation of Zeus.

Per installation instructions, the RPM sensor wiring on the alternator needs a fuse. At one point, I checked the fuse to see how I would change it, if needed. I pulled the fuse and put it back in, but I did not push the fuse in properly, which meant there was no connection. This caused Zeus to sense zero RPM, which caused it to not charge. When the fuse was pushed back in, all worked properly.

There was also an ignition sensor on the Zeus. As with the RPM sensor, the unit would not work without it. So the installer needed a way to tie the ignition of the main engine into the Zeus. A decision was made to pull the ignition voltage from the SeaFire panel. SeaFire is a fire safety system that shuts down the blowers, main engine, and generator if there is a fire detected from the fire suppression system.

I opened the ARCO Zeus App about two weeks ago for a couple reasons. First, someone was asking about what kind of configuration it had, so I wanted to get accurate details. Second, I actually wanted change the load charging curve because the parameters were a bit too aggressive on turning down the charge at lower RPMs. Things had been working fine for over 3,000 nm, but I wanted to adjust slightly as we will be potentially operating at lower speeds while in the shallow waters of the Bahamas. The app said a firmware update was required to make setting adjustments. I said no, and did NOT update. As a general rule, I do not want to do any updates without understanding exactly what the changes will be. It seems, however, the act of opening the software settings app on my phone did in fact set something different on the Zeus hardware box installed in the boat.

Back to today: I turned on the engine and the alternator was not generating power. I went to the Zeus App and it said zero RPMs. Since I had seen this before, I assumed it might be the fuse again, even though it seemed unlikely. I checked and even changed the fuse, but alas still no RPM. I wanted to look into the settings to see what else could be wrong. But it would not let me without the firmware update. We had a short departure window, and I wanted to have the alternator working. I figured a firmware update labeled 1.2 to 1.23 would be reasonable risk to take in order to troubleshoot better. Oh my, how wrong I was…

Right after the update, the RPM sensor started working properly. Success! However, this made me deduce that the Zeus App had somehow changed the state of the Zeus hardware box, telling it some kind of update was needed in order to operate. Yikes! If so, this was an unbelievably bad and unsafe design. To 1) not allow an app version to work with older firmware to at least check settings, and more importantly 2) change any state in the hardware without giving the user the option to say no. ARCO essentially created a mandatory stealth update to the hardware. This update both disabled existing operations, and — as you will discover as you read on — the update horribly malfunctioned…

Initially pleased that the RPM sensor was working, I then I got an alternator fault message from Zeus saying the voltage was too low. Of course, as noted above, the voltage was indeed zero from Zeus’s sensor perspective. Unfortunately, now with the new firmware, the voltage reading was stopping the Zeus from charging the batteries! And there were no firmware rollbacks. Not that a rollback would matter, as the Zeus was nonfunctional after merely checking for an update. Sigh.

I noticed in the Zeus App there was a minimum voltage setting. And the app let me set it to zero. Then the low voltage error cleared and current started flowing. I revved up the engine a bit, and charging looked to be normal. We were back in business! I thought I was both clever and quick enough to get us back on schedule. We cast off our lines. It was a beautiful morning, with less clouds than normal and an especially colorful pre-dawn. 

As you saw in the photo above, the channel into the marina was pretty tight, so I kept the boat at low RPM as we moved into the bay. Once we cleared a nearby reef, I set the auto pilot on our route to take us to sea, and then increased RPM to 1750 for cruising speed. I glanced at the alternator output amps and saw it at 160, which I thought was oddly high since the batteries were charged. Also, the generator was running for some air-conditioning, and was putting in a trickle charge of amps. I looked over at the ship’s voltage and did a double take as it read 15.1 volts, which is well above normal. I exclaimed, “Oh, fuck! The electronics are going to fry!”  

Then things got scary. Only a moment later, the main house DC power (from our LiFePo battery bank) cut off from the boat. This powers most of the boat’s systems. Navigation screens went dark, lights off. And the main engine and generator turned off, which I thought should have been impossible. The engine alarm blared, which is powered from another battery bank. I tried to start the main engine, but it would not start. In a procedure I have practiced in my head many times, I reached up and turned on the wing engine. The wing engine was a completely isolated propulsion system with the sole purpose of being available if there was a problem with the main propulsion system. It was the first time in nineteen-years that I have had to use the wing engine in an emergency, and I was certainly glad it was there. That gave me some steerage, and reduced some of the panic. The breakwater was close, an anchored sailboat was close, I’d just passed a reef, and there were a slew of sunken boats in the distance. Did those sunken boats have Zeus voltage regulators installed, as well?

Kosmos actually has four 12-volt battery banks with different amp hour (Ah) capacities and battery chemistries: 1) House (LiFePo chemistry, 1330Ah), 2) main engine (AGM chemistry, 255Ah), 3) generator/wing engine (AGM, 105Ah), 4) bow bank (thruster and windlass) (AGM, 510Ah). In an emergency, the banks can be connected in various ways. For powering the house load, the preference was to parallel in bank 4, the bow bank.

Well, I was in an emergency, so I flipped the switch to connect the bow bank to the house. Only the AC engine lights came on, no DC. But then the AC lights turned back off. Something was not right. I thought maybe it could be an issue with the interaction between bank 4 and the house bank battery manager, which was currently trying to reboot the system. Or maybe it was an issue between the DC to DC chargers that enabled the house bank to charge the bow bank. Or maybe it was the battery disconnect module that was in the house bank. Whatever the reason, the emergency backup system did not work as expected.

The house DC system then came back online after the Battery Manager System (BMS) did its reboot process. I tried the main engine, and it started. But it was short lived, as the exact same thing happened again — the house DC shut off and the engine stopped. Trying to protect itself from the unregulated voltage, the BMS had disconnected the batteries again.

The urgency to get the main engine started was that steering and maneuvering the boat with the wing engine was tricky. It had a sluggish response, and because the wing’s propeller was on the port side, it made the boat want to rotate on a starboard turn. This meant that the wing required full throttle to actually steer, and the steering was counter-intuitive.

We kept in mind that the situation could have been much worse: The sun had already risen and visibility was excellent, there was only about 5-10 knots of wind, and we were in a relatively protected bay with few boats anchored. But we started getting close to an anchored sailboat. I decided it was going to be tricky holding precise position and our best course of action was to drop anchor. The bow thruster and windlass battery bank were a separate bank and had power. I decided to undo the emergency switch from the bow bank to the house, not wanting to have any unexpected interaction there.

The LiFePo BMS had actually kept trying to reboot itself, and the house DC system came back online. Before attempting to start the main engine or drop the anchor, I went right to the Zeus App configuration and set the minimum voltage to 12. Then I started the main engine. Zeus reported a low voltage warning and did not turn on charging. I tested the main engine with a few circles while watching the voltages and the Zeus App. With the main engine operational again and Zeus disabled, I decided the best course of action now was to get back to a slip in the marina. We safely navigated the channel and returned to the slip we had vacated.

My first speculation was that the voltage traveled through the ignition sensing wire for the Zeus and killed the ignitions on the main and gen. My first order of business was to remove that connection, and come up with a work around.

Then I tweaked the parameters of the Zeus. Low load seemed to working, so I turned everything down to max 60% load. For the RPM we usually run at, it would stay at around 50%.

I called the engineer who’d installed the system. We each poured over the details and tried to figure out what was going on. He contacted ARCO on our behalf. Maybe there was some work around or a known issue, but ARCO stopped responding. The engineer and I were frustrated and angry at ARCO’s poor customer service.

It turned out the disconnect of our 12v house bank caused the main engine to shut down. At this point, I believe this is how SeaFire is wired. And since voltage is required to turn off the flow of fuel to the engine (a fuel shut off solenoid, normally open with no power, and power needed to close it), the power came from the engine battery bank. As soon as the house 12v was restored to the SeaFire system, SeaFire changed its state to normal, and stopped sending the engine battery bank voltage to shut off the fuel. I need to verify this, and I will update as we figure out the puzzle.

It also turns out that the battery disconnect module was the culprit for stopping instant emergency power to the main power distribution panel. It also probably helped save the electronics from triggering fuses. The main panel is where the navigation, lights, and pretty much most of the boat runs from. During the install, I had tested to confirm the emergency voltage was working to the distribution system, but I hadn’t tested the main panel. Sigh. The BMS told the battery disconnect module to disconnect voltage to the main power panel, and it would not proceed to reconnect until the BMS fully rebooted and determined it could safely energize the system. This meant no emergency power to the panel. So I bypassed the battery disconnect, and verified that we could quickly parallel the other battery banks to the power panel. The battery disconnect is there to protect the LiFePo batteries from getting too low, especially if no one is onboard to deal with the low battery situation. We might just live without it for now, since we are onboard every day and actively keep the batteries charged. Again, I will update as we figure out the puzzle.

We want to close by saying that, while we have lost all trust with ARCO and its Zeus product (mix of frustration, anger, dismay, etc.), we are grateful that the failure happened when it did. In different circumstances, this failure could have been a material loss, and life threatening. Had Zeus gone out while we were in the narrow channel, we likely would have crashed. Likewise, if we were closer to the reef, or if the anchorage was crowded, then there was a high chance of crashing. Had it been dark, it would have been scarier and more dangerous. And if we’d been in the open ocean with no stabilization, it could have been more difficult to deal with the situation.