Fix the pumps

Saturday, October 06, 2012

Isaac in New Orleans - what we know so far

The first round of primary documents from Hurricane Isaac's hit on New Orleans have arrived. They are the Sewerage and Water Board logs for drainage pump stations throughout the city.

I got the logs for Pump Stations 6 (at the south end of the 17th Street canal), 7 (at the south end of the Orleans Avenue canal), 3 (at the south end of the London Avenue canal), 4 (about half way along the east side of the London Avenue canal), and 1 (in the middle of the city, just lake side of the Broadmoor neighborhood, feeds station 6). Links for these files can be found at the end of this entry. The findings are depressingly predictable. But first let's review the information the Corps did not hold back.

17th Street canal - Still a probem seven years post-Federal Flood

The top news on the morning of August 29, 2012 - when Isaac was lashing the city - as it related to the drainage flow out of the city was that the Corps of Engineers lost communications with their lakefront pump station along the 17th Street canal, at the height of the heaviest rainfall. What emerged from coverage by the Times-Picayune (initial report at 7:41 AM, follow-ups at 8:30 AM, 9:41 AM, and 11:51 AM), the New York Times, and WWL-TV was the SCADA (Supervisory Control And Data Acquisition) system that allows the Corps to run their lakefront pumps remotely had failed early that morning. The Corps could not turn some of the lakefront pumps on, and in order to keep water below the safe water elevation for the canal, they refused to allow the Sewerage and Water Board to turn on more pumps at PS#6 as the water rose in the city's interior. Eventually, the Corps started their pumps at the lakefront manually (an hours-long delay in doing so - despite Corps procedures calling for staffing of the lakefront stations - was never explained publicly) and authorized the Sewerage and Water Board to pump as much as they liked.

The PS#6 logs for August 29th (here and here) confirm the outlines of this story, while filling in some detail. At 4:35 AM, the station staff asked to "load" (that is, bring to full capacity) 1100 cfs pump "F." They were refused. Then, at 4:55 AM the Corps told the S and WB not to "load" any more pumps in PS#6:
"Note: Army Corps of Engineers said we cannot load any more pumps. [T]hey are at their max of what they can pump"
At that time, the station was pumping aproximately 6200 cubic feet per second, or cfs. The station's capacity among its main rainstorm drainage pumps is 9200 cfs.

This initial notice to the S and WB to hold off on pumping appears to be 1.5 to 2 hours before the Corps alerted the media of pumping restrictions, according to the chronology of the NY Times article. Then, at about 6:30 AM, the station operators received another Corps warning not to load additional pumps beyond what was already running, despite the fact that in the intervening hours, water in the station's inlet basin had risen over four feet. Ten minutes later, the station received orders from S and WB's Central Control to break the prime on 1000 cfs pump "D," effectively losing most of its capacity and sending into a pump status called "light." So there were not only Corps orders to keep some pumps from running, but one pump was effectively ordered turned off with the water coming up. This left the station running at about 5200 cfs.

At 7:55 AM, with the water still very high in the inlet basin, the Corps 17th Street canal captain in the station (Ray Newman) gave an all clear to the S and WB personnel in the station to load additional pumps. Five minutes later, 1000 cfs pump "D" was authorized to be loaded by the S and WB Central, and was running in a loaded condition within a few minutes. Another 1000 cfs pump ("C") was brought on line shortly afterward, increasing the station's flowrate to 7200 cfs. Inlet basin levels dropped four feet in about an hour after those two pumps were added. This chronology roughly tracks what was playing out in the media that morning.

What also emerged from this story was that the 2011 remediation project along the 17th Street canal had failed to raise the Safe Water Elevation along the canal from 6 feet to 8 feet, as had been publicly promised repeatedly. Instead, it appears to have only been raised six inches, to 6.5 feet. This number was mentioned in the intial 17th Street reporting from the T-P on the morning of August 29th:
"Engineers were able to start the pumps manually, but because of the rainfall, the water piled up inside the canal a little higher than the corps would like to see, and is currently between six and six and a half feet."
Was further hinted at in subsequent T-P updates:
"The 17th Street Canal is now functioning properly, and water levels have dropped to safe levels... [W]ater levels in the canal have dropped from more than 6 feet to 4.3 feet at 9:30 this morning."
Was nearly confirmed by the NY Times later that afternoon:
"Chris Accardo, chief of operations for the district, sitting at a conference table with Colonel Fleming, quietly explained that gauges in the 17th Street Canal suggested the water had risen to six feet — maybe even six and a half — a height at or even beyond the margin of safety for those flood walls."
And was finally confirmed by WWL-TV the following morning:
"when water reached the 6½ foot maximum operating level in the canal, operators attempted to start the pumps from a remote control system but it didn’t work."

Had the 17th Street SWE been 8 feet, there may have been no crisis that morning. The remediation project on that canal had cost $15 million, and it all appears to have been wasted.

Many other Corps pumping restrictions revealed by logs

What has not been reported until now was that similar stories were also playing out at the other two outfall canals, with perhaps more spectacular events.

At the Orleans Avenue canal, the Corps had pulled two of their ten lakefront pumps out in early August, and had not repaired them by the time Isaac came ashore. In an article published in the Lens on Monday, August 27, 2012, the Corps gave some bland, content-free reassurances about how everything was fine with only 80% pumping capacity available at the lakefront:
"We’ve never had to close the (Orleans Avenue) gates during a storm. Not even during Ike, not even during Gustav," [Accardo] said. "So the threat of these pumps being down and causing any problems is extremely remote. There is hardly any threat to City Park or Mid-City.

"We are not where we want to be at Orleans currently. I would like to be a little higher, but it is not something people need to be worried about," he said. "My standard is to have as many pumps functional at all times."
[...]
"as long as I can stay ahead of the S and WB, it’s not a problem."
The first part of the quote was proven untrue just hours after the Lens published their article, when the gates were lowered at the Orleans Avenue site at around 11 PM on the 27th. That was the first time that had happened since the gates were put in place in mid-2006 and the Corps' pumps were accepted for service in September, 2007. The gates at Lomdon Avenue and 17th Street were dropped the following day, the 28th.

The truly worrisome activity began the morning of the 29th, during the heaviest of Isaac's rain. S and WB pump station 7 at the south end of the canal houses five pumps:

"A" (550 cubic feet per second, or cfs, powered by S and WB-generated 25 cycle power),
"C" (1000 cfs, 25 cycle),
"D" (1000 cfs, Entergy supplied 60 cycle, with diesel generator backup),
"1" (70 cfs, 25 cycle), a so-called constant-duty, or "CD" pump that normally only runs during dry times to pump out runoff from sprinklers and the like
"2" (70 cfs, 25 cycle), the other "CD" pump.

At about 2 AM on the 29th, as the water started rising in the stations' inlet basin, all five pumps were running, though only four were loaded. Pump "C" had yet to be loaded. 60 cycle pump "D" was running on generator backup, after Entergy power had been lost shortly after midnight.

According to the log, the station operators asked to load pump "C" at 2:20 AM, and were denied by the Corps of Engineers. Subsequent requests to load "C" at 3:05 AM and 4:46 AM were also denied, though the log seems to imply it was the S and WB's central control station ("c/c") in both those cases. However, it seems unlikely the S and WB do such a thing voluntarily, so it's sensible to presume the Corps refused to allow the city to run all the pumps at PS#7 for over two hours during Isaac (the results of FOIA rquests to the Corps will reveal the exact reason). Level gauges show the water levels in the canal were between 2 and 3 feet, well below the supposed canal Safe Water Elevation of 8 feet. Thus the only reason to cut back on the water going in the canal is because the Corps couldn't pump it out fast enough. This puts the lie to the Corps' Accardo when he said his pumps could stay ahead of the S and WB. They obviously couldn't.

But things would get even worse later that morning. At 9:40 AM, the log reported:
"Donald Constantine w/C.O.E. reported a Fire w/C.O.E pump"
You read that right.

At least one of the Corps' pumps Caught On Fire During Hurricane Isaac.

Thirteen minutes later, pump "C" - still unloaded - was turned off completely. The logs do not note who ordered that shutoff, but it's not hard to guess.

Then things went completely to pot in PS#7:
"9:55 AM: Lost 25 Hz power. Stop A, CD#1 and 2, MG 1, 2, 3 n/cc"
Since 25 cycle-powered pump C was already off, it didn't have to be shut down when they lost power. But all the other major equipment in the station was lost, including the three motor-generator ("MG") sets that provided power for supporting equipment.

At that point, the station was down from a potential flowrate of 2690 cfs to a generator-backup-powered 1000 cfs, or a 63% drop. I bet even the Corps' pumps could keep up with that.

The problems were not confined to just PS#7 though. Nearly the same script was playing out just to the east along the London Avenue. Two pump stations feed that canal: PS#3 and PS#4. Here's the major pump makeup at each:

PS#3
"A" (550 cfs, 25 cycle)
"B" (550 cfs, 25 cycle)
"C" (1000 cfs, 25 cycle)
"D" (1000 cfs, 25 cycle)
"E" (1000 cfs, 25 cycle)

PS#4
"1" (320 cfs, 60 cycle)
"2" (320 cfs, 60 cycle)
"C" (1000 cfs, 25 cycle)
"D" (1000 cfs, 25 cycle)
"E" (1000 cfs, 25 cycle, out of service long before Isaac approached)

So the potential total flow into the London Avenue canal was 6740 cfs. It should have been higher, but with pump "E" at PS#4 out of the game, the city was starting in the hole.

Nevertheless, the London Avenue pump operators were also refused the privelege of pumping water out of the city on the morning of the 29th. At 4:03 AM the operators in PS#3 logged the following:
"N/cc [notified S and WB central control] to load C pump (12' 25'5") [these numbers are the depths of the stations' inlet and outlet basins respectively], L/rain, Refused"
The operator also drew an arrow pointing to this entry and wrote "NOTE" in dark letters at the end of the arrow, highlighting the refusal to load pump C. Half an hour later, they were refused again. At that point, water in the inlet basin had risen another foot.

Meanwhile, just north at PS#4, the operators were also getting refusals. Remember that one of PS#4's three main pumps - 1000 cfs pump "E" - was already out of service before the storm. Also, they lost Entergy's 60 cycle power at 7:45 PM on the 28th, so their two 60 cycle pumps ("1" and "2") were also never turned on. I am unsure whether there is generator backup at PS#4, but it certainly doesn't look like it. Even with all these losses, when they lost load on pump "D" (that happens occasionally, sometimes because the pump prime is lost) and they called Central at 6:35 AM to reestablish the load, they were refused. A half hour later, however, they reloaded the pump.

What one notices when putting all these logs together was that requests to pump more water into the outfall canals from pump operators across all three canals were repeatedly refused at the height of the storm. Effectively, the Corps' inability to pump out the canals placed an hours-long cork in the city's drainage system. And this is the pumping system the Corps - because of cost reasons - has saddled the city with for years to come.

The exact reasons for individual refusals are unclear, but we can speculate. There's only two obvious causes to stop the S and WB from pumping into the canals: 1) Water levels were above the Safe Water Elevations in the canals; 2) the Corps' lakefront pumps could not keep up. (2) does not seem far fetched at all, considering we already know of Corps pump outages at 17th Street (SCADA system down) and Orleans Avenue (fire). (1) would come about if the 2011 remediation projects along the Orleans Avenue and London Avenue projects were also failures, and the Safe Water Elevations had not been reaised to the promised 8 feet, or had perhaps even been lowered. This also seems likely, since we know there is more remediation work coming on all three canals. I bet the reasons were a combination of both, with multiple Corps pump outages beyond what we know now.

Sewerage and Water Board power outage across three pump stations cripples pumping for hours

Though the refusals to authorize loading of city pumps throughout the early morning of the 29th are quite significant (and point to the hazard of the the Corps' permanent plan to keep the exactly same system in place for at least the next 5 decades), the effects paled in comparison to the loss of 25 cycle power later that morning. The same power outage that struck PS#7 at 9:55 AM also hit PS#3 and PS#4. This had the effect of shutting pumping at both those stations off completely. Among the three stations, only 1000 cfs was pumping, which was at PS#7. As at PS#6, inlet basin levels at all three stations shot up two to four feet very quickly with nowhere for the water to go.

The effects of the outage lasted different times at different stations. At PS#4, they were back up with their two functioning 1000 cfs pumps about an hour later. At PS#3 they were able to get everything running again in about 90 minutes. But at PS#7 it appears they were unable to restart pumps "A" and "C" until much later in the afternoon because of subsequent electrical problems.

In the middle of all that, the Corps reported at 11:10 AM they had put out the pump fire at the Orleans Avenue lakefront gates. But with all but one of the city's pumps in PS#7 down at that point, it was pretty irrelevant. The Corps' pumps were only receiving about 1000 cfs of water, so they could keep up with just five of their 10 lakefront pumps if they wished. So when the Corps reports they were able to keep up with what the city sent them, take it with a grain of salt. They couldn't, as the repeated refusals earlier in the morning of August 29th amply demonstrated.

Corps pumping refusals along Orleans Avenue canal continue nearly 1 day after initial incidents

One might think after all that activity, the worst had passed. And for the most part that was true. However, in the dead of night, at 12:55 AM on the 30th, the Corps once again stuck a plug in the city's Orleans Avenue canal drainpipe. From the PS#7 logs:
"O/ Donald Constantine w/C.O.E. Break prime on C pump. N/cc"
That is, the S and WB operators in PS#7 were ordered ("O/") directly by the Corps' Orleans Avenue canal captain to remove most of the flow from 1000 cfs pump "C," leaving just the flow from 550 cfs pump "A" (which had been loaded minutes earlier) and 1000 cfs pump "D." Inlet basin levels rose 2.5 feet in an hour. The station continued to run in this diminished state for four more hours, eventually lowering the inlet basin level back to where it had began before the Corps order to stop pumping. This is more evidence the Corps' Orleans Avenue canal lakefront pumps - two of which were removed from service - could not keep up with the city's pumps.

In fact, at no time during Isaac did city pump station PS#7 run at maximum capacity, a remarkable fact that blows to bits the ridiculous pre-storm reassurances from the Corps. That is, the city was refused permission by the Corps to run its drainage pumps at PS#7 as fast as it wanted to for the entire storm.

Wrapping up

In sum, as with every story involving the Corps of Engineers, there was much more happening behind the scenes than what the public was told during Hurricane Isaac's trudge across New Orleans. Restrictions on city pumping into the outfall canals - imposed by the Corps - occurred earlier than was previously reported and extended across all four major stations putting water into those canals. Specifically at the Orleans Avenue canal, the Corps appears to have limited city pumping for the entire duration of the storm, in part due one of the Corps' pumps catching on fire at the height of the storm. A Sewerage and Water Board power outage at the stations on the Orleans and London Avenue canals - also at the height of the storm - masked the effects of some of the pumping restrictions.

The Corps issued a $630 million contract on September 28, 2012 to construct permanent pump stations at the lakefront on all three outfall canals. The winner was a coalition called PCCP Constructors - formed specifically for the project - made up of Kiewit, Traylor Brothers, and local contractor M.R. Pittman . In part they will be supplying $70 million worth of Patterson Pumps, likely very similar to the direct drive pumps currently installed at the 17th Street and London Avenue gate structures and also supplied by Pittman and Patterson [edit, 10/8/12 - the 17th Street direct drive pumps are from Fairbanks-Morse, not Patterson]. The permanent stations are intended to replace the current unreliable stations by 2016. But the overall tandem pumping scheme, dubbed Option 1, will remain in place. As Isaac demonstrated amply, that is a dangerous way to operate. As more documents roll in, I will continue to add to this story.

Links to Isaac Sewerage and Water Board pump logs

Zipped files containing daily logs for each station from Sunday, August 26, 2012 through Tuesday, September 4, 2012:

PS#1 (in city interior, north of Broadmoor neighborhood)
PS#6 (at south end of 17th Street canal)
PS#7 (at south end of Orleans Avenue canal)
PS#3 (at south end of London Avenue canal)
PS#4 (half way along east side of London Avenu canal)

Individual daily log sheets for each station:

PS#1
PS#6
PS#7
PS#3
PS#4

Go to older posts Go to newer posts