Fix the pumps

Monday, July 13, 2015

A $6.68 million question for the Corps

Updated 7/24/15. See bottom of post.

Imagine you are a person, made of flesh and blood. You have hopes and dreams, likes and dislikes, needs and desires. Imagine further that you lived in the greater New Orleans area on August 29, 2005. Perhaps you lost your home, or your business, or worst of all, someone you loved. Before the levees failed you had a relationship of "protected" to the Corps of Engineers' "protector." Afterward, you struggled for months or years to return and then rebuild, and during that process you came to understand the Corps bore ultimate responsibility for your losses, since their storm protection failed well below their design limits. While you sweated, and cried, and laughed, and stressed through the rebuilding, you maybe - no, definitely - signed up for a class action lawsuit against the Corps.

As your personal recovery - and that of your neighbors and your city or parish - dragged on, you occasionally took note of the legal cases wending their way through the courts. Never was there talk of a global settlement; it was always the Corps fighting tooth and nail. Even when the head of the Corps, Carl Strock, admitted on June 1, 2006 they screwed up (but notably did not apologize), it didn't make a difference in the courts. Eventually all the cases died, leaving the Corps untouched. As a real flesh and blood person, you wondered how such a thing could happen.

Now imagine you are a person of a different sort - a corporation. Imagine further you're a corporation specializing in infrastructure construction with a lot of money coming in from government contracts. You too have hopes and dreams (beating analyst projections and having your patron agencies get large congressional appropriations), likes and dislikes (profits and costs), needs and desires (cash and more cash). Before the levee failures you had a similar relationship to the Corps as flesh and blood people. The Corps - and other federal agencies - protected you from the vicissitudes of the free market with a constant stream of contracts.

If you're big enough, the losses suffered by the unprotected human persons on August 29, 2005 turned out to be terrific opportunities to fulfill your desires. Unlike the flesh-and-bloods, your "recovery" began almost immediately, with four massive no-bid contracts going out two weeks after the levees failed. Those contracts got bigger and bigger, and if some of those millions of dollars were later deemed "wasteful," so be it. You stayed protected while you recovered.

So when - over four years after August 29, 2005 - another opportunity to enhance your "recovery" presented itself in the form of the permanent pumps project, you assumed you would remain safe under the watchful gaze of your protectors, the Corps. But when they inevitably screwed up and hurt you, you were sad. So like the flesh-and-bloods, you too went to court. But you're a better person than the puny humans, so of course you won. Repeatedly. And also unlike the flesh-and-bloods, the Corps felt bad about their mistakes and made up for your troubles in the form of taxpayers' cash. And you felt better and safe again.

Meanwhile the flesh and blood people again wondered how such a thing could happen. How could the Corps make mistakes that destroyed a city and its actual inhabitants, and not be found financially liable even a decade later, while when they made mistakes that harmed contractors' bottom lines, they chose to help out those contractors within months? Let's examine that.

First a little background...
The bid process for the permanent pump stations now under construction took place in two stages. In the first stage that ran from March, 2010 to June, 2010, anyone could submit a bid. At the end of that process, the Corps shortlisted five of the seven bidders (I know - not a very short list), who proceeded to the second stage where an eventual winner was chosen.

Dramatis Personae
The five bidders chosen for the second round were:

CBY Builders

This group included CDM Smith, Brasfield and Gorrie, and Yates Construction. According to Engineering News-Record, the three firms combined reported $3.1 billion in contracting revenue in 2011, the year the permanent pumps were originally awarded.

Bechtel team

Bechtel partnered with New Orleans-area engineering firms Waldemar S. Nelson; Brown, Cunningham and Gannuch (aka "BCG"); N-Y Associates; and Eustis Engineering. During the bidding process, Bechtel had a website up at, but it went away after the contract was awarded. It's still visible at the Internet Archive.

Bechtel is the largest contractor in the nation, reporting over $25 billion in 2011 contracting revenue according to ENR. They are a behemoth astride the planet. Fun fact: they hired the head of the Corps at the time of the Federal Flood, Carl Strock.

PCCP Constructors

PCCP Constructors is made up of Kiewit, Traylor Brothers, and the New Orleans-area based M.R. Pittman group. This is the same set of companies that made up the joint venture "Gulf Intracoastal Constructors," who had the prime contract for building the West Closure Complex. Kiewit and Traylor Brothers are national firms with a very large footprint. According to ENR, Kiewit - the number 3 contractor in the country in 2011, behind only Fluor and Bechtel - brought in $8.4 billion in contracting revenue that year, and Traylor Brothers generated an additional $470 million.

Louisiana Canal Constructors

"Louisiana Canal Constructors" appeared to be Alberici and Archer-Western (a subsidiary of Chicago-based Walsh Group), the same team that worked levee project LPV-111 along the Gulf Intracoastal Waterway in New Orleans East. Parsons was also involved. We know this because all three firms are listed on the Louisiana Secretary of State entry on LCC. Combined, Alberici and Parsons had just under $2 billion in contracting revenues in 2011.

Weston Solutions group

The Weston group also included design heavyweight HNTB. Weston brought in $338 million in 2011 contracting revenue

Altogether, the five bidders had over $39.2 billion in contracting revenue in 2011 alone. That is, all the bidders were national heavyweights used to going after huge projects, including two of the top three contractors in the country. They were used to getting their way.

A long and winding road

After almost a year of review and back and forth, the Corps made an initial $675 million award of the permanent pumps project to CBY in April, 2011. Nearly instantly, it was protested by two of the losing bidders - PCCP and Bechtel (Weston and LCC sat out the protest process). On a project this large, bid protests were almost a given. But the Corps didn't do themselves any favors. Through published rulings during the various protests, we learned a bunch about the inner workings of the bid process. Included in those documents were the following findings:

- A top civilian with intimate knowledge of the permanent pump project by virtue of being in charge of it, Richmond Kendrick, retired from the Corps and took a job with CBY during the bid process. The Corps - not an independent outsider - investigated themselves three times to determine whether there was a conflict of interest in Kendrick's move. Unsurprisingly, they exonerated themselves each time.

- The Corps review of the foundation engineering of the various bids appears to have been done with no engineering expertise whatsoever. A Corps witness at one bid protest hearing said the conversation about the bidders' foundation proposals took up no more than five minutes. More telling was the fact the Corps presented absolutely no contemporaneous documentary evidence at that protest hearing on how they arrived at their decision on who should get the job during their initial awarding. There were apparently no notes kept during the discussions, and the Corps could not come up with a structural engineer as a witness.

After a relatively brief protest process, and then a federal legal process that dragged on through the rest of 2011 and into 2012, the original award to CBY was thrown out and the bidding process restarted in May, 2012. This time, PCCP won in September, 2012. The new contract cost was revised downward to $629.5 million. This second award begat protests from CBY and Bechtel in October, 2012.

Now the good stuff

And this is where our story actually gets started. We know very little about what exactly happened after the September, 2012 award to PCCP. We can surmise there must have been deeper problems with the second round of bidding, because CBY's and Bechtel's October, 2012 protests resulted in the Corps very quickly agreeing to redo the bid process yet again, starting in November, 2012.

The activity in early 2013 is even murkier. All we know is that the Corps received yet another set of proposals from the bidders in February, 2013, and announced two months later that PCCP had won once again, albeit for a slightly smaller sum: $614 million. We'll get to the sequence of events surrounding the final award in a bit. But first some background on the dollars floating around this process before even the first shovel of dirt was turned.

Before the final award was announced in April, 2013, the Corps had already been paying bidders. Unmentioned in all the articles and press releases about the permanent pump project was a provision in the original solicitation which promised a $500,000 "stipend" to each of the bidders that made it to the second round but didn't win. Since there were five bids in the second round, and four of the bids would belong to the losers, $2 million total was promised to be sent out before construction began. However, that was not the only money transferred to bidders for non-construction reasons.

The Corps New Orleans District paid the legal costs for all the successful bid protests too. On January 15, 2013, the New Orleans District wrote a check for $244,179.06 to Bechtel (corrected downward on January 23 to $239,205.52 due to a clerical error). The Corps then cut another check, this time to PCCP (i.e. Kiewit) for $312,056.28 on February 23, 2013. And then they cut yet another check to PCCP - after awarding the $614 million contract to them - for more protest costs, this one for $134,679.27 on July 18, 2013. Altogether that's another $685,000 sent out to contracting firms as compensation for the Corps' errors, all before anything was built. To be clear, it appears successful protestors can file a claim to recover legal costs from protests, though the Corps is under no obligation to pay such costs, even if GAO recommends it (as they did for the 2011 round of protests). It is galling that the Corps chose to be more generous paying for their errors to well heeled contractors than to the hundreds of thousands of New Orleans, Jefferson, Plaquemines and St Bernard residents who suffered from much more significant mistakes in 2005.

That brings us to the very end of the bidding process. The chronology of the final award is of great interest, because it marks the high water mark of the Corps New Orleans District's largesse toward contractors. It's also typical of how the New Orleans District attempts to engage in image upkeep. So I've reconstructed the narrative after each bit of information was revealed, working from the most publicized chronology to the least publicized one.

Chronology 1, published in the Times-Picayune May 6, 2013
The official final contract award to PCCP Constructors came on April 17, 2013. The Corps announced it in a press release, and a corresponding article was written by the Times-Picayune's Mark Schleifstein.

On May 6, 2013, another Corps press release was issued announcing that the Corps had issued a Notice to Proceed to PCCP Constructors, the official green flag to start work. Schleifstein's companion article the same day had this to say about events between April 17 and that day:
"The Army Corps of Engineers has approved the start of construction of the $614.8 million permanent canal closures and pump stations at the mouths of the 17th Street, Orleans Avenue and London Avenue canals in New Orleans, after no objections were filed to the corps' April 17 award of the construction contract to PCCP Constructors... CBY and Bechtel had 10 days to appeal this latest corps contract decision."

Chronology 2, published by the Corps May 7, 2013
The next day, the Corps issued a much less noticed statement on the permanent pumps. In their May 7, 2013 Task Force Hope newsletter, they provided a chronology of the permanent pump bidding process. The next to last entry of that chronology reads,
"April 2013: Following the completion of corrective action procedures, the Corps reaffirms the award to PCCP JV, and their contract, previously awarded in Sept. 2012, is modified to incorporate the new PCCP JV Feb. 2013 revised proposal, and the sole Agency protest is voluntarily withdrawn."

[To clarify, there are two levels of protests bidders can file on contracts. They can go to the Government Accountability Office (GAO), which has a very formal, independent process and will publish notifications and decisions for all the public to see. Or they can also go to the Agency - the Corps in this case - where the protest is decided behind closed doors in a very informal process without any publication of results or even a notification a protest exists. In fact, even the protestors and other bidders themselves might be kept in the dark by the agency during an agency protest.]

This was the first indication there had been a protest of the final award, which contradicted the story given the Times-Picayune just one day earlier. Given the tenacity and infinitely deep pockets of the bidders throughout the two year long protest and rebid process, it seemed extremely unlikely one of them would "voluntarily" withdraw that same protest within days without receiving something in exchange.

Chronology 3, derived from FPDS-NG on August 1, 2013
We learned what that "something" was three months later. Due to the 90 day FPDS-NG publication hold on all Pentagon contracting actions, the public did not discover until early August that losing bidder Bechtel had been paid exactly $4 million on May 2, 2013, just four days before the Notice to Proceed was issued to PCCP Constructors. The description for the $4 million on FPDS was "PCCP Stipend Settlement for Unsuccessful Offeror." The use of the word "settlement" indicated something besides the standard $500,000 stipend was involved.

It seemed a good bet Bechtel was paid $3.5 million on top of their previously promised $500,000 in order to drop a protest, allowing the project to proceed more than three years after the original bid solicitation was issued. But without knowing what had happened with the other losing bidders, we could not confirm Bechtel was the bidder mentioned in the Corps May 7, 2013 newsletter who "voluntarily" withdrew a protest.

Chronology 4, derived from FPDS-NG on August 26, 2013
The final pieces of the puzzle were placed August 26, 2013, when the transactions from May 28, 2013 became public on FPDS-NG. Both CBY and Louisiana Canal Constructors had been paid their standard $500,000 stipends on May 28, 2013. With the already known information that Weston had gotten their $500,000 all the way back in January, 2012, this left Bechtel as the only firm that could have been the protestor. The only thing left was to get the documentation.

Which is what I did. It's here. The document confirms Bechtel was paid $4 million to "to settle and amicably resolve all differences, to avoid the uncertainties and expenses of litigation associated with the filing of its Agency Protest and any other litigation arising out of or related to the solicitation."

So, to summarize:
- The Corps appears to have withheld information from a reporter about how the final days of the permanent pumps bid process played out
- Then in their own newsletter, while revealing the protest, they claimed the firm (still unnamed) voluntarily dropped the protest without mentioning any recompense for the firm.
- Three months later the truth came out: It was largest U.S. contractor Bechtel that had protested, and the Corps had paid an extra $3.5 million above the standard $500,000 to make Bechtel drop its protest.
- In total, the Corps chose to pay out over $6.68 million in losing and winning bidders' protest costs, losing bidders' stipends, and losing bidders' settlement costs to companies whose combined annual contracting revenue was at least $39 billion.

For perspective on what $6.68 million can buy, each of the 54 rusting temporary pumps at the canals costs about $170,000 to refit for about two years' service. Given the Corps still has $3 million left in pump repair funds sitting unused, all of the existing pumps could have been made ready for the two extra years they must now serve with $6.68 million, with cash left over. Since the permanent pumps bid process took three years (two years longer than originally scheduled), that would have been the right thing to do to ensure proper storm protection. Instead, the Corps ensured contractor protection.

The question

Did you think no one would notice?

Did you think no one would notice nearly $7 million going out to bidders on the permanent pumps project - over $6 million of it to losing bidders - to cover legal bills, bidding incentives, and payoffs to make some of them just go away? Did you not think such grandiose treatment of multibillion dollar contractors was distasteful in light of your fighting against any compensation for those hundreds of thousands of actual human beings harmed due to your monumental screwups in 2005? Or did you think you could just hide the whole thing?

Update 7/24/15

The Times-Picayune's Mark Schleifstein wrote about this on July 22, 2015. Schleifstein got statements from Bechtel and the Corps. The Corps' statement was literally a copy-and-paste from the $4 million contract action, so that's useless. Bechtel's statement wasn't much better:
"Bechtel spokesman Warren Getler said Tuesday that the company challenged the award 'because of lingering concerns about the bid process.'
'Following discussion with the corps and to avoid further delay in this critical project, Bechtel accepted a stipend of $4,000,000, which included $500,000 for submitting a responsive bid and granting the corps the rights to all of Bechtel's work products in preparing its bid proposal,' Getler said in an email message responding to questions about the bid appeal.
Getler pointed out that a corps financial settlement with Bechtel was listed as one of a number of recommendations made by the Government Accountability Office in a 2011 report that recommended the corps throw out its first decision to award a $675 million contract to CBY Design Builders to build the pump stations.
'We also recommend that PCCP and Bechtel be reimbursed for the reasonable costs of filing and pursuing the protests, including reasonable attorneys' fees,' said the GAO opinion."

I call BS on Bechtel trying to characterize the $4 million as the same thing as the protest fee reimbursement. For one thing, Bechtel was already reimbursed $239,205.52 for those fees four months before the $4 million payoff. For another, the GAO caps the costs for legal fees at $150/hr. If we were to take Bechtel's statement at face value, this would mean Bechtel's lawyers had expended almost 27,000 hours on a protest which was filed on April 26, 2011 and decided 100 days later on August 4, 2011. That is, they would have had 11 attorneys working around the clock with no break for 100 days straight. Unlikely.

In an email exchange I had with Mr. Getler following the publication of the article, he did indeed backtrack from the linkage between the protest fees and the stipend:

"Thanks for asking for clarification on the reimbursement of legal fees. I did not mean to imply that the reimbursement was related to the $4 million stipend. As you know, it was a separate issue associated with the protest of the initial award of the contract. The reimbursement was recommended by the Government Accountability Office (GAO) in its decision sustaining Bechtel and PCCP’s protest."

When I asked for more detail regarding the $4 million payment, Mr. Getler demurred:

"Regarding Bechtel’s agency protest of the April 17, 2013, contract award, I don’t have anything to add to the information found in the government form that describes the terms of the stipend payment. If you have further questions, please direct them to the U.S. Army Corp [sic] of Engineers."

Finally, Schleifstein notes the most important detail: the extra $3.5 million came out of the money budgeted for the permanent pumps. Nice to know money meant to provide vital storm protection instead went to a multibillion dollar corporation that built no storm protection. 
[end 7/24/15 update]

Overall, I think Jeffrey over at Library Chronicles put it best when reacting to all this:
"Maybe there's stuff going on that I don't quite get. That does happen on a regular basis. But I swear it looks here like Bechtel had been holding the city's safety hostage and the Corps finally decided just to pay the ransom. But, again, this is all complicated engineering stuff and maybe I don't quite get it."

I think he got it just right.

Sunday, May 31, 2015

Rusty pumps keep rusting, Corps yawns

With the 2015 hurricane season upon us, and the continuously rusting pumps at the three interim closure structures (17th Street, Orleans Avenue, and London Avenue) well beyond their predicted life by any reasonable measure, I thought it would be time to check back in and see if the Corps has stepped up - or actually done any - preventative maintenance on them. For an overview of the rusty pumps saga from 2006 through 2012, see this wrap up post.

The pumps were put into the water in 2006, pulled out, and then put in for good in 2007. They were not expected to be in service more than a couple of years before they would get replaced by sturdier, bigger pumps. But planning for their replacement dragged on for years. The Corps, starting in 2009, stated the life of the entire structures - slyly without specifying the exact life of the pumps themselves - was "five to seven years." They said maintaining the pumps beyond 2013 would be difficult. They did this mostly to bring pressure to bear on locals to approve their permanent pumping scheme. But taking them at their word, if we started the lifespan clock in even 2007 - generous given construction of the structures began January of 2006 - we are now beyond even the worst case for the structures' and pumps' lives. One would have expected to see some effort toward life extension on these pumps.

And we did for a little while, with the replacement of many carbon steel components on the pumps with stainless steel between 2010 and 2012 (Note: Two pumps - both at the 17th Street site - remain untouched by those refit efforts). During that 2010-2012 period, the pumps were not expected to be needed past the 2014 storm season. However, in the intervening time the end date of the pumps has been extended even further into the future. Now they are expected to be in service until 2017, or 11 years after they first saw brackish lakewater. So what has been happening with the pumps since my last update during the 2012 hurricane season? Has the Corps done anything to make them last the additional three years?

In a word, no. Since 2012, the Corps has pulled six pumps across the three sites, almost all in reaction to oil leaks. There does not appear to have been any proactive moves to ensure the pumps can last until 2017. Here's what has happened:

2012-2013 off season: no pumps pulled.

May 20, 2013: Pump W10 at the 17th Street site sprung a hydraulic oil leak. The pump was pulled out a couple of weeks after the oil leak, just as the 2013 hurricane season was starting. It likely went back in the water some time in August, 2013. This failure was not a surprise, since W10 had not received the stainless steel upgrade in 2010-2012, but had been part of a group of six pumps given inferior carbon steel replacement repairs during a misbegotten effort in 2009.

July 2, 2013: Pump W9 at the 17th Street site sprung a leak. This failure is far, far more concerning. In fact, it's a canary in the coal mine. That's because W9 had been upgraded with the stainless steel suite of repairs during the summer of 2011, just two years earlier. Now in 2013 it was leaking once again, causing a third round of repairs (W9 also got the 2009 carbon steel repairs).

This is troubling in the extreme. It means the stainless steel upgrades are all suspect, which makes the integrity of all the pumps suspect. This would be confirmed just a few months later on...

December 19, 2013: Pump E8 at the 17th Street site sprung a leak. It was pulled out for repairs in January of 2014 and went back in the next month. The canary started chirping louder, since E8 had also received the stainless steel upgrade, in this case three years earlier, during the spring of 2010. While we don't know the exact problem in 2014, there is some indication the pump did not need a complete overhaul. The original task order for the 2013 E8 repair was for $172,934.74, but a modification subtracted $130,693.20 of that, Full pump overhauls usually cost around $160,000 to $170,000. If there was only $42,000 of work done on the pump - and considering some of that was for crane rentals on both ends of the repair - it makes sense to think the problem was relatively small but critical. There's really only a few possibilities: a) a piping failure, b) a seal failure, or c) a failure of the Rineer hydraulic motor. But it doesn't really matter, because there is no spare pumping capacity built into the interim closure structures, so a failure of any pump will hurt a storm response.

That was the only pump pulled in the 2013-2014 off season

And then late last year...

October 29, 2014: Two pumps at the London Avenue site sprung leaks. They were pulled out and repaired late in the year. The exact identification of the pumps is not known, but it is irrelevant, since all ten of the London Avenue hydraulic pumps underwent the stainless upgrade between 2010 and 2012. This means that the lifespan of the refitted pumps is about 3 years. Given that we are now in 2015, and the refits took place from 2010 to 2012, that means all the pumps - except the ones repaired since 2012 - are likely at the end of their lives due to corrosion. Again.

Finally, a sixth pump was pulled for repairs last December. There are no details on which pump this was, not even a spill report. It likely went back in this past February. There do not appear to have been any other pumps pulled, but Corps contracting records on FPDS only extend through the beginning of March, 2015

What's going on with these latest failures? Without the Conhagen repair reports, I can only venture a guess. There's the possibility of galvanic corrosion on the remaining carbon steel components, such as the deeply critical Rineer motors. Stainless steel plus carbon steel plus seawater sets up a battery, which can corrode things even faster than all-carbon steel units. There's also the possibility of inadequate amount of zinc anode protection placed during the refits. The anodes are supposed to corrode preferentially to the pumps, but who knows if that's taking place? The pumps aren't being pulled out preventatively to check such things, only when they are already corroded beyond use. Effectively, the Corps is driving their car with the engine light on.

So that's it. There does not seem to be any movement on the Corps' part to prepare the interim closure structure pumps for three more years' of service, let alone give any sense of confidence that the pumps furthest out from the stainless refits are not going to crap out this year. Further proof - in addition to the lack of activity at the site - is the fact the Corps' four year pump repair contract with Healtheon expires this year, and a solicitation to replace it has not been issued. In fact, of the $6.75 million on that repair contract, over $3 million ($3,068,911.61 to be exact) remains unspent as of March 1, 2015. At the going rate of $170,000 per pump refit, that $3 million could have gotten 18 pumps back into fighting shape for a couple of more years. Instead, the Corps New Orleans District just sat on their asses and didn't do anything proactive for the last three years.

This is beyond disappointing, since we now know the pumps remain ticking time bombs ready to fail at their first real test this year. I don't know how much more proof the Corps needs than at least three of their own pumps - refitted to their own specifications - failing within an average of three years after that refit. That doesn't even count the two remaining pumps at 17th Street (E5 and E7) which remain without the stainless refit entirely, though there exists the possibility the pump pulled out in December 2014 is one of them.

Mitigating this problem perhaps is the additional work performed on the London Avenue and 17th Street canals last year to Really Truly Be Able To Handle 8 Feet Of Water. The ability to have higher canal water levels means there is a small amount of breathing room between what the city sends the Corps' pumps and what the Corps' pumps must pump out. However, when one considers the large gap between what the Corps says (one day before Isaac in 2012: "Orleans Ave [canal] - Max. Water Level is 8.0.") and what they actually do (during Isaac in 2012: "we will discontinue one of the [city's] pumps [feeding the Corps' Orleans Avenue canal pumps] when we hit 3 to 4 foot"), their credibility isn't worth anything. The only thing you can trust is the dollars, and they haven't spent any money getting the outfall canal pumps ready, beyond fixing stuff that they found broken in the most obvious way - an oil slick on the water.

In fact, their lack of concern about the general end-of-life state of the interim closure structures is so huge, that the topmost Corps officials are even joking about it on camera. In WWL's "Eye on Hurricanes" special aired June 1st, Corps New Orleans District commander Richard Hansen compared the existing interim closure structures and pumps to the now-under-construction permanent pumping stations:
"Well I'd say the interim control structure is kinda like its backyard kitchen, okay? It's not fully protected from the weather. It can do its job out there, but the permanent features will be the Cadillac. That will be the feature designed to last for 100 years"

It's wildly inappropriate to joke about how inadequate your own storm protection structures are. In fact, the WWL reporter says in the very next breath:
"So the interim pump structures have been in since 2006 and have actually outlived their lifespan, but the Corps says it will not be a problem for them to remain in place and be working until the permanent pumps can be complete. And right now that's expected by the height of hurricane season in 2017"

"The Corps says" It's unbelievable this still is accepted as truth.

Based on the lack of any preventative maintenance activity, as well as the evidence of refitted pumps failing just two to three years after their refit, I would judge all but a handful of the the outfall hydraulic canal pumps in a stage of imminent failure. The drainage of the city during a major tropical event which includes large amounts of rainfall is now dependent on luck and hope that stuff won't rust through during a storm. That's very, very bad.

Sunday, July 06, 2014

Bricks, rebar, huge rocks inside the 17th Street canal levees (which is no surprise)

The photos showing the unearthing of massive amounts of unsuitable debris from the outfall canal levees this spring and summer bring into stark relief the true nature of flood protection in the greater New Orleans area. By the Corps' own standards, a deeply inadequate system remains in place - a system which under current design and construction guidelines would never get built today. We know this - not by having to plumb endless internal emails or looking through musty, decades-old records - but by simply comparing today's pictures to the records of the last 17th Street canal remediation project conducted just three years ago.

A new round of photos of the Corps of Engineers' 17th Street canal work site taken in late June detail more fully the debris that was piled into the 17th Street canal levees years ago:

Bricks and rebar:

...which we can see after moving closer to the area in red:

Bricks and large rocks:

...which are visible after zooming into the area in red:

None of this should come as a surprise, since the same junk was pulled out the levees a bit north of this year's work site three years ago during the first round of remediation (which I wrote about in 2012). That project offers insights applicable to this year's project.

Extensive "unsuitable" material found inside 17th Street canal levees in 2011

The 2011 17th Street canal remediation project consisted mostly of deep soil mixing of grout with existing soil to improve the strength of the soil along both east and west banks of the canal. Columns of dirt below the base of the levees were mixed with injected grout down to -20 to -40 feet. The contractor had to excavate a few feet off the top of the levee to give a flat platform on which to mount the drilling rigs and other work equipment. Here's the generic cross-sectional detail from the project issued-for-construction drawings showing this:

The excavated material is shown with the single hatching on top of the levee. It was expected to be reused as backfill for the same area at the completion of the project. That's not what happened.

Right from the start and throughout the entire 2011 project, the Corps' contractors continually found the levees on both sides of the canal were riddled with, shall we say, non-dirt objects. The greatest concentration appears to have been on the west - or Jefferson Parish - side. On the very first day of construction on the west bank (March 7, 2011, which was also Lundi Gras), the contractor dug into the levee near the intersection of Orpheum and Rosebud. The contractor's Quality Control report (all reports from the 2011 project are linked from this post) for that day said:
"Visually inspected the construction of the West Bank production pad. Material removed from the existing levee between stations 588+00 and 590+00 contained significant debris (rocks, concrete, bricks, shells, asphalt). Informed Bill and Derrick [Corps employees Bill Richardson and Derrick Parker] and they came over to look. We asked they to determine if this would be considered "unsuitable" to reuse in the construct of the levee"

Four days later, March 11th, a decision was made on this "significant debris:"
"Got a verbal answer from [Corps employee] Bill Richardson concerning the 'unsuitable material' encountered during the excavation of the West Bank platform. We are to excavate to the required depth to build the production platform only. All excavated material will be hauled offsite and replaced with suitable clay material from the Bonnet Carre borrow pit."

The 2011 project involved excavating the top of the west bank of the canal from Georgia Court to the Old Hammond Highway bridge, a distance of 3/4's of a mile. Thus, this statement implicitly said that the entire levee along that stretch  - at least as deep as the Corps wished to dig but no deeper - was made of dirt not suitable for levee construction:

The Corps' wink-and-a-nod instruction ("excavate to the required depth to build the production platform only") indicated they believed the rest of the levee was similarly riddled, but didn't want to bother dealing with it. That means there's likely still junk in the 17th Street canal levee today, just further down than where the 2011 project dug. Returning to the detail from the drawings, we can see this graphically:

Two weeks after the implicit acknowledgement of the unsuitability of the west side levees, the contractor made it explicit. In the March 24, 2011 Quality Control report they reported on excavation of the west bank levee along Orpheum between Chestnut and West Esplanade:
"Excavation (degrading/construction of production platform) on West side of canal today from station 581+80 to 579+00 to an elevation of +1.8 to +1.5'. All degraded material was hauled to laydown area on the East side of the canal. Material was considered unsuitable by USACE and will be stockpiled in laydown area on east side of the canal. The material will then be spread throughout the laydown yard at the end of the project per USACE direction."

As I noted when I wrote about this in 2012, at least the Corps didn't try to reuse this junk dirt. But they also didn't bother to tell the public and their government representatives in any meaningful fashion there were "rocks, concrete, bricks, shells, and asphalt" inside 3/4's of a mile of 17th Street canal levee that had been there since it was built decades ago. Or that such debris would remain within the levees even after the project was completed. They just proceeded to conduct business as usual, like it was a normal thing to discover - six years after the fatal failure of their levees - that the levees along the 17th Street canal were built out of trash. Day after day for months, they hauled dirt off the Jefferson Parish levee across the canal to their laydown yard on Bellaire Avenue along the east bank of the canal - in front of the 2005 New Orleans breach location - and told no one in the public the extent of the problem. Some of that bad dirt was spread out at the laydown yard during the summer, 2011 and literally buried under grass. The rest of it, including some stuff too big to be buried, was trucked offsite. The August 23, 2011 quality control report describes this disposition:
"East side of the canal - Spreading unsuitable material in laydown yard per USACE's direction. Large debris is being removed and stockpiled to be hauled off at later date."

Fortunately, three years later during another project involving digging into the Jefferson Parish side of the canal, we have pictures showing us just how poorly these levees were constructed - and indicating how much junk remains inside them today, so there's no way this information can be buried or hauled away again.

Debris so huge it couldn't be removed in 2011, or ever

Some of the junk was so huge in 2011 it couldn't even be moved out of the levee. An extreme example of this was documented by the contractor logs starting on March 25, 2011. That day, there came discovery of a very large concrete "object" just below the west bank levee near the intersection of Orpheum and West Esplanade. From the contractor's quality control report:
"During the construction of the production platform on the west side of the canal, a large concrete structure was exposed at approximately station 577+80. CKY excavated around object to try and see the size of the object and how to remove object. The size of the concrete object was found to be approximately 20'x13' with rebar protruding out of the concrete. Object was unable to be removed. USACE was contacted about the object and pictures were taken. GPS was used to mark the location of the object. The USACE will be asked in an RFI [request for information] if only elements ABC can be installed at this location due to the large concrete object."

The Corps quality assurance report has a little more detail:
"The concrete footing is quote [sic] thick, it could exceed 2 feet. The footing consists of spread footings that cover the area between stations 577+81.5 and 577+90.5. The depth, below levee grade, is near the elevation of the adjacent road."

That's right: a room sized chunk of concrete buried under the levee that didn't get discovered until the levee was dug out. This was so huge that even the Corps owned up to it, taking pictures two months later when they dealt with the thing:

This mass of concrete was in an area that had been extensively explored by Corps test borings. No fewer than 11 test bores and cone penetrometer tests had been drilled into the levee within 200 feet of this location, 10 of them since the 2005 levee failures, as seen on this detail:

One must ask how such a giant chunk of concrete could be unearthed as a surprise. It also raises the obvious question about the quality of the information gleaned from the test bores and other exploratory drilling the Corps does before a project.

In this particular case, the obstruction was so huge they actually left it in place. After covering it back up in March, they went back in late May. They uncovered the obstruction and then backfilled and compacted the affected area between May 24, 2011 and June 1, 2011. Presumably some work crew 20 years in the future will dig it out again.

Comparing the 2014 project to the 2011 project

Going back and reviewing all the records from the 2011 project has given me a little different perspective on what we can see in the 2014 project photos. During the 2011 project, all the construction on the levee was done with dirt from the levee. This included building of platforms and ramps - things occurring right now during the 2014 project. Clean dirt was not brought into the site in 2011 until all substantial work was completed. I have no reason to think the 2014 project is proceeding any differently.

So unlike what I wrote a couple of weeks back, I now believe that all the soil currently piled up on the levee came out the levee, including the stuff near the base that is acting as a "road" for the crane. This means giant boulders are also coming out of the levee, not just chunks of concrete, brick, and pieces of rebar. Here's a couple of the pictures from my prior entry, relabeled to include that conclusion:

In comparing the progress of the 2014 and 2011 projects, what do we make of the fact the Corps started removing bad dirt from the site within days of unearthing it in 2011, but the junk has remained on the levee for weeks in 2014? Some of the reason might simply be how this year's site is set up. The dirt at the bottom of the levee this year remains in use for access to the entire site, with only a single point of entry or exit for heavy equipment at the south end of the site. The crane using the platform at the levee bottom is needed for the entire duration of the project, which has not been completed yet (the contractor is moving from south to north). So the bad dirt making up the platform  probably can't be removed until all the sheet piling has been placed.

Debris throughout the entire 17th Street canal west side levees?

However, the logistics of this year's project are not nearly as important as the larger point: everywhere the inside of the 17th Street canal west side levee has been revealed - whether in 2011 or this year - massive amounts of debris have been found. And even more troubling, there are very long stretches which remain untouched:

Given the shallow depth it was found in 2011 and 2014, it makes sense some of the debris was part of the I-wall and levee construction in the early 1990's. That work did involve placement of dirt on top of the existing levees at the time. It extended the entire length of the canal (all the yellow and red lines shown above). Other, deeper debris has likely been there far longer. The records of nearly every soil boring taken in 1981 along the west bank - long before the I-walls were put in - have numerous mentions of shells, wood, bricks, and other "miscellaneous fill." Those records were used in the basis of design for the Corps' early 1990's work. This junk has been known to the Corps for a very, very long time. It also very likely extends the entire length of the canal on at least the Jefferson Parish side.

How it got there doesn't really matter now though, because all the junk is still there today, and the integrity of the outfall canals remains the responsibility of the Corps. As the decision recorded in the contractor quality control log of March 11, 2011 indicates, there is no way the 17th Street canal west bank levees would be built today with the debris-riddled dirt that is currently sitting in them. That dirt is - according to the Corps - "unsuitable." Yet there the levees sit.

Thus, this is what the Corps considers the recipe for sufficient levees along possibly the most well known stretch of levee in the nation:

These levees do not care what water presses against them - storm surge or rainwater. Thus it make no difference that the outlets of the outfall canals are "protected" by gates now. These levees still see pressure from stormwater during major rainfall events. One must wonder what local government officials and Corps personnel living behind these levees think seeing the true condition of the Corps' flood protection.

Labels: ,

Monday, June 23, 2014

Hidden in plain sight

[Update, 7/7/14: Based on photos taken after this post was published, and a review of the records of the 2011 17th Street canal remediation project, revisions have been made to this post. It is now believed the large rocks seen at the base of the 17th Street canal west side levee in photos taken June 15, 2014 did indeed come from within the levee. Text and photo captions have been adjusted accordingly. Further details and analysis are available at subsequent Fix the Pumps post published July 6, 2014.]

Unreported by anyone, the Corps has begun major work along two outfall canals in New Orleans in the last month. Very long sheet pile is being driven on the 17th Street and London Avenue canals, and rock is going to be placed at the base of the levees inside the canals starting in June or July. These are the same outfall canals that breached in three places in 2005 due to poor design. The ensuing floods of New Orleans devastated hundreds of thousands of lives throughout the greater New Orleans area and beyond.

The Corps' irrational need for secrecy about any work along the outfall canals is long known, but their actually beginning structural work without any public pronouncements, press releases, or whispers in the ears of the mainstream media is unprecedented, especially considering the scale of the work - which is extensive. I think they would rather nobody paid any more attention. But attention should be paid to work at the sites of the cause of the worst engineering disaster in American history.

The work underway already is for seepage control due to existing sheet piling being too short. Seepage happens when water works its way through sand below the bottoms of the I-walls along the canals. Should that water then percolate upward through the land-side (the side where people  live) soils, the levee and I-wall can fail. This was the primary mode of failure for the south breach of the London Avenue canal following the 2005 passage of Katrina dozens of miles to the east.

Problems have emerged just weeks into the project, as the soil being dug out of the levees appears to be packed with unacceptable debris like rocks and chunks of concrete. The Corps' own specifications forbid such debris in a levee since it decreases the strength of the soil. However, they have shown they are willing to ignore such specifications during many previous projects both before and since 2005, including a round of projects on all three outfall canals in 2011.

Those 2011 projects included similar seepage remediation work to this present project along the London Avenue canal and the Orleans Avenue canal. The 2011 17th Street project remediated against different modes of failure than seepage, and thus included different technologies than sheet piling.

For calculation purposes, the Corps and their consultants have broken each side of each canal into individual "reaches," based on differing soil and canal bottom conditions. The reaches are numbered sequentially, starting at the south end of the west side of each canal and finishing at the north end of the east side of each canal.

Here's the locations previously remediated along the London Avenue canal in 2011:

And here's the new locations in the 2014 project:

As you can see from the above map, after they are done with this round at London Avenue, nearly the entire canal north of Mirabeau will have had sheet piling placed outside the existing, too-short piling. It points up how poorly the I-walls were designed the first time around, and how inevitable their failure was.

The contract - which includes work at both the London Avenue and 17th Street sites - was awarded February 28, 2014 to Conquistador Dorado JV (aka Dorado Services) out of Sanford, Florida. There are indications Cycle Construction in Kenner is the local partner in the joint venture. The contract - actually a task order on a Service-Disabled Veteran-Owned Small Business (SDVOSB) contract awarded to a pool of four companies -  is worth $13,642,238.50 and is scheduled to be completed October 30, 2014, meaning construction will proceed throughout the 2014 hurricane season. Construction along the London Avenue canal started in late May with Reach 12, just north of Filmore Avenue on the west side of the canal.

I have had interested neighbors photographing the work as it proceeds. This photo is from what I believe was the first day piles were driven at the London Avenue canal site, May 29, 2014:

Sheet piling is being driven down to clay approximately 55 feet below sea level. The new sheet piling is being driven a few feet to the protected side of the existing I-wall. The existing sheet piling inside the I-walls goes down to sand about 15 feet below sea level. The Corps is using a pile pressing machine rather than a standard pile driving hammer. The pile presser supposedly creates fewer vibrations and is quieter. It's a machine they've used extensively in the dense urban environment of New Orleans and its surrounding metro area, especially where space is at a premium:

Here's piling being placed the following day:

To make room for the pile presser, the top couple of feet of levee soil next to the I-walls will be stripped off to make a level surface and the dirt stockpiled on the levee itself. Here's the excavators at the London Avenue site:

The sheet piling machine travels along the newly flat section and presses the piling in. This June 6, 2014 photo shows the first few linear feet of piling already driven:

Afterward, the soil will be placed back on top of the piling and grass will be planted. This was done at adjacent Reach 13 in 2011.

If we move closer to the stockpiled soil, details emerge. Worrisome details:

Here's enlargements of the photo above:

All the white bits are broken concrete. This next photo shows it best:

Here's the concrete debris highlighted:

The dirt coming out of the levee is absolutely shot through with the stuff. That's bad. Having debris like concrete in the clay decreases its strength because it cannot compact properly. Finding debris in these levees is not a surprise, since it was reported on either side of this stretch during the 2011 London Avenue project as well:

It is one thing to read about it months afterward, and another thing altogether to have pictures of it happening live.

After receiving the pictures of the London site, I asked for pictures of the 17th Street site. Based on those pictures - taken on June 15th, I believe the story is much the same.

The site location on the 17th Street canal is on the west side, just south of I-10:

It is reach 16, which I wrote about extensively 3 years ago. At that time, the Corps was considering placing dirt at the base of the levee on the protected side (outside) of the canal to prevent failure from underseepage. The idea was to provide enough weight on the landside to resist the upward motion of underseeping water. It's an indirect, cheaper way of addressing seepage, as opposed to the direct solution of cutting off the seepage with sheetpile driven into the clay below the sand. It's noteworthy nothing was done in the intervening three years. It's also noteworthy the Corps New Orleans District consultants' calculations in 2010-11 (as well as the Corps' District Engineering folks' marching orders to their consultants before any calculations were performed) claimed there were no seepage problems on reach 16 - or anywhere along the 17th Street canal. Because of that, this newest remediation project is the first one to address seepage concerns along the 17th Street canal. We'll get into what likely changed their mind in future postings.

Here's reach 16 in 2011 (photos taken from the June, 2011 Corps update on that year's remediation projects along the outfall canals):

Construction at the 17th Street canal site also started in late May. The contractor began at the southern end of the site, which is what is shown in the "before" pictures above.

Here's that same end of the site snapped June 15, 2014 (Update, 7/7/14: bottom caption used to read "Dirt placed to make 'road' for crane to travel. Dirt likely not from inside levee"):

I believe that the dirt toward the bottom of the levee is - I hope - fill brought from offsite and placed to allow the crane (which holds the piling vertically) to travel along the levee with the piling pressing machine. I do not think it, and the huge boulders in it, came out of the levee.

[Update, 7/7/14: Based on subsequent photos and a new review of the 2011 remediation project records, it is now believed the material at the bottom of the levee is from inside the levee. This includes the large boulders now visible.]

However, I believe the soil toward the top of the slope is from inside the levee. Even from a distance, we can see the chunks of debris. Getting a closer look shows a large amount of concrete and rocks (Update, 7/7/14: bottom caption used to read "Dirt placed to make 'road' for crane to travel. Dirt likely not from inside levee"):

Here's an enlargement of the above photo:

There's even a stump that appears to be about 2 feet long, just sitting on top of the levee. It seems very likely it came from within the levee (though I suppose it could have come from somewhere else, but then why would it be sitting on the top of the levee?):

Dirt taken out of the southern part of the site is also being stockpiled on the levee near the northern end of the site in front of the end of Lemon St. The copious debris - all the white bits - is obvious (Update, 7/7/14: bottom caption of first two photos used to read "Dirt placed to make 'road' for crane to travel. Dirt likely not from inside levee"):

By any standard, including the Corps', the dirt coming out of the levees at both these sites would be unsuitable for use in any levee. How do we know that? The Corps themselves said so just four years ago, during a different debris mess that made the news.

In late 2009 and early 2010, reports emerged about large amounts of concrete and other debris being found in the top layers of lakefront levee in Kenner during an enlargement project. That soil had been trucked there in the early 2000's and had not been systematically checked for debris. After a couple of Times-Picayune articles, the Corps held a public meeting on January 28, 2010. They showed the following slides as examples of unacceptable amounts of debris in the dirt:

Those photos look eerily similar to the pictures coming out of the London Avenue and 17th Street sites.

The 2010 slides explicitly called out concrete chunks as unacceptable:

In the 2009-10 lakefront case, the Corps' solution - after blaming the previous contractor and prior Corps employees and assuring the public these were isolated instances (which they weren't) - was to remove all the dirt with concrete in it. According to the Corps, the contaminated dirt went down 2.5 feet, so the Corps stripped it all out and replaced it with clean clay.

Here's the scary part: that might not be the plan for the outfall canals, for three reasons:

1) The bid specifications (as amended) for the outfall canal job call for the contractor to place most of the soil they remove from the levee during construction right back in the levee when the sheet piling work is done.

According to those amended specifications, 2350 cubic yards of soil is to be taken out of the levees and reused between the two canal sites. The Corps is only anticipating 650 additional cubic yards between the two sites to be "unsuitable." That is, they expect only one fifth of the total amount of dirt to come out of the levees to stay out, and the rest will go back in. That was the financial basis for awarding the job.

2) We know bad soil with debris just like what we're seeing this year was taken out of outfall canal levees and put back in during the 2011 projects. The Corps' own contractors' said so, doing so under orders from Corps employees.

3) Finally, we know the Corps has no compunction using or reusing debris-laden soil that violates their own specifications all over the greater New Orleans area. Besides the three outfall canal remediation projects in 2011, I counted 22 projects where it happened in the last few years:

For further information, you can consult my 14 (!) part series on debris in the levees. Part 1 is here, with the other parts linked from it. The Corps' use of unacceptable soil seems to be more the norm than the exception.

Why does this matter? After all, there's gates at the ends of the canals to protect them from hurricanes, right?

Well, the outfall canal walls and levees do not care where water comes from - whether it's the lake or the city's pump stations. Even if the lakefront gates are closed and secured during a high lake level event (a big "if," since during the last hurricane 5 of 11 gates at the London Avenue structure were not secured), the city still has to pump rainwater from within the city down the canals. That water level can get quite high - just as high as the water got from Katrina's surge. A levee with concrete in it instead of clay is simply not as strong, and could be prone to, um, difficulties.

In addition, there is the obvious question: if this is what the top couple of feet of the levees along both canals look like, what about the rest of the levee left untouched by these newest projects? Given that these stretches of levee have not been dug into since the failed I-walls were installed about 15 years before their 2005 failure, it is a reasonable concern that the entire levees - including previously remediated stretches - are just as bad as what we've seen here. Without exploratory trenching - as has been done on many other projects where debris was found - there exists a deep trust deficit between the Corps and the public who depend on these levees for their lives and livelihoods.

Labels: , ,

Wednesday, July 24, 2013

Isaac: the rest of the story, Part 3

Part 1
Part 2

NOTE: I am currently awaiting more material from my Isaac email FOIA request, including a PowerPoint presentation from the evening of August 30, 2012 which was supposed to include a candid assessment of pumping at the outfall canals. In the interim, I've pulled together what I could regarding events along the 17th Street canal during Isaac. This post may change as the remaining information is received.

When I placed my FOIA request for the Isaac emails, I had hoped to not only uncover unknown events during the storm, but also to get more details on the pump failures that occurred at the 17th Street canal on the morning of August 29th. Those failures were so extreme the Corps had to report them to the media. Here's the timeline from my October 6, 2012 entry, "Isaac in New Orleans - what we know so far:"

The PS#6 logs for August 29th (here and here) confirm the outlines of this story, while filling in some detail. At 4:35 AM, the station staff asked to "load" (that is, bring to full capacity) 1100 cfs pump "F." They were refused. Then, at 4:55 AM the Corps told the S and WB not to "load" any more pumps in PS#6:
"Note: Army Corps of Engineers said we cannot load any more pumps. [T]hey are at their max of what they can pump"
At that time, the station was pumping approximately 6200 cubic feet per second, or cfs. The station's capacity among its main rainstorm drainage pumps is 9200 cfs.

This initial notice to the S and WB to hold off on pumping appears to be 1.5 to 2 hours before the Corps alerted the media of pumping restrictions, according to the chronology of the NY Times article. Then, at about 6:30 AM, the station operators received another Corps warning not to load additional pumps beyond what was already running, despite the fact that in the intervening hours, water in the station's inlet basin had risen over four feet. Ten minutes later, the station received orders from S and WB's Central Control to break the prime on 1000 cfs pump "D," effectively losing most of its capacity and sending into a pump status called "light." So there were not only Corps orders to keep some pumps from running, but one pump was effectively ordered turned off with the water coming up. This left the station running at about 5200 cfs.

At 7:55 AM, with the water still very high in the inlet basin, the Corps 17th Street canal captain in the station (Ray Newman) gave an all clear to the S and WB personnel in the station to load additional pumps. Five minutes later, 1000 cfs pump "D" was authorized to be loaded by the S and WB Central, and was running in a loaded condition within a few minutes. Another 1000 cfs pump ("C") was brought on line shortly afterward, increasing the station's flowrate to 7200 cfs. Inlet basin levels dropped four feet in about an hour after those two pumps were added. This chronology roughly tracks what was playing out in the media that morning.

Oddly, there's zero Corps email traffic among the key personnel (Ray Newman, Don Schneider, and Chris Accardo) through that whole time. I would expect something to have been written by someone over that three-plus hour span, but perhaps they were just so busy they didn't have time to type on their Blackberrys. Or perhaps they were all communicating either over the radio or in person, so electronic communication was unnecessary. When there was a pump drive skid fire at the Orleans Avenue site a little over an hour after this event subsided, there was an email trail immediately created, so it's very strange there's no trail for the biggest event during the storm.

How high did the water get in the 17th Street canal on August 29, 2012?

The first question to be asked after an event like this is whether the 17th Street canal maximum operation water level of 6.5 feet (first revealed to the public during the press coverage of this mishap) was exceeded. If it was, that means the walls and levees were loaded beyond their safe capacity. To determine this, normally we would examine the outfall canal level gauge records kept at the Corps' website, but the Corps' level gauges along the canal were not responding reliably on August 29th.

However, there is another level gauge that did not fail: the Sewerage and Water Board's gauge at drainage pump station #6, located at the southern end of the canal.

That gauge uses a different datum for its zero point, an antiquated thing called "Cairo Datum." It must be converted to normal water levels, and sometimes figuring that conversion out is difficult. I've gotten different responses over the years. However, I just recently found the definitive source for the conversion. Take a look at this photo:

This was taken by an architectural graduate student affiliated with a group project called "Gutter to Gulf." They visited New Orleans in February, 2009 and stopped in at pump station 6. One of them snapped this photo inside the pump station control room. The digital readout on the right shows the water levels on the inlet (top row) and outlet (bottom row left) of the station. Of far more interest is the paper above that display:

There's our conversion factor for the discharge water levels at DPS#6: 6.0 feet in normal terms equals 27.6 feet in Cairo Datum terms. The safe water elevation was raised since 2009 to 6.5 feet, which was the number in effect when Isaac hit. So anything over 28.1 feet as measured by the DPS#6 discharge water gauge indicates the 17th Street safe water elevation was exceeded.

Examining the DPS#6 discharge level gauge readings on the morning of August 29, 2012, as recorded in the station's logs:

The logs show the discharge gauge read above 28.1 feet for almost two hours, at one point exceeding it by a full 0.5 feet, or six inches. That is, the water got to 7 feet in the canal, placing the walls and levees in danger. This is the first time such information has been reported publicly.

How bad were the Corps pump failures?

The next question to ask is how this happened. We know there were sizable pump failures at the Corps end of the canal, but how sizable were they? Let's go back two days before the storm.

Despite Corps Operations chief Chris Accardo's assurances to the locals and others in the days leading up August 29th, all was not well at the Corps' 17th Street site. Just before noon on August 27th, the Corps' 17th Street canal captain Ray Newman sent an email to Kirk Bowman of Point Eight Power in Belle Chasse, LA, in nearby Plaquemines Parish. Point Eight was (and remains) responsible for the pump controls on the 11 direct drive pumps at 17th Street and the 8 direct drive pumps at the London Avenue canal site. Newman's email reads,

This message is your authorization to have your field service tech. override the shutdown safety devices at our request, as listed below. These are temporary measures to provide vital pumping capacity for our operations during TS/Hurricane Isaac. Our operators will monitor the equipment manually while these shutdowns are inactive.

a) Override low gearbox oil pressure shutdown on Unit No. 19, 17th St. Canal
b) Override gearbox vibration shutdown on Unit No. 18, 17th St. Canal


This email is revealing in a number of ways. On the surface, it points to problems on two of the supposedly more reliable direct drive pumps just ahead of a hurricane. Overriding the automatic shutdown commands could point to the Corps wishing to temporarily bypass a nuisance interlock that was not indicating a real problem. That's the most generous explanation. The more likely - and less charitable - explanation was that pump W19 was experiencing low oil pressure on its' gearbox and pump W18 was experiencing high vibrations on its' gearbox. The Corps needed these pumps to run, so they told Point Eight to pull those interlocks out of the program.

But in a greater sense, it points to how thin the Corps itself is on expertise in their own equipment. For an experienced industrial programmer, bypasses like these should be a piece of cake. But the Corps obviously has no such person on staff. They had to summon a contractor to make programming changes on the eve of a major storm. We see this over and over - the Corps themselves doesn't just contract out duties like engineering, quality control, and quality assurance. They also farm out their emergency response activities under the pressure of a storm, leaving until the last minute the call to these folks, hoping the companies have decided to keep their employees around. The Corps can order their own personnel to stick around; their power over contractors is much more murky and lends a sense of unease to citizens depending upon the correct functioning of life-critical storm protection systems.

But back to the events surrounding the morning of August 29th. As noted above, S and WB drainage pumping station 6 was limited to 5200 cfs at one point. This means the Corps pumps were likely pumping at approximately the same rate. What does that mean in real terms?

Here's the layout of the pumps at the Corps 17th Street site:

Here's the numbering of each pump at 17th Street:

According to the Corps' "pump tracker" spreadsheet, the 18 "Phase 1" and "Phase 2" pumps, which are the 60" hydraulic pumps made and supplied by MWI, each have a nominal rated capacity of 200 cfs. The 11 larger direct drive pumps (abbreviated "DD" in Corps pump status emails) each have a nominal rated capacity of 364 cfs. The 14 "Phase 3," or "bridge," pumps - also hydraulically powered units made by MWI - each have a nominal rated capacity of 114 cfs.

We can get an idea of what was going on early on the morning of the 29th by examining what happened afterward. You see, the Corps' pump problems did not disappear at 8 AM. The problems just got small enough, and the rainfall lessened up enough, that the Corps pumps could barely squeak by. In fact, emails subsequent to the morning crisis reveal substantial problems remained. Just after 10 AM, Corps Operations Assistant Chief Jerry Colletti sent out an email requesting pump statuses from all three gate sites.

At 11:20 AM, the Corps' Michael Sullivan sent back the first update for 17th Street:

"No change.
ICS Capacity - 7700cfs
PS6 Capacity - 6200 cfs, will increase to 7300cfs w in 10 min."

Unfortunately, we don't have what Sullivan had possibly previously sent when he referred to "no change." Also, it is unclear if Sullivan, when he wrote "ICS capacity," was referring to the maximum capacity the Corps' station could pump, or what the station was currently pumping. However, the lack of clarity was eliminated in subsequent updates. A 12:03 PM email said:

"3 phase 1, 1 DD, 3 Bridge - DOWN.
ICS - 7894cfs
PS6 - 6200cfs, going to 7200cfs in 5."

So with 3 phase 1 pumps, 1 direct drive pump, and 3 bridge pumps down at noon on August 29, 2012, the theoretical capacity of the Corps 17th Street ICS was 9200 minus (3 times 200 + 1 times 364 + 3 times 114), or 7894 cfs, which matches what Sullivan reported. Put another way, the Corps's pump station - which does not have any spare capacity when the nearly 10,000 cfs city pump station #6 is going all out - was only running at 86% at noon on the 29th

What that means is that if the city pumps had needed to put more than 7894 cfs down the canal (and they can, and have before), the crisis would have restarted again.

From this sort of information, we can backtrack and figure out how poorly the Corps pumps were doing between 6:30 AM and 8 AM. We know they had 7 pumps down at noon and that resulted in a capacity of 7894 cfs. Let's round that to 7900 for ease of calculation.

Between 6:30 and 8 they couldn't pump more than between 5200 and 6200 cfs, or an additional 1700 to 2700 cfs deficit. At a minimum, this means at least 5 additional direct drive pumps, or some combination of an even larger number of pumps - some with smaller rated flows - were out of commission at that time.

Assuming the pumps offline at noon were also down earlier in the morning, this would mean at least 12 - a dozen - and possibly many more pumps were not available for nearly two hours during the height of Isaac. Given there's 43 pumps total at the site, just on a raw pump count basis that means nearly 28% of the pumps were non-functional. On a flowrate basis, it means the Corps was down between a third and nearly half of their rated throughput.

Nearly third to a half the Corps' pumping capacity was offline for hours... at the very time such pumps were needed. That allowed the 17th Street canal walls to have their safe level exceeded for hours as well. Let that sink in. At the moment the Corps was supposed to be providing maximum storm protection through pumping, they were failing so miserably the outfall canals were threatened with another breach.

Pumping failures continued throughout Isaac

Frankly, it was pure luck there wasn't more rain that day, and that is what saved the Corps' (and by extension, the city's) bacon. Because while the Corps got some of their stuff in order, after noon the news got worse at the 17th Street ICS.

At 2:04 PM, they lost another phase 1 hydraulic pump:

"4 phase 1, 1 DD, 3 Bridge - DOWN.
ICS - 2900 cfs
PS6 - 3000 cfs"

[Note this flow report for the ICS refers to Corps pumps turned on, not those actually ready to run]
Again - I cannot say this enough - it was pure luck that the city was only sending 3000 cfs down the canal at that time. The loss of a fourth hydraulic pump dropped the Corps' pumping capacity to 7694 cfs, or 83.6% - its lowest point outside the crisis period in the morning. The capacity at pump station 6 remained at its maximum of 9300 cfs - every city pump was available. The Corps pumps would never reach their maximum capacity during Isaac.

That situation continued for about 4 hours, during which the city fortunately pumped a steady 3000 cfs down the canal. The 6:01 PM update showed some improvement on the Corps end of things:

"(2) Phase 1 pumps out (change), (1) DD pump out, (2) Bridge pumps out (change)
17th ICS: 2700, 0.5 gage
PS 6: 2200, 0.7 gage"

So they got 2 of the phase 1 pumps and one of bridge pumps going, raising the ICS capacity to 8208 cfs, or 89%. The ICS remained at that pumping level until at least 10 PM.

Then, at 11:43 PM things appeared to get worse again. Donald Schneider sent the following email:

"11W and now 20W are now malfunctioning the same fault. Start the engine the point 8 screen goes to reboot and the engine dies. Never saw this one before. Gotta let point 8 wrestle this one its programing."

Once again, the Corps was dependent on a contractor - Point Eight Power - to make things run. This time - it was during the storm. As soon as they ran into the smallest problem that appeared to be "computerey," they were forced to throw up their hands and send up the bat-signal because they had no programming knowledge.

This problem is possibly the source of a report from WWLTV titled "17th street canal pumps forced to be started manually Tuesday night" Most of the article is a confused rehash of what had happened Tuesday morning. But the first line says:

"Pumps at the 17th Street canal failed to turn on automatically Tuesday night, forcing pump operators to manually start the system, the Corps of Engineers said Thursday."

Notice how the emergency role of contractors in bypassing the pump control programming was not even mentioned.

The final report on pump statuses came the following morning at 5:06 AM

Pumps out: 1 Phase 1, 2 DD, 2 Bridge
17th:1400 cfs, 0.2 gage
PS6: 2200 cfs, 0.2 gage

So even then, the Corps was down 5 pumps. This and the other updates through the afternoon and evening of August 29th completely contradict reporting by the New York Times' John Schwartz, who claimed regarding the pump status at 8 AM on the 29th: "Within a couple of hours, workers would get all of the pumps running and bring the water levels down, and the sense of potential crisis eased." The Corps never got "all" the pumps running. They got enough running at 8 AM to barely do the job at the same time rainwater in the city's drainage system was slackening off.

Other stuff going on at the same time

The events on the periphery of the Corps failures at the 17th Street canal - and they were failures, which were eventually only remedied by the luck of the weather - are just as interesting. I'd like to highlight two email chains from that day, both of which only came to light with the receipt of Chris Accardo's emails.

The first was an email chain involving the then-New Orleans District commander Colonel Ed Fleming. Fleming has since moved on to Corps HQ in Washington. That morning, Fleming responded to an inquiry from the Army in Washington DC asking what was happening at 17th Street:

"- We never had to tell SWB to slow down. They wanted to bring another 500cfs on line and we asked them to wait a little while - which they did. We started the rest of the pumps manually. Then we told them to give us everything they could.
- Our pumping percent capacity is interesting but not determining. Our goal is to stay ahead of SWB. If we are pumping at 80% capacity and they are pumping at 50% capacity then we are in good shape. Our goal is to out-pump them."

I'll deal with the misinformation in the first passage later. But first, I'd like to opine on the second statement that the Corps' "goal is to stay ahead of SWB." This reeks of the "it's close enough, we don't need any extra" philosophy that appears to permeate the New Orleans District. They always appear to be looking for a corner to cut or a penny to pinch, designing or building or operating things to the gnat's ass as if there's never going to be a circumstance where extra anything will be needed. Meanwhile, they ignore gaping holes (sometimes literally) and try to cover them with duct tape and press releases.

What happens when the SWB is pumping at 80% of capacity and the Corps only has 50% available? You get the freak-out we saw on the morning of August 29, 2012. The goal for the Corps' pumps should not be just to stay ahead of the SWB, because no one can predict the weather. The goal should be 100% reliable pumping at any time, for any duration, with spare capacity built in to overcome unforeseen failures. The SWB pump stations were built this way, but the Corps decided to go with "probably good enough" and "trust us." Their calculations for safe water elevations within the canals are of the same flavor - they only calculated the strength of the walls up to 10 feet, even though the walls are actually 15 feet or taller. What would happen in the case of a failure of a gate segment at the lake and the inrush of storm surge, as easily could have happened at London Avenue with the unpinned gates? Would the Corps say "we built stuff good enough to handle a scenario we deemed most likely, despite the dire circumstances a less likely scenario would present?" Phrased another way, that question would be "Katrina? Never heard of her."

At the very least, these massive pump failures show all the supposed exercising of pumps during and outside of storm season is clearly garbage. Under fewer than 12 hours of stress, their system nearly collapsed. The ability to run a single pump on a sunny day for ten minutes in front of TV cameras and a few dignitaries (as the Corps does every May) simply cannot compare with the need to run 43 pumps on the rainiest day of the year for 12 to 24 hours.

It's like someone telling you, "It'll work until it breaks." We saw how well that philosophy worked on August 29th, 2005, and it does not appear to have been excised from the New Orleans District at all.

Now, to the half-truths in Fleming's first passage. We know the Corps told the SWB to shut down 1000 cfs pump "D" in PS6 at about 6:40 AM on August 29th. That came after a an earlier refusal - at 4:35 AM - to allow the SWB to run 1100 cfs pump "F." A second refusal on "F" came at 6:50 AM. In addition, the SWB was told not to load any more pumps above the 5200 cfs they were flowing at 6:40. That's a potential 4000 cfs kept out of the canal, not 500 cfs.

Also, the first refusal came at 4:35 AM. The SWB was not allowed to turn on any pumps until 7:55 AM. That's well over three hours, not "a little while."

The second email chain of interest involves Corps engineering personnel within and outside the New Orleans District on August 29th.

At 9:33 AM, Nancy J. Powell, Chief of the Hydraulics and Hydrologic Branch at the Corps' New Orleans District, emailed Walter Baumy, the then-chief of the New Orleans' District's Engineering Division. Also copied was David Ramirez, the New Orleans District Chief of Water Management. By this time, multiple media reports were already mentioning the faulty level gauge readings within the canals, readings available to the public at During storms, Powell is a point person for storm modeling data, including predicted surge levels. She wrote:

"David can call MVR and see if they can stop malfunctioning gages from reporting"

"MVR" is the Corps' Rock Island District, headquartered in Rock Island, Illinois. They run the riverguages website. Powell was attempting to find out if the Corps could stop the bad readings from being reported to the public.

A little over two hours later, at 11:48 AM, Baumy forwarded Powell's email to Denny A. Lundberg, the engineering chief at the Rock Island District, with the additional message:

"Denny, potential issues on outfalls, especially 17th. Public calling on safe water elevations."

Oh no! The public was calling about safe water elevations! We can't have that.

Lundberg wrote back about an hour later, at 12:54 PM:


No one has called MVR as of an hour ago. Any station can be easily turned off using the RiverGages management page. Takes about 30 seconds. I'll have my WC [Water Control] staff contact your staff."

"Whoops" indeed.

Baumy wrote back a few minutes later with the last message we have, copying Chris Accardo, Powell, Ramirez, Emergency Operations honcho Michael Stack, and - most tellingly - two members of the New Orleans District's Public Affairs staff: chief Ken Holder and contractor Rachel Rodi:
"If outfall gages are unreliable, they should be turned off in my opinion.

Will need to defend the unreliability as public will want access. Please provide your assessment.


Based on the records on the rivergauges website, the Corps never carried out this plan. And with squinty eyes, I can see a sorta justification for not putting messed up information out on a public website. But the overriding concern here doesn't appear to be misinforming the public - or even what was being done to address the problem (something one would expect Baumy, the head of engineering for the District, to be asking), but rather exposure of the Corps' missteps to public scrutiny in real time. Having seen the Corps' penchant for keeping outfall canal information away from the public, my view may be a bit colored, but the plain language of the email exchange certainly doesn't cover the Corps in glory and worry for the flood-endangered public.


The events at the 17th Street canal on August 29, 2012 appear to be more serious and longer lasting than the reporting that morning and afterward indicated. The water level in the canal exceeded the Corps' safe water elevation for over two hours in the morning because approximately a dozen Corps pumps failed and could not remove the rainwater being put into the canal by city pumps. The Corps' pumping capacity was cut nearly in half at the height of the need for pumping. Those Corps pump failures prompted orders to the locals to not activate some pumps and in one case to turn one off, causing stormwater to back up into city streets.

After the heaviest rain abated and the Corps got a handfull of their pumps working, water levels inside the canal and in the city streets dropped. However, many Corps pumps continued out of commission throughout the event, raising the probability that if rainfall had again picked up, a crisis identical to that of the morning of August 29th would again occur.

Occurring in parallel with these events were Corps efforts to shape the narrative, often with faulty information. One effort even contemplated cutting the public (and the local pumping authority) off from level gauge data.

Strictly due to cost, the Corps has decided to make this tandem pumping system with weakened canal walls permanent, their so-called "Option 1." Ground has broken for permanent pump stations to replace the clearly faulty Interim Closure Structures, but no information has been released regarding further work to make the canals themselves sturdier. Taken together, the outfall canals remain a serious Achilles heel within the confines of the City of New Orleans and Jefferson Parish.

Go to older posts