Fix the pumps

Sunday, July 06, 2014

Bricks, rebar, huge rocks inside the 17th Street canal levees (which is no surprise)

The photos showing the unearthing of massive amounts of unsuitable debris from the outfall canal levees this spring and summer bring into stark relief the true nature of flood protection in the greater New Orleans area. By the Corps' own standards, a deeply inadequate system remains in place - a system which under current design and construction guidelines would never get built today. We know this - not by having to plumb endless internal emails or looking through musty, decades-old records - but by simply comparing today's pictures to the records of the last 17th Street canal remediation project conducted just three years ago.

A new round of photos of the Corps of Engineers' 17th Street canal work site taken in late June detail more fully the debris that was piled into the 17th Street canal levees years ago:

Bricks and rebar:


...which we can see after moving closer to the area in red:


Bricks and large rocks:


...which are visible after zooming into the area in red:


None of this should come as a surprise, since the same junk was pulled out the levees a bit north of this year's work site three years ago during the first round of remediation (which I wrote about in 2012). That project offers insights applicable to this year's project.

Extensive "unsuitable" material found inside 17th Street canal levees in 2011

The 2011 17th Street canal remediation project consisted mostly of deep soil mixing of grout with existing soil to improve the strength of the soil along both east and west banks of the canal. Columns of dirt below the base of the levees were mixed with injected grout down to -20 to -40 feet. The contractor had to excavate a few feet off the top of the levee to give a flat platform on which to mount the drilling rigs and other work equipment. Here's the generic cross-sectional detail from the project issued-for-construction drawings showing this:


The excavated material is shown with the single hatching on top of the levee. It was expected to be reused as backfill for the same area at the completion of the project. That's not what happened.

Right from the start and throughout the entire 2011 project, the Corps' contractors continually found the levees on both sides of the canal were riddled with, shall we say, non-dirt objects. The greatest concentration appears to have been on the west - or Jefferson Parish - side. On the very first day of construction on the west bank (March 7, 2011, which was also Lundi Gras), the contractor dug into the levee near the intersection of Orpheum and Rosebud. The contractor's Quality Control report (all reports from the 2011 project are linked from this post) for that day said:
"Visually inspected the construction of the West Bank production pad. Material removed from the existing levee between stations 588+00 and 590+00 contained significant debris (rocks, concrete, bricks, shells, asphalt). Informed Bill and Derrick [Corps employees Bill Richardson and Derrick Parker] and they came over to look. We asked they to determine if this would be considered "unsuitable" to reuse in the construct of the levee"

Four days later, March 11th, a decision was made on this "significant debris:"
"Got a verbal answer from [Corps employee] Bill Richardson concerning the 'unsuitable material' encountered during the excavation of the West Bank platform. We are to excavate to the required depth to build the production platform only. All excavated material will be hauled offsite and replaced with suitable clay material from the Bonnet Carre borrow pit."

The 2011 project involved excavating the top of the west bank of the canal from Georgia Court to the Old Hammond Highway bridge, a distance of 3/4's of a mile. Thus, this statement implicitly said that the entire levee along that stretch  - at least as deep as the Corps wished to dig but no deeper - was made of dirt not suitable for levee construction:


The Corps' wink-and-a-nod instruction ("excavate to the required depth to build the production platform only") indicated they believed the rest of the levee was similarly riddled, but didn't want to bother dealing with it. That means there's likely still junk in the 17th Street canal levee today, just further down than where the 2011 project dug. Returning to the detail from the drawings, we can see this graphically:


Two weeks after the implicit acknowledgement of the unsuitability of the west side levees, the contractor made it explicit. In the March 24, 2011 Quality Control report they reported on excavation of the west bank levee along Orpheum between Chestnut and West Esplanade:
"Excavation (degrading/construction of production platform) on West side of canal today from station 581+80 to 579+00 to an elevation of +1.8 to +1.5'. All degraded material was hauled to laydown area on the East side of the canal. Material was considered unsuitable by USACE and will be stockpiled in laydown area on east side of the canal. The material will then be spread throughout the laydown yard at the end of the project per USACE direction."

As I noted when I wrote about this in 2012, at least the Corps didn't try to reuse this junk dirt. But they also didn't bother to tell the public and their government representatives in any meaningful fashion there were "rocks, concrete, bricks, shells, and asphalt" inside 3/4's of a mile of 17th Street canal levee that had been there since it was built decades ago. Or that such debris would remain within the levees even after the project was completed. They just proceeded to conduct business as usual, like it was a normal thing to discover - six years after the fatal failure of their levees - that the levees along the 17th Street canal were built out of trash. Day after day for months, they hauled dirt off the Jefferson Parish levee across the canal to their laydown yard on Bellaire Avenue along the east bank of the canal - in front of the 2005 New Orleans breach location - and told no one in the public the extent of the problem. Some of that bad dirt was spread out at the laydown yard during the summer, 2011 and literally buried under grass. The rest of it, including some stuff too big to be buried, was trucked offsite. The August 23, 2011 quality control report describes this disposition:
"East side of the canal - Spreading unsuitable material in laydown yard per USACE's direction. Large debris is being removed and stockpiled to be hauled off at later date."

Fortunately, three years later during another project involving digging into the Jefferson Parish side of the canal, we have pictures showing us just how poorly these levees were constructed - and indicating how much junk remains inside them today, so there's no way this information can be buried or hauled away again.

Debris so huge it couldn't be removed in 2011, or ever

Some of the junk was so huge in 2011 it couldn't even be moved out of the levee. An extreme example of this was documented by the contractor logs starting on March 25, 2011. That day, there came discovery of a very large concrete "object" just below the west bank levee near the intersection of Orpheum and West Esplanade. From the contractor's quality control report:
"During the construction of the production platform on the west side of the canal, a large concrete structure was exposed at approximately station 577+80. CKY excavated around object to try and see the size of the object and how to remove object. The size of the concrete object was found to be approximately 20'x13' with rebar protruding out of the concrete. Object was unable to be removed. USACE was contacted about the object and pictures were taken. GPS was used to mark the location of the object. The USACE will be asked in an RFI [request for information] if only elements ABC can be installed at this location due to the large concrete object."

The Corps quality assurance report has a little more detail:
"The concrete footing is quote [sic] thick, it could exceed 2 feet. The footing consists of spread footings that cover the area between stations 577+81.5 and 577+90.5. The depth, below levee grade, is near the elevation of the adjacent road."

That's right: a room sized chunk of concrete buried under the levee that didn't get discovered until the levee was dug out. This was so huge that even the Corps owned up to it, taking pictures two months later when they dealt with the thing:




This mass of concrete was in an area that had been extensively explored by Corps test borings. No fewer than 11 test bores and cone penetrometer tests had been drilled into the levee within 200 feet of this location, 10 of them since the 2005 levee failures, as seen on this detail:


One must ask how such a giant chunk of concrete could be unearthed as a surprise. It also raises the obvious question about the quality of the information gleaned from the test bores and other exploratory drilling the Corps does before a project.

In this particular case, the obstruction was so huge they actually left it in place. After covering it back up in March, they went back in late May. They uncovered the obstruction and then backfilled and compacted the affected area between May 24, 2011 and June 1, 2011. Presumably some work crew 20 years in the future will dig it out again.

Comparing the 2014 project to the 2011 project

Going back and reviewing all the records from the 2011 project has given me a little different perspective on what we can see in the 2014 project photos. During the 2011 project, all the construction on the levee was done with dirt from the levee. This included building of platforms and ramps - things occurring right now during the 2014 project. Clean dirt was not brought into the site in 2011 until all substantial work was completed. I have no reason to think the 2014 project is proceeding any differently.

So unlike what I wrote a couple of weeks back, I now believe that all the soil currently piled up on the levee came out the levee, including the stuff near the base that is acting as a "road" for the crane. This means giant boulders are also coming out of the levee, not just chunks of concrete, brick, and pieces of rebar. Here's a couple of the pictures from my prior entry, relabeled to include that conclusion:


In comparing the progress of the 2014 and 2011 projects, what do we make of the fact the Corps started removing bad dirt from the site within days of unearthing it in 2011, but the junk has remained on the levee for weeks in 2014? Some of the reason might simply be how this year's site is set up. The dirt at the bottom of the levee this year remains in use for access to the entire site, with only a single point of entry or exit for heavy equipment at the south end of the site. The crane using the platform at the levee bottom is needed for the entire duration of the project, which has not been completed yet (the contractor is moving from south to north). So the bad dirt making up the platform  probably can't be removed until all the sheet piling has been placed.

Debris throughout the entire 17th Street canal west side levees?

However, the logistics of this year's project are not nearly as important as the larger point: everywhere the inside of the 17th Street canal west side levee has been revealed - whether in 2011 or this year - massive amounts of debris have been found. And even more troubling, there are very long stretches which remain untouched:


Given the shallow depth it was found in 2011 and 2014, it makes sense some of the debris was part of the I-wall and levee construction in the early 1990's. That work did involve placement of dirt on top of the existing levees at the time. It extended the entire length of the canal (all the yellow and red lines shown above). Other, deeper debris has likely been there far longer. The records of nearly every soil boring taken in 1981 along the west bank - long before the I-walls were put in - have numerous mentions of shells, wood, bricks, and other "miscellaneous fill." Those records were used in the basis of design for the Corps' early 1990's work. This junk has been known to the Corps for a very, very long time. It also very likely extends the entire length of the canal on at least the Jefferson Parish side.

How it got there doesn't really matter now though, because all the junk is still there today, and the integrity of the outfall canals remains the responsibility of the Corps. As the decision recorded in the contractor quality control log of March 11, 2011 indicates, there is no way the 17th Street canal west bank levees would be built today with the debris-riddled dirt that is currently sitting in them. That dirt is - according to the Corps - "unsuitable." Yet there the levees sit.

Thus, this is what the Corps considers the recipe for sufficient levees along possibly the most well known stretch of levee in the nation:


These levees do not care what water presses against them - storm surge or rainwater. Thus it make no difference that the outlets of the outfall canals are "protected" by gates now. These levees still see pressure from stormwater during major rainfall events. One must wonder what local government officials and Corps personnel living behind these levees think seeing the true condition of the Corps' flood protection.

Labels: ,

Monday, June 23, 2014

Hidden in plain sight

[Update, 7/7/14: Based on photos taken after this post was published, and a review of the records of the 2011 17th Street canal remediation project, revisions have been made to this post. It is now believed the large rocks seen at the base of the 17th Street canal west side levee in photos taken June 15, 2014 did indeed come from within the levee. Text and photo captions have been adjusted accordingly. Further details and analysis are available at subsequent Fix the Pumps post published July 6, 2014.]

Unreported by anyone, the Corps has begun major work along two outfall canals in New Orleans in the last month. Very long sheet pile is being driven on the 17th Street and London Avenue canals, and rock is going to be placed at the base of the levees inside the canals starting in June or July. These are the same outfall canals that breached in three places in 2005 due to poor design. The ensuing floods of New Orleans devastated hundreds of thousands of lives throughout the greater New Orleans area and beyond.

The Corps' irrational need for secrecy about any work along the outfall canals is long known, but their actually beginning structural work without any public pronouncements, press releases, or whispers in the ears of the mainstream media is unprecedented, especially considering the scale of the work - which is extensive. I think they would rather nobody paid any more attention. But attention should be paid to work at the sites of the cause of the worst engineering disaster in American history.

The work underway already is for seepage control due to existing sheet piling being too short. Seepage happens when water works its way through sand below the bottoms of the I-walls along the canals. Should that water then percolate upward through the land-side (the side where people  live) soils, the levee and I-wall can fail. This was the primary mode of failure for the south breach of the London Avenue canal following the 2005 passage of Katrina dozens of miles to the east.

Problems have emerged just weeks into the project, as the soil being dug out of the levees appears to be packed with unacceptable debris like rocks and chunks of concrete. The Corps' own specifications forbid such debris in a levee since it decreases the strength of the soil. However, they have shown they are willing to ignore such specifications during many previous projects both before and since 2005, including a round of projects on all three outfall canals in 2011.

Those 2011 projects included similar seepage remediation work to this present project along the London Avenue canal and the Orleans Avenue canal. The 2011 17th Street project remediated against different modes of failure than seepage, and thus included different technologies than sheet piling.

For calculation purposes, the Corps and their consultants have broken each side of each canal into individual "reaches," based on differing soil and canal bottom conditions. The reaches are numbered sequentially, starting at the south end of the west side of each canal and finishing at the north end of the east side of each canal.

Here's the locations previously remediated along the London Avenue canal in 2011:


And here's the new locations in the 2014 project:


As you can see from the above map, after they are done with this round at London Avenue, nearly the entire canal north of Mirabeau will have had sheet piling placed outside the existing, too-short piling. It points up how poorly the I-walls were designed the first time around, and how inevitable their failure was.

The contract - which includes work at both the London Avenue and 17th Street sites - was awarded February 28, 2014 to Conquistador Dorado JV (aka Dorado Services) out of Sanford, Florida. There are indications Cycle Construction in Kenner is the local partner in the joint venture. The contract - actually a task order on a Service-Disabled Veteran-Owned Small Business (SDVOSB) contract awarded to a pool of four companies -  is worth $13,642,238.50 and is scheduled to be completed October 30, 2014, meaning construction will proceed throughout the 2014 hurricane season. Construction along the London Avenue canal started in late May with Reach 12, just north of Filmore Avenue on the west side of the canal.

I have had interested neighbors photographing the work as it proceeds. This photo is from what I believe was the first day piles were driven at the London Avenue canal site, May 29, 2014:



Sheet piling is being driven down to clay approximately 55 feet below sea level. The new sheet piling is being driven a few feet to the protected side of the existing I-wall. The existing sheet piling inside the I-walls goes down to sand about 15 feet below sea level. The Corps is using a pile pressing machine rather than a standard pile driving hammer. The pile presser supposedly creates fewer vibrations and is quieter. It's a machine they've used extensively in the dense urban environment of New Orleans and its surrounding metro area, especially where space is at a premium:


Here's piling being placed the following day:


To make room for the pile presser, the top couple of feet of levee soil next to the I-walls will be stripped off to make a level surface and the dirt stockpiled on the levee itself. Here's the excavators at the London Avenue site:


The sheet piling machine travels along the newly flat section and presses the piling in. This June 6, 2014 photo shows the first few linear feet of piling already driven:


Afterward, the soil will be placed back on top of the piling and grass will be planted. This was done at adjacent Reach 13 in 2011.

If we move closer to the stockpiled soil, details emerge. Worrisome details:


Here's enlargements of the photo above:



All the white bits are broken concrete. This next photo shows it best:


Here's the concrete debris highlighted:

The dirt coming out of the levee is absolutely shot through with the stuff. That's bad. Having debris like concrete in the clay decreases its strength because it cannot compact properly. Finding debris in these levees is not a surprise, since it was reported on either side of this stretch during the 2011 London Avenue project as well:

It is one thing to read about it months afterward, and another thing altogether to have pictures of it happening live.

After receiving the pictures of the London site, I asked for pictures of the 17th Street site. Based on those pictures - taken on June 15th, I believe the story is much the same.

The site location on the 17th Street canal is on the west side, just south of I-10:


It is reach 16, which I wrote about extensively 3 years ago. At that time, the Corps was considering placing dirt at the base of the levee on the protected side (outside) of the canal to prevent failure from underseepage. The idea was to provide enough weight on the landside to resist the upward motion of underseeping water. It's an indirect, cheaper way of addressing seepage, as opposed to the direct solution of cutting off the seepage with sheetpile driven into the clay below the sand. It's noteworthy nothing was done in the intervening three years. It's also noteworthy the Corps New Orleans District consultants' calculations in 2010-11 (as well as the Corps' District Engineering folks' marching orders to their consultants before any calculations were performed) claimed there were no seepage problems on reach 16 - or anywhere along the 17th Street canal. Because of that, this newest remediation project is the first one to address seepage concerns along the 17th Street canal. We'll get into what likely changed their mind in future postings.

Here's reach 16 in 2011 (photos taken from the June, 2011 Corps update on that year's remediation projects along the outfall canals):





Construction at the 17th Street canal site also started in late May. The contractor began at the southern end of the site, which is what is shown in the "before" pictures above.

Here's that same end of the site snapped June 15, 2014 (Update, 7/7/14: bottom caption used to read "Dirt placed to make 'road' for crane to travel. Dirt likely not from inside levee"):


I believe that the dirt toward the bottom of the levee is - I hope - fill brought from offsite and placed to allow the crane (which holds the piling vertically) to travel along the levee with the piling pressing machine. I do not think it, and the huge boulders in it, came out of the levee.

[Update, 7/7/14: Based on subsequent photos and a new review of the 2011 remediation project records, it is now believed the material at the bottom of the levee is from inside the levee. This includes the large boulders now visible.]

However, I believe the soil toward the top of the slope is from inside the levee. Even from a distance, we can see the chunks of debris. Getting a closer look shows a large amount of concrete and rocks (Update, 7/7/14: bottom caption used to read "Dirt placed to make 'road' for crane to travel. Dirt likely not from inside levee"):


Here's an enlargement of the above photo:


There's even a stump that appears to be about 2 feet long, just sitting on top of the levee. It seems very likely it came from within the levee (though I suppose it could have come from somewhere else, but then why would it be sitting on the top of the levee?):


Dirt taken out of the southern part of the site is also being stockpiled on the levee near the northern end of the site in front of the end of Lemon St. The copious debris - all the white bits - is obvious (Update, 7/7/14: bottom caption of first two photos used to read "Dirt placed to make 'road' for crane to travel. Dirt likely not from inside levee"):





By any standard, including the Corps', the dirt coming out of the levees at both these sites would be unsuitable for use in any levee. How do we know that? The Corps themselves said so just four years ago, during a different debris mess that made the news.

In late 2009 and early 2010, reports emerged about large amounts of concrete and other debris being found in the top layers of lakefront levee in Kenner during an enlargement project. That soil had been trucked there in the early 2000's and had not been systematically checked for debris. After a couple of Times-Picayune articles, the Corps held a public meeting on January 28, 2010. They showed the following slides as examples of unacceptable amounts of debris in the dirt:



Those photos look eerily similar to the pictures coming out of the London Avenue and 17th Street sites.

The 2010 slides explicitly called out concrete chunks as unacceptable:


In the 2009-10 lakefront case, the Corps' solution - after blaming the previous contractor and prior Corps employees and assuring the public these were isolated instances (which they weren't) - was to remove all the dirt with concrete in it. According to the Corps, the contaminated dirt went down 2.5 feet, so the Corps stripped it all out and replaced it with clean clay.

Here's the scary part: that might not be the plan for the outfall canals, for three reasons:

1) The bid specifications (as amended) for the outfall canal job call for the contractor to place most of the soil they remove from the levee during construction right back in the levee when the sheet piling work is done.

According to those amended specifications, 2350 cubic yards of soil is to be taken out of the levees and reused between the two canal sites. The Corps is only anticipating 650 additional cubic yards between the two sites to be "unsuitable." That is, they expect only one fifth of the total amount of dirt to come out of the levees to stay out, and the rest will go back in. That was the financial basis for awarding the job.

2) We know bad soil with debris just like what we're seeing this year was taken out of outfall canal levees and put back in during the 2011 projects. The Corps' own contractors' said so, doing so under orders from Corps employees.

3) Finally, we know the Corps has no compunction using or reusing debris-laden soil that violates their own specifications all over the greater New Orleans area. Besides the three outfall canal remediation projects in 2011, I counted 22 projects where it happened in the last few years:



For further information, you can consult my 14 (!) part series on debris in the levees. Part 1 is here, with the other parts linked from it. The Corps' use of unacceptable soil seems to be more the norm than the exception.

Why does this matter? After all, there's gates at the ends of the canals to protect them from hurricanes, right?

Well, the outfall canal walls and levees do not care where water comes from - whether it's the lake or the city's pump stations. Even if the lakefront gates are closed and secured during a high lake level event (a big "if," since during the last hurricane 5 of 11 gates at the London Avenue structure were not secured), the city still has to pump rainwater from within the city down the canals. That water level can get quite high - just as high as the water got from Katrina's surge. A levee with concrete in it instead of clay is simply not as strong, and could be prone to, um, difficulties.

In addition, there is the obvious question: if this is what the top couple of feet of the levees along both canals look like, what about the rest of the levee left untouched by these newest projects? Given that these stretches of levee have not been dug into since the failed I-walls were installed about 15 years before their 2005 failure, it is a reasonable concern that the entire levees - including previously remediated stretches - are just as bad as what we've seen here. Without exploratory trenching - as has been done on many other projects where debris was found - there exists a deep trust deficit between the Corps and the public who depend on these levees for their lives and livelihoods.

Labels: , ,

Wednesday, July 24, 2013

Isaac: the rest of the story, Part 3

Part 1
Part 2

NOTE: I am currently awaiting more material from my Isaac email FOIA request, including a PowerPoint presentation from the evening of August 30, 2012 which was supposed to include a candid assessment of pumping at the outfall canals. In the interim, I've pulled together what I could regarding events along the 17th Street canal during Isaac. This post may change as the remaining information is received.

When I placed my FOIA request for the Isaac emails, I had hoped to not only uncover unknown events during the storm, but also to get more details on the pump failures that occurred at the 17th Street canal on the morning of August 29th. Those failures were so extreme the Corps had to report them to the media. Here's the timeline from my October 6, 2012 entry, "Isaac in New Orleans - what we know so far:"

The PS#6 logs for August 29th (here and here) confirm the outlines of this story, while filling in some detail. At 4:35 AM, the station staff asked to "load" (that is, bring to full capacity) 1100 cfs pump "F." They were refused. Then, at 4:55 AM the Corps told the S and WB not to "load" any more pumps in PS#6:
"Note: Army Corps of Engineers said we cannot load any more pumps. [T]hey are at their max of what they can pump"
At that time, the station was pumping approximately 6200 cubic feet per second, or cfs. The station's capacity among its main rainstorm drainage pumps is 9200 cfs.

This initial notice to the S and WB to hold off on pumping appears to be 1.5 to 2 hours before the Corps alerted the media of pumping restrictions, according to the chronology of the NY Times article. Then, at about 6:30 AM, the station operators received another Corps warning not to load additional pumps beyond what was already running, despite the fact that in the intervening hours, water in the station's inlet basin had risen over four feet. Ten minutes later, the station received orders from S and WB's Central Control to break the prime on 1000 cfs pump "D," effectively losing most of its capacity and sending into a pump status called "light." So there were not only Corps orders to keep some pumps from running, but one pump was effectively ordered turned off with the water coming up. This left the station running at about 5200 cfs.

At 7:55 AM, with the water still very high in the inlet basin, the Corps 17th Street canal captain in the station (Ray Newman) gave an all clear to the S and WB personnel in the station to load additional pumps. Five minutes later, 1000 cfs pump "D" was authorized to be loaded by the S and WB Central, and was running in a loaded condition within a few minutes. Another 1000 cfs pump ("C") was brought on line shortly afterward, increasing the station's flowrate to 7200 cfs. Inlet basin levels dropped four feet in about an hour after those two pumps were added. This chronology roughly tracks what was playing out in the media that morning.

Oddly, there's zero Corps email traffic among the key personnel (Ray Newman, Don Schneider, and Chris Accardo) through that whole time. I would expect something to have been written by someone over that three-plus hour span, but perhaps they were just so busy they didn't have time to type on their Blackberrys. Or perhaps they were all communicating either over the radio or in person, so electronic communication was unnecessary. When there was a pump drive skid fire at the Orleans Avenue site a little over an hour after this event subsided, there was an email trail immediately created, so it's very strange there's no trail for the biggest event during the storm.

How high did the water get in the 17th Street canal on August 29, 2012?

The first question to be asked after an event like this is whether the 17th Street canal maximum operation water level of 6.5 feet (first revealed to the public during the press coverage of this mishap) was exceeded. If it was, that means the walls and levees were loaded beyond their safe capacity. To determine this, normally we would examine the outfall canal level gauge records kept at the Corps' rivergauges.com website, but the Corps' level gauges along the canal were not responding reliably on August 29th.

However, there is another level gauge that did not fail: the Sewerage and Water Board's gauge at drainage pump station #6, located at the southern end of the canal.

That gauge uses a different datum for its zero point, an antiquated thing called "Cairo Datum." It must be converted to normal water levels, and sometimes figuring that conversion out is difficult. I've gotten different responses over the years. However, I just recently found the definitive source for the conversion. Take a look at this photo:


This was taken by an architectural graduate student affiliated with a group project called "Gutter to Gulf." They visited New Orleans in February, 2009 and stopped in at pump station 6. One of them snapped this photo inside the pump station control room. The digital readout on the right shows the water levels on the inlet (top row) and outlet (bottom row left) of the station. Of far more interest is the paper above that display:


There's our conversion factor for the discharge water levels at DPS#6: 6.0 feet in normal terms equals 27.6 feet in Cairo Datum terms. The safe water elevation was raised since 2009 to 6.5 feet, which was the number in effect when Isaac hit. So anything over 28.1 feet as measured by the DPS#6 discharge water gauge indicates the 17th Street safe water elevation was exceeded.

Examining the DPS#6 discharge level gauge readings on the morning of August 29, 2012, as recorded in the station's logs:



The logs show the discharge gauge read above 28.1 feet for almost two hours, at one point exceeding it by a full 0.5 feet, or six inches. That is, the water got to 7 feet in the canal, placing the walls and levees in danger. This is the first time such information has been reported publicly.

How bad were the Corps pump failures?

The next question to ask is how this happened. We know there were sizable pump failures at the Corps end of the canal, but how sizable were they? Let's go back two days before the storm.

Despite Corps Operations chief Chris Accardo's assurances to the locals and others in the days leading up August 29th, all was not well at the Corps' 17th Street site. Just before noon on August 27th, the Corps' 17th Street canal captain Ray Newman sent an email to Kirk Bowman of Point Eight Power in Belle Chasse, LA, in nearby Plaquemines Parish. Point Eight was (and remains) responsible for the pump controls on the 11 direct drive pumps at 17th Street and the 8 direct drive pumps at the London Avenue canal site. Newman's email reads,

"Kurt,
This message is your authorization to have your field service tech. override the shutdown safety devices at our request, as listed below. These are temporary measures to provide vital pumping capacity for our operations during TS/Hurricane Isaac. Our operators will monitor the equipment manually while these shutdowns are inactive.

a) Override low gearbox oil pressure shutdown on Unit No. 19, 17th St. Canal
b) Override gearbox vibration shutdown on Unit No. 18, 17th St. Canal

Thanks,
Ray"

This email is revealing in a number of ways. On the surface, it points to problems on two of the supposedly more reliable direct drive pumps just ahead of a hurricane. Overriding the automatic shutdown commands could point to the Corps wishing to temporarily bypass a nuisance interlock that was not indicating a real problem. That's the most generous explanation. The more likely - and less charitable - explanation was that pump W19 was experiencing low oil pressure on its' gearbox and pump W18 was experiencing high vibrations on its' gearbox. The Corps needed these pumps to run, so they told Point Eight to pull those interlocks out of the program.

But in a greater sense, it points to how thin the Corps itself is on expertise in their own equipment. For an experienced industrial programmer, bypasses like these should be a piece of cake. But the Corps obviously has no such person on staff. They had to summon a contractor to make programming changes on the eve of a major storm. We see this over and over - the Corps themselves doesn't just contract out duties like engineering, quality control, and quality assurance. They also farm out their emergency response activities under the pressure of a storm, leaving until the last minute the call to these folks, hoping the companies have decided to keep their employees around. The Corps can order their own personnel to stick around; their power over contractors is much more murky and lends a sense of unease to citizens depending upon the correct functioning of life-critical storm protection systems.

But back to the events surrounding the morning of August 29th. As noted above, S and WB drainage pumping station 6 was limited to 5200 cfs at one point. This means the Corps pumps were likely pumping at approximately the same rate. What does that mean in real terms?

Here's the layout of the pumps at the Corps 17th Street site:


Here's the numbering of each pump at 17th Street:



According to the Corps' "pump tracker" spreadsheet, the 18 "Phase 1" and "Phase 2" pumps, which are the 60" hydraulic pumps made and supplied by MWI, each have a nominal rated capacity of 200 cfs. The 11 larger direct drive pumps (abbreviated "DD" in Corps pump status emails) each have a nominal rated capacity of 364 cfs. The 14 "Phase 3," or "bridge," pumps - also hydraulically powered units made by MWI - each have a nominal rated capacity of 114 cfs.

We can get an idea of what was going on early on the morning of the 29th by examining what happened afterward. You see, the Corps' pump problems did not disappear at 8 AM. The problems just got small enough, and the rainfall lessened up enough, that the Corps pumps could barely squeak by. In fact, emails subsequent to the morning crisis reveal substantial problems remained. Just after 10 AM, Corps Operations Assistant Chief Jerry Colletti sent out an email requesting pump statuses from all three gate sites.

At 11:20 AM, the Corps' Michael Sullivan sent back the first update for 17th Street:

"No change.
ICS Capacity - 7700cfs
PS6 Capacity - 6200 cfs, will increase to 7300cfs w in 10 min."

Unfortunately, we don't have what Sullivan had possibly previously sent when he referred to "no change." Also, it is unclear if Sullivan, when he wrote "ICS capacity," was referring to the maximum capacity the Corps' station could pump, or what the station was currently pumping. However, the lack of clarity was eliminated in subsequent updates. A 12:03 PM email said:

"3 phase 1, 1 DD, 3 Bridge - DOWN.
ICS - 7894cfs
PS6 - 6200cfs, going to 7200cfs in 5."

So with 3 phase 1 pumps, 1 direct drive pump, and 3 bridge pumps down at noon on August 29, 2012, the theoretical capacity of the Corps 17th Street ICS was 9200 minus (3 times 200 + 1 times 364 + 3 times 114), or 7894 cfs, which matches what Sullivan reported. Put another way, the Corps's pump station - which does not have any spare capacity when the nearly 10,000 cfs city pump station #6 is going all out - was only running at 86% at noon on the 29th

What that means is that if the city pumps had needed to put more than 7894 cfs down the canal (and they can, and have before), the crisis would have restarted again.

From this sort of information, we can backtrack and figure out how poorly the Corps pumps were doing between 6:30 AM and 8 AM. We know they had 7 pumps down at noon and that resulted in a capacity of 7894 cfs. Let's round that to 7900 for ease of calculation.

Between 6:30 and 8 they couldn't pump more than between 5200 and 6200 cfs, or an additional 1700 to 2700 cfs deficit. At a minimum, this means at least 5 additional direct drive pumps, or some combination of an even larger number of pumps - some with smaller rated flows - were out of commission at that time.

Assuming the pumps offline at noon were also down earlier in the morning, this would mean at least 12 - a dozen - and possibly many more pumps were not available for nearly two hours during the height of Isaac. Given there's 43 pumps total at the site, just on a raw pump count basis that means nearly 28% of the pumps were non-functional. On a flowrate basis, it means the Corps was down between a third and nearly half of their rated throughput.

Nearly third to a half the Corps' pumping capacity was offline for hours... at the very time such pumps were needed. That allowed the 17th Street canal walls to have their safe level exceeded for hours as well. Let that sink in. At the moment the Corps was supposed to be providing maximum storm protection through pumping, they were failing so miserably the outfall canals were threatened with another breach.

Pumping failures continued throughout Isaac

Frankly, it was pure luck there wasn't more rain that day, and that is what saved the Corps' (and by extension, the city's) bacon. Because while the Corps got some of their stuff in order, after noon the news got worse at the 17th Street ICS.

At 2:04 PM, they lost another phase 1 hydraulic pump:

"4 phase 1, 1 DD, 3 Bridge - DOWN.
ICS - 2900 cfs
PS6 - 3000 cfs"

[Note this flow report for the ICS refers to Corps pumps turned on, not those actually ready to run]
Again - I cannot say this enough - it was pure luck that the city was only sending 3000 cfs down the canal at that time. The loss of a fourth hydraulic pump dropped the Corps' pumping capacity to 7694 cfs, or 83.6% - its lowest point outside the crisis period in the morning. The capacity at pump station 6 remained at its maximum of 9300 cfs - every city pump was available. The Corps pumps would never reach their maximum capacity during Isaac.

That situation continued for about 4 hours, during which the city fortunately pumped a steady 3000 cfs down the canal. The 6:01 PM update showed some improvement on the Corps end of things:

"(2) Phase 1 pumps out (change), (1) DD pump out, (2) Bridge pumps out (change)
17th ICS: 2700, 0.5 gage
PS 6: 2200, 0.7 gage"

So they got 2 of the phase 1 pumps and one of bridge pumps going, raising the ICS capacity to 8208 cfs, or 89%. The ICS remained at that pumping level until at least 10 PM.

Then, at 11:43 PM things appeared to get worse again. Donald Schneider sent the following email:

"11W and now 20W are now malfunctioning the same fault. Start the engine the point 8 screen goes to reboot and the engine dies. Never saw this one before. Gotta let point 8 wrestle this one its programing."

Once again, the Corps was dependent on a contractor - Point Eight Power - to make things run. This time - it was during the storm. As soon as they ran into the smallest problem that appeared to be "computerey," they were forced to throw up their hands and send up the bat-signal because they had no programming knowledge.

This problem is possibly the source of a report from WWLTV titled "17th street canal pumps forced to be started manually Tuesday night" Most of the article is a confused rehash of what had happened Tuesday morning. But the first line says:

"Pumps at the 17th Street canal failed to turn on automatically Tuesday night, forcing pump operators to manually start the system, the Corps of Engineers said Thursday."

Notice how the emergency role of contractors in bypassing the pump control programming was not even mentioned.

The final report on pump statuses came the following morning at 5:06 AM

Pumps out: 1 Phase 1, 2 DD, 2 Bridge
17th:1400 cfs, 0.2 gage
PS6: 2200 cfs, 0.2 gage

So even then, the Corps was down 5 pumps. This and the other updates through the afternoon and evening of August 29th completely contradict reporting by the New York Times' John Schwartz, who claimed regarding the pump status at 8 AM on the 29th: "Within a couple of hours, workers would get all of the pumps running and bring the water levels down, and the sense of potential crisis eased." The Corps never got "all" the pumps running. They got enough running at 8 AM to barely do the job at the same time rainwater in the city's drainage system was slackening off.

Other stuff going on at the same time

The events on the periphery of the Corps failures at the 17th Street canal - and they were failures, which were eventually only remedied by the luck of the weather - are just as interesting. I'd like to highlight two email chains from that day, both of which only came to light with the receipt of Chris Accardo's emails.

The first was an email chain involving the then-New Orleans District commander Colonel Ed Fleming. Fleming has since moved on to Corps HQ in Washington. That morning, Fleming responded to an inquiry from the Army in Washington DC asking what was happening at 17th Street:

"- We never had to tell SWB to slow down. They wanted to bring another 500cfs on line and we asked them to wait a little while - which they did. We started the rest of the pumps manually. Then we told them to give us everything they could.
- Our pumping percent capacity is interesting but not determining. Our goal is to stay ahead of SWB. If we are pumping at 80% capacity and they are pumping at 50% capacity then we are in good shape. Our goal is to out-pump them."

I'll deal with the misinformation in the first passage later. But first, I'd like to opine on the second statement that the Corps' "goal is to stay ahead of SWB." This reeks of the "it's close enough, we don't need any extra" philosophy that appears to permeate the New Orleans District. They always appear to be looking for a corner to cut or a penny to pinch, designing or building or operating things to the gnat's ass as if there's never going to be a circumstance where extra anything will be needed. Meanwhile, they ignore gaping holes (sometimes literally) and try to cover them with duct tape and press releases.

What happens when the SWB is pumping at 80% of capacity and the Corps only has 50% available? You get the freak-out we saw on the morning of August 29, 2012. The goal for the Corps' pumps should not be just to stay ahead of the SWB, because no one can predict the weather. The goal should be 100% reliable pumping at any time, for any duration, with spare capacity built in to overcome unforeseen failures. The SWB pump stations were built this way, but the Corps decided to go with "probably good enough" and "trust us." Their calculations for safe water elevations within the canals are of the same flavor - they only calculated the strength of the walls up to 10 feet, even though the walls are actually 15 feet or taller. What would happen in the case of a failure of a gate segment at the lake and the inrush of storm surge, as easily could have happened at London Avenue with the unpinned gates? Would the Corps say "we built stuff good enough to handle a scenario we deemed most likely, despite the dire circumstances a less likely scenario would present?" Phrased another way, that question would be "Katrina? Never heard of her."

At the very least, these massive pump failures show all the supposed exercising of pumps during and outside of storm season is clearly garbage. Under fewer than 12 hours of stress, their system nearly collapsed. The ability to run a single pump on a sunny day for ten minutes in front of TV cameras and a few dignitaries (as the Corps does every May) simply cannot compare with the need to run 43 pumps on the rainiest day of the year for 12 to 24 hours.

It's like someone telling you, "It'll work until it breaks." We saw how well that philosophy worked on August 29th, 2005, and it does not appear to have been excised from the New Orleans District at all.

Now, to the half-truths in Fleming's first passage. We know the Corps told the SWB to shut down 1000 cfs pump "D" in PS6 at about 6:40 AM on August 29th. That came after a an earlier refusal - at 4:35 AM - to allow the SWB to run 1100 cfs pump "F." A second refusal on "F" came at 6:50 AM. In addition, the SWB was told not to load any more pumps above the 5200 cfs they were flowing at 6:40. That's a potential 4000 cfs kept out of the canal, not 500 cfs.

Also, the first refusal came at 4:35 AM. The SWB was not allowed to turn on any pumps until 7:55 AM. That's well over three hours, not "a little while."

The second email chain of interest involves Corps engineering personnel within and outside the New Orleans District on August 29th.

At 9:33 AM, Nancy J. Powell, Chief of the Hydraulics and Hydrologic Branch at the Corps' New Orleans District, emailed Walter Baumy, the then-chief of the New Orleans' District's Engineering Division. Also copied was David Ramirez, the New Orleans District Chief of Water Management. By this time, multiple media reports were already mentioning the faulty level gauge readings within the canals, readings available to the public at rivergauges.com. During storms, Powell is a point person for storm modeling data, including predicted surge levels. She wrote:

"David can call MVR and see if they can stop malfunctioning gages from reporting"

"MVR" is the Corps' Rock Island District, headquartered in Rock Island, Illinois. They run the riverguages website. Powell was attempting to find out if the Corps could stop the bad readings from being reported to the public.

A little over two hours later, at 11:48 AM, Baumy forwarded Powell's email to Denny A. Lundberg, the engineering chief at the Rock Island District, with the additional message:

"Denny, potential issues on outfalls, especially 17th. Public calling on safe water elevations."

Oh no! The public was calling about safe water elevations! We can't have that.

Lundberg wrote back about an hour later, at 12:54 PM:

"Whoops

No one has called MVR as of an hour ago. Any station can be easily turned off using the RiverGages management page. Takes about 30 seconds. I'll have my WC [Water Control] staff contact your staff."

"Whoops" indeed.

Baumy wrote back a few minutes later with the last message we have, copying Chris Accardo, Powell, Ramirez, Emergency Operations honcho Michael Stack, and - most tellingly - two members of the New Orleans District's Public Affairs staff: chief Ken Holder and contractor Rachel Rodi:
"If outfall gages are unreliable, they should be turned off in my opinion.

Will need to defend the unreliability as public will want access. Please provide your assessment.

Walter"

Based on the records on the rivergauges website, the Corps never carried out this plan. And with squinty eyes, I can see a sorta justification for not putting messed up information out on a public website. But the overriding concern here doesn't appear to be misinforming the public - or even what was being done to address the problem (something one would expect Baumy, the head of engineering for the District, to be asking), but rather exposure of the Corps' missteps to public scrutiny in real time. Having seen the Corps' penchant for keeping outfall canal information away from the public, my view may be a bit colored, but the plain language of the email exchange certainly doesn't cover the Corps in glory and worry for the flood-endangered public.

Summary

The events at the 17th Street canal on August 29, 2012 appear to be more serious and longer lasting than the reporting that morning and afterward indicated. The water level in the canal exceeded the Corps' safe water elevation for over two hours in the morning because approximately a dozen Corps pumps failed and could not remove the rainwater being put into the canal by city pumps. The Corps' pumping capacity was cut nearly in half at the height of the need for pumping. Those Corps pump failures prompted orders to the locals to not activate some pumps and in one case to turn one off, causing stormwater to back up into city streets.

After the heaviest rain abated and the Corps got a handfull of their pumps working, water levels inside the canal and in the city streets dropped. However, many Corps pumps continued out of commission throughout the event, raising the probability that if rainfall had again picked up, a crisis identical to that of the morning of August 29th would again occur.

Occurring in parallel with these events were Corps efforts to shape the narrative, often with faulty information. One effort even contemplated cutting the public (and the local pumping authority) off from level gauge data.

Strictly due to cost, the Corps has decided to make this tandem pumping system with weakened canal walls permanent, their so-called "Option 1." Ground has broken for permanent pump stations to replace the clearly faulty Interim Closure Structures, but no information has been released regarding further work to make the canals themselves sturdier. Taken together, the outfall canals remain a serious Achilles heel within the confines of the City of New Orleans and Jefferson Parish.

Go to older posts