Flood mapping and forecasting in the Philippines - my first week

I've tried to arrive here with no preconceptions about what I'll be doing, and on reflection this has probably been the right thing to do. I've been amazed at the progress I've seen since i was first here around 18 months ago - there are large teams of keen graduates working on flood mapping and forecasting, who've already produced results for many river basins.

Merged LiDAR - sonar DTM for river in central Luzon

Some of the storm surge related forecasting is impressive too - they're not doing routine forecasts yet, but do run simulations for big events such as Haiyan, which produced such devastation in Tacloban in November 2013. Their model predicted a storm surge height very close to what was actually seen, and with a 6 hour lead time too. It's another story that this information wasn't acted upon until about 3 hours before the surge hit.

But scratch below the surface and there are still some big challenges - using the vast amount of data being collected is going to be hard, and will need some thinking about. The teams are under pressure from above to deliver products as quickly as possible, but without necessarily having the experience to balance the need for accuracy within the time available.

I guess the biggest problem for me is how to challenge the way things are being done, without seeming (overly) critical. They've had around 2 years of flood modelling experience, mine is closer to 20. So I'm seeing problems that other people won't pick up on, which could make a big difference to the quality of the final products. The sheer numbers of people killed and lives affected by big typhoon events over here means these are important decisions.

One of the desirable characteristics on the volunteer placement description was
"diplomatic" - which isn't me. I may have to wait for Al to arrive. He's worked a lot here before, so may be able to put things in a constructive way, which won't alienate the team here. 

Or perhaps we can do things the classic VSO way. Time to organise a workshop?

Horritt Consulting's 1st birthday

Welcome to 2013 - Horritt Consulting is now 1 year old.

It's been quite a successful year - but then my main aim for 2012 was to not end up homeless, so I set the bar pretty low. I've done a bit better than that - managing to find enough chargeable work for 85% of my time. My target was always to spend around 1 day a week doing stuff that didn't necessarily pay (business development, R+D, messing about on the internet), so by that measure I'm actually working a bit too much.

A couple of highlights. I was invited to give a week long training course in Manila for the Philippine Government who are setting up a national flood mapping and warning system. The Philippines have a big flood risk problem, with flood events killing hundreds or thousands of people happening every 2 years or so - and the people affected are, of course, typically the poorest. I'm hoping I can get out there again this year to help. And of course to see a bit more of the country too. 

A group led by Paul Bates at Bristol University, with Tim Fewtrell of huge insurance business Willis, and our very own Horritt Consulting, won the Lloyd's Science of Risk Prize. I'm spending the money on a folding bike and to support more work in the Philippines.   

Volcano, Talisay, Philippines

Stuff that could go better? I'm probably still reliant on too few clients - although the list is growing slowly, and I've got plans to pester more people in 2013 to see if I can help. I'd also like to get out into the real world a bit more - 95% of my time is spent looking at a computer screen rather than real water.

Severn Bore, seen from Minsterworth

What's my brain doing when I'm not looking?

What's the best way of problem solving? "I'll think about it" you say when confronted with a tricky problem - but it appears this is probably the worse thing to do.
There's an interesting extract in Saturday's Guardian (here) about how the different halves of the brain deal with information differently - left side is logical, right side is doing something much vaguer. Some neuroscientists are saying that this vague stuff the right side of the brain is doing is actually really very important in problem solving.

What the inside of my head feels like most of the time

When I'm stuck with a problem (which is usually in programming, maths or engineering - it's what I do), I do tend to sit down and think about it. Then stand up and think about it. Then get frustrated after an hour and go and do something else - and it's then that the solution often comes. And this is without trying - so there must be something going on in my head, which I'm not conscious of, that's making connections. I think this is the right side of my brain at work.
But sometimes it's slow - I still get small moments of revelation when something I learned at college suddenly makes sense - and I graduated almost 20 years ago.
This also explains why it's difficult to teach problem solving. You can go about it logically - but actually it's the weird connections the right side of the brain is making that finds the solution.
So "I'll sleep on it" may be a better approach when I've got a problem to solve. I wonder if I could get away with invoicing for time spent asleep...

Just when you think you know what's going on

So here's what's happened in the last week or two. Tories announce an unpopular budget; Tories embroiled in funding and influence scandal; Tories try to distract public by picking fight with union that isn't even on strike; incompetent cabinet office minister fuels panic buying with some ill chosen (or was it?) advice. Labour surge in the polls, despite rather than because of their leader; lib-dems remain dead men walking. Labour retain Bradford West with increased majority as a protest against the government.

The last bit didn't happen - instead George Galloway pulls off his election magic in another constituency with a big muslim population.

This was a surprise - not just to me, but to the media too. The World Tonight last night had very little (or even nothing?) about this item - devoting more time to the untimely passing of the banjo legend Earl Scruggs than the recently closed polls.

Galloway is a divisive figure - even in my own head. I agree with a lot of what he says, and he can say it very well. His powers of oratory are well known - but he is also comes across as a publicity seeking, self serving egotist. That doesn't make him wrong, just difficult to believe sometimes.    


So what does this mean for the bigger political picture in the UK? Probably not much. The Tories and Lib-dems have too much invested in the coalition to break it, as the outcome of any resulting general election would be too uncertain to call. If (big if) it were to happen, Labour would have to have changed their leader (perhaps find another Milliband from somewhere) to win a workable majority.

So it's probably a case of big news in Westminster, but we'll be stuck with the current lot until 2015.

2 months old, going on 87

Life as an independent consultant is a bit like being really old: you don't get out much, you eat a lot of soup, you worry about heating bills.

It's been two months since I started working as "Horritt Consulting", and things seem to be going pretty well. I've managed to invoice for 86% of my time so far - my target is 80% so I've one day a week to do admin, research, CPD, hoover the office and look for work.

Is eight cups of tea per working day excessive?

I'm finding it quite fun being everything from CEO up to office cleaner. I'm learning lots of new stuff, and using the stuff I learnt about geodatabases in Indonesia (there'll be an another exciting post about using Spatialite and PostGIS to process big spatial datasets soon...).

Having your own office space is great, as I used to hate working in a big open plan office, and the commute is down to around 2.5 metres. Takes me about 3 seconds if the traffic's not too bad.

There are some interesting jobs coming in too. I'm now looking at some potential work in the Philippines, which could be very exciting.

There's some things I've found difficult:
Not getting out much. Popping out for a paper is the highlight of the day. I'm trying to get round this by getting out and about to meet people at least once a week.
Getting paid 2 (or more) months in arrears. Fortunately I have a credit card, but living off one of those isn't ideal. Perhaps I should be chasing those invoices harder.
Focus (I should be working while I'm writing this) - giving up smoking hasn't helped. There's always something diverting to do - make another pot of tea, play a bit of guitar, listen to a really interesting programme on Radio 4 about fishing quotas, look out of the window.
Organising my time properly - never one of my strong points.

It'll be interesting to see how much work there is after March when the new financial year starts - lots of the stuff I've been doing is happening because people are trying to spend their budget for 2011-2. But at least I'll have had a few good months to begin with. 

Sunset from the office window

3's a bit of a silly name for a phone network

Because it's quite hard to Google. Lots of pages contain the digit "3" - especially ones about 3G phone networks.

I've moved into my new flat, but the broadband won't be working until the modem arrives next week (sluggish - but apparently it's a busy time for the Royal Mail). It's £50 for a 3GB pay as you go dongle in Maplin - so I bought one, after asking whether it would work with Windows XP (I've a virtual XP box). 

Youth in shop: "Says so on the box, innit." Well, that's certainly put my mind at rest. Lesson: don't ask anyone in a shop about computers, unless they have a beard. Or are at least old enough to have a beard. But I bought one anyway (a dongle, not a beard), and have spent around 10 hours trying to get it to work.

So just in case this will help any other frustrated Linux users, here's the useful stuff of how to get one of these working (on Xubuntu 10.04):

1. Spend 5-6 hours trying to get it to work on an XP virtualbox. Make sure you've plenty of swearwords handy - if not, go straight to step 2. 
2. Install some useful stuff from here - those nice people at Betavine have lots of things to make broadband dongles work on Linux. I got ozerocdoff, usb_modeswitch and vodafone-mobile-connect. I don't know which of these are necessary/sufficient.
3. Plug in your dongle. If you forget this, it's extremely unlikely that it will work.
4. With these installed, you should be able to right click on the network icon in the toolbar, go to the "Mobile Broadband" tab and Add, and then your dongle should appear in the dropdown list. 
5. HERE'S THE DIFFICULT BIT: Through a process of trial and error, inspired guessing and ouija board work, you have to fill in the options. Or copy them from here, where a lovely man/lady has done the hard work for you and posted the results. Here's what my options look like:
   
   
6. Right click on the network icon, enable mobile broadband, and Bob may well be your uncle, or at least your Auntie's favourite lodger - internet on the move. I got a staggering 1.4 Mbps, which is OK for general surfing. 
7. Marvel at how 3 (or Three, or III or whatever they're called) could provide this information quite easily, but don't.

National Flood Risk Assessment - Free

Following on from my last blog about open data from the ordnance survey, I've been thinking about what we can use all this free stuff for. So how about trying to estimate coastal flood risk for all England, Wales and Scotland?

As any fule kno, you need highly accurate elevation data, locations and types of residential and non-residential properties, effective flood propagation models and information on location and type of flood defences, as well as information on flood water levels from rivers and the sea. National agencies like the Environment Agency and SEPA spend a lot of money gathering this kind of data, and use it to estimate risk at the national scale.

Most of this data isn't available free - but how far can we get with the free data that is available? 

Here's a list of free stuff we can use for estimating flood risk in Great Britain (I can't seem to find equivalent open data for Northern Ireland):

Quantum GIS	Excellent open source GIS software.
Spatialite	Open source geodatabase for analysis of big data sets.
Postcode Open	Point data for postcodes - gives a point for each unit postcode, but not boundary polygons.
Landform Panorama	Elevation on a 50m grid. Accuracy is probably around 2-3m.
Estimates of Extreme Sea Conditions - Spatial Analyses for the UK Coast	Estimates of extreme water levels - doesn't actually cover NI despite the title.
The cost of the summer 2007 floods	Estimates of the damages per property from the 2007 floods.
Flooding in England - a national assessment of flood risk	The "official" estimate of national flood risk from the Environment Agency.
Flooding in Wales - a national assessment of flood risk	The "official" estimate of national flood risk from the Environment Agency, also available in Welsh.
National Flood Risk Assessment for Scotland	Some more "official" figures for Scotland from SEPA.

(I've used "official" in quotes because I don't want to place more trust in these figures just because they come from the appropriate government agency.)

And here's the recipe:

1. Sample elevation from the Panorama data in QGIS - there's a nifty plugin to do this called "Point Sampling Tool".
2. Load the postcode data into spatialite. Make life easier by deleting everything over 20m above sea level - we'll assume these will not be at risk from coastal flooding.
3. Load the Estimates of Extreme Sea Conditions data into spatialite - we get a series of points round the GB coastline, each with data on water levels for extreme event probabilities.
4. Use spatialite to join each postcode to the nearest sea conditions estimate point.
5. We now have a set of extreme water levels, and a ground elevation for each postcode. Use these to estimate the probability of flooding, using a log-probability model.
6. Count the number of properties in appropriate probability bands, by assuming there are roughly 16 properties per unit postcode (the UK has 1.7 million unit postcodes, and 27 million addresses).
7. Use the probability to estimate the annual average damages, assuming damages of £25k per property if it floods, taken from "The cost of the summer 2007 floods" report.

Locations of extreme water level estimates around the coast of Great Britain. Water levels have been grouped as high (red), medium (orange) or low (green). You can see higher levels in some places where we'd expect them - East Anglia and the Channel. But they're only medium around the Severn Estuary - which has the 2nd highest tidal range in the world. Mmm...

Cook all this up and you get results like the ones shown in the table below, showing how many properties have a probability of flooding greater than 1 in 200. 

P>0.005 (1 in 200)	Number of properties from this analysis	"Official" Estimate
England	1.4 million	1.2 million
Wales	100 000	140 000
Scotland	90 000	26 000

If we look at the number of properties at risk, we may have a reasonably good answer (or at least one that roughly agrees with the "official" figures), even using data that's not well suited to flood modelling. But using this technique to estimate annual average damages (if we measure the damages from flooding over many years, and calculate the average per year) gives around £30 billion for England, £2 billion each for Wales and Scotland. The "official" estimate is around £1 billion for England and £200 million for Wales (there are no figures for Scotland). So not so good if we start to think about cost of flooding, rather than the number of properties affected.

Postcode locations in colour coded flood risk bands - a lot of central London has an estimated flood probability of >1 in 75. Fortunately we have the Thames Barrier.

There are some huge gaps in this analysis - things we've ignored, rather than just represented badly. We ignore risk from rivers, the effects of flood defences, and we've assumed all the risk comes from residential properties. My gut feeling is that risk from rivers will roughly double the numbers we've come up with. Flood defences will tend to reduce the risk for properties indicated as flooding more often in this analysis - defences will tend to work for a 1 in 10 probability water level, but are less effective for a 1 in 200 probability event. I think this is why the estimate of annual damages is way too big - there are a lot of postcodes near sea level that are in reality protected by defences, but this analysis would have them flooding every year. Would you be still living in one of these houses? And clearly central London doesn't flood very often - we've got the Thames Barrier to stop that happening.

There are big contributions to the risk from business premises. They can be "hot spots" of risk - a bus garage in Carlisle 2005 and a caravan showroom in Tewkesbury in 2007 were significant contributions to damages. The estimates from the 2007 summer floods indicate that the if non-residential damages were accounted for, this would roughly double the risk.

The error in elevation data is also too big for this kind of application. The difference in water levels between 1 in 10 000 and 1 in 1 (i.e. we see a water level greater than this on average once a year) is as low as 1m in some places, so an error of 2-3m equates to a big error in flood probability.  

But despite the holes, I quite like this result - close enough to the "official" estimate to be interesting, but not suspiciously so. I wonder what would happen if we could get access to flood defence and better elevation data? I also want to look at the social aspects of this - are poorer communities more or less at risk? There's a load of UK Census data available that I want to use to try to answer this question, but first I need a holiday...