Thank you University of Maryland: AVHRR Land Cover Data

21 07 2008

Turns out that The University of Maryland Department of Geography has already done a land cover classification project, which highlights urban areas very well! You’d think that would make me feel quite bad, after spending much of last week trying to do this BUT unfortunately their project used AVHRR (Advanced Very High Resolution Radiometer) data which has a larger resolution than LandSat and the data is older. But there is still an awful lot of analysis I can do with 20 year old 1km resolution data, if nothing else I can try to use some of their methods with the more up to date LandSat data I was using before.

So this is how to download, import into Qgis and use the AVHRR Land Cover Data

Firstly go to for some info on the whole project, click on the Download via Search and Preview Tool (ESDI) if you want to browse the World and select the data you want OR head straight to the FTP server. Choose your area, projection and resolution (I’m using North America, LatLong and 1km), eventually you will be presented with a bunch of files from which we download the file ending asc.gz as Qgis can handle this not the bsq.gz. Extract the file after downloading and boot up Qgis. Now hit ‘Add a Raster Layer’ and select the .asc file you downloaded and extracted, the file will be pretty big (NA is 1.39 GB) so expect a wait. Now you will have a greyscale image of your region, the brightness depends on the land cover classification calculated by UMD. Here are the code values which can be found on their website.

Code Values for 1km and 8km data

Value Label RGB Red RGB Green RGB Blue
0 Water 068 079 137
1 Evergreen Needleleaf Forest 001 100 000
2 Evergreen Broadleaf Forest 001 130 000
3 Deciduous Needleleaf Forest 151 191 071
4 Deciduous Broadleaf Forest 002 220 000
5 Mixed Forest 000 255 000
6 Woodland 146 174 047
7 Wooded Grassland 220 206 000
8 Closed Shrubland 255 173 000
9 Open Shrubland 255 251 195
10 Grassland 140 072 009
11 Cropland 247 165 255
12 Bare Ground 255 199 174
13 Urban and Built 000 255 255

To make the map in Qgis look at all interesting we need to apply a colormap algorithm. You can select your own by right clicking the file and selecting properties and changing the settings or go to properties select ‘Load Style’ and load the style I created which can be downloaded here (right click and ‘Save target as’). Now you will have a wonderful map showing land use in your region, the area highlighted in white is what I am particuarly interested in and worked on for much of last week. The next step is to see if I can work out how to generate the code values easily and apply it to LandSat data, then compare to OSM.

Here are some image Qgis outputs for North America and San Francisco.

Another thing to look at while I’m at it: DMSP data

18 07 2008

Urban areas are likely to emit more artificial light, right? Well thats the idea behind this side-project anyway. The Defense Meteorological Satellite Program (DMSP) monitors meteorological, oceanographic, and solar-terrestrial physics for the United States Department of Defense and I’ve found this pretty cool high-res (8mb) image of the World at night on Wikipedia. If I can obtain some even higher resolution images like the LandSat data then I’m sure I could do some nice analysis as with the LandSat data, to predict where we should have more information in OpenStreetMap. So this is almost a plea, if you  find any more detailed data then please let me know!

Refining urban area highlighting: the previous method didn’t work around the World!

18 07 2008

So the previous method with the lovely flowchart and everything proved to be great for London but when I tried to use exactly the same method in New York it took far too much urban area away leaving us with a rather dull image when it should be very bright to indicate the NYC metropolis. Take a look at the difference between New York and London! So I essentially went back to the drawing board and thought, “okay what are the main features to a LandSat image?” answer: urban areas, rural areas and water! So that means I need to distinguish between urban and rural and then distinguish between urban and water, then somehow combine the two. Thats exactly what I did.

The method is very similar to before, except I’m now using both hue and saturation from the 742 false colour image instead of simply hue. The pretty flowchart for the process is shown below and for those you you who thought the last one was a mess wait ’til you see this one!

The method essentially works due to the different properties of hue and saturation. There is a great difference between the hue value for urban and rural areas, therefore hue is used to distinguish between rural and urban. Here is the hue component of the 742 image with a colour threshold to highlight urban areas (notice that that water is also heavily highlighted).Now we apply a colour threshold to the saturation layer and take advantage of the great difference in value for land and sea here. So now we have two layers one highlighting urban and sea and one highlighting urban and rural, we need to pick out the urban. This is done by using a clever layer mode algorithm in GIMP called ‘darken only’. This compares the pixel value in the two layers (hue and saturation) and displays the lower value pixel, in this way water and rural areas are removed from the image. Result!

Here are the resulting finished images for London and New York.

You can compare the previous method and the new one here:

Old London
Old New York
New London
New New York

By the way if anyone is really interested in this stuff and would like more guidance or some high resolution images just get in touch.

Some hypotheses that may help to measure how complete OpenStreetMap is

25 06 2008

These hypotheses below will hopefully help in measuring the completeness of OpenStreetMap. If anyone has any other ideas or comments please do let me know.

  • A complete map will have all of the roads in an area, but we cannot obtain stats on the amount of road in every area we may wish to test in OpenStreetMap. So an accurate way to predict the length of road in any area would be useful. My hypothesis is that the length of road in a given area is dependent upon population density. That is to say I expect that urban areas will have less road per person than rural areas.
  • For areas where OSM has aerial imagery I would like to compare the complexity (ie file size) of the Yahoo! jpeg and the corresponding OSM tiles. The hypothesis is that areas with very small aerial jpeg files (because they are simply one colour like the sea or vast expanses of desert) will have few if any entries on OSM, whereas areas with large file sizes (cities) will have a large density of nodes and ways in OSM and therefore large tile size. I do not have the technical knowledge to test this so any help would be great.
  • Another hypothesis is that more complete areas of OSM will have a higher level of edit activity. If no-one has ever edited an area then it may be unlikely that the map is complete there, obviously however there may just be nothing there, so this test could be used in conjunction with the Yahoo! Imagery test stated before. If we could produce some sort of heat map showing which areas are edited most frequently and monitor it over time this could certainly show us some interesting trends.
  • This is an attempt to solve the problem of missing roads. I would think it unlikely that there would a road which is completely cut off from others, or that there would be an entire settlement of roads not connected to rest of the country’s road network (as in the Madiera example shown below again). The hypothesis is that every road is connected to at least one other road of equal or higher classification. So if in OSM there are roads that are not, then maybe there are missing roads. This testing may require a lot of calculations and may not return that many missing roads. If some one can think of a way to do it simpy then I would very much like that input.