Map Maker but not Google’s

23 07 2008

While searching for urban area datasets I stumbled upon nationalatlas.gov which provides a whole host of information for the US. Examples include boundaries for counties, states, urban areas along with roads, railways and even time series data such as crime and population levels. All of this data can be viewed in your web browser in their easy to use interface Map Maker but much better than this it can all be downloaded as shapefiles and imported into Qgis or other GIS programs. Perhaps some of this data can even be uploaded to OpenStreetMap.





Digital Chart of the World and Vmap

22 07 2008

Unfortunately the urban areas classified in the UMD land cover classification dataset I referred to yesterday are not classified using the satellite imagery as I believed before, the data simply come from another data source the Digital Chart of the World (DCW). Today has seen some further research of DCW (renamed Vmap0) and I have found out that some of Vmap0 has already been used for import into OSM, in particular the urban areas that I was trying to pick out and test. I am still trying to find a decent data source for Vmap0, the only ones I can find at the moment require me to use ArcView, I want to use Qgis! If anyone knows of a good way to download DCW/Vmap data, please let me know.

Just because the UMD land cover data for urban areas was derived from another source, that doesn’t mean that the other data in the project isn’t useful. Their process obviously clearly identifies different types of forests, grasslands and bare land and it will be interesting to understand how it does this. The process is in fact rather complex, as can be seen by the published paper. They analysed data over a whole year and calculated Normalized Difference Vegetation Index (NDVI) figures. These compare the spectral reflectance in the red and near-infrared bands and essentially detect areas where there is vegetation, (I wonder if its possible to do something like this for urban!). Anyway the UMD data uses in total 41 metrics using 5 channels on radiation and the NDVI figures and then uses a flowchart to breakdown which land use there is. This could help us classify large rural areas in the World in OSM if we think that would be useful but it is not going to help with my model for urban areas.

The model I envisage will not simply classify areas but will hopefully be able to predict the length of road, or the amount of OSM data we can expect in an area. The processes discussed earlier can accurately distinguish urban areas from everything else, the question is whether there is a correlation between the brightness of a pixel and the amount of ‘stuff’ in that area. If there is, then my models will be a lot more useful than the UMD datasets I discovered yesterday.





Thank you University of Maryland: AVHRR Land Cover Data

21 07 2008

Turns out that The University of Maryland Department of Geography has already done a land cover classification project, which highlights urban areas very well! You’d think that would make me feel quite bad, after spending much of last week trying to do this BUT unfortunately their project used AVHRR (Advanced Very High Resolution Radiometer) data which has a larger resolution than LandSat and the data is older. But there is still an awful lot of analysis I can do with 20 year old 1km resolution data, if nothing else I can try to use some of their methods with the more up to date LandSat data I was using before.

So this is how to download, import into Qgis and use the AVHRR Land Cover Data

Firstly go to http://glcf.umiacs.umd.edu/data/landcover/ for some info on the whole project, click on the Download via Search and Preview Tool (ESDI) if you want to browse the World and select the data you want OR head straight to the FTP server. Choose your area, projection and resolution (I’m using North America, LatLong and 1km), eventually you will be presented with a bunch of files from which we download the file ending asc.gz as Qgis can handle this not the bsq.gz. Extract the file after downloading and boot up Qgis. Now hit ‘Add a Raster Layer’ and select the .asc file you downloaded and extracted, the file will be pretty big (NA is 1.39 GB) so expect a wait. Now you will have a greyscale image of your region, the brightness depends on the land cover classification calculated by UMD. Here are the code values which can be found on their website.

Code Values for 1km and 8km data

Value Label RGB Red RGB Green RGB Blue
0 Water 068 079 137
1 Evergreen Needleleaf Forest 001 100 000
2 Evergreen Broadleaf Forest 001 130 000
3 Deciduous Needleleaf Forest 151 191 071
4 Deciduous Broadleaf Forest 002 220 000
5 Mixed Forest 000 255 000
6 Woodland 146 174 047
7 Wooded Grassland 220 206 000
8 Closed Shrubland 255 173 000
9 Open Shrubland 255 251 195
10 Grassland 140 072 009
11 Cropland 247 165 255
12 Bare Ground 255 199 174
13 Urban and Built 000 255 255

To make the map in Qgis look at all interesting we need to apply a colormap algorithm. You can select your own by right clicking the file and selecting properties and changing the settings or go to properties select ‘Load Style’ and load the style I created which can be downloaded here (right click and ‘Save target as’). Now you will have a wonderful map showing land use in your region, the area highlighted in white is what I am particuarly interested in and worked on for much of last week. The next step is to see if I can work out how to generate the code values easily and apply it to LandSat data, then compare to OSM.

Here are some image Qgis outputs for North America and San Francisco.





Another thing to look at while I’m at it: DMSP data

18 07 2008

Urban areas are likely to emit more artificial light, right? Well thats the idea behind this side-project anyway. The Defense Meteorological Satellite Program (DMSP) monitors meteorological, oceanographic, and solar-terrestrial physics for the United States Department of Defense and I’ve found this pretty cool high-res (8mb) image of the World at night on Wikipedia. If I can obtain some even higher resolution images like the LandSat data then I’m sure I could do some nice analysis as with the LandSat data, to predict where we should have more information in OpenStreetMap. So this is almost a plea, if you  find any more detailed data then please let me know!





Refining urban area highlighting: the previous method didn’t work around the World!

18 07 2008

So the previous method with the lovely flowchart and everything proved to be great for London but when I tried to use exactly the same method in New York it took far too much urban area away leaving us with a rather dull image when it should be very bright to indicate the NYC metropolis. Take a look at the difference between New York and London! So I essentially went back to the drawing board and thought, “okay what are the main features to a LandSat image?” answer: urban areas, rural areas and water! So that means I need to distinguish between urban and rural and then distinguish between urban and water, then somehow combine the two. Thats exactly what I did.

The method is very similar to before, except I’m now using both hue and saturation from the 742 false colour image instead of simply hue. The pretty flowchart for the process is shown below and for those you you who thought the last one was a mess wait ’til you see this one!

The method essentially works due to the different properties of hue and saturation. There is a great difference between the hue value for urban and rural areas, therefore hue is used to distinguish between rural and urban. Here is the hue component of the 742 image with a colour threshold to highlight urban areas (notice that that water is also heavily highlighted).Now we apply a colour threshold to the saturation layer and take advantage of the great difference in value for land and sea here. So now we have two layers one highlighting urban and sea and one highlighting urban and rural, we need to pick out the urban. This is done by using a clever layer mode algorithm in GIMP called ‘darken only’. This compares the pixel value in the two layers (hue and saturation) and displays the lower value pixel, in this way water and rural areas are removed from the image. Result!

Here are the resulting finished images for London and New York.

You can compare the previous method and the new one here:

Old London
Old New York
New London
New New York

By the way if anyone is really interested in this stuff and would like more guidance or some high resolution images just get in touch.





Highlighting urban areas in LandSat

17 07 2008

The hypothesis I’m working on at the moment is that there should be more OpenStreetMap data (i.e. more nodes, more ways etc) in urban areas. To find out where these urban areas are I’m using freely available LandSat 7 data downloaded from http://www.landcover.org/data/landsat/. The data comes in the form of 8 different greyscale images corresponding to 8 different spectral bands ranging from visible blue light (0.45-0.52 µm) to thermal IR (10.40-12.5 µm).  Using bands 1,2,3 corresponding to blue, green and red a ‘true’ colour image of an area can be built, however this is not the best combination to use to highlight urban areas it turns out that the best combination is 7,4,2 for red, green and blue. Head over to my page on using LandSat imagery in GIMP to find a tutorial on all this.

What I’ve been working on over the past few days is how to use the false colour image I’ve produced using bands 7,4 and 2 to highlight the urban areas.Below is the 742 false colour image, and we can see that urban areas appear quite brown and purple, the aim is to extract that information and make everything else invisible. The problem is that some of the sea to the East is a similar sort of colour to the cities.

I decided the best way to extract the urban areas from  the image above would be to decompose the image into hue, saturation and value. The hue is the most interesting component, giving a good contrast between rural and urban areas, but as we can see there is little difference in colour still between some sea areas and some cities which could prove a problem in extracting just urban areas.

The next step is to apply a colour threshold on this image to try to pick out only urban areas and black out everything else. After a great deal of playing around with filters the urban threshold here appears to be within light levels 196-212, after applying this filter the image below is obtained.

As we can see the urban areas are nicely highlighted in white and everything else is black. Now the aim is to compare the brightness of every pixel in this image with OSM data. We know the coordinates of the image above and we kno that each pixel is (30m x 30m) so this should be easy enough using a simple bounding box query of the OSM database. Lets hope that there is a relationship now between OSM data and the brightness of the pixels above. I’ll be back with any results when I have them and then hopefully be rolling this out across the world.

Click on the image below for a nice flowchart of the whole process.





Have we hit the gem that will improve OpenStreetMap’s completeness

15 07 2008

OpenStreetBugs (http://openstreetbugs.appspot.com) is a wonderful application created by Xavier in Rennes, France which allows us to tag areas in OpenStreetMap which are incorrect, incoherent or incomplete. Its an idea I’ve been talking about since my project began, little did I know that this little gem was around, created about a month ago as can be seen from Xavier’s blog. If we all start to use this app I’m sure it will prove invaluable in our quest to make OSM complete.

Users can annotate OSM in pretty much anyway they want to, others can then come along and discuss the notes. Or even open Potlatch centered on a node or download a GPX dump of the visible nodes to load in JOSM (or in your GPS). With recent updates you can even obtain an RSS geo-feed for an area that you choose and monitor changes.

If OSB does become popular then it can become the one-stop shop for inaccuracies in OSM. People will be able to check how many flags there are in a particular areas and from that get an idea of how complete the map is in that area. It will also act as a guide for avid mappers who don’t know where to start next, just pick the area with lots of incomplete tags. Moreover we can encourage people to participate in the OpenStreetMap project just by visiting OSB and tagging everything they know is incomplete. That way people who feel a little bit frightened by the whole concept of going out mapping can still help improve the map. Perfect!








Follow

Get every new post delivered to your Inbox.