Cultivating Spatial Intelligence
Organizing Place-Based Data
Organizing Place-Based Data
In the study of a place we bring together infromation from many sources. Under our care, these diverse data come together to form a coherent collection that serves our purposes, can be backed up and moved from one computer to another, and shared with our collaborators or successors. Without a data management strategy our collections will be messy agglomerations of files that become increasingly burdensome and impossible to reorganize as a project progresses. This will lead to lost productivity in the course of a project and little of value when our piece of the project is finished. A shared data management strategy is especially important when there are multiple people using and contributing to a shared collection. For a general view of information management strategies in a research enterprise, see our page on Cultivating Design Information. The tutorial provided below provides a handbook of recommended procedures for compiling and organizing GIS and CAD data in an ongoing research project.
Click here to download the sample dataset. Follow the instructions on that page for installing the latest boston_metro_sources compilation.
After studying this tutorial, you should understand how build a collection of data and styling information from many sources and put it together with map documents that you can easily exchange with other people. Understanding how to do this involves knowing how to collect data and metadata, how to clip large image and feature datasets, how to capture layer styling information, and how to save maps so that their references to data remain intact when the collection is moved to different file storage locations. This tutorial takes off where the Tour of GIS Data and Metadata leaves off. Having explored this dataset, and understood the various forms of data, metadata and layer styling, will be ready to examine the structure of the collection and work through an exercise of extending it by adding some new data and maps of our own.
Overall Flow of this Tutorial
- Discuss the principles and structure of the data collection.
- Explore two modes of GIS Data assimilation: Downloads from the web, extracting subsets and layer styling from existing feature and raster datasets
- Save map documents using with relative pathnames to local data sources.
- Learn to check file references
- Learn to back up and restore your project folder.
Explore a Compilation of Data and Maps
Our objective over the next few weeks is to gather data together into a cohesive compilation that will serve as a base of resources that we will build on over the course of the semester. Our compilation will involve data from various sources and map documents that reference these data, and later we will add models to this collection as well. Because maps and models reference the data, it is best if the organization remains stable, since movimng data will cause the path-references in out maps and models to break. This structure will also help us to keep our data compilation backed up.
The folder greenline_project in our sample dataset is such a collection. You can open this folder in Windows Explorer and see that there is a folder structure that conforms to the structure as explained below:
This structure provides a predictable means of understanding where each piece of data should go when it is collected, and where it may be found later. It also allows the project collection to grow in terms of the number of collaborators and derived datasets without becoming unmanageable. Before the project compilation is shared, user data considered to be intermediate can be cleaned up. User documents that are to be shared with the project may be moved to the Project Documents folder.
Tips for Reliable Filesystems
- Keep your working files on the local hard drive. Working with data on network filesystems or usb drives is not only slower, but also is subject to all sorts of unpredictable behavior which we don't need complicating our lives.
- Don't work in your Backup Copy The tips in this tutorial will help you build a file structure that you can move to the local filesystem before starting to work, and back-up to the network or your USB drive when you are ready to take a break.
- Save backup copies of your map documents and your entire project as versions. Change the name of your zip file to reflect the date that you saved it. This way you can always go back to an earlier version of you find you have made a change that you regret!
- Don't work in folders that have spaces in their names. this in cludes the Desktop or My Documents folder.
- Never begin the name of a file folder or dataset with a numeral. its not clear why this is, but trust me, it can bring you very bad luck! It has something to do with the assumptions that are made by programming languages.
Metadata: Project, Folder and Node
Beginning a Data Compilation with a Vector Layer from the Web
So now that we see what a compilation of data looks like, lets pretend we are beginning our own from scratch. We will start by building a folder structure to hold our documents and our data. and then we will duiscover a few resources on the web and bring them onto our hard drive.
Getting a Roads Layer from the State GIS Agency
- Start by creating a project folder near the top of your filesystem C:\work.
- Within this folder make a subfolder for your project. You can call it union_square.
- Go to the website of Massachusetts GIS After visiting their Free Data Download page we find the descriptive information (metadata) for the Mass DOT Roads.
- Scan this metadata helps us to understand all of the critical things about the data. Note that the primary source of these data is The Massachusetts Department of Transportation -- NOT MassGIS!. The dataset is current as of 2008. All of this will be important to have for reference later.
- Click the link at the top of the page that brings us to the Download These Layers page. From here, we will scroll through the list and download the Somerville Roads layer (eotroads_274.exe).
- Download the EXE file, which is in this casde, a self-extracting aip archive containing a shapefile dataset.
- As you download this, we should create a new folder in our project to hold data from MassGIS and deposit the file there.
- Double-click the eotroads_274.exe file to extract its contents into your new massgis folder, and admire the contents.
- a shapefile is actually made of several files.
- Start ArcMap and add your new shapefile as a layer to your map.
- Examine the Attribute Table/ What do all of these attributes mean? Oh, yes... We need to go het the metadata.
- Go back to the Mass DOT Roads page and use the File->Save As dialog to save the web page into your massGIS folder as an HTML page. Most browsers let you save a "complete html page" which will also save the images that are embedded in the page.
- Now take a breath. YOu have now just captured your first GIS data set, with its metadata. And now whatever happens to MassGIS, you know that your data is safe!
Working with ArcGIS Layer Styling: Map Layers and .LYR Files
So you now have a shapefile dataset representing Somerville Roads. But it looks rather bland. When arcmap opens the dataset it creates some random symbolization for the features. We want to see if we can find the styling linformation (layer files) and, indeed we can find it referenced at the top of Download These Layers page. YOu won;t always be so lucky to find layer files with data that you download. But MassGIS is good.
Acquire and Integrate Styling Information
- Download the link labeled eotlegends.exe.
- Extract it into your massGIS folder. Lots of LYR files!!!
- Add these layer files in ArcMap.
- Oh No! Red exclamation marks. As you recall from our previous tutorial, these indicate broken references from the layer to the data.
- Check the layer properties (Source Tab) to see what the name and location is being referenced.
- Now click the Set Data Source button on this properties page.
- YOur new layer has lots of nice properties for Symbolization and Labeling which are quite elaborate. We will go into how these are created in a few weeks. For now just feel good that you were able to obtain the official styimng information for this dataset!
Extracting a Subset of Features from a Large Feature Class
A great starting place for any data compilation would be grabbing some of the data that comes with ArcGIS. At the GSD, this data collection is stored on-line in \\goliath\public\geo\esridata_9. These data came on 4 DVDs and cover the whole world. We want to keep our data collection focused on our area of interest and its greater context. If we aren't careful, we will end up with gigabytes of data that aren't of any use to us and our collaborators. So we will clip and export a subset of these features to our compilation folder.
- Displaying layers at certain scales
- Exporting Features
- Referencing Data in Maps
- The ArcMap Table of Contents
- Use Windows to go to \\goliath\geo\esridata_9. If your computer does not have a drive latter mapped to public on 'goliath' then you will need to follow these instructions for Mapping a Network Drive
- Look in the Streetmap USA folder. See that there is an ArcMap Layer file in there, named Streetmap USA.lyr
- When you try to find this layer with the Add Data button in ArcMap, you can't find the Goliath folder at all!
- Hover over the buttons in the Add Data dialog until you find the one that says Connect to Folder. And use this to add the folder Goliath/geo to your Arcmap catalog.
- Now you will be able to connect to and explore the GSD's collection of data.
- Use the Add Data button to add the layer named StreetmapUSA.lyr
- Note that the Streetmap USA layer is a Group Layer that contains a bunch of sub-layers, each referencing a dataset.
- note that as you zoom in and out some layers appear and dissapear. This is due to the visibility-scale property that you can examine and change from the General properties of each layer.
- Lets say we want to extract the major highways from the Streetmap Dataset to our compilation. We don;t want to extract all the streets in the world, so we will zoom in to the boston area.
- If necessary, we will right-click the Major Highways layer and click Visible Scale Range > Clear Scale Range so that we can see it.
- Now you can right click the Highways layer and use Data > Export Data to export just the data within current view extent into your project/streetmap folder. You may need to create this folder. When this finishes, it will ask you if you want to add this layer to your display, click Yes.
- Now you see you have managed to copy the features you need, but you have not got the nice symbology and labels. We would like to somehow copy the highways layer setting s frm streetmap and apply these to our clipped highways dataset.
- Right click on the highways layer and choose Copy then go to the data frame -- the thing named "Layers" at the top of your table of contents, and click Paste.
- Check the source properties of this new layer. Note that it still references daat on the Goliath folder.
- Click Set Data Source to link this layer to your clipped highways shape file.
- Right-click the new layer and Save Layer As to your compilation/streetmap folder.
- Repeat this procedure for each layer/dataset that you want to capture. When you are finished, you will have a bunch of layers in your map that have been dragged out of the streetmapUSA group layer. You can now delete the net-based group layer, and use the Source tab on your table of contents to see that all of the remaining layers are referencing local data.
- You may want to create your own streetmap group layer and put your captured layers into it.
- A final check would be to select all of your streetmap layers, right click and Zoom to Layers and make sure that none of them are un-clipped. The idea is to make a compact collection.
A note about Metadata for Data from the ESRI Data folder
Many of the layers from the ESRI Data folder have metadata encoded according to the Federal Geoographic Data Committee specification. These metadata files travel with the data -- even if we clip it!!! To view this data, you can right-click your highways layer and choose Data > View Metadata. Just for fun, you can find the actual metadata file itself as file with an .xml suffix associated with your clipped streetmap file.
Add an Image Service Layer and Save a Cropped Piece to Your Compilation
There are many ways to get areial photos. For example, you can find bunches of them at the MassGIS Web Site. Later on we will look at how to get aerial imagery from the US Geological Survey or from Google Earth Pro. Another way to access imagery can be through a web map service, like ArcMap Online. Using the on-line image service is very convenient, but, we do not want our project to be dependent on internet sites. We are building a laboratory that we want to be stable and predicatble. The next few steps will show you how to clip out and download screenfuls of imagery and save them at high resolution to your data collection.
Exporting a High Resolution Image of your Map Window
- Click File->Add Data from ArcGIS On-Line.
- When the Web page comes up, click the icon for "Imagery." This should begin a download that automatically inserts a new layer into ArcMap.
- Zoom way in to a place in Somerville. How does the image align with the roads from MassDOT? Pretty good!
- Zoom in until you can see what the smallest things are recognizeable in the image. You can measure something with the measure tool to get an idea of the limit to the resolution of the image. Althought we can;t zoom in so far as to see a pixel, it looks like features smaller than half a meter are not distinquishable.
- Now zoom out to an area of approximately 1 square kilometer. You can measure the extent of the screen with the measure tool
- Now choose File->Export Map. Set the Files of Type dialog to export a JPEG file.
- Click the options tag, and adjust the DPI until the pixel dimension is something like 2000 pixels. Since our map extent is about 1000 meters, this resolution would make the pixels of our saved image about one half meter apiece.
- Make sure that the checkbox for Write World File is checked. This world file will accompany our new image and will instruct arcmap how to shift, rotate and scale the image to land in the right place.
- Finally, click the Save button to save the image. You should follow our convention of making a new folder to save the imagery.
- as for metadata, it is not very good for the ArcMap imagery layer. The best description we can find for this layer is in the Properties for the World Imagery sub-layer. You can cut and paste this text into a file to save with your image. While you have the text file open, you may want to include the date that you downloaded the image. At least 20 years form now, you will know that the imagery can;t be any newer than that!
Absolute and Relative Path Names
You are now almost ready to go. You have created your compilation folder, you have gatherd some data from the web and organized it within your compilation. We have also captured the portrayal information that controls the grpajhical display of these data. And we have a compilation map document that references all of this. So we are almost ready to copy this compilation to our backup media and share it with our colleagues. There is just one more problem to solve!
If we copy our folder to another computer, it may not be in exactly the same place. That is, our collaborator may want to copy it to a folder other than c:\temp\yourusername. And if this happens, the file references between the map document and the data will be broken and instead of seeing nice portrayals of data, they will see the Dreaded Red Exclamation Marks!. Your final step before copying your work to your portable media is to set your map's file references to Relative Path Names, and then to check the source path forall of the files referenced on your map.
- Referencing Data on the Map
- The ArcMap Table of Contents
- Pathnames explained: Absolute, relative, UNC, and URL
About Data Source References
Moving Map Documents
If you want to move a map document to another folder, you should first open it in arcmap and make sure that it is working. Then use File->Save As to move it.-->
Saving your Map Document with all of its Portrayals to your Compilation
You have seen how layer files save portrayal information. It is also possible to save the whole map document with all of its layers to the docs folder of our compilation. To make sure that the compilation will work on another copmputer, we need to make sure that the map documents in our compilation folder do not reference datasets that aren't part of our compilation. For this, we will use the Source View of our ArcMap table of contents to view datasets according to what folders they reside in. Using this view we can eliminate any of the layers that are not in our compilation's gis folder. Then we set the Data Source Properties in ArcMap so that the document uses Relative Path Names to reference its data sources. Then we save the map document into the docs folder of our data compilation.
Always Check your References!!!
- Before closing your map, click the Source tab at the bottom of your table of contents to view your layers according to their file system location..
- Look for data that are referenced from file system locations that aren't within your data collection folder and delete them. If you need to move datasets, see below for advice on using ArcCatalog.
- Delete any data sources in your map that aren't in your portable folder and move datasets as necessary.
- Set File -> Document Properties -> Data Source Options to: Save Relative Path References. While you are here, make this the default.
- When saving your map, create a new folder named Map_Documents beside all of your source files in your project folder.
Always Check your Data Source Options and Paths
One of the mosrt common mistakes made by beginning ArcMap users is to give people maps that have all of their data references broken. This, of course is annoying to everyone concerned. One problem is that Absolute Paths is the default, and each computer you go to may have this setting, so you need to check it. Another thing is that your map may be referencing data that is not in the project folder. The only way to figure this out is to use the Source tab on the table of contents as described above. Finally, if you are burning your map to a CD it doesn't hurt to actually try to open your map document frm the CD.
Moving a Map Document
Now that you understand the issue of relative path references in map documents you should understand why it is a bad idea to move copies of map documents. If you move a map document to another folder it is very likely the references to all of the datasets will become invalid. Therefore if you want to move a map file, the best way to do this is to open it in ArcMap and use File -> Save As to save it to its new location.
Use ArcCatalog to Simplify the Task of Managing GIS Data
We have seen in our forrays into our data folders that GIS data files are complexes of files. A Shape File dataset is actually composed of 4 or 5 different files that all share the same file prefix. This presents problems for moving and copying files. The application ArcCatalog is a special file browser that makes this process simpler.
- Use ArcCatalog to explore the data (for example within the esri_streetmap folder.) Notice how it looks simpler.
- Use arcCatalog to look at the metadata for your streetmap/major_highways layer Since this dataset has metadata that complies with the Federal Geographic data comittee standard for geospatial metadata, ArcCatalog be used to view and manage it. If the layer is copied, ArcCatalog will also copy the metadata!
- Use arcCatalog to look in your massgis folder. Notice, that these datasets do not have
FGDC complient metadata. But if you look in the folder with Windows explorer you will see that
there are HTML documents in there that represent the MassGIS metadata. MassGIS downloads these files when we download the data, but we need to take care that the meartadata stays with the data when we move it around.