Python Standard Library – Book

Working in Python, I find myself googling for solutions to programming snags all the time. More and more often, my questions turn out to be about use of the Python Standard Library, be it hashlib for file comparison purposes or shutil for standard file system operations, or whatever. So I finally took the plunge and placed the Standard Library on my desk – the Addison-Wesley Professional/Developers Library text. I’ve said before, I learn more from reading 2 pages in a book than googling 20 pages.

This is not a beginners book on how to learn Python. This is the book you need, being the bookish type, if you use Python every day and would like to learn to finer details of how to use the standard modules effectively. This is a great book to just browse and start reading anywhere. I promise you that you will learn a lot.

 

Leave a Comment

Filed under Uncategorized

FME Desktop 2011 Class – Safe Software, Vancouver

I recently had the pleasure of escaping the Texas summer heat and traveling to Vancouver, BC, for a Safe Software training class. I had stopped at the Safe Software booth at ESRI UC for a few minutes this July, and one of their rep’s gave me a quick intro to FME Desktop. Well, one thing led to another, and so I was signed up for the class this month.

I haven’t taken a whole lot of similar classes but enough to know which ones are a failure, and which ones are worth your time. The one for FME Desktop 2011 was one of the best I’ve taken. In fact, I couldn’t help myself and had to rave to the friends at Safe about it. So let me quote from my own email:

First of all, let me share with you that I found the FME Desktop class to be top-notch ! I have taken a few similar classes or workshops in the past and this one stands out. Not only did I get the impression that Robyn really knows her stuff never mind the occasional FATAL ERROR :-)  but the format of the class (30 minutes teaching, 10 minutes hands-on), plus the quality of the manual, made for a real good package that I would recommend to anyone interested in FME. Well done ! —  That said, I think FME is fantastic. I think I mentioned to both of you that I have pulled my hair out trying to wrap my brain around ESRI’s ModelBuilder more than once. FME makes more sense to me, is a lot more intuitive to my thinking and the work flows I deal with …

So, there you have it. If you haven’t taken a look at FME and deal with lots of ETL (extraction-transformation-loading) procedures, then this may be for you. Funny thing is (and I think both instructor and manual shared this with us), they say, that 90% of all FME transformations consist of shapefile >>> shapefile conversion e.g. reprojection, adding attributes… What does that say about ArcGIS user friendliness or annoyance ? I’ve certainly tried to use ModelBuilder more than once, and every time I found that is was much easier and straight forward to just write things out in Python. With FME, I felt that – while maybe not easy to use – it’s very powerful and much more intuitive.

Leave a Comment

Filed under Uncategorized

Getting PETRA well bore paths into ArcMAP using Python

I recently had to bring in some IHS well data from Petra (or IHS Enerdeq) into ArcMap. While I endured 2 days of Introduction to Petra for Geoscientists this summer, I easy and expertise when it comes to using Petra is rather limited. I agree with many online posters who complain about how clumsy, cluttered, and counter-intuitive PETRA is. But that’s another story. I was told by a well seasoned PETRA user that PETRA does not export to SHP file format, upon which she exported a well data table with lat/long’s for me as XLS. Whether the SHP limitation is indeed the case, I have not further investigated. I simply created an ESRI feature class from the LL data. Another needed step was exporting the well bore paths for directional wells into a polyline feature class for use in ArcMap. Again, the only way to get the data out of PETRA appeared to be as a table. I chose CSV.

The table has the following headers:

UWI | NAME | MD | TVD | EWOFFSET | NSOFFSET | DIP | AZM | TVDSS | XPATH | YPATH | SURNAME | SURSTAT

and provides a row of data for each survey depth, with UWI(API) repeating for all rows belonging to the same well.

To read this into Python, I did this:

import csv

### Reference to Input CSV File
infile  = open('c:/temp/dirEXPORT.csv', "rb")
reader = csv.reader(infile)

rownum = 0
well = ""
Wells = []
points = []

### Read CSV File, Line by Line
for row in reader:

    if rownum == 0: # First line in file: read headers
        colcount = 0
        for column in row:
            print column,
            if column == "UWI":
                uwi = colcount
            if column == "XPATH":
                xpath = colcount
            if column == "YPATH":
                ypath = colcount
            if column == "MD":
                z = colcount
            colcount = colcount + 1
            rownum=+1

    else:

        print "\n"

        ### Check if row (line) is for the same well as row above
        ### if so, read coordinates from field and assign to xyz as list
        if row[uwi] == well:
            xyz = [row[xpath],row[ypath],row[z]]
            points.append(xyz)
            print row[uwi], xyz
            rownum=+1

        ### If row does not belong to the same well as the the last row,
### then append coordinate list and well name to Wells list, reset
### coordinates list (=[]), and assign the coordinates to the new well
        else:
            Wells.append([well,points])
            print "New Well"
            points = []
            well = row[uwi]
            xyz = [row[xpath],row[ypath],row[z]]
            points.append(xyz)
            print row[uwi], xyz
            rownum=+1
infile.close

Any remaining print statements allow me to watch what’s going on when the program is run. It’s my crude way of debugging code.
Once all the data is loaded into a list, I start some arcpy magic:

import arcpy
from arcpy import env
import os

env.overwriteOutput = True
env.workspace = "C:/temp"

cnt = 0

mypath = env.workspace
outputFile = r"wormtracks.shp"
template = r"c:\temp\template.shp"
fieldLength = 25

sr = arcpy.Describe(template).spatialReference

arcpy.CreateFeatureclass_management(mypath,outputFile,"POLYLINE","","","",sr)
arcpy.AddField_management(outputFile, 'API', "TEXT", "", "", fieldLength) # add API as new table field

rows = arcpy.InsertCursor(outputFile)
array_container = arcpy.Array()

for well in Wells[1:]: # Ignore first item in Wells list. Poor programming where an empty list is created.
    well_api = well[0] # First item in list is the API

    wellList = well[1] # 2nd item in list is a list of XYZ points

    for pt in wellList:
        point_object = arcpy.Point()
        point_object.X = float(pt[0])
        point_object.Y = float(pt[1])

        print pt[0]
        print pt[1]
        array_container.append(point_object) # I couldn't figure out the difference between array.append and array.add
        print point_object
        del point_object

    row = rows.newRow()
    row.shape = arcpy.Polyline(array_container) # Do not use arcpy.Multipoint which created a really messy polyline w/points ouf of sequence.
    row.API = well_api
    rows.insertRow(row)
    array_container.removeAll()

del row #unlock row
del rows #unlock table

del arcpy
del outputFile

 

2 Comments

Filed under Uncategorized

Book: Python Geospatial Development

Everywhere on the web, you find people developing new GIS tools for/in Python. But I’ve never really come across a book that ties all this together. All the standard Open Source GIS books and manuals (paper and online) have chapters about Python.

All the major packages have Python support, e.g. Mapserver (Python Mapscript) or the Quantum GIS Python Cookbook for QGIS. There a numerous other projects and packages.

So when I saw this new title: Python Geospatial Development, I got excited. It covers everything from basic GIS concepts to sources of geospatial/GIS data and imagery. It covers MySQL, PostGIS, and Spatialite, mapnik, and finally even setting up a Geodjango application.

This is probably not the kind of book to read from cover to cover if you’ve already worked with some of these projects, but it covers a wide range of available tools and is packed with online resources and links to even more tools and resources. I will try to remember to elaborate once I’ve read a little more.

.

1 Comment

Filed under Uncategorized

pmapper – a framework for Mapserver applications

After playing around with my own HTML/css templates for Mapserver, I started looking for a better solution. In fact, there are a number of frequently used frameworks (see http://maptools.org under Webtools).

I briefly looked at Dracones, because it’s written in Python. But I quickly got the sense that setup was less than straight forward. It sound like some of the adventures I had with getting any useful results out of (Geo)Django. Ka-Map appears to be pretty popular. But I didn’t like the look of their standard template, and wanted something to work out of the box.

So I turned to pmapper. It comes in a download-ready MS4W package that installs super easily – no problems at all if you’ve got a functioning version of MS4W version installed. I found it helpful to first get the sample project up and running – using the demodata provided. This works with a mapfile and some shapefiles of European countries and cities. Depending where you install (copy ‘n paste) your pmapper directory, you might have to adjust some of the Apache directives in httpd.conf. But if you’ve been through this for the MS4W installation, you won’t have any troubles.

Once you start using your own data, it might be easiest to work with the existing directory structure and slowly start changing/recycling the demodata mapfile. Anyway, now I have a great looking template with a somewhat dynamic legend, pan and continuous zoom, and more. The categories & groups references in config/config_default.xml confused me initially because groups actually refers to GROUP names in the MAPFILE. But with some trial and error, you get all your layers displayed the way you want.

 There  are still a few bugs, though, to be ironed out.

  1. The feature search still refuses to work properly.
  2. I think I may have made a mistake with my reference map, because the box highlighting my zoom area only adjusts in the north-south directions, but nevre east-west, and coincidentally, my panning now only works north-south but not east-west.

Since pmapper uses PHP and Javascript neither of which I understand well, the challenge now lies in adding some Python Mapscript capabilities so that I can extend the framework myself. I’m thinking about a little drop down input box where you select SHP data that’s added dynamically… I will get to that when I have time.

Leave a Comment

Filed under Uncategorized