Friday, July 29, 2016

GIS Programming with Python and QGIS - Part 1

Introduction

I will start by introducing the following key words: Programming, Python and QGIS

Programming:
There is high tendency that you as a geospatial professional/graduate/student, you already know what "programming" is all about?
Well, if you don't, here is a brief explanation according to WikiPedia: Computer programming (often shortened to programming) is a process that leads from an original formulation of a computing problem to executable computer programs. The purpose of programming is to find a sequence of instructions that will automate performing a specific task or solving a given problem.

Computer Programming is a much need skill for advanced GIS experts. These days, knowledge and competence in programming is a essential skill set and a critical requirement for most Geospatial job opportunities.  A vast majority of job openings in the Geospatial industry require some expertise in programming, application development and software management.

Some awesome articles I have come across on the web that talked about GIS Programming are listed below (I strongly reccomend you read them);-


Python:
Python is a widely used high-level, general-purpose, interpreted, dynamic programming language. Its design philosophy emphasizes code readability, and its syntax allows programmers to express concepts in fewer lines of code than possible in languages such as C++ or Java (Source WikiPedia).

Of all programming languages, many consider python programming language to be the front-runner in the Geospatial industry. Python has been a standard language in GIS because ESRI (ArcGIS) and open source (QGIS) tend to gravitate toward it. Python is very popular with Geospatial professionals working in the ESRI environment and is considered as the primary scripting language for ArcGIS.

Python provides many opportunities for integration within GIS computing systems. Cross-platform capabilities and ease of integration with other languages (C, C++, FORTRAN, and Java) - (CPython, IronPython, Jython, PyPy) mean that Python is most successful in gluing systems together. Because of the fluid language design, the development of large-scale applications is also easily supported. Many libraries and tools have already been developed for working with GIS data in Python. The basics are covered, including the manipulation of shapefiles, grids, and images, as well as more sophisticated stuff such as scripting pyQGIS API and interaction with Web services and databases.

Python can provide you with a complete set of tools for your GIS toolbox. In combination with QGIS, the possibilities are endless. New technologies, such as Web Map Service (WMS), Web Feature Service (WFS), and Web Coverage Service (WCS) are widely supported in Python. There are many online and paper resources to help you when developing Python scripts.

Other programming languages that are being used in the Geospatial industry includes C, C++, C#, Visual Basic.NET, Flex, Java, ActionScript, JavaScript,PHP, VBA for ArcObjects, etc.


QGIS:
QGIS (previously known as Quantum GIS) is a cross-platform free and open-source desktop geographic information system (GIS) application that provides data viewing, editing, and analysis (Source WikiPedia).

Similar to ArcGIS software, QGIS allows users to create maps with many layers using different map projections. Maps can be assembled in different formats and for different uses. QGIS allows maps to be composed of raster or vector layers.

QGIS integrates with other open-source GIS packages, including PostGIS, GRASS, and MapServer to give users extensive functionality. Plugins written in Python or C++ extend QGIS's capabilities. 


Overview of the QGIS Python Console

The QGIS Python Console is the environment where you execute python codes. This works without installing any additional python interpreter since QGIS installation comes with its own python interpreter.

So, by mere installation of QGIS, you are ready to execute python script as am going to demonstrate below.

To open the Python Console in QGIS, do any of the following:-
1) Go to the menu "Plugins >> Python Console"
2) Click on the "Python Console" icon on the tools bar
3) Press "Ctrl + Alt + P" on your keyboard


The Python Console should now appear at the bottom of the map view screen as seen above. The Python Console window can be moved and docked at any position on the screen.

On the QGIS Python Console you will find the following;-
~ the title bar: Minimize and close buttons are here. You can use the title bar to move the console to any position on the screen.
~ the toolbar: icons to the following commands are displayed (Clear console, Import Class, Run Command, Show Editor, Settings, and Help).
I will talk more about the toolbar soon.
~ the display window: the results of expressions are displayed here
~ the interactive prompt (>>>): where to type python expressions and press enter button to execute


From to tools on the tool bar, you can Clear console, Import Class, Run Command, Show Editor, Settings, and Help. Every tool on the tools bar is self explanatory. However, am going to discuss more on Show Editor and Settings.



Settings: This allows you to adjust the console and editor properties such as font style, font size, font color, autocompletion etc.
Lets increase the console and editor font's size. Click on the "Settings" icon on the tools bar, it will open a dialog box where you can specify the settings accordingly. Click on "OK" to apply the changes.


Show Editor: This will open up the editor panel. Here you can create, open, edit and save python modules (files) just as you can do in any text editor.
Lets open the "Editor" panel by clicking on "Show Editor" icon. It will show on the left side of the console as seen below, here you can create, open, edit, save and execute a .py file by making use of
the icons on the editor's tool bar. To close the editor window, click back on the "Show Editor" button to toggle the display back to hide.



At this point you need to be familiar with python scripting to use both the Console and Editor to execute python code. There are many resources on learn python on the internet. Check this resource for over 70+ materials for learning python programming.

Some other excellent resources for learning python using videos include: YouTube.com, Udemy.com, Coursera.com, Udacity.com etc.

If you are conversant with basics of python programming then you can move on with using the QGIS python console.


Executing Python script on QGIS python Console and Editor

To see the version of python installed with your QGIS software: type "import sys" and press enter/run button then type "sys.version" and press enter/run button again.
Anytime you typed an expression on the console, you most press enter/run button to execute it.


My QGIS python version is "2.7.5" as seen above. If we are to  use the editor, we will type and save the expressions before executing.

Now since you know the version of python installed with your QGIS, lets demonstrate the traditional "hello world program" on both the Console and Editor.

On the Console: type in " print 'Hello World, from QGIS Console' " and press enter/run button.

On the Editor: type in " print 'Hello World, from QGIS Editor' " then save it as (hello.py) in any folder of your choice and press "run script" button.

The advantage of using the editor is that you have your script file saved locally that you can refer to anytime. Unlike on the console, whenever you close the program, everything you typed disappears.


Getting Help with QGIS Python Console

You can use the help button to open the help window. Also by using the "help()" function you can get python specific help. For example, lets see what modules that are installed on our QGIS python installation.



Type " help('modules') ", this will display all the modules that are currently installed on the python that came with QGIS installation. My installed modules  are as listed below;-



Column 1 Column 2 Column 3 Column 4
ArgImagePlugin _symtable hotshot smtpd
BaseHTTPServer _testcapi htmlentitydefs smtplib
Bastion _threading_local htmllib sndhdr
BdfFontFile _warnings httplib socket
BmpImagePlugin _weakref httplib2 sqlite3
BufrStubImagePlugin _weakrefset ihooks sre
CGIHTTPServer _win32sysloader imageop sre_compile
ConfigParser _winreg imaplib sre_constants
ContainerIO _winxptheme imghdr sre_parse
Cookie abc imp ssl
CurImagePlugin adodbapi importlib sspi
DcxImagePlugin afxres imputil sspicon
DocXMLRPCServer aifc inspect stat
EpsImagePlugin algs io statvfs
ExifTags antigravity isapi string
FitsStubImagePlugin anydbm itertools stringold
FliImagePlugin argparse jinja2 stringprep
FontFile array json strop
FpxImagePlugin ast keyword struct
GbrImagePlugin asynchat lib2to3 subprocess
GdImageFile asyncore linecache sunau
GdalTools atexit lizmap sunaudio
GifImagePlugin audiodev locale symbol
GimpGradientFile audioop logging symtable
GimpPaletteFile backports macpath sys
GribStubImagePlugin base64 macurl2path sysconfig
HTMLParser bdb mailbox tablemanager
Hdf5StubImagePlugin binascii mailcap tabnanny
IcnsImagePlugin binhex markupbase tarfile
IcoImagePlugin bisect markupsafe telnetlib
ImImagePlugin bsddb marshal tempfile
Image bz2 math tests
ImageChops cPickle matplotlib textwrap
ImageCms cProfile md5 this
ImageColor cStringIO mhlib thread
ImageDraw calendar mimetools threading
ImageDraw2 certifi mimetypes time
ImageEnhance cgi mimify timeit
ImageFile cgitb mkgraticule timer
ImageFileIO chunk mmap toaiff
ImageFilter cmath mmapfile token
ImageFont cmd mmsystem tokenize
ImageGL code modeler tools
ImageGrab codecs modulefinder tornado
ImageMath codeop mpl_toolkits trace
ImageMode collections msilib traceback
ImageOps colorsys msvcrt tty
ImagePalette commands multifile types
ImagePath commctrl multiprocessing ui_frmEliminate
ImageQt compileall mutex ui_frmGeometry
ImageSequence compiler netbios ui_frmGeoprocessing
ImageShow console netrc ui_frmIntersectLines
ImageStat contextlib networkx ui_frmMeanCoords
ImageTk contour new ui_frmMergeShapes
ImageTransform cookielib nntplib ui_frmPointDistance
ImageWin copy nose ui_frmPointsInPolygon
ImtImagePlugin copy_reg nt ui_frmRandPoints
IptcImagePlugin core ntpath ui_frmRandom
JpegImagePlugin csv ntsecuritycon ui_frmReProject
McIdasImagePlugin ctypes nturl2path ui_frmRegPoints
MetaSearch curses numbers ui_frmSelectByLocation
MicImagePlugin datetime numericalDigitize ui_frmSimplify
MimeWriter dateutil numericalVertexEdit ui_frmSpatialIndex
MpegImagePlugin db_manager numpy ui_frmSpatialJoin
MspImagePlugin dbhash odbc ui_frmSubsetSelect
OleFileIO dbi ogr ui_frmSumLines
OpenGL dde opcode ui_frmVectorGrid
PIL decimal openlayers_plugin ui_frmVectorSplit
PSDraw difflib operator ui_frmVisual
PaletteFile dircache optparse unicodedata
PalmImagePlugin dis os unittest
PcdImagePlugin distutils os2emxpath urllib
PcfFontFile doDefineProj osgeo urllib2
PcxImagePlugin doEliminate osr urlparse
PdfImagePlugin doGeometry owslib user
PixarImagePlugin doGeoprocessing parser uu
PngImagePlugin doIntersectLines pct2rgb uuid
PpmImagePlugin doMeanCoords pdb voronoi
ProcessingPlugin doMergeShapes perfmon warnings
PsdImagePlugin doPointDistance pickle wave
PyQt4 doPointsInPolygon pickletools weakref
Qgis2threejs doRandPoints pipes webbrowser
Queue doRandom pkgutil whichdb
SgiImagePlugin doRegPoints platform win2kras
SimpleHTTPServer doSelectByLocation plistlib win32api
SimpleXMLRPCServer doSimplify pluginbuilder win32clipboard
SocketServer doSpatialIndex popen2 win32com
SpiderImagePlugin doSpatialJoin poplib win32con
StringIO doSubsetSelect posixfile win32console
SunImagePlugin doSumLines posixpath win32cred
TarIO doValidate pprint win32crypt
TgaImagePlugin doVectorGrid processing win32cryptcon
TiffImagePlugin doVectorSplit profile win32event
TiffTags doVisual pstats win32evtlog
UserDict doctest psycopg2 win32evtlogutil
UserList dumbdbm pty win32file
UserString dummy_thread py_compile win32gui
WalImageFile dummy_threading pyclbr win32gui_struct
WmfImagePlugin email pydoc win32help
XVThumbImagePlugin encodings pydoc_data win32inet
XbmImagePlugin epsg_tr pyexpat win32inetcon
XpmImagePlugin errno pygments win32job
_LWPCookieJar esri2wkt pylab win32lz
_MozillaCookieJar exceptions pyodbc win32net
__builtin__ expressions pyparsing win32netcon
__future__ fTools pyplugin_installer win32pdh
_abcoll filecmp pyspatialite win32pdhquery
_ast fileinput pythoncom win32pdhutil
_bisect fnmatch pytz win32pipe
_bsddb formatter pywin win32print
_codecs fpformat pywin32_testutil win32process
_codecs_cn fractions pywintypes win32profile
_codecs_hk ftools_utils qgis win32ras
_codecs_iso2022 ftplib qgis2leaf win32rcparser
_codecs_jp functools qgis2web win32security
_codecs_kr future_builtins qgiscloud win32service
_codecs_tw gc quopri win32serviceutil
_collections gcps2vec random win32timezone
_csv gcps2wld rasutil win32trace
_ctypes gdal re win32traceutil
_elementtree gdal2tiles regcheck win32transaction
_functools gdal2xyz regutil win32ts
_hashlib gdal_auth repr win32ui
_heapq gdal_calc rexec win32uiole
_hotshot gdal_edit rfc822 win32verstamp
_imaging gdal_fillnodata rgb2pct win32wnet
_imagingmath gdal_merge rlcompleter winerror
_io gdal_polygonize robotparser winioctlcon
_json gdal_proximity runpy winnt
_locale gdal_retile sched winperf
_lsprof gdal_sieve scipy winsound
_md5 gdalchksum script winxpgui
_msi gdalcompare select winxptheme
_multibytecodec gdalconst servicemanager wsgiref
_multiprocessing gdalident sets wx
_osx_support gdalimport sgmllib wxPython
_pyio gdalmove sha wxversion
_random gdalnumeric shapely xdrlib
_sha genericpath shelve xlrd
_sha256 getopt shlex xlwt
_sha512 getpass shutil xml
_socket gettext signal xmllib
_sqlite3 glob simplejson xmlrpclib
_sre gui sip xxsubtype
_ssl gzip sipconfig zipfile
_strptime hashlib sipdistutils zipimport
_struct heapq site zlib
_subprocess hmac six

As you can see from above, there are so much modules installed by default that we don't necessarily have to install another module to start GIS programming in QGIS. Some of the geospatial specific installed modules are shapely, matplotlib, gdalnumeric, gdal2xyz, gdal, numpy, gdal_polygonize, gdal_calc, pyqgis (QGIS API) etc.

If you want to know how any of the above module objects works, all you have to do on the console is call "help('object_name')". Also some other useful functions for getting help are "dir() - which displays all the object methods" and " 'object'.__doc__ - which displays documentation string of an object".

Lets clear the console screen and demonstrate the use of dir() and 'object'.__doc__

Assuming we need to learn more about the shapely module, we can type any of these on our QGIS console:-
1) help('shapely')
2) dir('shapely')
3) 'shapely'.__doc__






Thanks for reading.

1 comment:

  1. Great piece, Umar.
    Makes an enthusiast and n00b wanna really do the stuff.
    I'm motivated.

    ReplyDelete