Sunday, December 25, 2016

Nairaland Christmas Birthday Analyzed and Visualized in Python

Members of nairaland.com forum that celebrates their birthday on Christmas day. Happy Birthday Christmas to this Nairaland forum members

Which member of Nairaland.com forum is celebrating his/her birthdays today? Let's derive some insights from the Birthday dataset.

Friday, December 23, 2016

6 Python web scraping libraries you can't afford to ignore

Hello there,
In this post I will share with you some excellent Python Web Scraping Libraries. For the benefit of the reader who doesn't know what web scrapping is, here is quick introduction.

Web Scraping: is a technique employed to extract large amounts of data from websites whereby the data is extracted and saved to a local file in your computer or to a database in table (spreadsheet) format (source, webharvy.com). Web Scraping is also called Screen Scraping, Web Data Extraction, Web Harvesting etc.

Different programming languages have different techniques for doing web data scrapping. Here I will present to you some web data scrapping tools as used in the Python programming languages.



Wednesday, December 21, 2016

Copying tabular data to Excel/CSV file in Python

I know there are several ways of doing this task. But if you have very large number of tables to be copied into several spreadsheets, then the best approach is to automate the process using a scripting language like python.

One of the best python modules for doing this is called: pandas (an easy-to-use data structures and data analysis tools for the Python programming language)

If you don't already have pandas install, run "pip install pandas" from your command window.


Copying tabular data from PDF, HTML, Words etc to Excel/CSV spreadsheet file


With few lines of python pandas script, you can get this done. Then if you have multiple tables to copy into spreadsheets, you can use loop statement to go over all the tables.

I will give a very simple example here. Lets assume you have web page that contains a table and you want to save that table in an excel/csv file, here is how to go about doing it with python pandas module.

Tuesday, December 13, 2016

QGIS - Labeling a shapefile layer using two or more attributes information

Hello there,
There may be a situation whereby you want to display more than one attribute information or label of the shapefile map canvas you are working on. If this sounds like you, then you are reading the right tutorial.

In this post, I will discussed how to: Label a shapefile layer using two or more attributes information in QGIS.

Labeling a Shapefile Layer Using Multiple Attributes Information

Now, I assumed you already now how to label your shapefile map canvas by using a single attribute's information. This is fairly simple and it is what you will be taught in your first GIS training class.
What you probally don't know is that you can make use of more than on attribute information to label your map features in QGIS. This is made possible by the use of "QGIS label expression" command.
That is what QGIS expression in the label expression is used for.


The shapefile am going to use for this demonstration as seen above, has five (5) columns/fields of attributes which are: FID, State_Name, Area, Capital and Population. Now, lets label the map with as single attribute field then with multiple attribute fields.

Saturday, November 26, 2016

Advantages of GeoDatabase over Shapefile

Both GeoDatabase and Shapefile are excellent storage containers for Geospatial datasets. However, the former (geodatabase) has more robust advantages when handling big spatial data over the later (Shapefile).

Here are some of the reasons why you should move to using geodatabase asap when dealing with very big data:-

~ Allow multiple users management and privileges on big geospatial datasets

~ One-to-Many and Many-to-Many relationship between geodata sets.

~ A shapefile isn’t made of a single file. Shapefiles are quite unhandy when working with many of them very often. Four files at least makes a valid shape file (.shp, .dbf, .shx and .prj).

~ Shapefiles are binary files that can't be edited by hand using any text editor. They most be viewed and edited in a GIS software, that is not the case with Geodatabases (SpatiaLite and GeoJSON). You can edit them by hand (using text editor), rather than using GIS software, if you’re so need to do so.

~ Long attribute fileds/column names like “nig_geodb_majortrunck_RoadNetwork” can't be used as 10 characters at maximum is allowed for field name.

~ The maximum file size for shapefile (.shp / .dbf) is 2GB. With a geodatabase, you are not limited by the file size.

~ There are no much support for table field type. Numeric attributes are stored as characters (integers/floats), causing potential problems with rounding etc.

~ NULL values are interpreted differently between systems


Example to clarify the points above


Go to NaturalEarthData.com and download all vector geographical themes of the world as SHAPEFILE and GeoDataBase.

You will easily see that because the dataset contains over 100+ layer/themes, the GeoDB is easier to hand as it holds all the layers in one single file while the shapefile option has at least three files for each of the 100+ layers to deal with.



Thank you for following the blog post.

Tuesday, November 15, 2016

My Favorite GIS Blogs

Hello,

Today I want to share with you a list of "My Favorite GIS Blogs".

These are the GIS online blogs you should definitely keep your eyes on for getting latest up to date trends in the ever changing Geospatial industry...

These are my personal favorite GIS and related blogs:-


Be free to add anyone you feel is missing from the list in the comment section below.


Thanks for following the post.

Wednesday, November 9, 2016

Python List Comprehension Simplified

Hello,

This is going to be as simple introduction to Python List Comprehension.

A "list comprehension" in python provides a concise way to create lists. Basically, it nests 'expression', 'loops' and 'conditions' within a list on a single line. The basic structure of the python's list comprehension is like this: [*expression*  *for object in loop*  *if condition*] - without the "*" of-course.

Example


Assuming we have a list called "my_list" and we want to get the length of individual elements within the list, a 'for' loop can do it as follow:-


When you run above code it will output the result as: 4 10 2 5 1 2

Now lets use "list comprehension" to achieve the same result as above:-


The result returns a list of the lengths like so: [4, 10, 2, 5, 1, 2]

Note that we didn't add a condition to the list comprehension above. len(item): is the expression value and for item in my_list: is the loop part of the "list comprehension". We will add a condition in a moment.

List comprehension has two quick noticeable advantages over using the "for" loop namely:-
1) the code is shorter in length.
2) the result is in an object list instead of individual numbers as obtained using "for" loop.

To add a condition, we simply add an "if" statement to the end of the "list comprehension" as follow;-
Lets add a condition to display only length of items greater than four: if len(item) > 4. Add that syntax to the end of the "list comprehension".


The result returns a list of the lengths greater than four like so: [10, 5]


For more than one condition
Assuming we have to check for more than one conditions as follow; instead just having the lengths greater than four, we also want to consider the lengths less than four and equal to four. This makes it about three conditions to check as follow:-
Less than four = x
Greater than four = y
Equal to four = z


The syntax will be like this: [*expression* *if elif else condition*  *for object in loop* ]

In a normal forLoop it is like this....
for item in my_list:
    if len(item) < 4:
        print('x')
    elif len(item) > 4:
        print('y')
    else:
        print('z')

In list it will look like this....
print(['x' if len(item) < 4 else 'y' if len(item) > 4 else 'z' for item in my_list])
Note that 'elif' is not used in the list comprehension.



Conclusion


Even though "list comprehension" may sound very nice to use it has a little draw back in the sense that it can easily make your code very difficult to read in the nearest feature.

Imagine a situation where you compressed hundreds lines of code using "list comprehension" into a few lines. It become unreadable or difficult to read when time passes.

So use it with caution.


Thank you for following the blog post.

Saturday, November 5, 2016

Understanding the different concepts of GIS - GISystem, GIScience, and GIService

Difference between GISystem, GIScience and GIService


There are three major conceptual frameworks of the acronym "GIS" and without any doubt, the most common or most talked about is the GISystem - Geospatial/Geographic Information System.

However, GIScience and GIService have recently gained popularity within the geospatial industry. For example, in the academia GIS is often viewed as a GIScience while in the business market places, GIS is often viewed as a GIService. Casual GIS users often viewed it as a tool/system (i.e. GISystem) to support their spatial problem solving - GIS is fundamentally about solving real-world problems.


GISystem, GIScience, and GIService


Over the years, the concept of GIS has experienced continues changes, from Geographical Information System (GISystem) to Geographical Information Science (GIScience), and then the development of Geographical Information Services (GIServices).

Tuesday, November 1, 2016

GeoJSON in Python

Introduction


Few days ago, I wrote a blog post on using QGIS to convert a Shapefile to GeoJSON.

In this post, I will talk about working with GeoJSON object in Python programming language.

Basically we know a GeoJSON is an object contains spatially related datasets. Conventions used by GeoJSON are known to programmers, which include C, C++, Java, Python, Perl, etc.

GeoJSON has the benefit of having implementations in many languages (especially JavaScript), making it suitable for inter-application communication of spatial data.


GeoJSON in Python


There are many python libraries that allow you to load and manipulate JSON/GeoJSON objects within the python programming ecosystem. The common and default library is the JSON which comes by default with any python installation.

All you need to do to use it is to call this line "import json".

Code Demo


Am going to use a GeoJSON file containing polygon features and saved with the name "Polygon_3.geojson" on my local disk for this demonstration.



There are two most important functions in the json library for converting python objects to json/geojson or vise-versa named: dumps() and load() functions.

In this demo, I will just read in the geojson file above into the python development environment using the with open() python command. This will allow us have access to the individual polygon elements within the geojson file as follow:-


.As you can see from above, when you load a GeoJSON file using the JSON library, you get a dictionary that contains an entry features, which contains the list of features.

Each feature in turn consists of a dictionary, which among other things, contains an entry geometry.

The geometry is a dictionary containing the entries type (polygon in this case) and coordinates of the vertices. So you can traverse the GeoJSON file like this:

import json

with open('Polygon_3.geojson') as f:
    data = json.load(f)

for feature in data['features']:
    print (feature['geometry']['type'])
    print (feature['geometry']['coordinates'])



As you can see, the GeoJSON file contains five polygons of different number of vertices and shape as indicated by the amount of pairs of coordinates for each polygon.

That is it.
Thanks for reading.

Tuesday, October 25, 2016

Modern Database Models and Technologies

A data models define how the logical structure of a database is modeled. Data Models are fundamental entities to introduce abstraction in a Database Management Systems
 (DBMS). According to WikiPedia, a database model is a type of data model that determines the logical structure of a database and fundamentally determines in which manner data can be stored, organized, and manipulated. The most popular example of a database model is the relational model, which uses a table-based format.

(Source: WikiPedia)