Connecting to a Google Sheet from Python Script

It is very common when working with data sets in python to save processed data onto a local spreadsheet file. After which you will attached and send such a file to other users in some other locations. In a situation where you want the spreadsheet file to be readily available to those users as you push in process data, then you need to share a cloud hosted spreadsheet such as 'Google Sheet'.

So, in this post I will share with you how you can use python to connect to a 'Google Sheet' located in your Google Drive and push in data right from a python script running on your local machine. This post will guide you on loading data from a local python script into cloud based spreadsheet (Google Sheet).

Let's get started...

Python Google Sheet setup instructions

1)  Configuring Google account
a)  Go to: https://console.cloud.google.com/apis/dashboard and create a new project, give it a name and open it. You can use existing project if you already created one before.

Here my project is named “PySpreadSheetExample”.

b)  Open the project and enable “Google Drive API” and “Google Sheet API” by clicking on ‘Enable API and Services’ button.

You will search for “Google Drive API” and “Google Sheet API” and click on the ‘Enable’ button for each.

Geographic Coordinates Order - Latitude Longitude OR Longitude Latitude

There is this frustrating inconsistency in working with geospatial data as to whether an array of numbers like this "9.071, 7.499" means Latitude, Longitude OR Longitude, Latitude?

I have personally wasted valuable time when am using a new GIS tool trying to figure out the right coordinate order recognized by the tool.

Obviously, some GIS platforms use Latitude, Longitude while some uses Longitude, Latitude. The question now is which is the correct coordinates order? This is an opinion with no right answer. Vocal and written geographical theory favors Latitude, Longitude. Numerical and software prefer Longitude, Latitude.

It common to describe a location as *the Latitude and Longitude of ABC is "9.071, 7.499"*, thereby mentioning Latitude first instead of the Longitude. However technically speaking mathematics, Latitude represents Y coordinate while Longitude represents X coordinate. So Latitude, Longitude implies Y, X while Longitude, Latitude is X, Y.

If you include altitude or height (Z) of ABC location the two coordinates ordering above will read: Latitude, Longitude, Height (Y, X, Z) and Longitude, Latitude, Height (X, Y, Z) respectively.

The most common ordering is XYZ - Longitude, Latitude, Height. However, if you decide to adopt YXZ - Latitude, Longitude, Height you could still be right. The most important thing is for you to know the right order you need when working with a GIS platform especially a new one you haven't used before.

As an example, Google GIS products (such google maps, google earth, etc) use the format YXZ - Latitude, Longitude, Height. ESRI ArcGIS and QGIS adopted the format XYZ - Longitude, Latitude, Height.

This coordinate "9.071, 7.499" on Google maps is the Y, X- Latitude, Longitude values of "Millennium Park (Wupa River)", if you change the order like this "7.499, 9.071" (i.e X, Y - Longitude, Latitude) it points to a different location from the intended location in this case an Unnamed Road.

Millennium Park (Wupa River) - 9.071, 7.499 (Latitude, Longitude)

Unnamed Road - 7.499, 9.071 (Longitude, Latitude)

Python split list into sub-lists based on string value

I have a list that is randomly separated by a string value, now I want to make a sub-list at each of the random string that separates the whole list.


in other words as an example, the list below is separated by ':' at random intervals. So, at each occurrence of ':', I want to make a list of those elements.

mylist = [1, 'sistani', ':', 3, ':', 7, 9, 0, 'anita', ':', 20, 8, 4, ':', 12, 5, 10, ':', 56, ':', 6, 30, 56, 'usman', ':']
mysepstring = ':'

# Split list by value - python split list into sublists based on string value
def list_splitz(baseList, sepString):    
    group = []    
    for x in baseList:
        if x != sepString:
            group.append(x)
        elif group:
            yield group
            group = []
            
print(list(list_splitz(mylist, mysepstring)))

Hope this was useful.

How to Setup Geo-tagging on photos taken by Smart Phone Camera

Today, it is very common for people to use their smart phones to take pictures of locations they visited. However, very few of them know that they can actually tag the pictures with the latitude, longitude and altitude of the places.

In this post, I will show you how enable GPS on your smart phone so that the pictures you snap with the camera adds geo data (latitude, longitude and altitude) to the photos.

This feature is mostly disabled by default on most smart phones. There will be different ways to enable the feature on different phones, here I will use android device for my demonstration.

Android users follow these steps:

Step 1: Go to your phone "Settings".

Step 2: Under personal group of settings icons, select "Location" and turn it on.

Step 3: Now that you got your phone location sensor on, open you camera and select the option button. There you will find "GPS Location Info", turn it on if it is off.

Any picture you take with these settings above properly enabled, will link that picture with GPS coordinate of the location.

Geocoding and Reverse Geocoding with Python

Disclaimer: I originally submitted this article to DataCamp on "Jan 27, 2018". Since they didn't publish it on the platform, I have decided to do it here so that someone out there will find it useful.

Download the original files in HTML and Jupyter Notebook formats

DataCamp Tutorial - Geocoding and Reverse Geocoding with Python

The increasing use of location-aware data and technologies that are able to give directions relative to location and access geographically aware data has given rise to category of data scientists with strong knowledge of geospatial data - Geo-data Scientists.

In this tutorial, you will discover how to use PYTHON to carry out geocoding task. Specifically, you will learn to use GeoPy, Pandas and Folium PYTHON libraries to complete geocoding tasks. Because this is a geocoding tutorial, the article will cover more of GeoPy than Pandas. If you are not familiar with Pandas, you should definitely consider studying the Pandas Tutorial by Karlijn Willems so also this Pandas cheat sheet will be handy to your learning.

Tutorial Overview

• What is Geocoding?
• Geocoding with Python
• Putting it all together – Bulk Geocoding
• Accuracy of the Result
• Mapping Geocoding Result
• Conclusion

What is Geocoding?

A very common task faced by Geo-data Scientist is the conversion of physical human-readable addresses of places into latitude and longitude geographical coordinates. This process is known as “Geocoding” while the reverse case (that is converting latitude and longitude coordinates into physical addresses) is known as “Reverse Geocoding”. To clarify this explanation, here is an example using the datacamp USA office address:-

Geocoding: is converting an address like “Empire State Building 350 5th Ave, Floor 77 New York, NY 10118” to “latitude 40.7484284, longitude -73.9856546”.

Reverse Geocoding: is converting “latitude 40.7484284, longitude -73.9856546” to address “Empire State Building 350 5th Ave, Floor 77 New York, NY 10118”.

Now that you have seen how to do forward and reverse geocoding manually, let’s see how it can be done programmatically in PYTHON on larger dataset by calling some APIs.

Amazon EC2 - Using SFTP to download or upload files

In this post you will learn how to upload/download files to/from Amazon EC2 Instance using FileZilla and SFTP.

SFTP stands for: Secure Shell (SSH) File Transfer Protocol or Secure File Transfer Protocol.

Step 1:

The very first step is to download your AWS console key file in .pem and convert it to .ppk file.

The key file you download from AWS console will be a .pem file and filezilla doesn't read such a file. So you need to convert it to .ppk file. You can use PuttyGen (puttygen.exe) for this conversion from .pem to .ppk.

Step 2:
Download and install FileZilla client from here Download FileZilla.

Step 3:
Now, launch FileZilla and add your .ppk key file of your AWS instance for public key authentication as follow;

~ Go to Edit >> Settings

~ On the Settings dialog box, under 'Connection', select 'SFTP' then click on 'Add Key File...'. Navigate to where you saved your converted .ppk key file and upload it. Then click on 'Ok' button.

Step 4:
Lets login to the server. Go to File >> Site Manager

Step 5:
Click on 'New Site' button, under 'General' tab select 'SFTP' as the protocol and insert your instance username and password. Finally click on "Connect" button.

Host name is something like: ec2-21-xx-xx-xxx.compute-1.amazonaws.com
Username is: ec2-user
Password is optional.

You should now be connected as seen on the status...

Step 6:
You can now navigate your local site folders on the right side pane and upload to the remote site folder on the left pane. Similarly for the downloading of files.

Note: the files which are owned by root cannot be uploaded and downloaded. You will get permission denied error when you try to download a file owned by root.

So, if you want to download those files, you will use PuTTY to SSH into the machine and change the owner of that file to the normal user using the following command.

sudo chown user:user /folder/file

Replace "user:user" with the appropriate root user name.



That is it!
I hope you find it useful.

Python Programming for GIS Data Processing in QGIS

Python is a general purpose programming language that has a strong presence in the GIS industry as many house hold software such as ArcGIS and QGIS do expose their interfaces through python scripting.

Broadly speaking, python can be used for any of the following:-
1~ Scripting and Automation
2~ Web Development
3~ Data Science — including Machine Learning, Deep Learning, Data Analysis, and Data Visualization
4~ Embedded Applications - Hardware Programming
5~ Gaming
6~ Desktop Applications
7~ Network Programming

Here our focus will be to use python to automate GIS processes via well packaged video series as seen below. Python is excellent tool for doing GIS data wrangling, data manipulation and analysis processes. That is exactly what I set out to teach you in this video series.... By the end of the tutorial, you should be able to use python to:-

• Extend the capabilities of mapping applications
• Build a GIS workflow
• Automate redundant tasks
• Integrate maps or spatial data into other applications
• Build your own applications

Let's get our hand dirty manipulating GIS data with python starting with the basics off cause 😊.

Video 1: Getting Started Setting up working environment





Video 2: Writing and executing code





Video 3: How to Install Third-party Python Modules in QGIS 3.x





Video 4: Python Implementations, Comment, Variable, Interactive Help





Video 5: Python Data Types - Part 1





Video 6: Python Data Types - Part 2




Video 7: Python Data Types - Part 3


Coming soon...

How to Prepare A Site Plan

Before we see "How to a Site Plan can be Prepared", here is a little background to what it is.

Who is a Land Surveyor?
A land surveyor is person who is trained and licensed by a regulatory body to practice legal survey profession in a state. This person will be under the supervision of the state's surveyor general or the director/commissioner of surveys for that state.

One important document produced by land surveyors is the site plan (also called survey plan or professional survey or boundary survey or lot description or final survey or parcel demarcation etc).


What is a Site Plan?
A Site Plan is a graphical description of a parcel of land that is attached to the application of Right of Occupancy (RofO) or Certificate of Occupancy (CofO) over a landed property. It is a necessary legal document required by the authority to approve the development of a land parcel. It is a document that measures the boundary of a parcel of land. It gives an accurate measurement, description of the piece of land.

Information to be Shown on a Site Plan
A professional site plan drawn to scale should show the entire property (including all property lines) with the existing and proposed structures. Other information that must be shown include:
~ the name of the owner of the land surveyed
~ the address or description
~ the size and the drawn-out portion as mapped out on the survey plan document
~ the beacon numbers
~ the name of the Surveyor who drew up the survey plan
~ the date it was drawn and a stamp showing that the land is either free from government acquisition or not
~ the directional arrow (aka North Arrow)
~ the Scale at which the plan was drawn
~ the coordinates system used
~ the reference control beacons used

Why is a Site Plan Required?
The primary reason why a site plan is required is to get what is known as a "Land Information". Before a transaction is consummated over a plot of land, it is always advisable to prepare the site plan presented for the land to basically find out whether it falls under government acquisition or not and as well authenticate the claim of ownership to the land.
In the case of a new construction, building additions, substantial accessory structures, and parking must be submitted on a professional stamped survey. Land authorities will make the final determination whether your project met the requirements to be implemented on that land.

Can I Draw the Site Plan Myself?
Yes, if you have the skills to do it. However, if the purpose of the plan is for legal documentation a registered surveyor most endorses your work to be considered by any authority.

How to Prepare A Site Plan

You don’t have to be a genius to prepare a site plan, but you do have to be able to work with the tools that helps you collect data from site and draw them on a paper.

Basically, there are two phases to producing a site plan namely:-

Phase 1: Site work
Here is where the surveyor goes to the field for the purpose of data collection. Depending on the purpose and the accuracy of the survey plan, different kind of instruments are used for the site data collection. These instruments include:-
a) Chain instrument
b) Compass
c) Theodolite
d) Total station
e) GPS

Whatever the instrument used, you will collect data such as distance, bearings and coordinates that will be used to draw the survey plan in the next phase.

Phase 2: Office Work
This is where you do the actual drawing of the survey plan. There are different kind of instruments used for plotting/charting survey plan, however the most commonly use software is the AutoCAD drafting software.

Learn to draw/plot survey plans using a FREE software similar to AutoCAD from here.

Thank you for reading.

Install Third-party Python Modules in QGIS 3.x on Windows 10 platform

Here is how you can install third-party Python Modules in QGIS 3.x on Windows 10 platform. First let's see one very important module that is not installed with python setup that comes with QGIS. This module we will look at is "Pandas".

If you launch QGIS Python Console, and import 'pandas', you will get error

Steps to solving the problem

Step 1: Search for "OSGeo4W Shell"

Step 2: Right click on it and select "Run a s Administrator"

Step 3: Ensure you can run pip from the command line. You may encounter error that says: No module named pip

In my case I ran python -m ensurepip --default-pip to get pip working. You can learn more from the link above. The "-m" flag import the pip module or package and then run as a script (it ensures that module is ran as script on cmd).

Note: If you want to switch the shell from python 2.7 to python 3.x, simply enter this command: py3_env. I will advice you do that because that is what the QGIS 3.x python console is using by default. So, just enter py3_env to switch to python 3 shell, then install your third-party modules there.

Step 4: Run: python -m pip install pandas to get pandas module installed as seen above.
Now you can import and use pandas in QGIS Python Console for further data manipulation without any error as seen below. The "-m" flag import the pip module or package and then run as a script.

That is it!


A video explanation of the steps above is here below:-

How to save High Resolution Image from Google Earth Pro

I have seen many "Google Earth Pro" users taking screenshots of the images instead of saving it directly in high resolution, probably because the "Save Image" function is a little bit obscured. This post will guide you through saving a high resolution image from "Google Earth Pro" released in the year 2018.

At first when you click on the save button, nothing seems to happen aside the "Untitled Map", "Compass", "Scale" and empty "Legend" boxes that pop-up on the screen.

If you look closely, just under the "Untitled Map" box, there appear the "Save Image" tool bar. On that very tool bar, you will see the "Save Image" button. That is the button you actually needed to save the image.

But before you save your image, you will definitely like to update the map title and do other settings. This can be done from the "Map Options" button on the "Save Image" tool bar.

You could also select your preferred image resolution from the tool bar.

Thank you for reading.