Wednesday, October 9, 2019

Calculating the total size of zip code maps


On this US printable zip codes maps page, there is a list of all the US states zip code maps with their respective sizes in braces like this "Alabama ZIP Code Map (3.59MB)" as seen below...



Lets calculate the total size of all the maps using python scripting!

Off course, there are several or even better ways to get this done. But here we want to test our python skills on this, let us stick to using python 😏.

Some other reasons it is good idea we use python is that we can easily use our python skill to:-
1) make HTTP request to scrape/download the map data
2) generate the download links on the fly
3) create a bot to monitor change in map size (which could indicate the map has been updated).
4) visualization of the string including map/geographic visualization.

The list can go on and on, but I will keep it simple here to just calculate the total sum the map sizes.

Step 1:
First thing is to get the string/text off the web page into our python environment. There are several ways to do this as I have mentioned above, but I will just select, copy and paste it in a CSV file as seen below.



Step 2:
Read the CSV file in python. Here I will use the pandas module to read the CSV file, could have also used the CSV module to do this.



Monday, October 7, 2019

Filtering Missing Zip codes out of master Zip codes list

Here I have a list of zip codes, I want to know the missing zip code from the given list (these are the postal code in Texas, USA).




List 'available_zipcodes' contains the master zip codes and list 'given_zipcodes' contain the provided or working zip codes. Now I want check and filter out those zipcode that are NOT in the master zip codes.

These three lines of python code below will do it. It uses the 'for' loop with and 'if' statement. Basically, we loop through the list of 'given_zipcodes' and if it is not in the 'available_zipcodes', then we print it out.




If you care to run the script and don't want to type all that out, here below is the Code is...

available_zipcodes = [77389, 77086, 77346, 77018, 77040, 77388, 77065, 77080, 77041, 77396, 77385, 77354, 77382, 77067, 77066, 77090, 77345, 77355, 77373, 77339, 77043, 77302, 77304, 77070, 77375, 77095, 77433, 77069, 77038, 77091, 77380, 77092, 77316, 77429, 77377, 77379, 77064, 77088, 77338, 77449, 77386, 77381, 77493, 77356, 77068, 77014, 77084, 77055, 77301, 77303, 77384]

given_zipcodes = [77325, 77339, 77345, 77346, 77380, 77381, 77382, 77383, 77384, 77385, 77386, 77301, 77302, 77303, 77304, 77316, 77354, 77356, 77389, 77014, 77018, 77038, 77040, 77041, 77043, 77055, 77064, 77065, 77066, 77067, 77068, 77069, 77070, 77080, 77084, 77086, 77088, 77090, 77091, 77092, 77095, 77375, 77377, 77379, 77388, 77429, 77433, 77449, 77493, 77373, 77338, 77347, 77391, 77396, 77355]


for zipcode in given_zipcodes:
    if zipcode not in available_zipcodes:
        print(zipcode)

In the case above, the missing zip codes are: 77325, 77383, 77347, 77391

Note: In a production job. these zip codes will probably come in a text file, just read the file into python lists and loop through as seen above.

That is it!

Tuesday, September 24, 2019

Limitations of a Shapefile

For along time, shapefile has being my primary GIS file for working with vector data. I have never had any reason to look beyond shapefile for handling my GIS vector datasets not until recently when I have a need to store some large quantity of text string in the attribute table.

Before I share my story, let make a point to what a shapefile is just in case you don't know it.

Shapefile is a file type developed by ESRI to handle vector map data in the form of points, polylines and polygons. More details can be found on the Wikipedia page as summarized in the picture below, also on the 'Shapefile Technical Description' document.



Limitations of a Shapefile
Specifically, I was trying to convert a KML file to shapefile. Then one of the columns that had alot of text/string content gets truncated when converted to shapefile. I couldn't figure out why and what caused that until I found this website (Switch from Shapefile) that listed listed some its limitations and that one that affected my situation directly was that the maximum characters is 254.


No way! My attribute table has way more than 254 characters. Then I had to look beyond a shapefile. I actually settled with a GeoJSON file type.

Once again as listed on the website Switch from Shapefile, other limitations include:-
~ No coordinate reference system definition.
~ It's a multifile format.
~ Attribute names are limited to 10 characters.
~ Only 255 attributes. The DBF file does not allow you to store more then 255 attribute fields.
~ Limited data types. Data types are limited to float, integer, date and text with a maximum 254 characters.
~ Unknown character set. There is no way to specify the character set used in the database.
~ It's limited to 2GB of file size. Although some tools are able to surpass this limit, they can never exceed 4GB of data.
~ No topology in the data. There is no way to describe topological relations in the format.
~ Single geometry type per file. There is no way to save mixed geometry features.
~ More complicated data structures are impossible to save. It's a "flat table" format.
~ There is no way to store 3D data with textures or appearances such as material definitions. There is also no way to store solids or parametric objects.
~ Projections definition. They are incompatible or missing.
~ Line and polygon geometry type, single or multipart, cannot be reliably determined at the layer level, it must be determined at the individual feature level.


Now you know some troubles you may encounter with you shapefile data are due to some of these limitations, so no need to full your hair just switch to a more advanced GIS file type.

Thursday, September 19, 2019

QGIS Calculate the Mid Coordinates of Polygons

In QGIS field calculator, you can calculate the center point of all polygons within a polygon layer.

Formula 1:
x($geometry), y($geometry)

Formula 2:
xmin(centroid($geometry)), ymin(centroid($geometry))

Formula 3:
x(centroid($geometry)), y(centroid($geometry))


Note that: 'x' standards for Longitude while 'y' standards for Latitude. $geometry represent the variable polygon geometry.


As you can see the preview result for the three formulas are the same.

Sunday, September 1, 2019

Map from GIS to CAD

Introduction

No doubt, on the desktop ESRI ArcGIS is the top GIS software while AutoDesk AutoCAD is the top CAD software.

Both are capable of making maps and in this article, I will demo how to convert existing map in ArcGIS to AutoCAD. But before I go into that, lets get to know what GIS and CAD mean.



What is GIS and CAD?

GIS = Geographic Information System
CAD = Computer Aided Design



What is the Difference between GIS and CAD?

Both GIS and CAD can be used for making maps however, they are very different technologies with different applications.

GIS: analyzing/visualizing map data
CAD: creating/editing accurate map data

GIS allows data to be attached to the points, lines, and polygons used in the map. This makes GIS the best tool for analyzing and visualizing data through the use of a map.

CAD easily allows a user to create a very accurate drawing whether it is a map, site plan, profile etc. CAD allows the drawing of maps by the use of coordinates or through distances/bearings in different types of unit.


Map displayed in ArcGIS



Map displayed in AutoCAD




How to converting map data from GIS to CAD and vice versa

GIS to CAD:
In ArcGIS, you use the command at: ArcToolBox >> ConversionTools >> To CAD to concert map layer to CAD.





CAD to GIS:
In AutoCAD you simply save the map as .dxf or .dwg file to have it usable in GIS.




That is it!

Wednesday, August 28, 2019

QGIS Remove Black Background Boarder from Raster Image


Often times, you are left with black boarder around an image you manipulated in QGIS as seen below. This is often cause because there is no data to display around data part of the image.



Here is how to get ride of the black background in QGIS 3.

Open the raster image layer property window and select the 'Transparency' tab. Then enter '0' under No data value >> Additional no data value.



Click 'Ok' to apply the changes. Your raster image should now have no black background color surrounding it as seen below.




That is it.

Monday, August 26, 2019

Get the row count of multiple excel spreadsheet files

Here I have many excel spreadsheet files within a folder as seen below...


The task is to return the number of rows in each of the excel files. I can go manually, open each file, scroll to the bottom and note down the row number. That will be cumbersome and time consuming given that number of files I have to cover.

So, I have to write a simple script in python that will handle this boring task accordingly as follow:-

Step 1: First things first, lets find a way to read all the .xlsx files. Here I used the glob module to handle this.

import glob

folder_xlsx = r"C:\Users\Yusuf_08039508010\Desktop\my-xlsx-folder"

# read all the individual order xlsx files
xlsx_files = glob.glob(folder_xlsx + '/*.xlsx')
what I have above is a list that contains path to all the excel files in the folder. Lets move on...


Step 2: Next step is to read each excel file into a pandas dataframe and use a function to count the number of rows in the dataframes. There are many functions to count the number of rows as seen below, but I will use this function 'len(df.index)'.


Here is the solution for the fisrt dataframe.

df = pd.read_excel(xlsx_files[0])

row_count = len(df.index)

To do for the whole excel files, we just write a for loop and save the into a list as seen below. Noticed that I used rsplit() function to get the file names to print it along its corresponding row count.

import pandas as pd
row_count_list = []
for xls_file in xlsx_files:
    df = pd.read_excel(xls_file)
    row_count = len(df.index)
    
    file_name = xls_file.rsplit('\\', 1)[1]
    
    file_details = file_name, row_count
    
    row_count_list.append(file_details)
    
print (row_count_list)



That is it!


P.S: You could easily extend the script above to do many other thing with the files. An example will be to merge all the files into one file using the pandas concat() method. So, instead of appending the file names and the row counts, we will simply append the dataframe as seen below.

df_list = []
for xls_file in xlsx_files:
    df = pd.read_excel(xls_file)
    
    df_list.append(df)
    
merge_df = pd.concat(df_list)

Thursday, August 8, 2019

Split string at the last occurrence of a string


I have a list of strings with varying length. However, the each string always end with certain same information (country in this case) as seen below.


data_list = ['Adams Smith, white, UK', 
             'Samuel Tom, Black, 29 leen st. NY, USA', 
             'Yaks Ramson, New Student, Yet to register, Romania']
    

As you can see, there are three items in the list and each item ends with a country name after a comma (,) sign.

When you loop through the items, you can split each item by comma like this: item.split(','). However, this isn't what I wanted, I want to split just at the last comma. In other words, I want to plit each of the string at the last occurrence of the comma (,) sign.

So, here the solution is to use a list method call rsplit(',', 1), which accept a second argument that tells how many times you want to split a string. Here I want to split the string just once, so my script will look like this...

data_list = ['Adams Smith, white, UK', 
             'Samuel Tom, Black, 29 leen st. NY, USA', 
             'Yaks Ramson, New Student, Yet to register, Romania']

item_list = []
for item in data_list:
    item_1 = item.rsplit(',', 1), # Not item.split(',')
    
    item_list.append(item_1)

Now, each item is split into two and you can access the individual countries as seen below:-


Sunday, July 28, 2019

Ways to create and add SVG maps to a web page

SVG stands for 'Scalable Vector Graphics' which defines vector-based graphics in XML format. It is just another format for displaying images on the web and every element and every attribute of its can be animated. It have greater advantages over other image file types such as PNG, JPG, GIF, BMP, etc.

Some of its notable advantages are:-
~ It is scalable. That is it doesn't loss quality when stretched or compressed.
~ It has interactive ability with CSS and JS
~ It can be created and edited with any text editor
~ It can be searched, indexed and scripted
~ It can be printed with high quality at any resolution

Ways to create SVG image maps

SVG maps can be created with either of the two ways namely; text editor or drawing program. The examples of each is given below:-
1) Text editor (Code): any text editor such as notepad++, sublime, atom, etc can be used.
2) Drawing tools/program: Inkscape, Adobe Illustrator, etc. While most of these tools can save SVG files directly, it is worth noting that there are some such as ArcGIS and QGIS that edit maps in other formats such as shapefile then other online such as Mapshaper,  Mapstarter, Geoconverter, etc are used to convert the shapefile to SVG file.

Creating SVGs with Code allows you to understand the different svg elements and the attributes that make up the file. This is very import when you want to manipulate and interact with the SVG using CSS and JS scripting.

If your SVG is a complex one such as a state map administrative boundaries, you are better off creating the SVGs with drawing tools.

Ways to add SVG image maps into HTML web page

1) Using inline SVG tag (<svg>...</svg>)

2) Using image tag (<img src='filename.svg' >)

3) Using CSS background-image property
body{
background: url(filename.svg);
}

4) Using HTML object, iframe or embed tags

Using inline SVG tag to add SVG images are the most powerful and flexible, as it allows certain CSS and JS operations with SVG that other ways don't allow. This method also helps with very fast loading speed of the web page.

On the other hand, a major draw back for using the inline SVG tag is that it is very poor in "search engine indexing".


That is it, hope it was useful.
Thank you for reading.

Saturday, July 6, 2019

Toggle cell line numbers in Jupyter notebook

When you are switching from a text editor to jupyter notebook environment for writing python code, you will definitely wish to see line numbers on jupyter notebook cells. And one obvious reason is that it makes it easier to trace errors as seen below.



In the script pictured above, there is an 'ElementNotVisibleException' error on line 86. If the cell line number was off, it will be very difficult to count and locate the line where the error occurred. But will line number enabled, we just scroll to the line number as seen below.



How to enable cell line numbers in Jupyter notebook

To toggle cell line numbers in Jupyter notebook, you can use two keyboard shortcuts in "Command Mode" as follow:-
1) L - to toggle line numbers
2) Shift-L - to toggles line numbers in all cells, and persist the setting



Note that the Jupyter Notebook has two different keyboard input modes.
Edit mode allows you to type code or text into a cell and is indicated by a green cell border.



Command mode binds the keyboard to notebook level commands and is indicated by a grey cell border with a blue left margin.




That is it!

Tuesday, June 18, 2019

How to work with Geographical APIs

Introduction
API - is the acronym for "Application Programming Interface" which basically means some sets of actions that allow interaction with some bunch of data hosted somewhere on a remote server. These interactions are commonly achieved by sending HTTP request to the server where the data resides. When such a data is related to some geographic location, then it is said you are working with a geographical API.

Imagine there is database of all elevation/heights of points around the world which you can access by providing the location's latitude and longitude. If you want to obtain an elevation of a point, you don't need to travel to that location just to get the elevation value. You will simply send a request to the API of the organization that has the database to retrieve the value.

Or on another hand, there is a database of latitude and longitude values of places around the world. If you need the latitude and longitude of a place you just send a request to with the name of that place to retrieve the latitude and longitude value. This is what the Google Geocode API does, we will see more of other APIs in the coming sections.


Who maintains Geographical API?
An API can be developed and maintained by any individual or organization. The bottom line is that you have access to some data legally and you want users to have access to them by sending HyperText Transfer Protocol (HTTP) requests. Then you will host the data on a server and define some instructions for accessing the data.

To make further clarifications, many organizations such as NASA, Google, ESRI, Mapbox, Government Organisations, etc who have access to large geographic data often have API for accessing and interacting with the data. At the time of writing, the above list are major providers of geo-data API.

There are several method available for the HTTP request, however the most commonly used once are the GET and POST methods. Others are described in the table below:-

SNMethod and Description
1GET
The GET method is used to retrieve information from the given server using a given URI. Requests using GET should only retrieve data and should have no other effect on the data.
2HEAD
Same as GET, but transfers the status line and header section only.
3POST
A POST request is used to send data to the server, for example, customer information, file upload, etc. using HTML forms.
4PUT
Replaces all current representations of the target resource with the uploaded content.
5DELETE
Removes all current representations of the target resource given by a URI.
6CONNECT
Establishes a tunnel to the server identified by a given URI.
7OPTIONS
Describes the communication options for the target resource.
8TRACE
Performs a message loop-back test along the path to the target resource.



Where to fine Geographical API?
Just about any individual or organisation that maintains a large volume of geographical data would probably has an API where such data is exposed to developer. However, there are few places to look when searching for geographical API services as listed below:-

1) Rapid API



2) ProgrammableWeb Mashup & API Directory



3) Any API



Above are few places where various APIs are listed and you can use the keywords "Geography", "Location" or "Mapping" to narrow down the search list. In few minutes, I will get some APIs from the directories for demonstration in the section below.

As you can see from each API description, there are literally hundreds of geographical API from different organizations that you can use for different kind of data exploration. You just need to know what the API can do and use it according to there terms of usage as I will explain in the next section.


How to work with Geographical API
1) The first thing is to know what kind of data you want, then head over to the API directories above and search for it availability.

2) If it is available, read and study the documentation provided for the API and use it accordingly.

3) Use any programming language to send/make the HTTP requests.


Example: Lets say we want to work with "Elevation" data across the globe in our project. Obviously we can travel all the globe to collect elevations of points locations, then what do we do? The solution lies in a API, we need an API the provide accurate elevations of places on the globe.

Let's search for such a API...



There are many API providers the can provide "Elevations", some are: Google Elevation API, Open-Elevation API, ElevationAPI.ioBing Maps Elevations etc.

Now, we have several options to choose from. Here I am going to work with ElevationAPI.io. Lets read and study its documentation.

From the doc, you will see that to retrieve the elevation of a point you need to provied the point's latitude and longitude pair in this HTTP GET request format: 'https://elevation-api.io/api/elevation?points=(39.90974,-106.17188),(62.52417,10.02487)&key=YOUR-API-KEY-HERE'.




Further study of the API doc reveals that the API_Key is an optional parameter in the GET url. So, it means we can obtain an elevation of a point at 5km resolution like this: https://elevation-api.io/api/elevation?points=(39.90974,-106.17188)'


The output result is in JSON format and you can parse this easily in python or similar programming language or use an online tool like 'JSON Editor Online' to view the result in a friendly form.



The above concept is the same for any other type of API data you want to work with begin it: Geocode API, Distance API, Satelite Image API, IPGeolocation API, Drone UAV API, etc just to name a few.

That is it!

Saturday, June 8, 2019

Documenting a python script and REST API with 'PyDoc' and 'Swagger OpenAPI Specification' respectively

Documenting a python app appropriately goes along way in communicating what the app does and what is expected of it.

In this post, I will introduce you to two tools you can easily use to create and keep appropriate documentation of your python app.

The first is the PyDoc module which is a standard documentation module in python programming language similar to PerlDoc and JavaDoc for Perl and Java programming languages respectively. With PyDoc, we can generate text and HTML pages with documentation specifics.

The second tool is the Swagger (aka OpenAPI ) which is an open-source software framework backed by a large ecosystem of tools that helps developers design, build, document, and consume RESTful Web services.


PyDoc: Documenting a python script

Pydoc is a Python documentation tool. On cmd enter this: python -m pydoc
You will see that all that the pydoc module is capable of doing as explained below....



1) pydoc <name> ...
    Show text documentation on something.  <name> may be the name of a
    Python keyword, topic, function, module, or package, or a dotted
    reference to a class or function within a module or module in a
    package.  If <name> contains a '\', it is used as the path to a
    Python source file to document. If name is 'keywords', 'topics',
    or 'modules', a listing of these things is displayed.

As an example enter: python -m pydoc pandas to see the documentation on the pandas module (note you should have already install "pandas" for you to see its documentation).



2) pydoc -k <keyword>
    Search for a keyword in the synopsis lines of all available modules.

An example to search for the keyword 'sql' is: python -m pydoc -k sql
All the keywords related to 'sql' on your python environment will be returned as seen below.



3) pydoc -p <port>
    Start an HTTP server on the given port on the local machine.  Port
    number 0 can be used to get an arbitrary unused port.

For example, python -m pydoc -p 0 started an arbitrary unused port '61281' http://localhost:61281 in my case. Yours will definitely be on a different port.

On the cmd use letter 'b' to browse and 'q' to quit/stop the HTTP server.



4) pydoc -b
    Start an HTTP server on an arbitrary unused port and open a Web browser
    to interactively browse documentation.  The -p option can be used with
    the -b option to explicitly specify the server port.

Example is: python -m pydoc -b will start the server doc automatically, it is just a handy shortcut for the above.


5) pydoc -w <name> ...
    Write out the HTML documentation for a module to a file in the current
    directory.  If <name> contains a '\', it is treated as a filename; if
    it names a directory, documentation is written for all the contents.

This enable you generate a HTML doc for a module or a script you have written. Example is: python -m pydoc -w XXX where XXX can be a module or a script file name.

This is useful if you want to share or host the doc, you simply share the resulting html file with others. Below is a written sample script I did like to document. Save it as 'testScript.py' and run the command from the folder that contains the script: python -m pydoc -w testScript

This will generate a html doc for the script that you can share with other developers as seen below. The script contains classes, methods and function with multi line comments (doc strings) in them, the html doc is generated based on those multi line comments (doc strings).

'''
Author: Umar Yusuf
Date: 2019/04/01
This python script, demonstrates how a module is documented for easy future reference.
Hope you like it!
'''

class ClassA(object):
    """Here is the docstring for ClassA"""
    def __init__(self, arg):
        super(ClassA, self).__init__()
        self.arg = arg

    def myMethod(self):
        '''A function in a class is a know as a METHOD'''
        pass
        


class ClassB(ClassA):
    """Here is the docstring for second ClassB. It inherits from ClassA"""
    def __init__(self, arg):
        super(ClassB, self).__init__()
        self.arg = arg
        

# --------------------
# These are functions since they are outside the class definition
# Python program to multiply two numbers

def multiply(a, b):
    '''
    This function takes two numbers in the form of input and multiplies them.
    It displays the multiplication as the result.
    '''
    print("Result= ", (a * b))


# functions calling
multiply(5, 2)





Recap
a) Show module's doc: python -m pydoc XXX 
b) Search for keyword: python -m pydoc -k XXX
c) Start python doc on http port: python -m pydoc -p 0
d) Shortcut for http server doc: python -m pydoc -b
e) Generates HTML file for modules or written scripts: python -m pydoc -w XXX


Sunday, May 19, 2019

Connecting to a Google Sheet from Python Script

It is very common when working with data sets in python to save processed data onto a local spreadsheet file. After which you will attached and send such a file to other users in some other locations. In a situation where you want the spreadsheet file to be readily available to those users as you push in process data, then you need to share a cloud hosted spreadsheet such as 'Google Sheet'.

So, in this post I will share with you how you can use python to connect to a 'Google Sheet' located in your Google Drive and push in data right from a python script running on your local machine. This post will guide you on loading data from a local python script into cloud based spreadsheet (Google Sheet).

Let's get started...

Python Google Sheet setup instructions

1)  Configuring Google account
a)  Go to: https://console.cloud.google.com/apis/dashboard and create a new project, give it a name and open it. You can use existing project if you already created one before.


Here my project is named “PySpreadSheetExample”.

b)  Open the project and enable “Google Drive API” and “Google Sheet API” by clicking on ‘Enable API and Services’ button.



You will search for “Google Drive API” and “Google Sheet API” and click on the ‘Enable’ button for each.




Wednesday, May 1, 2019

Geographic Coordinates Order - Latitude Longitude OR Longitude Latitude

There is this frustrating inconsistency in working with geospatial data as to whether an array of numbers like this "9.071, 7.499" means Latitude, Longitude OR Longitude, Latitude?

I have personally wasted valuable time when am using a new GIS tool trying to figure out the right coordinate order recognized by the tool.

Obviously, some GIS platforms use Latitude, Longitude while some uses Longitude, Latitude. The question now is which is the correct coordinates order? This is an opinion with no right answer. Vocal and written geographical theory favors Latitude, Longitude. Numerical and software prefer Longitude, Latitude.

It common to describe a location as *the Latitude and Longitude of ABC is "9.071, 7.499"*, thereby mentioning Latitude first instead of the Longitude. However technically speaking mathematics, Latitude represents Y coordinate while Longitude represents X coordinate. So Latitude, Longitude implies Y, X while Longitude, Latitude is X, Y.

If you include altitude or height (Z) of ABC location the two coordinates ordering above will read: Latitude, Longitude, Height (Y, X, Z) and Longitude, Latitude, Height (X, Y, Z) respectively.

The most common ordering is XYZ - Longitude, Latitude, Height. However, if you decide to adopt YXZ - Latitude, Longitude, Height you could still be right. The most important thing is for you to know the right order you need when working with a GIS platform especially a new one you haven't used before.

As an example, Google GIS products (such google maps, google earth, etc) use the format YXZ - Latitude, Longitude, Height. ESRI ArcGIS and QGIS adopted the format XYZ - Longitude, Latitude, Height.

This coordinate "9.071, 7.499" on Google maps is the Y, X- Latitude, Longitude values of "Millennium Park (Wupa River)", if you change the order like this "7.499, 9.071" (i.e X, Y - Longitude, Latitude) it points to a different location from the intended location in this case an Unnamed Road.







Friday, April 19, 2019

Python split list into sub-lists based on string value


I have a list that is randomly separated by a string value, now I want to make a sub-list at each of the random string that separates the whole list.


in other words as an example, the list below is separated by ':' at random intervals. So, at each occurrence of ':', I want to make a list of those elements.


mylist = [1, 'sistani', ':', 3, ':', 7, 9, 0, 'anita', ':', 20, 8, 4, ':', 12, 5, 10, ':', 56, ':', 6, 30, 56, 'usman', ':']
mysepstring = ':'
# Split list by value - python split list into sublists based on string value
def list_splitz(baseList, sepString):    
    group = []    
    for x in baseList:
        if x != sepString:
            group.append(x)
        elif group:
            yield group
            group = []
            
print(list(list_splitz(mylist, mysepstring)))


Hope this was useful.

Monday, April 1, 2019

How to Setup Geo-tagging on photos taken by Smart Phone Camera

Today, it is very common for people to use their smart phones to take pictures of locations they visited. However, very few of them know that they can actually tag the pictures with the latitude, longitude and altitude of the places.

In this post, I will show you how enable GPS on your smart phone so that the pictures you snap with the camera adds geo data (latitude, longitude and altitude) to the photos.

This feature is mostly disabled by default on most smart phones. There will be different ways to enable the feature on different phones, here I will use android device for my demonstration.

Android users follow these steps:

Step 1: Go to your phone "Settings".



Step 2: Under personal group of settings icons, select "Location" and turn it on.




Step 3: Now that you got your phone location sensor on, open you camera and select the option button. There you will find "GPS Location Info", turn it on if it is off.



Any picture you take with these settings above properly enabled, will link that picture with GPS coordinate of the location.

Friday, March 29, 2019

Geocoding and Reverse Geocoding with Python

Disclaimer: I originally submitted this article to DataCamp on "Jan 27, 2018". Since they didn't publish it on the platform, I have decided to do it here so that someone out there will find it useful.

Download the original files in HTML and Jupyter Notebook formats

DataCamp Tutorial - Geocoding and Reverse Geocoding with Python

The increasing use of location-aware data and technologies that are able to give directions relative to location and access geographically aware data has given rise to category of data scientists with strong knowledge of geospatial data - Geo-data Scientists.
In this tutorial, you will discover how to use PYTHON to carry out geocoding task. Specifically, you will learn to use GeoPy, Pandas and Folium PYTHON libraries to complete geocoding tasks. Because this is a geocoding tutorial, the article will cover more of GeoPy than Pandas. If you are not familiar with Pandas, you should definitely consider studying the Pandas Tutorial by Karlijn Willems so also this Pandas cheat sheet will be handy to your learning.

Tutorial Overview

  • What is Geocoding?
  • Geocoding with Python
  • Putting it all together – Bulk Geocoding
  • Accuracy of the Result
  • Mapping Geocoding Result
  • Conclusion

What is Geocoding?

A very common task faced by Geo-data Scientist is the conversion of physical human-readable addresses of places into latitude and longitude geographical coordinates. This process is known as “Geocoding” while the reverse case (that is converting latitude and longitude coordinates into physical addresses) is known as “Reverse Geocoding”. To clarify this explanation, here is an example using the datacamp USA office address:-
Geocoding: is converting an address like “Empire State Building 350 5th Ave, Floor 77 New York, NY 10118” to “latitude 40.7484284, longitude -73.9856546”.

Reverse Geocoding: is converting “latitude 40.7484284, longitude -73.9856546” to address “Empire State Building 350 5th Ave, Floor 77 New York, NY 10118”.
Now that you have seen how to do forward and reverse geocoding manually, let’s see how it can be done programmatically in PYTHON on larger dataset by calling some APIs.

Monday, March 25, 2019

Amazon EC2 - Using SFTP to download or upload files

In this post you will learn how to upload/download files to/from Amazon EC2 Instance using FileZilla and SFTP.

SFTP stands for: Secure Shell (SSH) File Transfer Protocol or Secure File Transfer Protocol.




Step 1:
The very first step is to download your AWS console key file in .pem and convert it to .ppk file.

The key file you download from AWS console will be a .pem file and filezilla doesn't read such a file. So you need to convert it to .ppk file. You can use PuttyGen (puttygen.exe) for this conversion from .pem to .ppk.


Step 2:
Download and install FileZilla client from here Download FileZilla.


Step 3:
Now, launch FileZilla and add your .ppk key file of your AWS instance for public key authentication as follow;

~ Go to Edit >> Settings



~ On the Settings dialog box, under 'Connection', select 'SFTP' then click on 'Add Key File...'. Navigate to where you saved your converted .ppk key file and upload it. Then click on 'Ok' button.



Step 4:
Lets login to the server. Go to File >> Site Manager



Step 5:
Click on 'New Site' button, under 'General' tab select 'SFTP' as the protocol and insert your instance username and password. Finally click on "Connect" button.



Host name is something like: ec2-21-xx-xx-xxx.compute-1.amazonaws.com
Username is: ec2-user
Password is optional.

You should now be connected as seen on the status...



Step 6:
You can now navigate your local site folders on the right side pane and upload to the remote site folder on the left pane. Similarly for the downloading of files.

Note: the files which are owned by root cannot be uploaded and downloaded. You will get permission denied error when you try to download a file owned by root.


So, if you want to download those files, you will use PuTTY to SSH into the machine and change the owner of that file to the normal user using the following command.

sudo chown user:user /folder/file

Replace "user:user" with the appropriate root user name.



That is it!
I hope you find it useful.