Thursday, July 6, 2017

Setting up virtual environment (virtualenv) for python app development

When working on fairly large app in python, it is a good practice to create/setup an isolated python environment (called virtual environment - virtualenv) for development.

There are many long term reason for setting up a python's virtual environment (virtualenv) for any project you are working on. The notable once include:
1) When you have to experiment with something new.
2) When you want to use two different versions of package and compare them.
3) Virtualenv really shines when you have a number of projects, and don't want them to all share the same Python installation. For example, you could have two project with conflicting requirements.
4) Virtualenv will let you define isolated Python runtime environments without harming the root or system python installation. This comes in handy when deploying several python applications and you want to isolate the different environments. Typically this is needed when runtime dependencies differ between frameworks or libraries in different applications.

Creating a virtual environment

Step 1: Create a folder where you want to have the python virtual environment
Step 2: Open CMD or terminal window in that folder
Step 3: Enter "virtualenv name_of_folder" on the cmd/terminal

At this point, you did have a virtual environment created.


Using a virtual environment

Step 1: Open CMD or terminal window in that folder
Step 2: Enter "folder_name\Scripts\activate"
Step 3: To deactivate, simply enter "deactivate"
Step 4: To install packages enter "pip install package_name"
Step 5: To view the install packages enter "pip freeze"
Step 6: To save the install packages to file, enter "pip freeze > file_name.txt"



That is it!

Wednesday, July 5, 2017

QGIS - Restore Missing Vector menu such as: Geometry Tools

Hello,

There is a bug in the 2.16.0 Nodebo version of QGIS. This bug hides some vector processing tools (GDAL tools) found under the "Vector" menu. As already reported here... on the QGIS issues tracking page. This is caused as a result of installing "processing tools" as an external plugin.



Here is how to fix it

To fix this, go to "Plugins" menu and search for "Processing" plugin and enable it.



That is it!
Happy mapping...:)

Thursday, June 22, 2017

How to make Custom Google Map of Nigeria displaying the state capitals

In this post, I will guide you to make a customized Google map displaying the capitals of all the states in Nigeria with additional information attached to each state capital.

I will also show you how to embed it to any html web page.

Let's get started...


Requirements
1) A Google account
2) CSV file containing the states information including longitude and latitude of the state capitals



Friday, May 26, 2017

Elshayal Smart GIS and Google Earth Compatibility Error [FIXED]

There has been serious complains on this error: *** Please be sure that Elshayal Smart GIS and Google Earth are opened and running as the same, either as adminstrators or as not adminstrators. ***

With this error, there are fear that the Elshayal software is not longer useful for downloading of free images from Google Earth. That is wrong, and here I am going to share with you the solution to fix this error.


This error is largely due to the usage of non-compatible versions of Elshayal Smart GIS and Google Earth.

At the moment of writing this post, the latest compatible versions of the software are:
Elshayal Smart GIS = 17.011
Google Earth = 7.1.7.2606








You can get the latest compatible versions from the official blog here.


Are you still having troubles? Feel free to let me know. I will help you download your images, contact me by email or phone: umaryusuf49@gmail.com or +2348039508010


Thanks for following

Sunday, May 14, 2017

How to setup a Geospatial Programming Blog - Served with "Static Website Generators"

How to setup a Geo-spatial Programming portfolio Blog that scales better than the traditional blogs

Blogging is one of the most important things that any aspiring GIS programmer or Geo-data scientist should be doing on a regular basis. It can be a fantastic way to demonstrate once skills and learn topics in more depth.

There are various software for creating your Geo-Data Science blog. The software that helps you create and publish your blog is called a Content Management System (CMS), Some are free and some cost money.

One common thing with CMS is that, they heavily depend on server to deliver contents. That is a request has to be sent to a database and responses are sent in return each time a use perform an action on the blog. This can be time consuming and over killing for a static geo-data content blog.

As a Geospatial programmer, if you really want to setup a GIS scripting oriented blog that scales very well. Then look beyond server centric CMS blog engines such as Blogger, Joomla, WordPress, Drupal etc. Instead choose a "Static Website Generators" that gives room for increasing both speed and uptime, and managing static sites with version control systems like Git means the process of creating and updating sites is highly efficient.

Static site generators allow a user to create HTML files by writing in a markup language and coding template files. A static website generator combines a markup language, such as Markdown or reStructuredText, with a templating engine such as Jinja, to produce HTML files.



Open-Source Static Site Generators

There are several of them, a comprehensive list is available at: www.staticgen.com

Since this blog is about python Programming, I will list those static site generators written in Python programming language below. Later on, I will cover how to create a blog using Python, how to create posts using Jupyter notebook, and how to deploy the blog live using Github Pages.


List of Python powered static site generators

1. Pelican - www.blog.getpelican.com/
2. MkDocs - www.mkdocs.org/
3. Cactus - www.github.com/koenbok/Cactus/
4. Lektor - www.getlektor.com/
5. Hyde - www.hyde.github.io/
6. Nikola - www.getnikola.com
7. Sphinx - www.sphinx-doc.org/
8. Frozen-Flask - www.pythonhosted.org/Frozen-Flask/
9. Blogofile - www.blogofile.com
10. Acrylamid - www.posativ.org/acrylamid/
11. Tinkerer - www.tinkerer.me
12. Tags - www.tags.brace.io/
13. staticjinja - www.staticjinja.readthedocs.org/en/latest/
14. Urubu - www.urubu.jandecaluwe.com
15. Grow - www.grow.io/
16. Tarbell - www.tarbell.io
17. wok - www.wok.mythmon.com/
18. Statik - www.getstatik.com/
19. QPage - www.qpage.ir
20. drupan - www.github.com/fallenhitokiri/drupan
21. Prosopopee - www.github.com/Psycojoker/prosopopee
22. PyKwiki - www.pykwiki.nullism.com
23. PieCrust2 - www.bolt80.com/piecrust
24. Poole - www.bitbucket.org/obensonne/poole/
25. Pagegen - www.pagegen.phnd.net
26. django-distill - www.github.com/mgrp/django-distill
27. Halwa - www.github.com/mhlakhani/halwa
28. Blended - www.jmroper.com/blended/
29. Blo - www.github.com/savuir/blo
30. YASBE - www.github.com/underr/yasbe
31. Bang - www.github.com/squdle/Bang
32. Rocadocs - www.rocadocs.com
33. BootDown - www.project.geekweaver.com/



Creating a Geo-Data Science portfolio blog will improved your career more than anything else and land you your dream job while doing the work you love.


Thanks for following.

Saturday, April 29, 2017

How do I do that in ArcGIS and QGIS - Learn both ArcGIS and QGIS Simultaneously

According to WikiPedia, there are two major categorization of Desktop GIS software namely:-
1- Commercial or Proprietary GIS software
2- Open Source GIS software

Over the years, the leading Desktop GIS software for the two categories above are ESRI ArcGIS and QGIS.

In this blog post, I will show you how to do basic GIS operations in both ArcGIS and QGIS. If you are interested in knowing the difference between the two software, here are some interesting articles on the internet I have found:-
a) QGIS v ArcGIS
b) 27 Differences Between ArcGIS and QGIS – The Most Epic GIS Software Battle in GIS History
c) ArcGIS vs QGIS – 10 Most Important Differences Between ArcGIS and QGIS
d) How does ESRI ArcGIS (proprietary) compare with Quantum GIS (QGIS) (open source)?
e) GIS lounge - QGIS versus ArcGIS

So, this post isn't about which is the best GIS software, instead it will address common questions in the form of "How do I do that in ArcGIS and QGIS". I will go over how you can perform some of the very basic GIS tasks in both ArcGIS and QGIS, and you will decide at the end which software approach is easier and better for you.

Along the way, you will be learning both ArcGIS and QGIS software simultaneously.

Lets get started...




Sunday, April 23, 2017

Installing wxPython on Python 3.x

Few years ago I wrote this blog post on "How to install wxPython (Phoenix) on Python 3.x". Since then, a lot has happened including the release of "wxPython 4.0.0a1"


So, today I decided to install the latest alfa release on my python 3 anaconda setup. I went to wx PyPi url downloaded the wheel file locally and ran: pip install wheel_file_name to install successfully as seen below.



I tested the installation, by importing wx and it work perfectly.



Congrats WX is now officially on python 3

Monday, April 10, 2017

Cleaning big data CSV file with Python

Hello there,

I know I can open a CSV file in a spreadsheet software like MS Excel or a text editor like NotePad++ and edit/clean its contents (by removing unwanted contents).

In a situation where by the CSV file is very big (big data - it contains millions of rows and columns), if you try to open such file in a spreadsheet software or a text editor there is possibility of memory error from the software you are using as seen in the image below. This excel error says: "File not loaded completely", the file appeared to be beyond MS excel capabilities.



This is where using a data oriented language such as Python comes in handy. Python pandas module can efficiently load/open and process such heavy duty big data files without much difficulties.

Saturday, April 1, 2017

Windows PostgreSQL installation Error - Unable to write inside TEMP environment variable path

Here is the solution to the "PostgreSQL installation Error - Unable to write inside TEMP environment variable path" on windows PC.

I downloaded the latest version of PgAdmin4 for PostgreSQL 9.6 from EnterpriseBD. When I tried to install it, I got this error massage: There has been an error, Unable to write inside TEMP environment variable path.




Solution to the Error


Now, to solve this error on your windows machine proceed as follow:-
1- Locate your windows sytem32 folder at: C:\Windows\System32



2- Inside the folder, find regedt32.exe file and run/open it


3- Navigate to this folder: HKEY_CURRENT_USER >> Software >> Microsoft >> Windows Script Host >> Settings



4- Right click on the "Enable" icon and select "Modify"



5- Change"Value Data" from 0 to 1 and click on "OK"



6- Close the Registry Editor and you are done.



Thanks for following.

Python break, continue and pass within Try... Except block

Hello there,

When using loops in python, there are situations when an external factor may influence the way your program runs.  You may want your program to exit a loop completely, skip part of a loop before continuing, or ignore that external factor. You can do these actions with break, continue, and pass statements respectively.

The "pass" statement is a no-operation (meaning that it doesn't do anything). The program just continues to the next statement, which is why you get an error.

The "break" statement exits the loops and continues running from the next statement immediately after the loop. In this case, there are no more statements, which is why your program terminates.

The "continue" statement restarts the loop but with the next item.

Sample Code

for letter in 'Python':
   if letter == 'h':
      break # continue pass
   print ('Current Letter :', letter)


Testing the above code for: break, continue and pass will yield different results as seen below:-





If we introduce a "Try ... Except" block and intentionally call an error, only pass will print the error statement at the end of the block statement.




Thanks for reading.

Sunday, March 26, 2017

Assigning single attribute to multi geometric features in QGIS

There are situations when you will want as many as possible geometric features within a shapefile layer to share the same attribute information within the attribute table.

This kind of scenario is commonly seen when representing Islands. The attribute data is the same, but the geometric representation of the Islands (in polygon) will be separated from each other to give way too water bodies.

As an example, am going to use a Nigerian state the has Islands (Bayelsa state). So, to digitize Bayelsa into wards polygons we have to deal with Islands. And here I have three polygons representing the same ward with ward_code 12 as seen below.

So, the task here is to merge or combine these same attributes into one and all the three polygons will share that same attribute information/row.



In QGIS, to make multiple polygons share the same attribute proceed like thus:-
1) First digitize the Islands polygons and assign the same attributes as usual
2) Go to Vector Menu >> Geometry Tools >> Singleparts to Multiparts


3) The next window will prompt you for the input layer (from a drop-down list) and the Unique ID field (Ward_Code) and you can specify an output file name. Select the appropriate information and click on run.


4) You should now have multi polygons sharing one row of your attribute table


Alternative method


Alternatively, you could use the "Merge Selected Features" command available in the Advanced Digitizing Toolbar as seen below:-


Select the polygons you want to merge there attributes and click on "Merge Selected Features" within the "Advanced Digitizing Toolbar", configure the attributes accordingly and click on "Ok".


That is it.
Thanks for following.

Wednesday, March 22, 2017

Using Python OpenCV to detect human face on Nigeria currency notes

Hello,

Below is a short video clip of a project am working on. It shows you about: Using Python OpenCV and Haar Cascade for Detecting human face on Nigeria currency notes




Happy viewing!

Saturday, March 11, 2017

How to merge multiple shapefile layers into one big shapefile layer

In QGIS, there are two common ways to merge multiple vector layers into one big vector shapefile layer. Namely:-

1) Using "Merge Vector Layers"
2) Using MMQGIS Plugin

I will only talk about the first option, since it is the default option made available in QGIS. For the second option, you will have to install the MMQGIS Plugin accordingly.


Merging multiple vector layers using "Merge Vector Layers"


This function is available under the menu Vector | Data management tool. You can merge all files in the input directory or select specific files in the input directory.






This algorithm combines two vector layer of the same geometry type into a single one. If attributes tables are different, the attribute table of the resulting layer will contain the attributes from both input layers.

The same applies as for MMQGIS Plugin: merged layers must all be of the same geometry type. Also, if the source layers have different attributes fields, the merged file will contain all fields, but with NULL values inserted when a source layer does not have a specific field.


Thanks for following.

Saturday, March 4, 2017

How to download Satellite or Google earth Images with High Resolution

Hey there,
Few years ago, I published a YouTube video demonstrating how to do this using El-Shayal GIS software.


Since then, many things have changed including OS etc. So today, I decided to add more options to the list.

There are multiple solutions for downloading a satellite image at high resolution. Some commonly used solutions are listed below:-

1- AutoGR software

This a powerful geometry coordinate and statistical software.




2- Map Puzzle software

If you want to download some map location from Google Maps, Bing Maps or any other map service, Map Puzzle is just what you need! It will allow you to save your map locations into high resolution images (jpg, gif, png, bmp, tiff). The only thing you need to do is to enter GPS coordinates or address by GeoCoding, hit "Download" and you are done! The map has been saved into to your computer. It works on Windows XP, Windows Vista, and Windows 7. Requires no installation and its FREE.




3- Elshayal software

Elshayal Smart is a freeware map editor app filed under mapping and made available by Elshayal Smart Web on Line Software for Windows.




4- SAS planet software

SASPlanet is a program designed for viewing and downloading high-resolution satellite imagery and conventional maps submitted by such services as Google Maps, DigitalGlobe, Kosmosnimki, Yandex.Maps, Yahoo! Maps, VirtualEarth, Gurtam, OpenStreetMap, eAtlas, Genshtab maps, iPhone maps, Navitel maps, Bings Maps (Bird's Eye) etc., but in contrast to all these services all downloaded images will remain on your computer and you will be able to view them, even without connecting to the internet. In addition to the satellite-based maps you can work with the political landscape, combined maps and maps of the Moon and Mars.




5- Google map downloader software

Google Maps Downloader helps users design custom maps by using geographical coordinates. Maps downloaded through this app can be arranged together to form one large map, or viewed individually. The Google Maps Downloader interface is practical and user-friendly.



6- Universal Maps Downloader

Universal Maps Downloader is a tool that help you to get small tile images from Google Maps, Bing maps, OpenStreet Maps, Yandex Maps, Ovi Maps, MapQuest Maps, etc. All downloaded small images are saved on the disk. You can view downloaded maps by Maps Viewer Or you can combine them into a big bmp, jpg or tif file by Maps Combiner, and also you can export all tiles into a MBTiles format database.




7- Global Mapper

Global Mapper is an affordable and easy-to-use GIS application that offers access to an unparalleled variety of spatial datasets and provides just the right level of functionality to satisfy both experienced GIS professionals and beginning users.




8- USGS Earth Explore





9- BigEast Map Downloader



There are definitely more than those mentioned above. If you know any that is worth knowing feel free to list it in the comment box below.


Thanks for reading

Thursday, February 16, 2017

How I was hacked and lost my 2017 freelance earnings

This was really painful, I was hacked and my freelance 2017 earnings were stolen away! I will share my experience and how to protect yourself from potential hacking attacks.

I work as freelancer and used Payoneer to collect my earnings. On the 14/02/2017 (coincidentally, on Valentine's day of 2017) a supposed potential client contacted me with a project request and sent me an MS document file supposedly to contain more detailed description about the project. But unfortunately, the client was a fake and he was just a hacker looking to collect innocent peoples password to there online bank accounts.



The MS document file he sent to me actually contains some VB macro scripts that runs in the background of a PC to extract/grab login credentials saved on the web browser.

Few hours after I opened the file, I got notification that my funds were been transferred out of my account  an unknown email account tied to payoneer and that was how it all happened.

Honestly it sucks to see a potential client turning out to be a hacker. See (image above) the message he sent to me and I was looking up to working with a legitimate buyer, but unfortunately I was wrong.

The hacker transferred or stole my funds to this payoneer email account at: baland.v@yandex.ru

I know my God will never allow him go free! I have reported this issue already to my God and other relevant authorities.


How to protect yourself from potential hacking attacks


1- Never save important logins details on your web browser, even if you are working from your personal computer. Device a way of saving/remembering your passwords that no script can read remotely.

2- Never trust any MS doc file that contains VB macros. If you most open such file make sure you are not connected to the internet or at least use a different trash PC to open it.

3- Always use anti virus to scan files you received (especially from first time clients) and change your login credentials regularly.


I hope this safes someone out there from being the next victim.
Stay safe!

Wednesday, February 1, 2017

Essential tools or Websites to help you understand, test and generate regular expressions

Hello there,

Recently, I was working on a project that requires me to search through bunch of text and identify some matched patterns within the text, this is what "regex" is good at. So I thought of finding a tool that will ease my way through the project and fortunately, there are many of such tools on the internet.

I wrote this post just to highlight and share with you some of the regex tools I have tested. Just in case you didn't know what regex is here is a quick intro....

RegEx written in full as "Regular Expression" is a pattern that consist of rules used to match a certain set of strings. Regular Expressions are an essential part of any programmer’s toolkit. They can be very handy when you need to identify, replace or modify text, words, patterns or characters.

RegEx is a concept that is found in any programming language. The syntax of regular expressions is the same for all programming and script languages.
Below, you will find tools that are primarily associated with a programming language while others are generic to all languages.

Here is the list:-


  • http://pythex.org
  • http://regexpal.com/
  • https://regex101.com
  • http://www.regextester.com
  • http://regexr.com
  • https://www.regexbuddy.com
  • http://regular-expressions.mobi
  • http://regexlib.com
  • https://regexone.com
  • http://www.regexter.com/
  • http://www.rubular.com
  • http://kodos.sourceforge.net
  • http://www.txt2re.com
  • http://buildregex.com
  • http://www.regexe.com
  • http://www.regextester.com
  • http://www.phpliveregex.com
  • http://regexstorm.net





Useful Tutorials:

1- What are the best resources for learning RegEx?
2- Essential Guide To Regular Expressions: Tools and Tutorials
3- Regular Expressions: 30 Useful Tools and Resources


Thanks for reading

Saturday, January 28, 2017

Understanding the system of land measurements in Nigeria

Let's talk about how plots of land are measured in Nigeria.

Just like in other parts of the world, in Nigeria landed properties  are either measured in Hectares, Acres, Meters or Feet. The value of these measurements are affected by  human and environmental factors.

For example 1-Hectare (1ha) of land in the heart of the federal capital Abuja will not be of the same value as 1-Hectare (1ha) of land located in the heart of any states capital.

Generally speaking, Systems of measurement in modern use include the metric system, the imperial system, and United States customary units.

Nigeria is among the countries which have officially adopted the metric system.

Metric system adoption map (In Green - Countries which have officially adopted the metric system)


That not withstanding, many Nigerians tend to utilize other system of measurements when referring to the size of land or parcels.


Hectare
A hectare (ha) is a metric unit of square measure, equal to 100 ares (2.471 acres or 10,000 square metres).


Acre
The acre (acre) is a unit of land area used in the imperial and US customary systems. It is defined as the area of 1 chain by 1 furlong (66 by 660 feet), which is exactly equal to 1⁄640 of a square mile, 43,560 square feet, approximately 4,047 m2, or about 40% of a hectare.


Meter
The metre (m) is a unit of length in the metric system, and is the base unit of length in the International System of Units (SI).


Feet
The feet "foot - singular" (ft) is a unit of length in the imperial and US customary systems of measurement. Since 1959, both units have been defined by international agreement as equivalent to 0.3048 meters exactly.


From above explanation, we can that there are two metric (ha and m) and tow imperial (arce and ft) systems of land measurement commonly used in Nigeria.


What Is The Size Of A Standard Plot Of Land In Nigeria?


The size vary from state to state. But generally, the size will likely be measured in feet as 50x50ft, 50x100ft or 100x100ft.

In relation to Nigeria landed property measurements, the above definitions means a plot can be and of these size:-

1) 50 x 100ft / 15.2 x 30.5m - 463.6sqm (which means there are 8.7 plots of land measuring 463.6sqm in an acre)

2) 60 x 120ft / 18.3 x 36.6m - 669.8sqm (which means there are 6 plots of land measuring 669.8sqm in an acre)

3) 100 x 100ft / 30.5 x 30.5m 930.25sqm (which means there are 4.3 plots of land measuring 930.25sqm in an acre)



Thanks for reading.

Sunday, January 22, 2017

Combining multiple tabular data files together (excel, csv, html)

Introduction


If you work with any of these files "excel, csv, or html" regularly, you have probably encounter a situation where you have data in same file type say excel for instance and you needed to merge or combine those data from multiple excel files into a one big master file.

The traditional approach to such task is to open up all the files and then copy and paste their contents into the master file. This approach isn't efficient as it is slow and prone to errors.

Lets use the tools in python programming that is much more powerful than the manual copying and pasting most people do in excel.

Thursday, January 19, 2017

Add Multiple map images into one PDF file per page

Bulk images to pdf Tutorial

In this post I will show you how to create an atlas like of Nigeria LGA maps. Let me explain better by using the map of Abuja as an example.

This is a simple script procedure to collect images from a folder and add into a pdf file with each per page.



Abuja (FCT) has six municipal area councils that is the equivalent of LGA for other states. Now I want to create a pdf file to contain or display the map of Abuja and the six municipal area councils with each map on a new page.

This task could be done manually, but since I intend to do the same for all the states, that is about 700+ maps for the whole LGAs. The process become tedious to accomplished, so I use python pdf package (called reportlab) to automate the process as follow.

The maps above are to be added to a single pdf file with each map on a new page. I use python glob module to read in the map images, use for loop to iterate the files and used python reportlab module to generate the pdf file.

The final code is as seen below:-


Note: the above script was written and tested on Python 2. You will need to modify some lines to run/use it with Python 3.

The resulting pdf and other files used for the script are here.


That is it!

Wednesday, January 11, 2017

GIS programming in "pure Python" Vs "GIS-software"

Hello there,
Thank you for stopping by to read about "Python GIS programming" on my blog!

Basically, Python GIS programming is either done in "pure Python" or "GIS-software". By saying "pure Python" I mean using python GIS modules/packages such as the once listed below. And by saying "GIS-software", I am referring to software that supports python GIS scripting such as ArcGIS - ArcPy, QGIS - PyQGIS, etc.


List of python GIS modules/packages:

  • GDAL –> Fundamental package for processing vector and raster data formats (many modules below depend on this). Used for raster processing.
  • Geopandas –> Working with geospatial data in Python made easier, combines the capabilities of pandas and shapely.
  • Shapely –> Python package for manipulation and analysis of planar geometric objects (based on widely deployed GEOS).
  • Fiona –> Reading and writing spatial data (alternative for geopandas).
  • Pyproj –> Performs cartographic transformations and geodetic computations (based on PROJ.4).
  • Pysal –> Library of spatial analysis functions written in Python.
  • Geopy –> Geocoding library: coordinates to address <-> address to coordinates.
  • GeoViews –> Interactive Maps for the web.
  • Networkx –> Network analysis and routing in Python (e.g. Dijkstra and A* -algorithms), see this post.
  • Cartopy –> Make drawing maps for data analysis and visualization as easy as possible.
  • Scipy.spatial –> Spatial algorithms and data structures.
  • Rtree –> Spatial indexing for Python for quick spatial lookups.
  • Rasterio –> Clean and fast and geospatial raster I/O for Python.
  • RSGISLib –> Remote Sensing and GIS Software Library for Python.

Tuesday, January 10, 2017

Wiki Map of Nigeria


This wiki map allows you to learn more about Nigeria states by clicking on the map you will be taken to Wikipedia page for that state. You will learn more on the wiki page!

Feel free to click and explore any of the states.


Happy viewing

Wednesday, January 4, 2017

How to make Maps of Nigeria - Country, States, LGA, and Towns

Hi,
I often come across this question on how to make a customized Map of Nigeria that shows the Country boundary, the States boundaries, the Local Government Area (LGA) boundaries, and Towns or Cities around the country?

To demonstrate what am referring to, below is an example of a customized map of Nigeria, map of Nasarawa state, Nasarawa state LGAs and some towns within the state:-