Tuesday, July 5, 2022

How to execute a string of code as a code in Python

 Lets say you got a string representation of a python syntax that you actually want to execute/run not just display it on the console, how do you execute/run such a string code?

print( "print('Hello World')" )

Above code will print print('Hello World') instead of 'Hello World'. Also the code below will display len('Hello World') instead length of the characters as 11.

print( "len('Hello World')" )

There are two function that can help you get the task done. They are the exec and the eval functions. See difference between exec and eval here


# Using exec function
exec( "print('Hello World')" )
exec( '''print( "print('Hello World')" )''' )

# Using eval function
eval( "print('Hello World')" )
eval( '''print( "print('Hello World')" )''' )


print( "len('Hello World')" ) # Using exec function exec( "len('Hello World')" ) exec( '''len( "print('Hello World')" )''' ) # Using eval function eval( "len('Hello World')" ) eval( '''len( "print('Hello World')" )''' )

I found this very useful when I want to evaluate a dynamically generated Python expression.
Happy coding.

Friday, July 1, 2022

Python GIS Data Wrangling - U.S. Drought Monitor

 This post was inspired by John Nelson YouTube video on "How to Make This Drought Map Pt 1: DATA WRANGLING", where he manually wrangled the dataset for the year 2018.

What he did was great if you are just doing it for a single year. If you intend to repeat the workflow for several years, then the process can be time consuming and prone to mistakes. For this reason, I will recreate the workflow using python scripting and the whole process can be automated with few button clicks.


More specifically, I will cover the following processes:-

  1. Download and extract the zip folder
  2. Combine the shapefiles into a single folder
  3. Merge the shapefiles into shapefile


Lets get started.

Sunday, June 26, 2022

QGIS Select by attribute - specific character or text string

 Lets say you want to perfect attribute selection where the entry contains a specific word or letter, this can get complicated if the word is not alone in the table cell.

There are couple of ways to solve this kind of query. The most advance way is using regular expression, I have written about regular expression on the page titled: Working with Regular Expression in QGIS

An easier approach would be to use the LIKE operator together with the % wildcard symbol. For example, lets say you want to search for ward names that contain the letter "o", the expression will look like this:-

"ward_name" LIKE '%o%'

You just need to add the text string you want to search for in between %...% as seen below:-

"ward_name" LIKE '%Add Something To Search%'

That is it!

Saturday, June 25, 2022

Useful code snippets to Clip polygon with another polygon and Convert Vector file in Geopackage to Shapefile in QGIS

 Clip polygon with another polygon using processing algorithm in QGIS

# Clip polygon with another polygon using processing algorithm in QGIS

ovly = 'C:\\Users\\Yusuf_08039508010\\Desktop\\Working_Files\\...\\Name_Bouganville Lower - N.gpkg'

outfile = ovly.split('\\')[-1].replace('Name_', '').replace('.gpkg', '')
print('Processing...', outfile)

parameters = {'INPUT':'C:\\Users\\Yusuf_08039508010\\Desktop\\Working_Files\\...\\Results\\SHP\\Woody_02.shp',
'OVERLAY':ovly,
'OUTPUT':f'C:/Users/Yusuf_08039508010/Desktop/Working_Files/.../Results/SHP/Study Location Individual Files/SHP with Holes/{outfile}.shp'}

processing.run("native:clip", parameters)



# -----------------------------------------------
# Bulk Clip polygon with another polygon using processing algorithm in QGIS

import glob

overlay_file = glob.glob('C:\\Users\\Yusuf_08039508010\\Desktop\\Working_Files\\Fiverr\\2022\\06-June\\Masking None Vegetation Ground Covers\\Results\\SHP\\Study Location Individual Files\\*.gpkg')

failed = []
for ovly in overlay_file:
    try:
        outfile = ovly.split('\\')[-1].replace('Name_', '').replace('.gpkg', '')
        print('Processing...', outfile)

        parameters = {'INPUT':'C:\\Users\\Yusuf_08039508010\\Desktop\\Working_Files\\Fiverr\\2022\\06-June\\Masking None Vegetation Ground Covers\\Results\\SHP\\Woody_02.shp',
        'OVERLAY':ovly,
        'OUTPUT':f'C:/Users/Yusuf_08039508010/Desktop/Working_Files/Fiverr/2022/06-June/Masking None Vegetation Ground Covers/Results/SHP/Study Location Individual Files/SHP with Holes/{outfile}.shp'}

        processing.run("native:clip", parameters)
    except Exception:
        failed.append(ovly)

print('Done...')



Convert Vector file in Geopackage to Shapefile

# Convert Vector file in Geopackage to Shapefile


gpkg = 'C:\\Users\\Yusuf_08039508010\\Desktop\\Working_Files\\...\\Name_Bouganville Lower - N.gpkg'

# Read vector layer and test it is valid....
input_file = QgsVectorLayer(gpkg, "polygon", "ogr")
input_file.isValid()

# Construct output file name...
out_filename = gpkg.split('\\')[-1].replace('.gpkg', '')

# Write the file to disc...
out_folder = r"C:\Users\Yusuf_08039508010\Desktop\Working_Files\...\Results\SHP\Study Location Individual Files\Failed files"
QgsVectorFileWriter.writeAsVectorFormat(input_file, f"{out_folder}\\{out_filename}.shp", "UTF-8", input_file.crs(), "ESRI Shapefile")
print('Done...') # ----------------------------------------------------------- # Bulk Convert Vector file in Geopackage to Shapefile import glob gpkg_folder = r"C:\Users\Yusuf_08039508010\Desktop\Working_Files\Fiverr\2022\06-June\Masking None Vegetation Ground Covers\Results\Deliverables\Shp\gpk" gpkg_files = glob.glob(f'{gpkg_folder}\\*.gpkg') for gpkg in gpkg_files: # Read vector layer and test it is valid.... input_file = QgsVectorLayer(gpkg, "polygon", "ogr") input_file.isValid() # Construct output file name... out_filename = gpkg.split('\\')[-1].split('_')[-1].replace('.gpkg', '') # Write the file to disc... out_folder = r"C:\Users\Yusuf_08039508010\Desktop\Working_Files\Fiverr\2022\06-June\Masking None Vegetation Ground Covers\Results\Deliverables\Shp" QgsVectorFileWriter.writeAsVectorFormat(input_file, f"{out_folder}\\{out_filename}.shp", "UTF-8", input_file.crs(), "ESRI Shapefile") print('Done...')


Friday, June 24, 2022

Extracting 'Standardized Precipitation-Evapotranspiration Index (SPEI)' data from netCDF file into CSV spreadsheet

 In this post, we shall learn how to use python to extract netCDF data into CSV.


What is NetCDF?

NetCDF stands for "network Common Data Form" and it is a file format for storing multidimensional scientific data (variables) such as temperature, humidity, pressure, wind speed, precipitation and direction. There are several versions of netCDF and the version we are using here is version 4 (netCDF4).


The Dataset

The dataset am going to use is the Global 01-month 1901-2020 'Standardized Precipitation-Evapotranspiration Index (SPEI )' provided by Digital.csis



Read more details on this dataset on: SPEI Global Drought Monitor Database

How to open netCDF dataset

The netCDF data can be visualized using QGIS software. This will give us a quick overview as to what variables and attributes are contained in the file.


Some GIS packages allow reading netCDF data, such as QGIS, ArcGIS and IDRISI Taiga. There are other netCDF viewers, such as Panoply (developed in Java), ncBrowse, ncview, and nCDF_Browser.

Tuesday, May 17, 2022

Arcpy and python scripts for Arcmap desktop GIS software

 Listed below are some scripts in python/acrpy for performing various useful tasks in Arcmap/ArcGIS desktop software.


Selected and save shapefile based on attribute

# Code to save selected shp based on attribute...

import arcpy

# Path to input shp...
nig_shp = r"C:\Users\Yusuf_08039508010\Desktop\...\Data\New Data from Scratch\data\NIG_ADM.shp"

# Path to output shp...
out_put_path = r"C:\Users\Yusuf_08039508010\Desktop\...\Data\New Data from Scratch\data\out\nez_3.shp"

# Select and save polygons where "Weight"=5...
arcpy.Select_analysis(nig_shp, out_put_path, '"Weight"=5')

# Select and save polygons where "geographic"='NEZ'...
arcpy.Select_analysis(nig_shp, out_put_path, '"geographic"=\'NEZ\'')


Create new attribute field in bulk

# Create new attribute field

import arcpy


shp = r"C:\Users\Yusuf_08039508010\Desktop\...\Data\New Data from Scratch\data\NIG_ADM.shp"

field_name = 'Yr_2025'
field_type = 'Integer'

arcpy.AddField_management(shp, field_name, field_type)

print('New field has been added...')



#Exercise: Modify the arcpy code snippet above to create attribute fields from years from 1980 to 2020.

# Solution....
for year in range(1980, 2021):
        print('Processing....', year)

        field_name = 'Yr_{}'.format(year)
        field_type = 'Integer'

        arcpy.AddField_management(shp, field_name, field_type)
# Another version of bulk creation of attribute fields...
import arcpy

for x in range(1980, 1990):
    print('Creating field...', x)

    field_name = 'AA_{}'.format(x)
    field_type = 'Integer'

    arcpy.AddField_management('NIG_ADM', field_name, field_type)


Buffer at multiple distances

# Buffer at multiple distance

import arcpy

my_buffer_dist = [5000, 8000, 11000, 14000, 17000, 20000, 23000, 26000, 29000, 32000]

for distance in my_buffer_dist:
    print('Processing...', distance)

    out_file_name = "Buffer_{}m".format(distance)

    arcpy.Buffer_analysis("Lagos_to_Kano", out_file_name, distance)


Select features from provided list

import arcpy
from arcpy import env


# Get number of features in shp
feature_count = int( arcpy.GetCount_management(nig_shp).getOutput(0) )

# Define custome workspace directory...
env.workspace

states = ['Abia', 'Adamawa', 'Akwa Ibom', 'Anambra', 'Bauchi', 'Bayelsa', 'Benue', 'Borno', 'Cross River', 'Delta', 'Ebonyi', 'Edo', 'Ekiti', 'Enugu', 'Abuja', 'Gombe', 'Imo', 'Jigawa', 'Kaduna', 'Kano', 'Katsina', 'Kebbi', 'Kogi', 'Kwara', 'Lagos', 'Nasarawa', 'Niger', 'Ogun', 'Ondo', 'Osun', 'Oyo', 'Plateau', 'Rivers', 'Sokoto', 'Taraba', 'Yobe', 'Zamfara']

out_folder = r"C:\Users\Yusuf_08039508010\Desktop\...\Data\New Data from Scratch\data\out"

for f in states:
    print('Processing object...', f)
    out_put_path = out_folder + '\{}.shp'.format(f)
    arcpy.Select_analysis(nig_shp, out_put_path, '"state_name"=\'{}.format(f)\'')


Label based on column with varying font size and color

# Label based on column with varying font size and color (to be used in label EXPRESSION)
def FindLabel ( [state_name] ): len_of_name = len( [state_name] ) if len_of_name < 6: return " <CLR red='255' green='0' blue='0'> <FNT size = '{}'> {} </FNT> </CLR> ".format( len_of_name, [state_name] ) elif len_of_name >= 6: return " <CLR red='0' green='0' blue='255'> <FNT size = '{}'> {} </FNT> </CLR> ".format( len_of_name, [state_name] )


Multiline label with varying font size

# Multiline label with varying font size (to be used in label EXPRESSION)

def FindLabel ( [state_name], [Weight] ):
  return [state_name] +  "\n"  +  "<FNT size = '14'> {} </FNT>".format([Weight])



Return random value from a list into cell of an attribute field
# Return random value from a list (to be used in field Calculator)

import random
meters_per_hr = [30000, 35000, 40000]

def random_km():
    return random.choice(meters_per_hr)




Hope you find it useful for you next project.

Monday, January 31, 2022

LeafletJS Vs Python Folium Web map

 In this post, I will show how various components are made in both LeafletJS and Python Folium.


First map: Initialize a map with center, zoom and openstreetmap background



LeafletJS
<!DOCTYPE html>
<html>
<head>
	<script src="https://cdnjs.cloudflare.com/ajax/libs/leaflet/1.0.0-beta.2.rc.2/leaflet.js"></script>
	<link href="https://cdnjs.cloudflare.com/ajax/libs/leaflet/1.0.0-beta.2.rc.2/leaflet.css" rel="stylesheet" />
	<script src="https://cdnjs.cloudflare.com/ajax/libs/leaflet.draw/0.2.3/leaflet.draw.js"></script>

	<link href="https://cdnjs.cloudflare.com/ajax/libs/leaflet.draw/0.2.3/leaflet.draw.css" rel="stylesheet" />

	<meta charset="utf-8">
	<title>Web map....</title>
</head>

<style type="text/css">
	html, body, #map { margin: 0; height: 100%; width: 100%; }
</style>


<body>



  <div id='map'></div>



  <script>
  	// center of the map
	var center = [8.242, 7.671];

	// Create the map
	var map = L.map('map').setView(center, 7);

	// Set up the OSM layer
	L.tileLayer(
	  'https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png', {
	    attribution: 'Data © <a href="http://osm.org/copyright">OpenStreetMap</a>',
	    maxZoom: 18
	  }).addTo(map);





  </script>

</body>
</html>


Folium

import folium

# initialize a map with center, zoom and openstreetmap background...
mapObj = folium.Map(location=[8.242, 7.671],
                     zoom_start=7, tiles='openstreetmap')


mapObj






Draw point



LeafletJS
l


Folium

f






point

Draw line



LeafletJS
l


Folium

f






point

Draw polygon



LeafletJS
l


Folium

f






point

Plot geojson data



LeafletJS
l


Folium

f






point

Add layer control



LeafletJS
l


Folium

f






point

Add HTML



LeafletJS
l


Folium

f









Tuesday, January 18, 2022

Geopandas Vs Folium - Generate Web map from data

 The code snippet below will demonstrate how to create an interactive choropleth web map using Geopandas and Folium libraries.


The latest version of Geopandas has the explore() method which can create a leafletjs map as seen above. 


import geopandas as gpd

# Read shp...
gdf = gpd.read_file(r"NGA_adm1.shp")

# Create web map obj...
mymap = gdf.explore(column='geographic')

# Save to file...
mymap.save('map.html')







import folium
import geopandas as gpd

zones = {'NEZ':1, 'SEZ':2, 'SSZ':3, 'SWZ':4, 'NCZ':5, 'NWZ':6}

# Read shp...
gdf = gpd.read_file(r"NGA_adm1.shp")

gdf.reset_index(level=0, inplace=True)
gdf['Weight'] = gdf['geographic'].map(zones)
gdf['index'] = gdf['index'].apply( lambda x: str(x) )

# Create folium map obj...
mymap = folium.Map(location=[8.67, 7.22], zoom_start=6)

folium.Choropleth(
    geo_data=geo_json_str, 
    data=gdf,
    name = 'Choropleth Map',
    columns = ['index','Weight', 'state_name'],
    key_on = 'feature.id',
    fill_color = 'YlGnBu', # RdYlGn
    legend_name = 'Name of Legend...',
    smooth_factor=  0
    
    ).add_to(mymap)

mymap










Sunday, January 9, 2022

Keeping track on some favorite developers websites

 There are many developer authors who publish useful content on their blog on a regular basis.

As a learning fan, it is a great idea to use the skills you learnt from them to keep track of what is new on their blogs.

The two most common ways for achieving this are API and Scrapping. So, you will research if the author's blog has API service and in the case where it doesn't exist then you will think about using web scraping.

The authors I want to lookup in this post are: Renan MouraWilliam Vincent and Flavio Copes

As at the time of writing, the above authors don't have an API implemented on their respective websites, so we will use web scraping to keep track of the latest post on their blogs. So, basically we will write a scrapper to store the data in a file then compare it with feature scraped data to get the latest or newest entries on the blogs.

There are several libraries for scraping websites, here I will use python requests/selenium, beautifulsoup and pandas to get the job done.


Let's get started...


1- Renan Moura


From Renan Moura's  blog, I will like to keep track of the following post variables: Category, title, title url, published date and updated date.

Using requests library, I got "406 Not Acceptable client error response". Which means that there is a bot manager on the server where the website is hosted that prevents bots from accessing the website. To overcome this, we can either use request with a user-agent or selenium to access this website.

import requests
import pandas as pd
from bs4 import BeautifulSoup

url = 'https://renanmf.com'
# Get user-agent from: http://www.useragentstring.com/
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}

response = requests.get(url, headers=headers)
html = response.text

soup = BeautifulSoup(html, 'html.parser')
article = soup.find_all("div", {'class':'card-content'})

print(len(article))


import requests
import pandas as pd
from bs4 import BeautifulSoup
from selenium import webdriver


url = 'https://renanmf.com'

driver = webdriver.Chrome('chromedriver.exe')
driver.get(url)

html = driver.page_source


soup = BeautifulSoup(html, 'html.parser')
article = soup.find_all("div", {'class':'card-content'})

print(len(article))


From any of the methods above, we can now loop through the columns we wanted as seen below:-

data_list = []
for art in article:
    category = art.find("li", {'class':'meta-categories'}).text
    title_txt = art.find("h2", {'class':'entry-title'}).text
    title_link = art.find("h2", {'class':'entry-title'}).find('a')['href']
    pub_date = art.find("li", {'class':'meta-date'}).text
    updated_date = art.find("li", {'class':'meta-updated-date'}).text
    
    data = category, title_txt, title_link, pub_date, updated_date
    
    data_list.append(data)

# ------------------------
data_list_df = pd.DataFrame(data_list, columns=['Category', 'Title', 'Title URL', 'Published Date', 'Updated Date'])




2- William Vincent


Here, we will get the following post variable: title, title url and published date

import requests
import pandas as pd
from bs4 import BeautifulSoup


url = 'https://wsvincent.com/'

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36'}

response = requests.get(url, headers=headers)
html = response.text

soup = BeautifulSoup(html, 'html.parser')
article = soup.find_all("li")

# ------------------------


data_list = []

for art in article:
    title = art.find('h2').text
    title_link = art.find('h2').find('a')['href']
    pub_date = art.find('span', {'class':'post-meta'}).text
    
    data = title, title_link, pub_date
    
    data_list.append(data)
    
# ------------------------

    
data_list_df = pd.DataFrame(data_list, columns=['Title', 'Title URL', 'Published Date'])




3- Flavio Copes


Flavio's blog is similar to William Vincent above, we will get the following post variable: title, title url and published date.


url = 'https://flaviocopes.com'

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36'}

response = requests.get(url, headers=headers)
html = response.text

soup = BeautifulSoup(html, 'html.parser')
article = soup.find_all("li", {'class':'post-stub'})
# ---------------

data_list = []

for art in article:
    title = art.find('h4').text
    title_link = art.find('a')['href']
    pub_date = art.find("time", {'class':'post-stub-date'}).text
    
    data = title, title_link, pub_date
    
    data_list.append(data)
    

    
data_list_df = pd.DataFrame(data_list, columns=['Title', 'Title URL', 'Published Date'])

data_list_df    




Happy scrapping!

Saturday, January 1, 2022

Make a WordCloud in Python

 Here is how to make something like this image below in python with less than ten lines of code. It is called "WordCloud" and it is a visual representations of words that give greater prominence to words that appear more frequently.


You need to install WordCloud and MatPlotLib libraries to run the code blow.

Make a list of text you want to use for the word cloud and generate it as seen below.

# Libraries
%matplotlib notebook
from wordcloud import WordCloud
import matplotlib.pyplot as plt
 
# Create a list of word
text=("Umar Umar Umar Matplotlib Matplotlib Seaborn Network Plot Violin Chart Pandas Datascience Wordcloud Spider Radar Parrallel Alpha Color Brewer Density Scatter Barplot Barplot Boxplot Violinplot Treemap Stacked Area Chart Chart Visualization Dataviz Donut Pie Time-Series Wordcloud Wordcloud Sankey Bubble")
 
# Create the wordcloud object
wordcloud = WordCloud(width=480, height=480, margin=0).generate(text)
 
# Display the generated image:
plt.imshow(wordcloud, interpolation='bilinear')
plt.axis("off")
plt.margins(x=0, y=0)
plt.show()

That is it!