Assign color to vector layer based on HTML Notation (HEX color) codes in attribute table

 Here, I have a polygon layer with 'color' attribute column which contains HEX color codes as seen below.

We want to assign each polygon to its attribute color value, for example: Delta = #6808C9.

This is close to 'Categorized Symbology', but the only difference is that the colors are from the attribute column. To do this, we have to edit the 'Fill Color' expression to read the color column.

This way, each polygon is given a HEX color code that corresponds to the value on the attribute column.

Happy Mapping!

Map two data frames base on common column

 The aim is to enrich a dataframe from a second dataframe. In other words, copy corresponding cell in a dataframe onto a second dataframe based on their common columns.

This is very common task that is difficult to explain! This works like a "vLookUp" function found in MS excel. Where we have two tables and we want to update the second table using correspond values based on column that relates them.

If the two tables have same name for the relating/common column, then the merge() function can be used like so: -

df2.merge(df1, on="common_column_name", how='inner')

On the other hand, if the relating/common column have varying column names and you don't want rename them. The below example would be more flexible:-

We create dictionary from columns in df2 and map it to df1

import pandas as pd

df1 = pd.DataFrame({'lkey': ['foo', 'bar', 'baz'],
                    'value0': [1, 2, 3]})

df2 = pd.DataFrame({'rkey': ['foo', 'bar', 'baz'],
                    'value1': [5, 6, 7],
                   'value2': [8, 9, 10]})

# This is a list of columns in: df2
s = ['rkey', 'value1', 'value2'] # Don't want to create it manually? then use: s = df2.columns.tolist()

# loop through all columns of df1 and map to df2
for column in s:
    # make a mapping dictionary from df...
    mydict = dict(zip(df2['rkey'], df2[column]))
    
    # create new column based on the map column and dictionary
    df1['new_'+column] = df1['lkey'].map(mydict)

df1

Here below is a real world example:-

Below are two tables with different attribute columns from countries in the world. The 'Country Name' and 'Country Code' columns are the same for both tables, so we need to merge them together.

Since, the two table have the same column names we can use this approach...

df1.merge(df2, on="Country Name", how='outer')

In case, the result isn't what you expected. Then play around with the parameters to get what you wanted.

Enjoy!

QGIS - Solve Launch alert on "python modules numpy and matplotlib both not installed"

When you start QGIS and got a screen start error that reads:-

The contour plugin is disabled as it requires python modules numpy and matplotlib which are not both installed.

This is custom error caused by a deprecated matplolib import command "from matplotlib.mlab import griddata". You can click on 'Ok' button to continue loading the QGIS software.

However, clicking on that button is somewhat annoying and you will definitely like to fix the error permanently.

The error comes from the 'contour' plugin that was installed in the plugins directory at: 
C:\Users\YourPcName\AppData\Roaming\QGIS\QGIS3\profiles\default\python\plugins\contour

Open the python file "ContourDialog.py" in your text editor. You would find and comment out the import statement "from matplotlib.mlab import griddata" and save the file.

This should fix the warning error massage from launching the software when you restart it.

The warning message box that pops up above is actually located in this same file somewhere below.

That is it!

Creating Zip file for items within multiple folders

 The script below will access all children folders within a parent directory then zip the contents and return back to the parent directory before moving to the next child folder.

The process continue until all children folders are processed.

# list of folders to zip their contents...
folders = glob.glob('*')

# Change dir to the first folder, zip its contents and return back to parent dir...
for folder in folders:
    # Change directory to folder in list of folders...
    os.chdir(folder)
    
    list_files = glob.glob('*')
    with zipfile.ZipFile(folder+'.zip', 'w') as zipF:
        for file in list_files:
            zipF.write(file, compress_type=zipfile.ZIP_DEFLATED)
    
    # Change directory back to main folder...
    os.chdir(r"C:\Users\Yusuf_08039508010\Documents\DandR")

print('Done...')

Note that the script uses the folder's name to name the resulting zip file as seen below.

That is it!

Count Numbers of Rows from bulk excel files

Here I got many excel files in a folder and I wanted to know the number of rows in each file.

import glob
import pandas as pd

files = glob.glob("C:\\Users\\Yusuf_08039508010\\Desktop\\LinkedIn Profile Scrapping\\Old Project\\Second Project\\New folder\\*.xlsx")

# Get file names
file_names = [os.path.basename(x) for x in files]
# ---------------------

count_list = []
i = 0

for f in files:
    df = pd.read_excel(f)
    # count rows based on column index...
    rows_count = df.count()[0] # this will not count NaN

    data = file_names[i], rows_count # Combine filename and row count into a dataframe
    
    count_list.append(data)
    
    i = i+1

df_count = pd.DataFrame(count_list, columns=['File', 'Count'])

df_count

 This will return a table with each files name and corresponding number of rows in the file as seen below.

Enjoy!

Python - Separating photos into subfolders

Read Images in a Folder and Move to Sub folder Based on given Condition

Few weeks ago on this blog post, we downloaded pictures of all the US presidents. On this post, we want to separate the photos into subfolders depending on whether the president is from 'Democratic' or 'Republican'. 

If this sounds interesting to you, then keep reading!

Google Apps Script (GAS) - Make GoogleForms from GoogleSheets

On this post, I will demonstrate how tom make a GoogleForms from GoogleSheets using Google Apps Script (GAS). There are several advantages of using GAS to automate your form creation over manual procedure.

GAS is a Googles owned scripting language that is flavor of a modern JavaScript, so if you already know javascript, then you should be confortable with GAS. With GAS, you can do more with Google apps — like GoogleCalendar, GoogleDocs, GoogleDrive, GoogleGmail, GoogleSheets, and GoogleSlides.

The form we will make is a multi choice question, and the obvious reason why we will use GAS to make it is because it has a lot questions to be added. Hence, it makes sense to prepare the questions and there option in a GoogleSheets, then generate the form from there. That is the approach we are going to take.

Create a GoogleSheet where the first column A is the question and the columns B, C and D are the answer options (only one option is the correct answer) as seen below.

Pandas Dataframe to JSON

 Pandas has a function (.to_json()) that takes a series/dataframe and converts it into JSON object.

It has an interesting parameter (orient) that allow different orientations of the JSON string format. Allowed values are: {'split','records','index','columns','values','table'}. Let see how it works.

Assuming we have the table below in out dataframe and now we want it as a JSON data for use on another web platform (JSON work better on web than tabular data).

As usual, we read the data table into a dataframe variable (here it is called: json_df).

Then the .to_json() function is called on the dataframe to convert it to JSON like this:-

json_df.to_json()

There are many parameters we can pass into the function such as path to save the JSON and orient to orient the JSON in different formats. Below, we will explore the various orient options available.

As you can see, each orient value has its own different JSON format or structure.

print(json_df.to_json(orient='columns'))
output = {"Question":{"0":"In what country is the city 'Tokyo'?","1":"In what country is the city 'Delhi'?","2":"In what country is the city 'Cairo'?","3":"In what country is the city 'Chongqing'?","4":"In what country is the city 'Orlando'?","5":"In what country is the city 'Abuja'?"},"Option 1":{"0":"Japan","1":"India","2":"Argentina","3":"Philippines","4":"Mexico","5":"Spain"},"Option 2":{"0":"India","1":"United States","2":"Bangladesh","3":"Pakistan","4":"Angola","5":"Nigeria"},"Option 3":{"0":"Brazil","1":"China","2":"Egypt","3":"China","4":"United States","5":"Canada"}}


print(json_df.to_json(orient='split'))
output = {"columns":["Question","Option 1","Option 2","Option 3"],"index":[0,1,2,3,4,5],"data":[["In what country is the city 'Tokyo'?","Japan","India","Brazil"],["In what country is the city 'Delhi'?","India","United States","China"],["In what country is the city 'Cairo'?","Argentina","Bangladesh","Egypt"],["In what country is the city 'Chongqing'?","Philippines","Pakistan","China"],["In what country is the city 'Orlando'?","Mexico","Angola","United States"],["In what country is the city 'Abuja'?","Spain","Nigeria","Canada"]]}


print(json_df.to_json(orient='records'))
output = [{"Question":"In what country is the city 'Tokyo'?","Option 1":"Japan","Option 2":"India","Option 3":"Brazil"},{"Question":"In what country is the city 'Delhi'?","Option 1":"India","Option 2":"United States","Option 3":"China"},{"Question":"In what country is the city 'Cairo'?","Option 1":"Argentina","Option 2":"Bangladesh","Option 3":"Egypt"},{"Question":"In what country is the city 'Chongqing'?","Option 1":"Philippines","Option 2":"Pakistan","Option 3":"China"},{"Question":"In what country is the city 'Orlando'?","Option 1":"Mexico","Option 2":"Angola","Option 3":"United States"},{"Question":"In what country is the city 'Abuja'?","Option 1":"Spain","Option 2":"Nigeria","Option 3":"Canada"}]


print(json_df.to_json(orient='values'))
output = [["In what country is the city 'Tokyo'?","Japan","India","Brazil"],["In what country is the city 'Delhi'?","India","United States","China"],["In what country is the city 'Cairo'?","Argentina","Bangladesh","Egypt"],["In what country is the city 'Chongqing'?","Philippines","Pakistan","China"],["In what country is the city 'Orlando'?","Mexico","Angola","United States"],["In what country is the city 'Abuja'?","Spain","Nigeria","Canada"]]


print(json_df.to_json(orient='table'))
output = {"schema": {"fields":[{"name":"index","type":"integer"},{"name":"Question","type":"string"},{"name":"Option 1","type":"string"},{"name":"Option 2","type":"string"},{"name":"Option 3","type":"string"}],"primaryKey":["index"],"pandas_version":"0.20.0"}, "data": [{"index":0,"Question":"In what country is the city 'Tokyo'?","Option 1":"Japan","Option 2":"India","Option 3":"Brazil"},{"index":1,"Question":"In what country is the city 'Delhi'?","Option 1":"India","Option 2":"United States","Option 3":"China"},{"index":2,"Question":"In what country is the city 'Cairo'?","Option 1":"Argentina","Option 2":"Bangladesh","Option 3":"Egypt"},{"index":3,"Question":"In what country is the city 'Chongqing'?","Option 1":"Philippines","Option 2":"Pakistan","Option 3":"China"},{"index":4,"Question":"In what country is the city 'Orlando'?","Option 1":"Mexico","Option 2":"Angola","Option 3":"United States"},{"index":5,"Question":"In what country is the city 'Abuja'?","Option 1":"Spain","Option 2":"Nigeria","Option 3":"Canada"}]}


print(json_df.to_json(orient='index'))
output = {"0":{"Question":"In what country is the city 'Tokyo'?","Option 1":"Japan","Option 2":"India","Option 3":"Brazil"},"1":{"Question":"In what country is the city 'Delhi'?","Option 1":"India","Option 2":"United States","Option 3":"China"},"2":{"Question":"In what country is the city 'Cairo'?","Option 1":"Argentina","Option 2":"Bangladesh","Option 3":"Egypt"},"3":{"Question":"In what country is the city 'Chongqing'?","Option 1":"Philippines","Option 2":"Pakistan","Option 3":"China"},"4":{"Question":"In what country is the city 'Orlando'?","Option 1":"Mexico","Option 2":"Angola","Option 3":"United States"},"5":{"Question":"In what country is the city 'Abuja'?","Option 1":"Spain","Option 2":"Nigeria","Option 3":"Canada"}}

The structure you will adopt will depend on where you want to use it. Usually, you would find sample JSON structure from the platform you intend to use. Compare the option and adopt the one that suit your project.

That is it!

Pandas dataframe.append() function - Append 'Jeo Biden' to US presidents dataframe

Here we got a dataframe of 45 USA presidents with four columns as seen below.

The dataframe is this wikipedia table that lists the presidents of the United States. As you already saw, the wikipedia table  has been updated. So, we need to update our dataframe with 46th president.

Now we got a new president-elect (Joseph Robinette Biden Jr.), we will create a new dataframe for him and use the append() function to add it to the end of our dataframe.

To create the Biden's dataframe, we store his data in a dictionary and use it to make a one row dataframe to be appended to the main dataframe above.

biden_data = {
            'Name':['Joseph Robinette Biden Jr.'], 
              'Image':['https://upload.wikimedia.org/wikipedia/commons/9/99/Joe_Biden_official_portrait_2013_cropped_%28cropped%29.jpg'], 
              'Party':['Democratic'], 
              'Presidency Period':['20-Jan-21 – Incumbent']
             }

biden_df = pd.DataFrame(biden_data)
biden_df

Call the append function on the USA presidents dataframe like this:- You'll notice that the index is '0' for the total line. We want to change that from '0' to '46' using rename.

us_presidents_df.append(biden_df).rename(index={0:"45"})

That is it!

Mass make folders from list of states

 I have written on this before at How to make Multiple Folders/Directories in Python. Refer to that page for more detailed instructions.

Anyway, here I have a list of states in Nigeria that I want to generate folder for each.

import os
states = ['Ebonyi', 'Edo', 'Ekiti', 'Enugu', 'Fct', 'Gombe', 'Imo', 'Jigawa', 'Kaduna', 'Katsina', 'Kebbi', 'Kogi', 'Kwara', 'Lagos', 'Nasarawa', 'Niger', 'Ogun', 'Ondo', 'Osun', 'Akwa Ibom', 'Oyo', 'Plateau', 'Rivers', 'Sokoto', 'Taraba', 'Yobe', 'Zamfara', 'Anambra', 'Bauchi', 'Bayelsa', 'Benue', 'Borno', 'Cross River', 'Delta']

for n in states: # Looping over the individual state to make folder name
    if type(n) == str:
        print ('Making directory for.... ', n)

        directory = n + '__state'
        if not os.path.exists(directory):
            os.makedirs(directory)

The resulting output is seen below:-

Enjoy!