Saturday, May 15, 2021

Python Django Setup Template

Here listed below are code snippets for creating a "PROJECT_NAME" Django Project with "APP_NAME" App.

1) Initial Set Up

  • mkdir My_Django_Folder
  • cd My_Django_Folder
  • pipenv install django
  • pipenv shell
  • django-admin startproject PROJECT_FOLDER_NAME .
  • python manage.py migrate
  • python manage.py createsuperuser
  • python manage.py startapp APP_FOLDER_NAME
  • python manage.py runserver

mkdir My_Django_Folder
cd My_Django_Folder
pipenv install django
pipenv shell
django-admin startproject PROJECT_FOLDER_NAME .

python manage.py migrate
python manage.py createsuperuser

python manage.py startapp APP_FOLDER_NAME
python manage.py runserver

Your project folder structure should look like this when you type 'tree' on the command prompt:-


It should also contain two files ("Pipfile" and "Pipfile.lock") that makes the virtual environment work properly.

If everything went successfully, you should get the welcome screen that congratulates you when you access the localhost with the given port number.





2) Add HTML/CSS/JS templates

At this point, you want to display a custom front-end the user instead of the default seen above. A front-end framework such as Bootstrap can be use, however, I will keep it simple by using a simple html/css/js files.

Recall from the step above, we create a django PROJECT and APP. Now, we need to edit some files within both folders like this:-
  • PROJECT folder: here we will edit these files - settings.py and urls.py
  • APP folder: here we will edit/create these files - urls.pyviews.py, and models.py. Note we could also add our templates folder here for the index.html but I prefer to create a root templates folder and point to it by updating the TEMPLATES list in PROJECT settings.py.

The following code snippets will add HTML/CSS/JS templates
  • mkdir templates
  • echo > templates/index.html (touch templates/index.html)
  • add your html code to the templates\index.html file
  • Update INSTALLED_APPS and TEMPLATES from PROJECT's settings.py
  • Update urls.py file from PROJECT's folder
  • Create a new urls.py file in app folder 'APP_FOLDER_NAME' and add view path
  • Update views.py file from the app folder 'APP_FOLDER_NAME' by defining the view added above. Since the name was called 'home', it most be the same in the views.py file.


If you do python manage.py runserver, this should load our html page from the templates folder as seen below:-


Notice that this is a very basic web page that didn't make use of a database, hence we didn't talk about the models.py file.

To add more pages for example, lets add contact page. We will create the contact.html file in the templates folder and update urls.py and views.py file in the app folder.



Note: If you don't want the .html extension to appear in the url like this "http://localhost:8000/contact.html", the edit urls.py file to remove the extension so that the path is like so... "http://localhost:8000/contact"



3) Adding Static file

In django, to use external CSS, JS, or Image files within HTML document you need to tell it how to get them by creating a root folder named "static" to house all the static files. The steps involved are as follow:-
  1. First thing is to update PROJECT's settings.py file by adding STATICFILES_DIRS


  2. Create a root folder named "static" and create another sub-folder with name of the app 'APP_FOLDER_NAME' 
  3. Add your static files there, for example I added an image 'buhari.jpg'
  4. Update the html file by first typing "{% load static %}" on the very first line. Then point to the file using this "{% static 'APP_FOLDER_NAME/buhari.jpg' %}"

The new home page should look like this...



This is the django way of doing things! In normal html, the inserted image tag will just look like this: <img src="APP_FOLDER_NAME/buhari.jpg " />

Likewise inserting a css or js file like this: <link rel="stylesheet" type="text/css" href="style.css"> in normal html will look like this: <link rel="stylesheet" type="text/css" href="{% static 'APP_FOLDER_NAME/style.css' %}"> in django way.

Monday, May 10, 2021

Extracting Information from PDF with Python

There are several python libraries available that allows the extraction of pdf data. Some notable once are listed below:-

1) PyPDF2
2) Tabula-py (requires JAVA installed)
3) Zamzar.com API
4) Camelot
5) TIKA
6) PDFMiner

All the above are great have there pros and cons which depends on the pdf file in question. You will need to try each one out to find out the one that works best for your specific case.

In my own case, TIKA has been the best so far for all the pdf I have worked with. So in this blog post, I will demonstrate how the 'tika' module is used to extract information from a pdf file.

Lets get started....

To install this module, you most has java 7 or above installed on your machine. Then using it is just as easy as this:-

from tika import parser # pip install tika (need Java 7+ installed)

raw = parser.from_file('C:/Desktop/pdf_file_name.pdf')
print(raw['content'])

The content can then be stored in a variable for further manipulation.



Example

Lets extract the content of this PDF file:-



The result will look like this below. Note that I added .strip() function that will remove extra white spaces at the beginning and end of the content result.







That is it!

Thursday, May 6, 2021

Python script to separate shapefile based on type (Point, Line and Polygon)

 Here I got a folder that contains several shapefiles of mixed types (i.e Point, Line and Polygon shapefiles all in the same folder), now I want separate each type of shapefile into one separate folder. That is all Point shapefiles into same folder, all Line shapefiles into same folder, and all Polygon shapefiles into same folder.


Here we got twelve shapefiles that look like this:-


To complete this manually, we have to open all the twelve files and save each to its respective folder. This is not we wanted and it is time consuming, so I will use python scripting to complete this.

Let's get our hands dirty...

Monday, May 3, 2021

Geo-calibrating an SVG map for use in mapsvg.com wordpress plugin

 Scalable Vector Graphic (SVG) file isn't the best for creating geospatial referenced maps. However, it is possible to maintain map data that have geo-like coordinates that are close to the geolocation they represent. SVGs are not commonly used for geo-enable maps on the web, instead they are commonly used for drawing maps just for visual purpose on web pages.

If you really want to work with geo-enabled map on a website page where spatial reference system is of paramount importance, then you should use file format such as GeoJSON. Or store your map in a spatial database.

There are many reasons why you would want to make your SVG file have spatial capability. A common reason is when you want to overlay your SVG drawing on a geographical base map as used in MapSVG wordpress plugin. The mapsvg plugin allows you to add custom maps in svg format to wordpress web pages. If you want the map overlay the svg map on Google maps background, then you must geo-calibrate the map as we will see in a moment.

The plugin isn't free, at the time of writing it is about $49. However, there is a demo admin panel (with the following login details: demo / demo123) where you can try it out for free. We will use this to upload a custom map that is geo-enable.

Wednesday, April 28, 2021

Merge images to PDF Using 'Python Imaging Library' (PIL)

 This script will merge any given amount of images into a single PDF file.



from PIL import Image


# List of images... 4 images in this case, so we should get a pdf with 4 pages.
image_files = [r"C:\Users\Yusuf_08039508010\Desktop\L1.jpeg", r"C:\Users\Yusuf_08039508010\Desktop\L2.jpeg", r"C:\Users\Yusuf_08039508010\Desktop\L3.jpeg", r"C:\Users\Yusuf_08039508010\Desktop\L4.jpeg"]

imagelist = []
for img in image_files:
    print('Processing...', img)
    # read in image
    img1 = Image.open(img)
    
    # convert to RGP
    img2 = img1.convert('RGB')
    
    imagelist.append(img2)
    
    print('Done for...', img.split('\\')[-1])
    
# To prevent appending last image to the beginning...
start_img = Image.open(image_files[0])
start_img = start_img.convert('RGB')

# Delete first image from list 'imagelist'...
del imagelist[0] # imagelist.pop(0) or imagelist[1:]

# merge images to pdf...
start_img.save(r'merged_L1-4.pdf', save_all=True, append_images=imagelist)




That is it!

Tuesday, April 13, 2021

Testing Google Maps Geocoder API by Geocoding COVID-19 Vaccination sites in Lagos, Nigeria

 Recently, the Lagos state government listed 88 COVID-19 vaccination sites in the state. These sites cut across 20 local government areas.

Lets test how good Google Maps covers Nigeria by using the geocoder API to retrieve the 'Latitudes and longitudes' of those vaccination sites so we can plot them on Lagos state map.

Lets get stated, the list of the sites is seen below.




Friday, April 9, 2021

Extracting data from .HAR file

 A HTTP Archive file (shorten as 'HAR file'), is a JSON format used for tracking information between a web browser and a website. The common extension for these files is '.har'.

In python there is third party module called "Haralyzer" developed for getting useful stuff out of HAR files.

Since HAR files are in JSON formats, I will not use the "Haralyzer" module instead I will read the .har file and extract data from the text. Another reason I don't want to use the library is that I don't want to install new third party library on my machine most especially that the haralyzer module depends on another third library "six".

Other than that nothing wrong in using a library that reads the .har file directly.

Let's get our hands dirty...


How to get a HAR file

Practically, any website that uses JSON format as its data communication pipeline will generate a .har file on clients browser which can be accessed from the browser's developer tool.

Lets use this use this website on Earthquake data by USGS. Open the website and go to your browser developer tool, then select 'Network' tab >> XHR >> Expot HAR...


This will download a HAR files that contains JSON representation of the earthquake data as seen below...


You can save the file with any name in a location you can remember, we will use it in the next section. Note that the file is a GeoJSON with Padding.

Thursday, April 1, 2021

How to make black and white Road network map in Mapbox Studio

Mapbox Studio is a platform used by developers to prepare maps for mobile, desktop and web applications.

In this post, we are going to prepare a black and white road network map similar to what you see below.


There are many startup web applications that allow you create this kind of map and download it for a fee! In few moments, you will learn how to make your own maps and you don't have to spend money buying maps.

See an example below that cost $59.


Step-by-step instructions



  • Create a new map by clicking on "New Style" button.


  • Select from the map style templates (here we will use the Blank Template)


  • Rename the map to a suitable name and add map components and layers.


To achieve this type of map, we need to add the following map components and layers:-

  1. Road Network
  2. Land, Water and Sky
  3. Administrative Boundaries

On each component/layer adjust the settings to fit what you wanted. For example, I set the administrative boundaries base to white etc.


After which you can publish and share the map as WMTS for use in a desktop software like QGIS for further map processing as you will see below in a moment.


Wednesday, March 24, 2021

Looping over an iterable (array/list) in JavaScript Vs Python

 Lets see what it takes to loop over an iterable using for-loop in both JavaScript Vs Python. By the way, an iterable is an object capable of returning its members one at a time, permitting it to be iterated over in a for-loop.

Assuming we have this iterable : m = [3.23, 4.56, 5.3, 2.44, 6.7, 12.4, 566] and we want to perform some math operation on each element (in this case: 2 to the power of element divided by 2).

The math formulae is as follow:-

For JavaScript

Math.pow(2, element) / 2


For Python

(2**element)/2


The solutions

JavaScript Solution

m = [3.23, 4.56, 5.3, 2.44, 6.7, 12.4, 566]

for (let i=0; i<m.length; ++i){
console.log(Math.pow(2, m[i]) / 2);
}


m = [3.23, 4.56, 5.3, 2.44, 6.7, 12.4, 566]

for (let i in m){
console.log(Math.pow(2, m[i]) / 2);
}

Python Solution

m = [3.23, 4.56, 5.3, 2.44, 6.7, 12.4, 566]

for i in m:
    print((2**i)/2)




That is it!

Thursday, March 18, 2021

Python script - Merge PDF files into a single file

 This script make use of the PyPDF2 library to merge list of pdf files into one big file.


from PyPDF2 import PdfFileMerger
from os import listdir

input_dir = r"C:\Users\Yusuf_08039508010\ND 2 WAEC Result Check\Result" 
#your input directory path


merge_list = []

for x in listdir(input_dir):
    if not x.endswith('.pdf'):
        continue
    merge_list.append(input_dir +'\\'+ x)

merger = PdfFileMerger()

for pdf in merge_list:
    merger.append(pdf)

merger.write(input_dir + "\pdf_file_name.pdf") #your output directory and pdf_file name
merger.close()

print('Finished...')


Enjoy!