Generating all possible two letter strings to scrape web data

 Recently, I encountered a situation where I have to generate all possible two letter combinations that makes up web urls I had to scrape.

Here you find the python code that does exactly that:-

# Generate all possible two letter strings
from itertools import product
from string import ascii_lowercase

keywords = [''.join(i) for i in product(ascii_lowercase, repeat = 2)]
len(keywords)

My use case what the generate URLs like this:-

url_list = []
for kwd in keywords:
    for i in range(1, 11):
        main_url = f'https://www.fiverr.com/search/users?query={kwd}&page={i}'
        print(main_url)
        url_list.append(main_url)

That is it!

Wrangling and Mapping ‘list of colleges teaching MBBS in India’ with Python

 In this post, I will demonstrate how to wrangle and map the ‘list of colleges teaching MBBS in India’. The list is available on this webpage.

First I will collected the dataset, clean it into a GIS friendly format, geocode the colleges, then map it on the map of India sourced from GADM. This is a project you could 100% complete without coding using tools like QGIS or ArcGIS, but here I will use python code to do it from scratch.

Data collection

Lets collect the list of data from the web page into CSV file. There are ways to do this in python like using requests module, selenium module, beautifulsoup module, scrapy module etc.

In this situation, I will copy the html element the represent the table into a local file, then extract the table from the local CSV spreadsheet using python.

Using pandas library, with few lines of code we got the list into a CSV file as seen below;-

mbbs_df = pd.read_html(r"mbbs.html")
mbbs_df[0].to_csv(r"mbbs.csv", index=False)
mbbs_df = mbbs_df[0]

mbbs_df

Convert Sentinel-2 Tile KML to CSV

 Lets convert this KML file to CSV or other GIS friendly file.

In this case we just needed the tile number and the multipolygon objects, hence we will have tow columns in the CSV file or table.

The open() function can be used to read the kml file, then the required objects read line by line.

kml_file = r"S2A_OPER_GIP_TILPAR_MPC__20151209T095117_V20150622T000000_21000101T000000_B00_2.kml"
kml_txt_file = open(kml_file)

cnt = 0
tile_number_list = []
tile_wgs_polygon_list = []

for line in kml_txt_file:
    # print(line)
    if 'MULTIPOLYGON' in line:
        tile_number = line.split('<font COLOR="#008000">')[1].split('</font>')[0]
        tile_wgs_polygon = line.split('<font COLOR="#008000">')[-1].split('</font>')[0]

        tile_number_list.append(tile_number)
        tile_wgs_polygon_list.append(tile_wgs_polygon)
        cnt += 1
    # break
    
print(f'{cnt} records found...')
kml_txt_file.close()

To convert to geo dataframe, the script is like so;-

sentinel_df = pd.DataFrame([tile_number_list, tile_wgs_polygon_list]).T
sentinel_df.rename(columns = {0:'TileNumber', 1:'geometry'}, inplace = True)

# Use shapely well known text (wkt) format to create geometry column...
from shapely.wkt import loads
sentinel_df['geometry'] = sentinel_df['geometry'].apply(lambda x: loads(x))

sentinel_gdf = gpd.GeoDataFrame(sentinel_df, crs='epsg:4326')

To convert to csv file;

sentinel_gdf.to_csv('folder/SEn-2.csv')

To convert to shapefile;

sentinel_gdf.to_file('folder/SEn-2.shp')

If you visualize the converted file, the sentinel-2 tiles the covers Nigeria are seen below:-

31NDE, 31NDF, 31NDG, 31NDH, 31NDJ, 31NEE, 31NEF, 31NEG, 31NEH, 31NEJ, 31NFE, 31NFF, 31NFG, 31NFH, 31NFJ, 31NGE, 31NGF, 31NGG, 31NGH, 31NGJ, 31NHE, 31NHF, 31NHG, 31NHH, 31NHJ, 31PDK, 31PDL, 31PDM, 31PDN, 31PDP, 31PDQ, 31PDR, 31PEK, 31PEL, 31PEM, 31PEN, 31PEP, 31PEQ, 31PER, 31PFK, 31PFL, 31PFM, 31PFN, 31PFP, 31PFQ, 31PFR, 31PGK, 31PGL, 31PGM, 31PGN, 31PGP, 31PGQ, 31PGR, 31PHK, 31PHL, 31PHM, 31PHN, 31PHP, 31PHQ, 31PHR, 32NKK, 32NKL, 32NKM, 32NKN, 32NKP, 32NLK, 32NLL, 32NLM, 32NLN, 32NLP, 32NMK, 32NML, 32NMM, 32NMN, 32NMP, 32NNK, 32NNL, 32NNM, 32NNN, 32NNP, 32NPK, 32NPL, 32NPM, 32NPN, 32NPP, 32NQK, 32NQL, 32NQM, 32NQN, 32NQP, 32NRK, 32NRL, 32NRM, 32NRN, 32NRP, 32PKA, 32PKQ, 32PKR, 32PKS, 32PKT, 32PKU, 32PKV, 32PLA, 32PLQ, 32PLR, 32PLS, 32PLT, 32PLU, 32PLV, 32PMA, 32PMQ, 32PMR, 32PMS, 32PMT, 32PMU, 32PMV, 32PNA, 32PNQ, 32PNR, 32PNS, 32PNT, 32PNU, 32PNV, 32PPA, 32PPQ, 32PPR, 32PPS, 32PPT, 32PPU, 32PPV, 32PQA, 32PQQ, 32PQR, 32PQS, 32PQT, 32PQU, 32PQV, 32PRA, 32PRQ, 32PRR, 32PRS, 32PRT, 32PRU, 32PRV, 33NTE, 33NTF, 33NTG, 33NTH, 33NTJ, 33NUE, 33NUF, 33NUG, 33NUH, 33NUJ, 33NVE, 33NVF, 33NVG, 33NVH, 33NVJ, 33PTK, 33PTL, 33PTM, 33PTN, 33PTP, 33PTQ, 33PTR, 33PUK, 33PUL, 33PUM, 33PUN, 33PUP, 33PUQ, 33PUR, 33PVK, 33PVL, 33PVM, 33PVN, 33PVP, 33PVQ, 33PVR,

That is it!

How to find Sentinel-2 satellite image for your study area using GoogleEarthEngine

Unlike the Landsat satellite, Sentinel-2 satellite doesn't use Path/Row naming convention to define its scenes instead it uses granules or tiles of 110km x 110km ortho-images in UTM/WGS84 projection.

The tiles are numbered from 1 to 60 followed by three letters, so an example of a valid Sentinel-2 tile number are: 01CCV, 01CDH, 01CDJ, etc.

Now the problem is how do you know the number that covers your study area! Well there are several ways of solving this, one is by visualizing the granules or tiles KML file in you favorite tool.

If you work solely in Google Earth Engine, then I would be a bad idea to load that KML file due to its size. So, an approach to use is to load/import image collection and filter based on region of interest or other criteria and print the 'id' using the id() function as demonstrated in the code snippet below:-

var dataset = ee.ImageCollection('COPERNICUS/S2')
  .filterDate('2022-01-01', '2022-12-31')
  .first();
  

print(dataset.id())

This will print the ID on the console as seen below:-

According to the documentation, 'the first numeric part represents the sensing date and time, the second numeric part represents the product generation date and time, and the final 6-character string is a unique granule identifier indicating its UTM grid reference'.

If you want to get more specific, you could a .filterBounds(...) function to your region of interest. Here I added filterBounds to a location in Abuja, Nigeria.

Note that the image collection will return 'Null' if there is no image available for your filter criteria.

That is it!

Using Python Interpret Landsat File Naming Convention

 If you have ever worked with Landsat images, you would have noticed that the image files usually have a long name made-up of letters and numbers that probably looks like this: LC08_L2SP_188054_20221120_20221129_02_T1.

In this article, we will use python to extract the meaning of the file name.

After downloading a Landsat file from the USGS Earth Explorer, there’s surprisingly a lot of information you can get from just the file name. The following web pages provide detailed information on the file naming convention:-

• https://www.usgs.gov/faqs/what-naming-convention-landsat-collections-level-1-scenes
• https://gisgeography.com/landsat-file-naming-convention/

The Landsat satellite series is a long-running satellite program that provides high-resolution imagery of the Earth's surface. The images taken by the Landsat satellites are named using a standardized naming convention that includes information about the satellite, the path and row of the image, the date the image was taken, and the band of the image.

Here is an example of a Landsat image file name: LC08_L1TP_012030_20170604_20170615_01_T1

This file name can be interpreted as follows:

LC08: The satellite that took the image (Landsat 8)

L1TP: The processing level of the image (Level 1 Terrain Corrected)

012030: The path and row of the image (path 01, row 20)

20170604: The date the image was taken (June 4, 2017)

20170615: The date the image was processed (June 15, 2017)

01: The band of the image (band 1)

T1: The product type of the image (Type 1)

You can use Python to parse and interpret Landsat image file names by splitting the file name string using the split() method and extracting the relevant pieces of information using string slicing and indexing. For example:

filename = 'LC08_L1TP_012030_20170604_20170615_01_T1'

satellite = filename[:4]  # 'LC8'
processing_level = filename[5:9]  # 'L1TP'
path_row = filename[10:16]  # '012030'
date_taken = filename[17:25]  # '20170604'
date_processed = filename[26:34]  # '20170615'
band = filename[35:37]  # '01'
product_type = filename[38:]  # 'T1'


print(f'''
      The satellite name is: {satellite}, 
      The processing level of the image is: {processing_level}, 
      The path and row of the image is: {path_row}, 
      The date the image was taken on: {date_taken}, 
      The date the image was processed is: {date_processed}, 
      The band of the image is: {band}, 
      The product type of the image is: {product_type}
      ''')

The code use string slicing to get the required information from the file names.

SVG remove polygon using JS

 In this post, you will find a code snippet that will remove a state polygon on mouse click. The tech stack used are: HTML, CSS, JS and SVG.

The result will look like below image.

First step is to setup a basic HTML page and use the 'object' tag to embed the SVG file into the HTML body like this:-

<object id="svg-object" data="svg_map_filename.svg" type="image/svg+xml"></object>

Then we will update the SVG file with CSS and JS code as follow. In the '.svg' document, we need to put the styles declaration inside cdata, as follows;-

<style type="text/css" media="screen"> <![CDATA[

/*Add CSS code here...*/

  ]]> </style>

The CSS style is added at the top of the svg file just after the opening <svg ...> tag.

Lastly we will add JS code at the end of the  svg file just after the closing <svg ...> tag. The script code must be wrapped inside cdata, as follows;-

<script> <![CDATA[

    // Add JS code here...

  ]]> </script>

Wrapping the CSS or JS codes in the cdata is important so that characters like > don't conflict with XML parsers. Using cdata is highly recommended for embedding style or script in SVG, even if the conflicting character is not given in the style sheet or script file.

That is it!

Python workout in JavaScript

 In this post, I will convert some python code from the book "Python Workout by Reuven M. Lerner" into JavaScript (you may get a copy of the book from his website). The code used in the book are publicly available on his github page.

Lets get started...

Task 1: Generate a random integer from 1 to 100.

Ask the user repeatedly to guess the number. Until they guess correctly, tell them to guess higher or lower.

<!DOCTYPE html>
<html>
<head>
	<meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
	<meta name="viewport" content="width=device-width, initial-scale=1">
	<title>Random Number Generator | Guess the number...</title>

<!-- ***************** CSS ********************* -->
	<style type="text/css">
		body{
			background-color: #db20d96e;
		}

		h2, #num, #submit, #result {
			font-size: 2em;
		}

		h1{
			margin: 0;
			color: #f94711;
		}
	</style>

</head>
<body>



<!-- ***************** HTML ********************* -->
<center>
	<h2>Am thinking of a number, what is it?</h2>

	<input type="number" name="num" id="num" />
	<button id="submit">Submit</button>
	<br>
	<br>

	<label type="text" name="result" id="result" />
</center>



<!-- ***************** JS ********************* -->
<script>

// Funtion to get random number between two nums...
	function getRandomIntInclusive(min, max) {
		  min = Math.ceil(min);
		  max = Math.floor(max);
		  return Math.floor(Math.random() * (max - min + 1) + min); // The maximum is inclusive and the minimum is inclusive
		}
// Guessed number is...
	let guessed_num = getRandomIntInclusive(1, 100);


// Get form entry and compare to the user's guessed number....
	let btn = document.getElementById('submit');

	btn.onclick = (e) => {
		let guess_num = document.getElementById('num').value;
		let result = document.getElementById('result');

		console.log(guess_num);

		if (guessed_num == guess_num){
			// console.log('Correct Guess...');
			result.innerHTML = `Correct Guess... The number in my mind was: <h1>${guessed_num}</h1>`

		} else if (guessed_num < guess_num) {
			// console.log(`Guessed number is too HIGH; try again...`);
			result.innerHTML = `Guessed number is too HIGH; try again...`

		} else if (guessed_num > guess_num) {
			// console.log(`Guessed number is too LOW; try again...`);
			result.innerHTML = `Guessed number is too LOW; try again...`
		}

	}
	
// `${guessed_num}`
</script>

</body>
</html>

Task 2: Summing numbers

Define "mySum". Accepts any number of numeric arguments as inputs. Returns the sum of those numbers. If invoked without any arguments, returns 0.

<!DOCTYPE html>
<html>
<head>
	<meta charset="utf-8">
	<meta name="viewport" content="width=device-width, initial-scale=1">
	<title>Summing Numbers in an array....</title>


<!-- ***************** CSS ********************* -->
	<style type="text/css">
		body{
			margin: 50px;
			background-color: burlywood;
		}

		center{
			background-color: grey;
			border: solid 10px;
			border-radius: 25px;
		}

		h2, #num, #submit, #result {
			font-size: 2em;
		}

		h1{
			margin: 0;
			color: #f94711;
		}

		#num, #submit{
			border-radius: 25px;
		}		
	</style>


</head>
<body>


<!-- ***************** HTML ********************* -->

<center>
	<h2>Enter your list of numbers like this: <i>1, 2, 3, 4, 5</i></h2>

	<input type="text" name="num" id="num" />
	<button id="submit">Submit</button>
	<br>
	<br>

	<label type="text" name="result" id="result" />
</center>



<!-- ***************** JS ********************* -->

<script type="text/javascript">

// Function to sum array of numbers.... https://www.educative.io/answers/how-to-get-the-sum-of-an-array-in-javascript
	function mySum(myarray) {
		  let sum = 0;

		  for (let i = 0; i < myarray.length; i += 1) {
		    sum += (myarray[i] * 1); // multiply by 1 to convert to number: https://www.freecodecamp.org/news/how-to-convert-a-string-to-a-number-in-javascript/
		  }
		  
		  return sum;
		}

// Get form entries....
	let btn = document.getElementById('submit');
	let result = document.getElementById('result');

// Perform the magic...
	btn.onclick = (e) => {
		let list_of_num = document.getElementById('num').value;
		// console.log(typeof(list_of_num))

		// Replace spaces....
		list_of_num = list_of_num.replace(/\s+/g, '');

		// Split by ','...
		list_of_num = list_of_num.split(',');

		// Use the function above to sum the list...
		let sum_of_list_of_num = mySum(list_of_num)

		// Display result...
		if (isNaN(sum_of_list_of_num)){
			result.innerHTML = `<h1>Invalid</h1> Your entries most all be numbers.`;
		} else {
			result.innerHTML = `The Sum of ${list_of_num} is: <h1>${sum_of_list_of_num}</h1>`;
		}
		


		// console.log(`${list_of_num}`)
		// console.log(mySum(sum_of_list_of_num))
	}


</script>


</body>
</html>

Task 3: Running Time Average

Here we will use javascript to dynamically add the run time input box.

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Running Time Average...</title>

<!-- ------------------------ CSS -------------------------------- -->
  <style type="text/css">
    #num{
          height: 50px;
          width: 320px;
          font-size: xx-large;
        }

    #submit, #addmore{
      font-size: x-large;
      height: 70px;
    }

    #result{
      color: green;
      font-size: xxx-large;
    }


  </style>

</head>
<body>


<!-- ------------------------ HMTL -------------------------------- -->

<center>
  <h1>Enter the time run for the 10km: </h1>

  <input type="number" name="num" id="num" class="num" />
  <button id="submit">Submit</button>
  <button id="addmore">Add More Time</button>

  <br>
  <br>

  <label type="text" name="result" id="result" />
</center>



<!-- ------------------------ JS -------------------------------- -->


<script type="text/javascript">


  // Select the button element
  const submitButton = document.querySelector('#submit');
  const addmoreButton = document.querySelector('#addmore');

  const result = document.getElementById('result');



  // Add a click event listener to the addmore button
  addmoreButton.addEventListener('click', function() {
    // Create a new element
    const elem0 = document.createElement('center');
    const elem1 = document.createElement('input');
    const elem2 = document.createElement('br');


    // Set the element's content and style
    elem1.style.color = 'red';
    elem1.style.width = '600px';
    elem1.style.height = '50px';
    elem1.style.fontSize = 'xx-large';

    elem1.type = 'number';

    // Add a class and id named 'num' to the element
    elem1.className = 'num';
    elem1.id = 'num';


    // Append the 'input' to the 'center' element
    elem0.appendChild(elem1);

    // Append the element to the DOM
    document.body.appendChild(elem0);

    // document.body.appendChild(elem1);
    document.body.appendChild(elem2);

  });




// ---------------------------------------------

 // Add a click event listener to the submit button
  submitButton.addEventListener('click', function() {


    // const all_time = document.getElementsByClassName('num');
    const all_time = document.querySelectorAll('.num');


    const averageTimeRan = Array.from(all_time).reduce((total, elem) => total + parseFloat(elem.value), 0) / all_time.length;

    result.innerHTML = `The average of total time ran is: ${averageTimeRan.toFixed(2)}`;

  });



</script>


</body>
</html>

Wrangling the 'Database of health facilities in Nigeria' using QGIS and Python

 A database of health facilities in Nigeria would be a valuable resource for both healthcare providers and the general public. It could provide information about the types of facilities available, their locations, and the services they offer. This information could help healthcare providers refer patients to appropriate facilities, and it could help the public find the nearest and most suitable facilities for their needs.

There are many different types of health facilities in Nigeria, including hospitals, clinics, and pharmacies. Some facilities may specialize in a particular area of healthcare, such as maternal and child health, while others may offer a range of general medical services. The availability and accessibility of these facilities can vary greatly depending on where you are in the country.

Having a comprehensive database of health facilities in Nigeria would allow for better planning and resource allocation by the government and healthcare organizations. It could also help to ensure that underserved areas are not overlooked and that there is an adequate distribution of healthcare resources throughout the country.

In addition to providing information about health facilities, such a database could also include information about the availability of healthcare professionals, including doctors, nurses, and other medical staff. This could be particularly useful for rural areas where access to healthcare professionals may be limited.

Overall, a database of health facilities in Nigeria would be a valuable tool for improving the country's healthcare system and ensuring that all citizens have access to the healthcare services they need.

Python for GIS data wrangling

 Python is a powerful and widely used programming language that is well-suited for GIS data wrangling tasks. There are a number of libraries and tools available in Python that can be used to perform a wide range of GIS data wrangling tasks, including reading and writing various geospatial data formats, performing spatial transformations and projections, and performing spatial analyses and visualizations.

Some of the most commonly used libraries for GIS data wrangling in Python include:

pandas: A powerful data manipulation and analysis library that is widely used for GIS data wrangling. It provides a range of functions for reading and writing geospatial data formats, including CSV, shapefile, KML, and GeoJSON.

geopandas: A library built on top of pandas that provides additional functionality for working with geospatial data. It allows you to manipulate and visualize geospatial data using familiar pandas dataframes, and includes support for geographic projections and spatial operations.

fiona: A library for reading and writing geospatial data formats, including shapefiles, GeoJSON, and KML. It can be used in combination with other libraries, such as geopandas, to perform advanced GIS data wrangling tasks.

shapely: A library for working with geometric objects such as points, lines, and polygons. It provides functions for creating, manipulating, and performing spatial operations on these objects, and can be used to clean and transform geospatial data.

Other useful libraries for GIS data wrangling in Python include rasterio, gdal, and pyproj, among others. These libraries can be used to perform a wide range of GIS data wrangling tasks, including reading and writing various geospatial data formats, performing spatial transformations and projections, and performing spatial analyses and visualizations.

Over the past few years, I have used python to wrangle different kinds of GIS data and some example can be reviewed below:-

1- Python GIS data wrangling - Harris County Appraisal District

2- GIS data wrangling with python - Gridded Street Finder

3- Geocoding and Reverse Geocoding with Python

4- Python GIS data wrangling - Mapping supper eagles head coaches since 1949

5- Python GIS Data Wrangling - U.S. Drought Monitor

6- Python Programming for GIS Data Processing in QGIS

7- GIS data wrangling with python - case study of 'African Journals Online' featured countries

If you need help with wrangling your GIS data, you may contact me via email/phone details on my website:  UmarYusuf.com

Cheers!

Using chatGPT (Generative Pretrained Transformer) to map first 100 regions to welcome the new year

 ChatGPT is a variant of the GPT (Generative Pretrained Transformer) language model that is specifically designed for generating human-like text in a conversational context. It is not a tool for creating maps or visual representations of data.

How do we now use it to make maps?

The answer is simple, we will use ChatGPT to generate text that lists the first 100 regions to welcome the new year by providing the model with a prompt in the form of a natural language question or statement, such as "Can you provide a list of the first 100 regions to welcome the new year?", and then allowing the model to generate the text.

We can also ask ChatGPT to geocode the places for us and that will return the geographic coordinates as seen below;-

Now, we got all we needed to map the first 100 regions to welcome the new year. Copy the chatGPT text into your GIS software of choice for the mapping, of course the text will be formatted appropriately as required be the GIS software you want to use.

As for me I use QGIS software and here is how it looks:-

The New Year is celebrated on January 1st in most countries around the world. However, different countries have different time zones, so the New Year is actually celebrated at different times in different parts of the world.

The new year is celebrated at different times around the world. celebrate the new year is typically Samoa and Christmas Island in Kiribati, which are the first to enter the new year due to their location west of the International Date Line. The last place to celebrate the new year is usually Baker Island and Howland Island, which are the last to enter the new year due to their location east of the International Date Line. However, these locations are uninhabited, so the last place where the new year is celebrated with a significant population is usually American Samoa.

Thank you for following.