Wednesday, May 17, 2023

Generating all possible two letter strings to scrape web data

 Recently, I encountered a situation where I have to generate all possible two letter combinations that makes up web urls I had to scrape.

Here you find the python code that does exactly that:-


# Generate all possible two letter strings
from itertools import product
from string import ascii_lowercase

keywords = [''.join(i) for i in product(ascii_lowercase, repeat = 2)]
len(keywords)


My use case what the generate URLs like this:-

url_list = []
for kwd in keywords:
    for i in range(1, 11):
        main_url = f'https://www.fiverr.com/search/users?query={kwd}&page={i}'
        print(main_url)
        url_list.append(main_url)



That is it!

No comments:

Post a Comment