Python Generator functions

Have you ever stumbled upon RAM exhaustion issues when your application has to read records from a large dataset? Or any complex operations where a continuous flow of data from the Source occurs and in turn demands the upkeep of an internal state on regular time intervals, for example, a stream of data in socket connections, maintaining system logs?

Yes!!, then in such cases, Python Generator is a tailor-made function that produces continuous results based on your requirement.

Generator functions are also of a great boon in reducing the load on the machine memory and computational time while we work on big data especially.

Before I could mention the definition of Generator and how it works, let’s take a basic example with for loop, yes yes the same traditional way of iterating over the objects like lists, tuples, sets, dictionaries, strings, etc. (In short, anything you can loop over is iterable.)

Suppose we want to return a list of numbers from 0 to 9, then the function goes like this:

def mylist():
    n =0
    numbers = [] 
    while n < 10:
        # print statement to have a check in While loop
        print("In loop", n)
        numbers.append(n) 
        n += 1
    return numbers
for num in mylist():
    print("Got number from the list", num)
OUTPUT:
In loop 0
In loop 1
In loop 2
In loop 3
In loop 4
In loop 5
In loop 6
In loop 7
In loop 8
In loop 9
Got number from the list 0
Got number from the list 1
Got number from the list 2
Got number from the list 3
Got number from the list 4
Got number from the list 5
Got number from the list 6
Got number from the list 7
Got number from the list 8
Got number from the list 9

On the same example program, we will try to give an integer value to a for loop instead of a list. Oops!!, we ended up getting an error now.

def mylist():
    n =0
    while n < 10:
        # print statement to have a check in While loop
        print("In loop", n) 
        n += 1
    return n
for num in mylist():
    print("Got number from the list", num)
OUTPUT:
In loop 0
In loop 1
In loop 2
In loop 3
In loop 4
In loop 5
In loop 6
In loop 7
In loop 8
In loop 9
Traceback (most recent call last):
    for num in mylist():
TypeError: 'int' object is not iterable

Python threw an error because, in the program from num, python understood integer as one entity and left with nothing to iterate. Back to basics, one of the conditions for iteration is what next operation to be performed. Yes, on the list, this requirement has met. The list tells for loop that the next item to iterate on is in the index+1 from the current one (1 comes after 0).To overcome this, Python has offered Generators.

Now we are at a point where we have a vague idea of when we can use Generators!!! Let's explore more.

Alright, What are Generators?

Generators are functions that generates objects or items from the given sequence. These functions don’t produce all items simultaneously but only one at a time.

Generators are also called as lazy iterators ,unless we asks for the values clearly with next() method, generators wont give the results and they also do not store their contents in memory.

A generator function is just like a regular function but with a key difference: the yield keyword replaces return. The additional capability is -when the generator function is called in between ,it doesnt start all over but from where it was left-off in the last call. This is what makes generators to stand out when compared with normal functions.

Don’t think too much if it sounds confusing. This is simpler when we see how generators work.

Let's use the same example function which we had seen earlier, to get finite numbers i.e. 0 to 9, but this time as a generator function

def mylist():
    n = 0
    while n < 10:
      n += 1
      # yield the number
      yield n
gen=mylist()
print(gen)
# values
print(next(gen))
print(next(gen))
OUTPUT: 
<generator object generator at 0x0000025090522A48>
1
2

Initially, when the object for mylist() generator function is created, it initializes n variable, yield generates the values. Now, when the next() method is called on the object, the generator function computes the values and returns the output, while at the same time remembering the state of the function.

The function will keep on generating values every time it is asked by the next() method until n becomes greater than 10, at which point, a StopIteration error will be raised as shown below.StopIteration will automatically be raised when a generator stops yielding. It's part of the protocol of how generators work.

def generator():
    n = 0
    while n <10:
      n += 1
      # yield the number
      yield n
gen=generator()
print(gen)
print(next(gen))
print(next(gen))
print(next(gen))
print(next(gen))
print(next(gen))
print(next(gen))
print(next(gen))
print(next(gen))
print(next(gen))
print(next(gen))
OUTPUT:
<generator object generator at 0x0000025090522A48>
1
2
3
4
5
6
7
8
9
10
Traceback (most recent call last):
    print(next(gen))
StopIteration

So, to draw the conclusion, we have learned that with the help of generators how we can pull the data when we want in bits rather than pulling at one stretch. I hope Generator feature of python comes as a solution to someone if in case they struggle to iterate their massive dataset.