*args and **kwargs

Python has two very useful idioms for passing a variable number of arguments to functions, *args and **kwargs. These arguments are used extensively when developing general-purpose libraries, decorators or proxies. But what are these these two mysterious words of pirate sounding origin?

  • *args is a tuple of positional arguments (i.e. “fred”, “tom”, “betty”)
  • **kwargs is a dict of keyword arguments (i.e. name=”sonia”, age=20)

Argument Types

Before we begin, note that Python has two different types of arguments:

  • positional – these arguments contain a value, such as “fred”, their ordinal position matters
  • keyword – these arguments are of the form key=value, such as mobile=”00-01234567″

Positional arguments must be declared before keyword arguments. They are often called value arguments. They are assigned values according to their ordinal position. i.e. the first argument gets the first value, the second argument gets the second, etc:

def display_student(student_id, name, age):
    print(student_id, name, age)

display_student(1001, "fred", 20)

# output: 1001  fred   20

Keyword arguments can be provided in any order so long as they follow positional arguments:

def display_student(student_id, year, name, age):
    print(student_id, year, name, age)

display_student(name="tom", year=7, student_id=1001, age=12)

# output: 1001 7 tom 12

Each key must be unique, you can’t specify name=“fred” and name=“tom” in the same call:

# SyntaxError: keyword argument repeated
display(year=7, name="tom", student_id=1001, age=12, name="fred")

There is also the following distinction:

  • required arguments – these can be positional or keyword arguments
  • optional arguments – this includes *args, and **kwargs

Required arguments can be given a default value, in which case you do not need to supply a value for them when making a function call:

def display_student(student_id, name="tbd"):
    print(student_id, name)

# no need to supply name, it has a default value
display_student(1001)

# output: 1001  tbd

With a positional argument that has a default value, always pay attention to the order of assignment. For example, feeling tired I notice that student_name has a default value and since I don’t know the real name at this stage, I decide to just provide the student_id and form_teacher values. Unfortunately, the student’s name is now professor jon smith:

def display_student(student_id, student_name="tbd", form_teacher="tbd"):
    print(student_id, student_name, form_teacher)

display_student(1001, "professor jon smith")

# outputs: 1001   professor jon smith     tbd

Most developers avoid this kind of mistake, but here’s a more subtle situation: when *args is used. Jumping ahead of ourselves for a moment, consider the following example:

def display_student(student_id, name="tba", *args):
    print(student_id, name, args)

# thinking name has a default value, I mentally ignore it
display_student(1001, "cricket", "soccer")

Remember positional arguments are always assigned values in order, so in this case 1001 is assigned to student_id, “cricket” would be assigned to name and *args would only contain “soccer”.

Generally, you should avoid using default values for positional arguments if you are using *args.

Positional arguments with a default value must follow those with no default value. For keyword arguments, a default value has no impact on the ordering – but remember keyword arguments must appear after all positional arguments. This is the correct order:

  1. positional arguments with no default value
  2. positional arguments with a default value (avoid this if you are using *args)
  3. *args (optional positional arguments)
  4. keyword arguments
  5. **kwargs (optional keyword arguments)

Or more simply, declare your function arguments in this order:

def my_func(Positional, *args, Keyword, **kwargs):

And call them in the same order:

my_func(Positional, *args, Keyword, **kwargs

**args

With that behind us, let’s push on with our look at *args and **kwargs.

Let’s consider *args first. The name args is a convention, it could be anything, what’s important is the asterisk (*) operator. For example, I am using it with hobbies rather than args:

def display_student(student_id, name, *hobbies):
  print("id:      ", student_id)
  print("name:    ", name)
  print("hobbies: ")

  for hobby in hobbies:
    print("\t", hobby)

display_student(1001, "fred", "cricket", "soccer", "gaming")

# outputs:

id:      1001
name:    fred
hobbies:
  cricket
  soccer
  gaming

*hobbies contains a variable number of positional arguments, it is a tuple. All the non-required arguments are collected in this tuple. This might look familiar to you if you code in C#:

public Student(int studentId, string name, params object[] hobbies)

or Java:

public String(int studentId, String name, Object... hobbies)

It’s always best to stick with conventions, so I’ll use *args from here on in.

As mentioned *args must appear after all the other positional arguments and before any keyword arguments. Let’s add a keyword argument to our student function:

def display_student(student_id, name, *args, mobile):
  print("id:      ", student_id)
  print("name:    ", name)
  print("mobile:  ", mobile)
  print("hobbies: ")

  for hobby in args:
    print("\t", hobby)

display_student(1001, "fred", "cricket", "soccer", mobile="00-01234567")

# outputs:

id:      1001
name:    fred
mobile:  00-01234567
hobbies: 
  cricket
  soccer

We can see that mobile is a required keyword argument, if you omit either the key (mobile) or its value(“00-01234567”) you will get an error. You can have as many required keyword arguments as you like, but they should appear after all positional arguments, including *args.

Taking a step back, let’s see how Python interprets various ways of calling our display function:

# we'll use a decorator to identify positional and keyword arguments

from functools import wraps

def inspect(f):
    @wraps(f)
    def inspect_f(*args, **kwargs):

        if args:
            print(f'positional: {args}')
        if kwargs:
            print(f'keyword:    {kwargs}')

        f(*args, **kwargs)

    return inspect_f

@inspect
def display(year, student_id, name, age):
    print('student:     ', student_id, year, name, age)

# let's call the display function in all sorts of ways

display(7, name="tom", student_id=1001, age=12)
display(year=7, name="tom", student_id=1001, age=12)
display(name="tom", year=7, student_id=1001, age=12)
display(7, 1001, "tom", 12)
display(7, 1001, name="tom", age=12)
display(7, 1001, "tom", age=12)

# following are the identified argument types for each call

# output for (7, name="tom", student_id=1001, age=12)

positional:   (7,)
keyword:      {'name': 'tom', 'student_id': 1001, 'age': 12}
student:      1001 7 tom 12

# output for (year=7, name="tom", student_id=1001, age=12)

keyword:      {'year': 7, 'name': 'tom', 'student_id': 1001, 'age': 12}
student:      1001 7 tom 12

# output for (name="tom", year=7, student_id=1001, age=12):

keyword:      {'name': 'tom', 'year': 7, 'student_id': 1001, 'age': 12}
student:      1001 7 tom 12

# output for (7, 1001, "tom", 12)

positional:   (7, 1001, 'tom', 12)
student:      1001 7 tom 12

# output for (7, 1001, name="tom", age=12)

positional:   (7, 1001)
keyword:      {'name': 'tom', 'age': 12}
student:      1001 7 tom 12

# output for (7, 1001, "tom", age=12)

positional:   (7, 1001, 'tom')
keyword:      {'age': 12}
student:      1001 7 tom 12

Python interprets the function call arguments into positional and keyword, and then does its best to make sense of what we want. It is very flexible.

However, one rule that is not flexible is that once you use a keyword argument you cannot use subsequent positional arguments:

# SyntaxError: positional argument follows keyword argument 
display(year=7, name="tom", 1001, age=12)

And as soon as we use *args we need to be more precise, the arguments proceeding it must be positional. In the next example, year must be specified as 7 not year=7:

def display(year, *args, student_id, name, age):
    print(student_id, year, name, age)
    if args:
        print(args)

# ok
display(7, "cricket", "soccer", name="tom", student_id=1001, age=12)

# SyntaxError: positional argument follows keyword argument
display(year=7, "cricket", "soccer", name="tom", student_id=1001, age=12)

If you follow the conventional ordering as listed earlier (Positional, *args, Keyword, **kwargs) in both your function declaration and calling convention you’ll avoid this sort of confusion:

def display(year, *args, student_id, name, age):
    print(student_id, year, name, age)
    if args:
        print(args)

# ok
display(name="tom", year=7, student_id=1001, age=12)

# ok
display(7, "cricket", "soccer", name="tom", student_id=1001, age=12)

# TypeError: student() missing 3 required keyword-only arguments: 'student_id', 'name', and 'age'
display(7, 1001, "tom", 12)

# display() got multiple values for argument 'year'
display("cricket", "soccer", year=7, name="tom", student_id=1001, age=12)

**kwargs

Similar to args, kwargs is a convention, it is the double-asterisk (**) operator that matters. In the following example, we could have used**contacts but we’ll stick with convention:

def display(student_id, name, *args, mobile, **kwargs):
    print("id:      ", student_id)
    print("name:    ", name)
    print("mobile:  ", mobile)

    print("hobbies: ")
    for hobby in args:
        print("\t", hobby)

    print("contacts: ")
    for name, phone in kwargs.items():
        print("\t", name, phone)

display(1001, "fred", "cricket", "soccer", mobile="12", jill="45", tom="78")

# outputs:

id:       1001
name:     fred
mobile:   12
hobbies: 
	 cricket
	 soccer
contacts: 
	 jill 45
	 tom 78

Here we have:

  • two required positional arguments: student_id and name
  • a tuple of positional arguments collected in *args (cricket, soccer)
  • a required keyword argument: mobile (mobile=”12″)
  • a dict of keyword arguments collected in **kwargs (jill=”45″, tom=”78″)

Let’s specify a default value for our mobile keyword argument. Notice in the following example how the age keyword argument appears after the mobile keyword argument – which has a default value of “123”. And notice in the function call age is specified between the optional arguments. Python is very smart, it assigns all the keyword arguments it can identify, then collects the remainder into **kwargs:

def display(student_id, name="tba", *args, mobile="123", age, **kwargs):
    print("id:      ", student_id)
    print("name:    ", name)
    print("mobile:  ", mobile)
    print("age:     ", age)

    print("hobbies: ")
    for hobby in args:
        print("\t", hobby)

    print("contacts: ")
    for name, phone in kwargs.items():
        print("\t", name, phone)

display(1001, "cricket", "soccer", jill="45", tom="67", age=20, fred="78")

# ouputs:

id:       1001
name:     cricket
mobile:   123
age:      20
hobbies: 
	 soccer
contacts: 
	 jill 45
	 tom 67
         fred 78

Argument Forwarding

It’s also possible to pass *args and **kwargs from one function to another. You can do so by using the argument-unpacking operators * and ** when calling the function you want to forward arguments to:

def bar(required, *args, **kwargs):
    print(required)
    if args:
        print(args)
    if kwargs:
        print(kwargs)

def foo(required, *args, **kwargs):
    # use the argument-unpacking operators * and **
    bar(required, *args, **kwargs)
 
foo('hello', 1, 2, 3, name="fred", age=20)

# outputs:

hello
(1, 2, 3)
{'name': 'fred', 'age': 20}

Examples

Here are some examples to demonstrate how *args and **kwargs might be used. First up, a simple example of passing required and optional arguments at the same time:

def foo(required, *args, **kwargs):
    print(required)
    if args:
        print(args)
    if kwargs:
        print(kwargs)
    print("-" * 30)

foo('hello')
foo('hello', 1, 2, 3)
foo('hello', 1, 2, 3, name="fred", age=20)
foo('hello', name="fred", age=20)

# outputs: 

hello
------------------------------
hello
(1, 2, 3)
------------------------------
hello
(1, 2, 3)
{'name': 'fred', 'age': 20}
------------------------------
hello
{'name': 'fred', 'age': 20}
------------------------------

Using a simple decorator to implement singletons:

def singleton(cls):
    instance = None

    def on_call(*args, **kwargs):
        nonlocal instance
        if not instance:
            instance = cls(*args, **kwargs)
        return instance

    return on_call

@singleton
class Contractor:
    def __init__(self, name, rate):
        self.name = name
        self.rate = rate

    def __str__(self):
        return f'{self.name} rate: {self.rate}'

c1 = Contractor('Bob', 40)
print(c1)

c2 = Contractor()
print(c2)

c3 = Contractor('Betty', 45)
print(c3)

# outputs:

Bob rate: 40
Bob rate: 40
Bob rate: 40

Here is an example of a trace decorator:

from functools import wraps

def trace(f):

    @wraps(f)
    def decorated_function(*args, **kwargs):
        print(f'-> {f.__name__}: args={args} kwargs={kwargs}')

        try:
            result = f(*args, **kwargs)
            if result:
                print(f'<- {f.__name__}: {result}')
            else:
                print(f'<- {f.__name__}')
            return result
        except Exception as e:
            print(f'error: {str(e)}')
            raise e

    return decorated_function

@trace
def get_highlight(text):
    return "!!! " + text + " !!!"

@trace
def get_greeting(greeting, name):
    text = '{} {}'.format(greeting, name)
    return "*** " + get_highlight(text) + " ***\n"

get_greeting('Hello', 'Bob')
get_greeting(greeting='Hi', name='bob')
get_greeting('Hola', name='bob')

# outputs:

-> get_greeting: args=('Hello', 'Bob') kwargs={}
-> get_highlight: args=('Hello Bob',) kwargs={}
<- get_highlight: !!! Hello Bob !!!
<- get_greeting: *** !!! Hello Bob !!! ***

-> get_greeting: args=() kwargs={'greeting': 'Hi', 'name': 'bob'}
-> get_highlight: args=('Hi bob',) kwargs={}
<- get_highlight: !!! Hi bob !!!
<- get_greeting: *** !!! Hi bob !!! ***

-> get_greeting: args=('Hola',) kwargs={'name': 'bob'}
-> get_highlight: args=('Hola bob',) kwargs={}
<- get_highlight: !!! Hola bob !!!
<- get_greeting: *** !!! Hola bob !!! ***

Here’s an example where we wrap a function call to add more arguments:

def bar(required, *args, **kwargs):
    print(required)
    if args:
        print(args)
    if kwargs:
        print(kwargs)

def foo(required, *args, **kwargs):
    # add a new item to the dictionary
    kwargs['teacher'] = 'Professor Smith'

    # tuples are immutable, so we need to create a new tuple
    new_args = args + (4, 5)

    # using the argument-unpacking operators call function bar
    bar(required, *new_args, **kwargs)

foo('hello', 1, 2, 3, name="fred", age=20)

# outputs:

hello
(1, 2, 3, 4, 5)
{'name': 'fred', 'age': 20, 'teacher': 'Professor Smith'}

Finally, here’s a very basic and contrived example of using **kwargs for metaprogramming:

class Animal:
    def __init__(self, name, **kwargs):
        self.name = name

        if kwargs:
            for name, value in kwargs.items():
                setattr(self, name, value)

    def __str__(self):
        return str(self.__dict__)

cat = Animal("Cat", speak="meow", colour="white")
dog = Animal("Dog", speak="woof-woof", colour="golden", hobby="sleeping")

print(str(cat))
print(str(dog))

# outputs:

{'name': 'Cat', 'speak': 'meow', 'colour': 'white'}
{'name': 'Dog', 'speak': 'woof-woof', 'colour': 'golden', 'hobby': 'sleeping'}

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s