Python Remove Duplicates from a List

There are many ways to remove duplicates from a Python List.

• Using a temporary List and Looping
• Using set() built-in method
• Using Dictionary Keys
• List count() function
• List Comprehension

Removing Duplicates from a List

Python list can contain duplicate elements. Let’s look into examples of removing the duplicate elements in different ways.

1. Using Temporary List

This is the brute-force way to remove duplicate elements from a list. We will create a temporary list and append elements to it only if it’s not present.

ints_list = [1, 2, 3, 4, 3, 2]

temp = []

for x in ints_list:
    if x not in temp:
        temp.append(x)

ints_list = temp

print(f'Updated List after removing duplicates = {temp}')

Output: Updated List after removing duplicates = [1, 2, 3, 4] Recommended Reading: Python f-strings

2. set() function

Python set doesn’t have duplicate elements. We can use the built-in set() function to convert the list to a set, then use the list() function to convert it back to the list.

ints_list = [1, 2, 3, 4, 3, 2]

ints_list1 = list(set(ints_list))
print(ints_list1)  # [1, 2, 3, 4]

3. List elements as Dictionary Keys

We know that dictionary keys are unique. The dict class has fromkeys() function that accepts an iterable to create the dictionary with keys from the iterable.

ints_list = [1, 2, 3, 4, 3, 2]

ints_list2 = list(dict.fromkeys(ints_list))
print(ints_list2)  # [1, 2, 3, 4]

4. List count() function - not recommended

The list count() method returns the number of occurrences of the value. We can use it with the remove() method to eliminate the duplicate elements from the list.

ints_list = [1, 2, 3, 4, 3, 2]

for x in ints_list:
    if ints_list.count(x) > 1:
        ints_list.remove(x)
print(ints_list)  # [1, 2, 3, 4]

NOTE: As pointed out in the below comment, using the count() function is not advisable while removing the element from the same iterator because it can lead to unwanted results. For example:

values = [87, 94, 45, 94, 94, 41, 65, 94, 41, 99, 94, 94, 94]

for x in values:
    if values.count(x) > 1:
        values.remove(x)
print(values)  # [87, 45, 65, 41, 99, 94, 94] - 94 is still present twice

5. List Comprehension

We can create a list from an iterable using the list comprehension. This technique is the same as using the temporary list and the for loop to remove the duplicate elements. But, it reduces the number of lines of the code.

int_list = [1, 2, 3, 4, 3, 2]
temp = []
[temp.append(x) for x in ints_list if x not in temp]
print(temp)  # [1, 2, 3, 4]

Best Way to Remove Duplicates from a List

If you don’t want duplicate elements, you should use Set. But, if you have to remove the duplicate values from a list, then I would prefer count() function because it doesn’t create another temporary set or list object. So, it’s more memory efficient.