Remove Duplicates from an Array in Java

How to Remove Duplicates from an Array in Java

Once you know how to find duplicates, the natural next question is how to actually remove them. Java's fixed-size arrays introduce an interesting limitation: you can't literally shrink an array, so removing duplicates really means creating a new array (or list) that contains only distinct values. This guide covers the most practical approaches for both sorted and unsorted arrays.

Problem Statement

Given an array like {4, 2, 7, 4, 9, 2, 8}, the goal is to produce an array containing only unique values while preserving the original order of first appearance:

{4, 2, 7, 9, 8}

Method 1: Using LinkedHashSet (Order-Preserving, Unsorted Arrays)

import java.util.LinkedHashSet;
import java.util.Set;

int[] numbers = {4, 2, 7, 4, 9, 2, 8};

Set<Integer> uniqueSet = new LinkedHashSet<>();

for (int num : numbers) {
    uniqueSet.add(num);
}

int[] result = uniqueSet.stream()
                        .mapToInt(Integer::intValue)
                        .toArray();

System.out.println(java.util.Arrays.toString(result)); // [4, 2, 7, 9, 8]

LinkedHashSet is the ideal choice because it automatically removes duplicate elements while preserving their original insertion order.

Method 2: Two-Pointer Technique (Sorted Arrays)

If the array is already sorted (or can be sorted first), duplicate values become adjacent, allowing an efficient in-place solution.

import java.util.Arrays;

int[] numbers = {2, 2, 4, 4, 7, 8, 9};

int writeIndex = 1;

for (int readIndex = 1; readIndex < numbers.length; readIndex++) {
    if (numbers[readIndex] != numbers[writeIndex - 1]) {
        numbers[writeIndex] = numbers[readIndex];
        writeIndex++;
    }
}

int[] result = Arrays.copyOf(numbers, writeIndex);

System.out.println(Arrays.toString(result)); // [2, 4, 7, 8, 9]

This classic interview technique runs in O(n) time while using only O(1) extra space (excluding the final trimmed copy).

Method 3: Java Streams (distinct())

import java.util.Arrays;

int[] numbers = {4, 2, 7, 4, 9, 2, 8};

int[] result = Arrays.stream(numbers)
                     .distinct()
                     .toArray();

The distinct() operation removes duplicate elements while preserving encounter order, making it one of the cleanest solutions for modern Java code.

Step-by-Step Explanation

LinkedHashSet Method

Every element from the array is inserted into a LinkedHashSet.

Automatic Duplicate Removal

Whenever a duplicate element is encountered, the set ignores it automatically because sets only store unique values.

Preserving Original Order

Unlike HashSet, LinkedHashSet remembers the insertion order, so the first occurrence of every element remains in its original position.

Converting Back to an Array

After removing duplicates, the set is converted back into a primitive int[] using a stream.

Internal Working (Memory View)

For the sorted array:

{2, 2, 4, 4, 7, 8, 9}

The two-pointer method works as follows:

writeIndex = 1

readIndex = 1
2 == 2 → Skip

readIndex = 2
4 != 2 → Write at index 1
Array: [2, 4, 4, 4, 7, 8, 9]
writeIndex = 2

readIndex = 3
4 == 4 → Skip

readIndex = 4
7 != 4 → Write at index 2
Array: [2, 4, 7, 4, 7, 8, 9]
writeIndex = 3

readIndex = 5
8 != 7 → Write at index 3
Array: [2, 4, 7, 8, 7, 8, 9]
writeIndex = 4

readIndex = 6
9 != 8 → Write at index 4
Array: [2, 4, 7, 8, 9, 8, 9]
writeIndex = 5

Final result:
[2, 4, 7, 8, 9]

Notice that the original array is modified in-place. The values after writeIndex become irrelevant, which is why Arrays.copyOf() creates the final trimmed array.

Real-Life Analogy

Imagine creating a guest list from a sign-in sheet where some guests accidentally signed in multiple times. As you read each name, you add it to your final guest list only if it hasn't already been added. By the end, every guest appears exactly once and in the same order they first arrived.

Best Practices

• Use LinkedHashSet when working with unsorted arrays and preserving insertion order is important.
• Use the two-pointer technique for sorted arrays because it provides the best memory efficiency.
• Use Stream.distinct() for concise and readable modern Java code.
• Remember that Java arrays have a fixed size, so removing duplicates always produces a new array or a trimmed copy.

Common Mistakes

1. Using HashSet instead of LinkedHashSet when the original order needs to be preserved.
2. Assuming Java arrays can shrink dynamically after removing duplicates.
3. Applying the two-pointer technique to an unsorted array, which produces incorrect results.
4. Forgetting to trim the array after the two-pointer process, leaving unwanted leftover elements.

Expert Tips

• The read/write pointer technique used here is widely applicable to many in-place array modification problems, such as removing specific values or moving zeroes to the end.
• For arrays containing custom objects, ensure the class properly overrides equals() and hashCode() so that duplicate detection works correctly.
• When memory usage is critical, the sorted two-pointer approach is generally the most efficient solution.

Comparison Table

Frequently Asked Questions

Can I remove duplicates without sorting the array first?

Yes. Both LinkedHashSet and Stream.distinct() work with unsorted arrays while preserving the original order.

Why should I use LinkedHashSet instead of HashSet?

LinkedHashSet maintains insertion order, whereas HashSet does not guarantee any ordering.

Can Java arrays be resized after removing duplicates?

No. Arrays are fixed-size. You must create a new array, usually using Arrays.copyOf(), to store only the unique elements.

What is the most memory-efficient way to remove duplicates?

The two-pointer technique on a sorted array uses only O(1) extra space (excluding the final trimmed copy).

Does distinct() preserve the original order?

Yes. Stream.distinct() preserves encounter order for ordered streams, including arrays processed with Arrays.stream().

How do I remove duplicates from an array of Strings?

The same approaches work. For example, use LinkedHashSet<String> or:

Arrays.stream(stringArray)
      .distinct()
      .toArray(String[]::new);

What if my array contains custom objects?

Ensure your class correctly overrides equals() and hashCode() because both LinkedHashSet and distinct() depend on them.

Is the two-pointer technique useful only for removing duplicates?

No. The same read/write pointer pattern is commonly used for removing specific values, compacting arrays, moving zeroes, and many other in-place array transformation problems.