Validate items uniqueness in z.array

[ramos-ph]

I was checking the docs to see if there was any way to validate if there was any duplicate items on an array, perhaps we could implement a unique() function that does just that for arrays?

const schema = z.array(z.string().unique());

schema.parse(['a', 'b', 'c']); // passes
schema.parse(['a', 'a', 'c']); // fails: 'a' is included twice

Or for more complex arrays:

const schema = z.array(z.object({
  prop: z.string().unique()
}));

schema.parse([{ prop: 'a', prop: 'b' }]); // passes
schema.parse([{ prop: 'a', prop: 'a' }]); // fails: prop: 'a' is included twice

Is this doable or is it out of scope?

[joaoGabriel55]

Would be very useful!

[HassanZahirnia]

Having a unique() would really be nice!
For now I'm using this as an alternative:

z.array(z.string()).refine(items => new Set(items).size === items.length, {
    message: 'Must be an array of unique strings',
})

[MentalGear]

If you need to reuse it across various fields, use this wrapping function:

function uniqueArray(schema: ZodType) {
  return z
    .array(schema)
    .refine((items) => new Set(items).size === items.length, {
      message: 'All items must be unique, no duplicate values allowed',
    });
}
  

const schema = {
  setString: uniqueArray(z.string().minLength(87)),
  setNumber: uniqueArray(z.number()),
  setBoolean: uniqueArray(z.boolean()),
 }

[eli-rlima]

I agree @joaoGabriel55, it would be very useful!

[aarontravass]

Has there been any update on this? Would be super useful to validate a unique array of primary keys without hitting the db layer.

[joaoGabriel55]

I'm working on a solution for that.

[ValTashkov]

Hello @joaoGabriel55

The method looks dope but I don't see how one can get info about which specific elements are duplicated. I think that would be a great addition. Let me give an example:

// We have the following schema
const fruitsEnum = zod.enum([ 'apple', 'banana', 'grapes' ])

// ...and let's say a DTO (for example for an endpoint that will receive fruits idk)
const fruitDto = z.object({
    fruits: z.array(fruitsEnum).uniqueness()
})
 
// ...and we want to send this array to our endpoint
const fruitsArrayWithDuplicates = [ 'apple', 'banana', 'grapes', 'apple' ]

const fruitsRequestObject = {
    fruits: fruitsArrayWithDuplicates
}

// when we send the array with duplicates
postFruits('/fruits', fruitsRequestObject)

// We should throw an error, because we have the fruit 'apple' twice

Following the example. What I am interested in is to have a way of informing the user Hey, you have sent the fruit 'apple' twice, AKA know exactly which element(s) have been duplicated so we can inform the user.

Currently I don't see that implemented as you are just converting the array to a Set and checking the size against the original array (correct me if I am wrong)

It would be a great addition to the method (or maybe another method specifically checking which elements get duplicated).

I would love to hear your feedback on this proposal. I can't think rn how an API for that would look like exactly, and I am not interested in the concrete implementation, but rather if you think it would make sense to have it implemented

Cheers,
Valentin

[aarontravass]

@ValTashkov this sounds great! One nitpick tho. uniqueness should take a configuration which asks the user whether he would like to view duplicate objects since I believe displaying it by default is computationally more expensive than just saying duplicate items were found. What do you reckon?

[ValTashkov]

Yes, an option that accepts a callback that has an array of the duplicated elements as first argument and returns a custom message with the duplicates should do the trick. I imagine something like this:

const users =[
   {
      id: 1,
      name: "Pesho"
    },
 {
      id: 2,
      name: "Gosho"
    }
{
      id: 1,
      name: "Pesho"
    }
]
z.array(users).uniquness(
      (user) => user.name,

     // duplicatedUsersArray = [ { id: 1, name: 'Pesho' } ]
     (duplicatedUsersArray) => `Duplicated users found: ${duplicatedUsersArray.map(user=>user.name).join(', ')}` 
)

  // And the error would look like this:
  // Duplicated users found: Pesho

I know the example is a bit complex, but wanted to show that this solution can also work with objects as well

[aarontravass]

I see one issue with this. If the array consisted of large and deep objects, JSON.stringify would be slow and subsequently, logging the entire object wouldn't be performant. Maybe instead of printing the object itself, the object's index in the array can be printed.

[ValTashkov]

I don't see where we would need JSON.stringify in this example, even with nested objects

We add second argument to the method, that is a callback that takes as first argument an array of the duplicated items. After that you can do whatever you like. In my case I am mapping the array of user to just the names

[aarontravass]

No I meant that if the duplicate items array has many dense objects, logging them would be expensive. On Sep 14, 2023, at 2:22 AM, ValTashkov ***@***.***> wrote:  I don't see where we would need JSON.stringify in this example, even with nested objects We add second argument to the method, that is a callback that takes as first argument an array of the duplicated items. After that you can do whatever you like. In my case I am mapping the array of user to just the names — Reply to this email directly, view it on GitHub<#2316 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AKO6F3JCQXNYPDLWTJD6ZN3X2KPDPANCNFSM6AAAAAAWZBHTTA>. You are receiving this because you commented.Message ID: ***@***.***>

[ValTashkov]

I believe this is customer's concern, not zod's

[christophemarois]

Made a simple helper:

import z from 'zod';

function zUniqueArray<
  ArrSchema extends z.ZodArray<z.ZodTypeAny, 'many'>,
  UniqueVal
>(
  uniqueBy: (item: z.infer<ArrSchema>[number]) => UniqueVal,
  schema: ArrSchema
) {
  return schema.superRefine((items, ctx) => {
    const seen = new Set<UniqueVal>();
    for (const [i, item] of items.entries()) {
      const val = uniqueBy(item);
      if (seen.has(val)) {
        ctx.addIssue({
          code: z.ZodIssueCode.custom,
          message: `Unique property validation failed`,
          path: [i],
        });
      } else {
        seen.add(val);
      }
    }
  });
}

const uniqueIdSchema = zUniqueArray(
  (obj) => obj.id,
  z.array(z.object({ id: z.number() }))
);

// succeeds
console.log(uniqueIdSchema.safeParse([{ id: 1 }, { id: 2 }]));
// fails
console.log(uniqueIdSchema.safeParse([{ id: 1 }, { id: 1 }]));

[ThePiyushAggarwal]

I didn't read the discussion much. I am using this
uniq function is from lodash

z.array().transform((data) => uniq(data))

[ootkin]

It seems to work, but its a transformation stage, not a validation. Right now thanks to your suggestion I'm using uniqueBy instead to check only a specified attribute in the object:

import {uniqBy} from 'lodash';

z.array(
  z.object({
    name: z.string(),
    locale: z.string(),
  })
).transform((value) => uniqBy(value, 'locale'));

[tangaye]

Another way is to use superRefine as per the docs: https://zod.dev/?id=superrefine

    const Strings = z.array(z.string()).superRefine((val, ctx) => {

              if (val.length !== new Set(val).size) {
                         ctx.addIssue({
                                 code: z.ZodIssueCode.custom,
                                 message: `No duplicates allowed.`,
                         });
                }
     });

[Manuel-Bauer]

Another way is to use superRefine as per the docs: https://zod.dev/?id=superrefine

    const Strings = z.array(z.string()).superRefine((val, ctx) => {

              if (val.length !== new Set(val).size) {
                         ctx.addIssue({
                                 code: z.ZodIssueCode.custom,
                                 message: `No duplicates allowed.`,
                         });
                }
     });

This works. How would you then validate each item of the array, to e.g. display an error message at that input field that had caused the "duplication"?

[akellbl4]

@Manuel-Bauer here is my solution with errors assigned to fields

z.array({
      value: z.string()
}).superRefine((items, ctx) => {
      const uniqueValues = new Map<string, number>()
      items.forEach((item, idx) => {
        const firstAppearanceIndex = uniqueValues.get(item.value)
        if (firstAppearanceIndex !== undefined) {
          ctx.addIssue({
            code: z.ZodIssueCode.custom,
            message: `Value must be unique`,
            path: [...ctx.path, idx, 'value'],
          })
          ctx.addIssue({
            code: z.ZodIssueCode.custom,
            message: `Value must be unique`,
            path: [...ctx.path, firstAppearanceIndex, 'value'],
          })
          return
        }
        uniqueValues.set(item.value., idx)
      })
    })

[m-abboud]

z.array({
      value: z.string()
}).superRefine((items, ctx) => {
      const uniqueValues = new Map<string, number>()
      items.forEach((item, idx) => {
        const firstAppearanceIndex = uniqueValues.get(item.value)
        if (firstAppearanceIndex !== undefined) {
          ctx.addIssue({
            code: z.ZodIssueCode.custom,
            message: `Value must be unique`,
            path: [...ctx.path, idx, 'value'],
          })
          ctx.addIssue({
            code: z.ZodIssueCode.custom,
            message: `Value must be unique`,
            path: [...ctx.path, firstAppearanceIndex, 'value'],
          })
          return
        }
        uniqueValues.set(item.value., idx)
      })
    })

I had to change this part from path: [...ctx.path, idx, 'value'], to just path: [idx, 'value'], to get it to work but sweet ty

[FraternalFiber]

You can combine _.isEqual and _.uniqBy functions from Lodash.

const itemSchema = z.object({
    prop: z.string()
})

items: z
    .array(itemSchema)
    .refine(items => _.isEqual(items, _.uniqBy(items, 'prop')))

[MentalGear]

A more generalized version (based on the past answer of @HassanZahirnia).
If you need to reuse it across various fields, use this wrapping function:

function uniqueArray(schema: ZodType) {
  return z
    .array(schema)
    .refine((items) => new Set(items).size === items.length, {
      message: 'All items must be unique, no duplicate values allowed',
    });
}
  

const schema = {
  setString: uniqueArray(z.string().minLength(87)),
  setNumber: uniqueArray(z.number()),
  setBoolean: uniqueArray(z.boolean()),
 }

[MentalGear]

I just found out that zod has Sets, which might be more what someone would like to use instead of array with uniquness.

[FelixZY]

Zod sets didn't exist when I originally upvoted this question a while back IIRC. Glad to see they do now! 🎉

[DibyodyutiMondal]

Zod sets is amazing for simple native types

But what if we want to customize what "unique" means?

Suppose I have an array of objects with an id field and I want to make sure that the ids do not repeat in the array, then zod Sets does not fulfill the criteria, unless I missed some part of the documentation

[BrendanC23]

You can write a custom validation function and use refine or superRefine.

schema.superRefine((data, ctx) => {
    const ids = data.items.map(item => item.id);
    const uniqueIds = new Set(ids);
    
    if (ids.length !== uniqueIds.size) {
        ctx.addIssue({
            code: z.ZodIssueCode.custom,
            message: `Items must have unique ids.`,
            path: ["items"],
        });
    }
});

[DibyodyutiMondal]

Another problem with z.sets I ran into was that it ran into "expected set, received array" errors when trying to parse data from an api.
I'm not sure if that is a bug, but I use the following now:

export function zodUniqueArray<Schema extends ZodSchema, K>(
  schema: Schema,
  idSelector: (item: Schema['_output']) => K = item => item
) {
  return z.array(schema)
    .refine(
      items => {
        const ids = new Set<K>(items.map(i => idSelector(i)));
        return ids.size === items.length;
      },
      'array-items-must-be-unique'
    );
}

Usage without a id selector

suppose I have nested records and I need the leaf string arrays to have unique values

const domainSoftwareMap: Record<string, Record<string, string[]>> = {};

Usage with an id selector

Suppose I have a list of objects, and I need to check uniqueness by the _id field

const roles: { _id: string, name: string, descripton: string }[]= [];

[taig]

Validating that an array has no duplicates and transforming into a Set:

const set = <A extends ZodTypeAny, B extends z.ArrayCardinality>(array: z.ZodArray<A, B>) =>
  array.transform((items, context) => {
    const set = new Set(items)

    if (set.size !== items.length) {
      context.addIssue({
        code: z.ZodIssueCode.custom,
        message: "uniqueItems",
      })

      return z.NEVER
    }

    return set
  })