useProxy() hook without the macro

[sarimarton]

Hey guys,

Ever since I started using valtio, I wondered why there's this separation between the proxy object and the snapshot in a react component. For a while I thought it's a deliberate API decision for some kind of reading/writing separation pedantry, and I wasn't aware that there's actually a transpiler macro which simplifies the usage to my desired shape.

Now I learned that given that we have this macro, it should've been a technical limitation rather than by choice. And indeed, having to deal with the snapshot reference is pretty annoying. I have to find out a name, now we have this somethingSnap naming convention which is pretty lengthy, and it's still just a convention; then sometimes I mix up the proxy/snapshot references, so it's a source of error as well...

I started thinking of what the technical limitation can be here, and why don't we have a runtime useProxy hook which returns a kind of "meta proxy" which simply gives back the snapshot on reads, and the real proxy on writes.

I quickly came up with an implementation which worked on the basic cases, and then I started inserting it in my apps. It turned out that my meta proxy returns new references each time the component renders, and we need a stable reference in order to make it work with useEffect etc. But it was a fixable issue, so after adding some cache mechanism, this problem has been eliminated. I replaced useSnapshot to this all across my apps, and after a few corrections, it works just fine for me.

Do I miss something here? Or should I open a PR? Obviously there might be better intergrations, because I just copied some parts from valtio source files.

Here is the code:

import { useMemo } from 'react'
import { snapshot, useSnapshot } from 'valtio'

// Copied from https://github.com/pmndrs/valtio/blob/main/src/vanilla.ts#L64
const isObject = (x: unknown): x is object => typeof x === 'object' && x !== null

const canProxy = (x: unknown) =>
  isObject(x) &&
  // !refSet.has(x) &&
  (Array.isArray(x) || !(Symbol.iterator in x)) &&
  !(x instanceof WeakMap) &&
  !(x instanceof WeakSet) &&
  !(x instanceof Error) &&
  !(x instanceof Number) &&
  !(x instanceof Date) &&
  !(x instanceof String) &&
  !(x instanceof RegExp) &&
  !(x instanceof ArrayBuffer)

// Typing is just to make it work on the consuming side...
function createMetaProxy<T>(proxy: T, snap, plainSnap, cache): T {
  return new Proxy(snap, {
    get: (target, prop) => {
      if (canProxy(plainSnap[prop])) {
        if (!cache.has(snap[prop])) {
          cache.set(
            snap[prop],
            createMetaProxy(proxy[prop], snap[prop], plainSnap[prop], new WeakMap())
          )
        }
        return cache.get(snap[prop])
      }

      return snap[prop]
    },
    set: (target, prop, value) => {
      proxy[prop] = value
      return true
    }
  })
}

// Valtio useProxy hook which acts the same as the official useProxy macro
// (https://github.com/pmndrs/valtio#useproxy-macro), except that it's not a
// macro.
export const useProxy = <T extends object>(proxy: T) => {
  const snap = useSnapshot(proxy)

  // 'metaProxy' is a proxy which returns the snapshot on reads and updates the
  // proxy on writes. It keeps referencial equality.
  const metaProxy = useMemo(() => {
    return createMetaProxy(proxy, snap, snapshot(proxy), new WeakMap())
  }, [snap])

  return metaProxy
}

[dai-shi]

Separation of proxy and snapshot is "the reason" I started developing valtio.
It's necessary to support concurrent react. The separation is kind of my invention, and if it didn't come to me, I wouldn't have considered developing valtio as the difference from mobx would be mostly syntactic.

Having that said, what you tried is proposed several times. For example, see #38 and #238.

It works. The only concern is mutating a state multiple times.

const state = proxy({ count: 0 });

const Component = () => {
  const writableSnapshot = useProxy(state);
  const handleClick = () => {
    ++writableSnapshot;
    ++writableSnapshot;
    // we expect state.count becomes `2`, but actually it's `1`
  }
  ...
};

If we know this behavior, it's actually fine. But, this is the reason why such function isn't provided in valtio repo.

My suggestion for you is to develop a third-party library.
I'm pretty sure you can build it only with public valtio APIs.
You can use https://www.npmjs.com/package/on-change or createMetaProxy approach should also be fine if done properly.
If it goes well, we can consider promoting it in the valtio repo.

Looking forward to it.

[sarimarton]

Yeah, it was a weird realization that top level referential stability for refs doesn't matter actually (too much). For equality checking it's assumed to be the same, and for usage, we always get/set subkeys, and in those cases it's stable. Even if react returned with a new proxy ref on useRef with a get/set handler, I guess nobody would notice it. It would maybe only matter if someone passes it down as a prop, and then uses memo for render optimization, but useCallback suffers from this too.

[dai-shi]

Yeah, I wouldn't let people use useProxy blindly, they probably should still understand state/snapshot separation.

[sarimarton]

But do you plan, or consider adding this to the official repo? In any form, it might be just some "nice to have" thing, it doesn't matter to me how it's communicated, my only concern is that if it's gonna get some kind of official approval or not. The reason I'm asking is that it makes a difference when I'm trying to sell Valtio in the team that I need to attach my own stuff or not.

[dai-shi]

Yeah, at least I think it's much better than useProxy macro.

Can you open a PR to add your implementation in ./src/react-utils/useProxy.ts? I'll add some changes then. How it's communicated is my concern. Will see how to do it.

[sarimarton]

Cool, thanks @dai-shi. #626

[dai-shi]

https://github.com/pmndrs/valtio/releases/tag/v1.9.0 is released which includes new useProxy() hook, and deprecates old useProxy() macro.

Someone asked this:

Is there’s an explanation on when to use the util?

Short answer

Please try it. Just avoid mixing useProxy() and useSnapshot() in the same component (and all its hooks).

Long answer

useProxy() hook is a simple wrapper around useSnapshot(). You should read the source code.
It hides the separation of proxy states and snapshots. It would be easier for beginners and maybe experienced devs too.

As being a simple wrapper, not many things go wrong, and we highly recommend trying out the new hook. If you understand the implementation correctly, there's no risk using it in production.

Unless people try the new hook, we don't know if there are unknown pitfalls. So, it's encouraged from the library maintainer perspective. At the same time, keeping using useSnapshot is 100% okay and I personally prefer it (as I don't update state directly in a component anyway).

Known limitations

Here are some known limitations of useProxy() hook:

1. It uses an extra Proxy, which is a trivial overhead, but not zero.
2. useLayoutEffect shows console warning in SSR (in old react?) which is a false positive.
3. useProxy returns a new reference every time unlike useSnapshot, but that's actually fine because relaying on the reference stability is 99% misusage.
4. It only wraps the top proxy object, so nested objects in render will just be snaphosts. (This is probably solvable with cost.)
5. TypeScript type doesn't reflect read-only snapshots. (This is technically impossible as types are static.)

For 4, here's an example.

const state = { nested: { count: 1 } }

const Component = () => {
  const combined = useProxy(state)
  const { nested } = combined // this is a snapshot
  const handleClick = () => {
    ++nested.count // causes a runtime error
  }
  ...
}

[sarimarton]

const combined = useProxy(state)
const { nested } = combined
...
++nested.count

Hmm, I never thought of this use case... We could create an eslint plugin which only allows mutating from the top object property chain of a useProxy() instance. I don't think it's solvable other ways. In order to tell apart render/non-render usage, we have to sacrifice referential stability. (Or access react internals, but that's for another day.) The novelty of the new hook comes from the realization that the top object is mostly sacrificable for this purpose. I would name it a "switch proxy" which switches the property chain into snapshot in render, or primary proxy otherwise.

Maybe "useSwitchProxy" would be a clearer name. And along this line, maybe "useTrackingProxy" or "useTrackingSnapshot" would be a clearer name for useSnapshot, but this goes into a fancy direction :)

[dai-shi]

It's actually solvable if we return the switching proxy for nested objects. Yeah, but unless we see reasonable use cases, top level switching seems fine.

[sarimarton]

But then we'd lose reference stability for the whole tree.

[dai-shi]

true

[shamilovtim]

@dai-shi you say:

Separation of proxy and snapshot is "the reason" I started developing valtio.
It's necessary to support concurrent react. The separation is kind of my invention, and if it didn't come to me, I wouldn't have considered developing valtio as the difference from mobx would be mostly syntactic.

But my understanding is that MobX requires wrapping every React component that uses state with an observer() HOC. This is a non-trivial cost because component creation and update in React is expensive and useless components (like the host components created by HOCs like observer()) both degrade performance and bloat devtools.

To me, this is a non-trivial difference between Valtio and MobX. Personally, I actively avoid libraries that use HOCs because HOCs collide with the disadvantage's of React's rendering model and make devtools hideous and unpleasant to use.

In a Solid.js world perhaps MobX does not have this disadvantage but in React the difference between Valtio and MobX is not trivial.

[dai-shi]

To me, this is a non-trivial difference between Valtio and MobX.

Yes.

It's necessary to support concurrent react.

So, that wasn't precise. Separation of proxy and snapshot allows hooks instead of HoCs. (and hooks are designed for concurrency.)

[shamilovtim]

Now to speak on the biggest "gotcha" in my opinion to Valtio that everyone I know who uses Valtio has run into, it deals with the work OP has done. Valtio's insistence that useSnapshot and the proxy are separated is extremely confusing. The docs don't comment on whether the following is ok to do (and its very common code!):

// counterAtom.ts
export const counterAtom = proxy({ count: 1 });

// MyComponent.tsx
import { counterAtom } from './';

const MyComponent () => {
  const counterAtomSnapshot = useSnapshot(counterAtom):

  return 
    <>
      <Text>{counterAtomSnapshot.count}</Text>
      <Button onPress=(() => counterAtom.count++) />
    </>
}

Firstly, it is not obvious to both absolute beginners and advanced users whether using both the snapshot and the atom in the same component is even ok. Is it a footgun? Has the dev done something wrong or implemented an anti-pattern? They don't know, and the docs don't clarify.

Second, if it weren't for Typescript this would be so cumbersome as to be unmanageable. Even with Typescript when working heavily with Valtio I find myself accidentally writing to the snapshot constantly and getting red squigglies from Typescript or runtime errors from React Native.

It sounds like this useProxy is an imperfect solution because it comes with performance implications and it doesn't proxy nested elements. However I did want to touch on that the docs are currently lacking a best practice for dealing with both the atom and the snapshot at the same time and any API that simplifies this pain point (while preserving performance) would make Valtio a premiere choice for React state management. I'm not interested in MobX for the reasons I stated above and it's reliance on weird HOCs and ES6 classes which I view as a fundamentally broken and half finished API.

[sarimarton]

First I wanted to work on an eslint plugin to quickly tackle with the hook limitations, but I ended up with a new useProxy() implementation which I think mitigates all the currently known problems:

• referentially unstable root object (barely a real problem)
• only the root object being a switch proxy, so deconstructing it for further usage in the render fails with mutations in a callback

For the new version, I combined the very first demo (I fixed its issues) in this thread about a referentially stable deep meta proxy, and the idea of passing along the closures a stable ref of isRendering which flips back-and-forth in render/non render. I replaced the previous simple useProxy() with this new one, and my apps work fine. It passes the render optimization test, as well as the double assignments. It has the same usage as the previous one. This deep switch proxy only traps get, otherwise it falls back to the primary proxy, so it allows setting in render.

@dai-shi, this might be the ultimate solution?

(NB., I ran into an issue of an unexpectedly renewing snapshot reference, but it is unrelated to this, as I reproduced that with useSnapshot as well. I'll open a bug ticket about that.)

// Copied from https://github.com/pmndrs/valtio/blob/main/src/vanilla.ts#L64
const isObject = (x: unknown): x is object => typeof x === 'object' && x !== null

const canProxy = (x: unknown) =>
  isObject(x) &&
  // !refSet.has(x) &&
  (Array.isArray(x) || !(Symbol.iterator in x)) &&
  !(x instanceof WeakMap) &&
  !(x instanceof WeakSet) &&
  !(x instanceof Error) &&
  !(x instanceof Number) &&
  !(x instanceof Date) &&
  !(x instanceof String) &&
  !(x instanceof RegExp) &&
  !(x instanceof ArrayBuffer)

function createSwitchProxy<T extends object>(
  proxy: T,
  snap,
  plainSnap,
  cache,
  isRendering
): T {
  return new Proxy(proxy, {
    get: (target, prop) => {
      if (!isRendering.current) {
        return proxy[prop]
      } else if (!canProxy(plainSnap[prop])) {
        return snap[prop]
      } else {
        if (!cache.has(snap[prop])) {
          cache.set(
            snap[prop],
            createSwitchProxy(
              proxy[prop],
              snap[prop],
              plainSnap[prop],
              new WeakMap(),
              isRendering
            )
          )
        }
        return cache.get(snap[prop])
      }
    }
  })
}

/**
 * Takes a Valtio proxy and returns a new proxy which you can use in both react
 * render and in callbacks. Referentially stable. For the best ergonomics, you
 * can export a custom hook from your store, so you don't have to figure out a
 * separate name for the hook reference. E.g.:
 *
 * export const store = proxyWithComputed(initialState, {})
 * export const useStore = () => useProxy(store)
 * // in the component file:
 * function Cmp() {
 *   const store = useStore()
 * }
 *
 * @param proxy Valtio proxy
 * @returns A new proxy which you can use in the render as well as in callbacks.
 */
export const useProxy = <T extends object>(proxy: T) => {
  const snap = useSnapshot(proxy)

  const isRendering = useRef()

  isRendering.current = true

  useLayoutEffect(() => {
    isRendering.current = false
  })

  return useMemo(() => {
    return createSwitchProxy(
      proxy,
      snap,
      snapshot(proxy),
      new WeakMap(),
      isRendering
    )
  }, [snap, proxy, isRendering])
}

[sarimarton]

useRef allows there to have a per-component reference from which the traps can read the per-render isRendering value. That must be a per-component ref, otherwise you can't make it available in memo'ed closures without messing up the proxy ref stability.

const s1 = useProxy(state);
const s2 = useProxy(state);
console.log(s1.obj === s2.obj);

This one is a good point which I'll get back to below.

new Proxy(Object.freeze({ a: {} }), { get() { return {} } })

Ugh, I wouldn't even consider supporting such cases.

---

Now we have a shallow switch proxy and a deep switch proxy implementation. 100% agree that the ultimate simplicity of the shallow one is appealing, and the top object's ref stability is not a concern. However, a double hook on the same target doesn't seem like a real use case (nor would do if someone declared a separate switch proxy on a subkey (a = useProxy(foo); b = useProxy(foo.bar)). Whereas deconstructing objects is pretty common to shorten up repetitive long prop chains.

In either case, I summed up what IMO could do a good service to complete the viable options:

• useSnapshot:

  • eslint rule to only allow useSnapshot variable in render
  • docs/naming: referring back to @shamilovtim, the name "useSnapshot" somewhat unfairly silences its tracking purpose. Although I understand that it rhymes well to the vanilla "snapshot" API, and it's a snappier (sorry) name than something like "useTrackingProxy". But it's a bit inconsistent too: what if someone wants to get a real POJO snapshot from a useSnapshot proxy, for example, to pass it down as a prop without tracking, or any other purpose snapshot() could be used? They would need to "create a snapshot from a snapshot", which makes little sense. Whereas "untracking/snapshoting a tracking proxy" makes more sense semantically, and it cleans up the concept of the "snapshot" term to what it generally means: a non-proxy, dumb, POJO object.

• shallow switch proxy (current useProxy):

  • eslint rule to disallow mutating from a non-root key. Might be tricky to implement the rule, but definitely doable.

• deep switch proxy:

  • eslint rule to prevent declaring separate hook variables for proxies which are either the same or are in the same hierarchy. (It could be hijacked with separate named exports, but who cares.)

• For all of the above:

  • eslint rule to allow enforcement of a naming paradigm, e.g.: eslint-valtio-hooks/hookname: ['warn', { prefix: '$' }] would mandate const $state = useSnapshot(state). Or eslint-valtio-hooks/hookname: ['warn', { postfix: 'Snap' }] would mandate const stateSnap = useSnapshot(state)

[dai-shi]

useRef allows there to have a per-component reference

My understanding is it should be a per-render reference.

  const isRendering = { current: true }
  useLayoutEffect(() => {
    isRendering.current = false
  })

[sarimarton]

Then the wrapper object were meaningless, we get back to the const isRendering = false scenario which is sufficient for the shallow switch proxy, but not for the deep one. The shallow switcher has a new reference on render, so everything is per-render. In contrast, the deep switcher is able to retain its reference and tie it to the snapshot reference recursively - with useMemo on the top object (and a passed WeakMap on the lower ones).

return useMemo(() => {
    return createSwitchProxy(
      proxy,
      snap,
      snapshot(proxy),
      new WeakMap(),
      isRendering
    )
  }, [snap, proxy, isRendering])

Here, I only put isRendering in the dep array to silence the linter. Semantically, it's more like a [snap, proxy] or [snap], because isRendering never changes. If isRendering were a new reference sometimes, but snap keeps the same, then the stable content of useMemo would read the old closure with a stale { isRendering } object. The linter would be actually correct spotting a bug. And if you put an unstable isRendering in the useMemo dep list, then useMemo is meaningless, and nothing is stable in the switcher tree.

With useRef, we can control reference stability. In fact, we can make the shallow switcher stable with a useRef too:

export const useProxy = <T extends object>(proxy: T) => {
  const snapshot = useSnapshot(proxy)

  // stable info object
  const isRendering = useRef()
  
  isRendering.current = true

  // per-render update
  react.useLayoutEffect(() => {
    isRendering.current = false
  })

  return useMemo(() => new Proxy<T>(proxy, {
    get: function (target, prop) {
      // always reads the latest render value, no matter how old this closure is
      return isRendering.current ? snapshot[prop] : target[prop]
    }
  }), [snapshot]) // or [snapshot, isRendering] to silence the linter, it doesn't matter
}

In earlier comments you were actually correct by saying:

dai-shi
It's actually solvable if we return the switching proxy for nested objects. Yeah, but unless we see reasonable use cases, top level switching seems fine.

And I was wrong by responding:

sarimarton
But then we'd lose reference stability for the whole tree.

[dai-shi]

// always reads the latest render value, no matter how old this closure is

that's actually bad for concurrency.
but, i think i get your point. hm, actually i'm not sure if it's solvable in a way i wanted.

The current useProxy impl is very elegant, as it turns out.

[sarimarton]

that's actually bad for concurrency.

Yes, I see now.

[valstu]

What are your thoughts, should useProxy also support disabling batching with similar syntax to useSnapshot, so
useProxy(state, { sync: true })?

[valstu]

Oh and when I tried to use this with nested state it didn't work and I found this thread. I think this "limitation" would be worth adding to the useProxy docs.

[dai-shi]

Having the same option as useSnapshot is reasonable.

Yeah, we should have something in docs to keep readme small. On second thought, it's rather a design choice than a limitation. But, it should be documented clearly. Would you like to work on it?

[valstu]

Made a PR #639, not sure if the wording is clear enough?

[dai-shi]

Looks okay at a glance. You can mutate nested objects. Let's look for some compact wording.

We want to make readme super compact. We can't explain everything. What do you think about adding a new doc in docs folder?