How to handle large pulls of data?

[maXXCZ1]

Hello everyone,

we built app with watermelonDB that works with large SQL database. We can manage (communicate with user) that initial sync is long, but our main issue is that after user is not using app for some time (we are talking days, not months) he/she receives quite a lot of new data and the biggest problem is that watermelon is using JS thread to write it down all at once. This leads to complete freeze of UI after the start which can last from second to tens (!) of seconds. As you can imagine, that is unacceptable.
Originally i searched within issues and found this topic below:
#1183
If we understood it correctly, that is not exactly our problem. We have issues purely on user's side when WM is writing into DB. For small amount of data the lag is negligible, but that is sadly not our case very often.

First thought was same as in mentioned issue, periodical sync every xy hours even when app is in quit state. Problem is, that after research we found out that especially for iOS any solution is hacky at best to put it mildly.
I feel like we are missing something as this would likely be issue in many more applications of WMDB, aren't we?

Did you find any way to don't spoil user experience after larger sync of data other than only syncing very minor part of recently added records or some kind of UI that manually starts pull from server?
Can _unsafeBatchPerCollection help in this case?

[lucaswitch]

Go paginated!
Hey if you're dealing on react-native/reactjs and using reactjs you can solve this using this simple hook:
So the data is fetched and you can use the next, prev functions to handle the pagination

export function useWatermelonModelsPage({
  collection,
  database,
  onChange,
  query,
}) {
  const LIMIT = 20;
  const HALF_LIMIT = Math.ceil(LIMIT / 2);
  const [cPage, setCPage] = useState(0);
  const [count, setCount] = useState(0);

  /**
   * Go to next page.
   * @type {(function())|*}
   */
  const next = useCallback(
    function () {
      const lastPage = Math.ceil(count / HALF_LIMIT);
      if (cPage < lastPage) {
        let current = cPage + 1;
        setCPage(current);
      }
    },
    [count, HALF_LIMIT, cPage],
  );

  /**
   * Go to prev page.
   * @type {(function())|*}
   */
  const prev = useCallback(
    function () {
      if (cPage !== 0) {
        const current = cPage - 1;
        setCPage(current);
      }
    },
    [cPage],
  );

  /**
   * Computes the total count.
   * @type {(function())|*}
   */
  const computeCount = useCallback(
    async function () {
      const current = await database
        .get(collection)
        .query(Q.where('is_deleted', false))
        .fetchCount();
      setCount(current);
    },
    [collection, database],
  );

  /**
   * Computes the page items.
   * @type {(function())|*}
   */
  const computeItems = useCallback(
    function () {
      const observable = database.collections
        .get(collection)
        .query(...query)
        .observe();
      const observer = observable._subscribe({
        next: onChange,
        error(err) {
          console.error(err);
        },
        complete() {},
      });

      return observer.unsubscribe;
    },
    [collection, database.collections, onChange, query],
  );

  useEffect(
    function () {
      computeCount();
      return computeItems();
    },
    [computeItems, computeCount],
  );

  return {count, prev, next};
}

// Then you can call it on a reactjs/react-native component like this:

const query = useMemo(
    function () {
      const conditions = [
        Q.where('is_deleted', false),
        Q.sortBy('name', Q.asc),
      ];

      if (search.enabled) {
        const value = search.value || '';
        if (value.length > 0) {
          conditions.push(Q.where('name', Q.like(`%${value}%`)));
        }
      }
      return conditions;
    },
    [search?.enabled, search.value],
  );
  const {next, prev} = useWatermelonModelsPage({
    database,
    collection: User.table,
    onChange: setUsers,
    query,
  });

[maXXCZ1]

Thank you for your suggestion.

What i was not fully aware of is fact that there is additional computing when (after) writing records into database after we pull changes from server so it causes significant delay. So for the record we came back to idea of periodical updates while looking into optimizing the process of writing.