Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add offline module #200

Draft
wants to merge 9 commits into
base: main
Choose a base branch
from
Draft

Add offline module #200

wants to merge 9 commits into from

Conversation

ziyuan-linn
Copy link
Member

The PR implements an offline version of the ml5.js library.

Webpack will now build ml5-offline.js and ml5-offline.min.js in addition to ml5.js and ml5.min.js. The file ml5-offline.js and ml5-offline.min.js is self-contained and will not make additional network requests. Once a user downloads ml5-offline.min.js, the ml5 library can run entirely without internet.

The entry point for the offline model is src/offline/index.js. It imports the ml5 object, injects a loadOfflineModel function into each ml5 model's prototype, and exports the ml5 object with the modified models. The approach keeps the offline implementation mostly separate from the main source code.

Changes:

  • add a script that downloads the pre-trained models from Kaggle.com.
    • The script converts the model JSON and binary files to JS files.
    • model.json is written to a JS file.
    • binary files converted to Uint8Array and then written to JS files.
    • This is done so that Webpack can pack the model files and source code into a single file.
  • implement a loadOfflineModel function.
    • This function loads the above data into File objects and creates object URLs for them.
    • The object URLs are then injected into the "model URL" fields (e.g. landmarkModelURL) of the ml5 model.
  • For the ml5-offline.js build, add the loadOfflineModel functions into object prototypes of each ml5 model.
  • Call the loadOfflineModel function from each ml5 model during initialization if the function is defined.

To test this implementation, find an example sketch and change ml5.js to ml5-offline.js in index.html.

Note: This is WIP, ml5-offline.js only works for HandPose with modelType: "lite".

@shiffman
Copy link
Member

This is exciting to see thank you @ziyuan-linn for working on this! One consideration is that running a stand-along node.js script for downloading the models may involves concepts/steps unfamiliar to beginners especially those used to working only in the p5.js web editor. I think it's ok for this to be the primary way to get the models since this is an "edge" use case, however, we should probably include some detailed documentation for total beginners. I could also record a video tutorial on this topic. Alternatively we could consider providing links to some model downloads directly but this might be hard to maintain.

@ziyuan-linn
Copy link
Member Author

@shiffman Thank you for the review! The node.js script for downloading models is meant to be only called by the ml5.js maintainers. I could also set it up as a postinstall script that automatically runs after yarn so we never have to worry about it. The purpose of the script is to download the models so WebPack can bundle the model files into the ml5 library file itself.

As far as the users are concerned, they only need to change ml5.js to ml5-offline.js in the script tag. If they want to go fully offline, they can also download the p5.js and ml5-offline.js scripts and put them in the sketch folder. We can definitely have a quick guide/documentation for this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants