Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(); eth balance & docs #118

Open
wants to merge 25 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 9 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,12 @@
# Messari Substreams
# Messari Substreams • [![GitHub license](https://img.shields.io/badge/license-MIT-blue)](https://github.com/messari/substreams/blob/master/LICENSE) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://github.com/messari/substreams/compare) [![Issues Report](https://img.shields.io/badge/issues-report-yellow.svg)](https://github.com/messari/substreams/issues/new)

<p align="center">
<img src="./docs/images/logos/messari-logo.png" alt="Messari Logo" width="460" />
</p>

Messari substreams aim to contextualize on chain data using [streamingfast](https://www.streamingfast.io/)'s [substreams](https://substreams.streamingfast.io/) 🚀

> Learn more about messari substream development in our [`./docs`](./docs)

## Pre-requisites

Expand Down
17 changes: 17 additions & 0 deletions common/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Common Definitions

This folder contains common definitions for protobufs used more than once. We intentionally have this in order to standardize the way we define the data to make it easier for the end user to "plug-n-play".

We do NOT want to "over-standardize" on a substream level. Substreams works with raw, lower-level data. We do not want to lose any descriptiveness at this layer, therefore it is important to not generalize any data to try and fit a standard.

## Common Proto

TODO

## DEX AMM Proto

TODO

## EVM Token Proto

This proto is designed to support fungible tokens on evm chains. In order to preserve the data from each chain we have separate definitions for tokens across chain implementations.
30 changes: 19 additions & 11 deletions common/proto/erc20.proto
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ message ERC20Token {
string name = 2;
string symbol = 3;
uint64 decimals = 4;
string tx_created = 5;
uint64 block_created = 6;
}

message TransferEvents {
Expand All @@ -19,20 +21,26 @@ message TransferEvents {

message TransferEvent {
string tx_hash = 1;
uint32 log_index = 2;
uint64 log_ordinal = 3;
string token_address = 4;
string from = 5;
string to = 6;
string amount = 7; // BigInt, in token's native amount
uint64 block_number = 2;
uint64 timestamp = 3;
uint32 log_index = 4;
optional uint64 log_ordinal = 5;
string token_address = 6;
string from = 7;
string to = 8;
string amount = 9; // BigInt, in token's native amount
repeated TokenBalance balance_changes = 10;
}

message TokenBalance {
string token_address = 1;
string balance = 2; // BigInt, in token's native amount
optional uint64 log_ordinal = 1;
ERC20Token token = 2;
string address = 3; // account address of the balance change
optional string old_balance = 4; // BigInt, in token's native amount
string new_balance = 5; // BigInt, in token's native amount
optional int32 reason = 6;
}

message Account {
string address = 1;
repeated TokenBalance balances = 2;
message TokenBalances {
TokenBalance items = 1;
}
14 changes: 11 additions & 3 deletions common/src/codegen.rs
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,9 @@ pub fn generate_pb(out_dir: Option<&str>) -> Result<(), Error> {
// parse version from file name
let filename = file.split('.').collect::<Vec<&str>>();
// let package_name = filename[0];
if filename.len() < 2 {
continue;
}
let name = filename[1].to_string();
let version = filename[2];
pb_files_hash
Expand Down Expand Up @@ -146,15 +149,20 @@ pub fn generate_pb(out_dir: Option<&str>) -> Result<(), Error> {
let pb_file_content = pb_files
.into_iter()
.map(|(filename, versions)| {
let mut name = "messari".to_string();
if filename == "entity".to_string() {
name = "substreams".to_string();
}
let (mod_content, registration_content): (Vec<String>, Vec<String>) = versions
.into_iter()
.map(|version| {
(

format!(
"#[rustfmt::skip]\n\
#[path = \"../{}/pb/messari.{}.{}.rs\"]\n\
pub(in crate::pb) mod {1}_{2};\n",
out_dir, filename, version
#[path = \"../{}/pb/{}.{}.{}.rs\"]\n\
pub(in crate::pb) mod {2}_{3};\n",
out_dir, name, filename, version
),
format!(
" pub mod {} {{\n \
Expand Down
8 changes: 4 additions & 4 deletions compound-v2/src/pb.rs
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
#[rustfmt::skip]
#[path = "../target/pb/messari.compound.v1.rs"]
pub(in crate::pb) mod compound_v1;
#[path = "../target/pb/messari.erc721.v1.rs"]
pub(in crate::pb) mod erc721_v1;

pub mod compound {
pub mod erc721 {
pub mod v1 {
pub use super::super::compound_v1::*;
pub use super::super::erc721_v1::*;
}
}
3 changes: 3 additions & 0 deletions docs/DECODING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Decoding Data

Firehose delivers data in a very raw format and in order to make use of the data we need to be able to decode it. This file will go over examples of where/how you might need to do this.
9 changes: 9 additions & 0 deletions docs/STANDARDS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Standards

Standardizing where it makes sense and early pays dividends in terms of developer time in the long run. This file will explore where we standardize and why those decisions were made.

## Protobuf Definitions

Where the same data structures are used multiple times, it makes sense to make a [common](../common) protobuf definition. It shows the user this definition can be found in multiple places. It also makes the code base cleaner. You can find these under [`../common/proto`](../common/proto).

> You can learn more details about this [here](../common/README.md)
Binary file added docs/images/logos/messari-logo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion erc20-holdings/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,4 @@ build:

.PHONY: run
run:
substreams run -e mainnet.eth.streamingfast.io:443 substreams.yaml map_block_to_erc20_contracts -s 1
substreams run -e mainnet.eth.streamingfast.io:443 substreams.yaml map_block_to_erc20_contracts -s 6082465 -t +1
8 changes: 8 additions & 0 deletions erc20-holdings/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# ERC20 Holdings

This substream will find all of the ERC20 tokens on ethereum with metadata. In addition it will map transfers. Balance can be derived from this. Then the `store_balance_usd` module combines chainlink data to store the prices as well.

## Notes

- The `map_block_to_erc20_contracts` module should be a `store` module since multiple other modules will want to use it as input to get ERC20 token metadata.
- `map_block_to_erc20_contracts` gets ERC20 metadata some of the time. It should grab all ERC20 contracts, however the metadata is not always populated. I am not quite sure why this is.
Loading