Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add reference id for idempotent refence counter operations #100

Draft
wants to merge 15 commits into
base: master
Choose a base branch
from
Draft

add reference id for idempotent refence counter operations #100

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
953263b
add reference id
xiegeo Jun 17, 2020
e86f5e7
add counter to dag removal
xiegeo Jun 18, 2020
7c6df84
update dependencies
xiegeo Jun 18, 2020
ae81063
more updates
xiegeo Jun 18, 2020
46ef6ec
Merge branch 'master' into delete-id
xiegeo Jun 18, 2020
ff2e8cf
pull in more updates
xiegeo Jun 19, 2020
78475c4
fix repeated id
xiegeo Jun 19, 2020
36beaa3
generate go bindings
xiegeo Jun 19, 2020
23123e1
add progressive option and generate all code except for java
RT-nilPointer Aug 30, 2020
f2b2674
go mod tidy
RT-nilPointer Aug 30, 2020
a0ad397
Merge branch 'master' into delete-id
RT-nilPointer Sep 3, 2020
252a3bf
prevent tool installation from upgrading dependencies to keep grpc at…
RT-nilPointer Sep 8, 2020
b7e0579
downgrade go-libp2p-core because v0.7.0 has compatibilty problems for…
RT-nilPointer Sep 26, 2020
1c3b690
update deps
RT-nilPointer Sep 26, 2020
7ee9da6
add replace functionality to RefID
RT-nilPointer Nov 19, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .script/protoc-java.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ mkdir -p build
PLUGIN_EXE=protoc-gen-grpc-java-$VERSION-$PLATFORM.exe
OUT=build/protoc-gen-grpc-java
echo Downloading $PLUGIN_EXE to $OUT...
curl https://search.maven.org/remotecontent?filepath=io/grpc/protoc-gen-grpc-java/$VERSION/$PLUGIN_EXE \
curl -C - https://search.maven.org/remotecontent?filepath=io/grpc/protoc-gen-grpc-java/$VERSION/$PLUGIN_EXE \
-o $OUT

echo Setting plugin as executable...
Expand Down
2 changes: 1 addition & 1 deletion .script/protoc-js.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ mkdir -p build
OUT=build/protoc-gen-grpc-web
PLUGIN_EXE=protoc-gen-grpc-web-$VERSION-$PLATFORM
echo Downloading $PLUGIN_EXE to $OUT...
curl -L https://github.com/grpc/grpc-web/releases/download/$VERSION/$PLUGIN_EXE \
curl -C - -L https://github.com/grpc/grpc-web/releases/download/$VERSION/$PLUGIN_EXE \
-o $OUT

echo Setting plugin as executable...
Expand Down
20 changes: 10 additions & 10 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,16 +43,16 @@ fmt:
# install supplementary tooling
.PHONY: install
install:
go get -u github.com/gogo/protobuf/protoc-gen-gogoslick
go get -u github.com/gogo/protobuf/protoc-gen-gogofaster
go get -u github.com/gogo/protobuf/protoc-gen-gogofast
go get -u github.com/gogo/protobuf/protoc-gen-gogo
go get -u github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway
go get -u github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger
go get -u github.com/mwitkow/go-proto-validators/protoc-gen-govalidators
go get -u github.com/gogo/protobuf/proto
go get -u github.com/gogo/protobuf/gogoproto
go get -u github.com/pseudomuto/protoc-gen-doc/cmd/protoc-gen-doc
go get github.com/gogo/protobuf/protoc-gen-gogoslick
go get github.com/gogo/protobuf/protoc-gen-gogofaster
go get github.com/gogo/protobuf/protoc-gen-gogofast
go get github.com/gogo/protobuf/protoc-gen-gogo
go get github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway
go get github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger
go get github.com/mwitkow/go-proto-validators/protoc-gen-govalidators
go get github.com/gogo/protobuf/proto
go get github.com/gogo/protobuf/gogoproto
go get github.com/pseudomuto/protoc-gen-doc/cmd/protoc-gen-doc
npm install -g ts-protoc-gen
npm install -g grpc
npm install -g @grpc/proto-loader
Expand Down
102 changes: 72 additions & 30 deletions doc/PROTO.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,32 +14,27 @@
- [REFREQOPTS](#pb.REFREQOPTS)
- [REFREQTYPE](#pb.REFREQTYPE)


- [AdminAPI](#pb.AdminAPI)


- [file.proto](#file.proto)
- [Blob](#pb.Blob)
- [DownloadRequest](#pb.DownloadRequest)
- [DownloadResponse](#pb.DownloadResponse)
- [RemoveRequest](#pb.RemoveRequest)
- [RemoveRequest.RefIDsEntry](#pb.RemoveRequest.RefIDsEntry)
- [RemoveResponse](#pb.RemoveResponse)
- [UploadOptions](#pb.UploadOptions)
- [UploadRequest](#pb.UploadRequest)



- [FileAPI](#pb.FileAPI)


- [namesys.proto](#namesys.proto)
- [NameSysPublishRequest](#pb.NameSysPublishRequest)
- [NameSysResolveRequest](#pb.NameSysResolveRequest)
- [NameSysResolveResult](#pb.NameSysResolveResult)



- [NameSysAPI](#pb.NameSysAPI)


- [node.proto](#node.proto)
- [Block](#pb.Block)
- [BlockstoreRequest](#pb.BlockstoreRequest)
Expand Down Expand Up @@ -77,10 +72,8 @@
- [KSREQTYPE](#pb.KSREQTYPE)
- [P2PREQTYPE](#pb.P2PREQTYPE)


- [NodeAPI](#pb.NodeAPI)


- [pubsub.proto](#pubsub.proto)
- [PubSubMessage](#pb.PubSubMessage)
- [PubSubPeer](#pb.PubSubPeer)
Expand All @@ -89,10 +82,8 @@

- [PSREQTYPE](#pb.PSREQTYPE)


- [PubSubAPI](#pb.PubSubAPI)


- [replication.proto](#replication.proto)
- [AddrInfo](#pb.AddrInfo)
- [Replication](#pb.Replication)
Expand All @@ -102,29 +93,20 @@
- [Subscription](#pb.Subscription)
- [SubscriptionUpdate](#pb.SubscriptionUpdate)



- [replicator](#pb.replicator)


- [status.proto](#status.proto)
- [StatusResponse](#pb.StatusResponse)
- [VersionResponse](#pb.VersionResponse)

- [APISTATUS](#pb.APISTATUS)


- [StatusAPI](#pb.StatusAPI)


- [util.proto](#util.proto)
- [Empty](#pb.Empty)
- [PutResponse](#pb.PutResponse)





- [Scalar Value Types](#scalar-value-types)


Expand Down Expand Up @@ -287,7 +269,7 @@ Blob is a chunk of binary data
| Field | Type | Label | Description |
| ----- | ---- | ----- | ----------- |
| content | [bytes](#bytes) | | content is the actual binary data contained in this message |
| rangeStart | [uint64](#uint64) | | Range start and end mirrors developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Range. If both is zero, the blobs streams contents of the file from start to finish. The unit of range is alway in bytes. Currently, DownloadResponse support blob range. |
| rangeStart | [uint64](#uint64) | | Range start and end mirrors developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Range. If both is zero, the blobs streams contents of the file from start to finish. The unit of range is always in bytes. Currently, DownloadResponse support blob range. |
| rangeEnd | [uint64](#uint64) | | |


Expand All @@ -307,7 +289,7 @@ there may be some undefined behavior
| ----- | ---- | ----- | ----------- |
| hash | [string](#string) | | hash is the ipfs hash/cid (content identifier) of the data to download |
| chunkSize | [int32](#int32) | | chunkSize specifies the size of chunks to be sent to the client it allows us to efficiently control incoming data amounts which is useful on machines with low-memory |
| rangeStart | [uint64](#uint64) | | Range start and end mirrors developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Range. If both is none zero, only data within range is requested. The unit of range is alway in bytes. If used, please check the returned range values in blobs to make sure this feature is supported. |
| rangeStart | [uint64](#uint64) | | Range start and end mirrors developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Range. If both is none zero, only data within range is requested. The unit of range is always in bytes. If used, please check the returned range values in blobs to make sure this feature is supported. |
| rangeEnd | [uint64](#uint64) | | |


Expand All @@ -331,6 +313,52 @@ which allows the gRPC server to stream blobs to the client



<a name="pb.RemoveRequest"></a>

### RemoveRequest
UploadRequest is used to decrease the reference count on UnixFS objects


| Field | Type | Label | Description |
| ----- | ---- | ----- | ----------- |
| refIDs | [RemoveRequest.RefIDsEntry](#pb.RemoveRequest.RefIDsEntry) | repeated | refIDs is a map of reference IDs to hash/cid of objects to remove those reference counts |






<a name="pb.RemoveRequest.RefIDsEntry"></a>

### RemoveRequest.RefIDsEntry



| Field | Type | Label | Description |
| ----- | ---- | ----- | ----------- |
| key | [string](#string) | | |
| value | [string](#string) | | |






<a name="pb.RemoveResponse"></a>

### RemoveResponse
RemoveResponse contains the response to a remove request


| Field | Type | Label | Description |
| ----- | ---- | ----- | ----------- |
| count | [uint64](#uint64) | | The number of removal operations performed. A missing count is because the refID to hash pair was already removed or was never added |






<a name="pb.UploadOptions"></a>

### UploadOptions
Expand All @@ -339,9 +367,12 @@ UploadOptions allows controlling the parameters of a file upload

| Field | Type | Label | Description |
| ----- | ---- | ----- | ----------- |
| multiHash | [string](#string) | | specifes the multihash function to use |
| multiHash | [string](#string) | | specifies the multihash function to use |
| layout | [string](#string) | | specifies the dag layout (balanced, tricklet) |
| chunker | [string](#string) | | specifies the chunker type (rabin, default, etc...) |
| refID | [string](#string) | | optional reference ID to tag the file with. If set, the same reference ID must be used for deletion |
| progressive | [bool](#bool) | | if refID is set, allows progressive upload |
| replace | [bool](#bool) | | if refID is set, remove the any existing uploads with same refID |



Expand All @@ -357,7 +388,7 @@ UploadRequest is used to upload data as a UnixFS object
| Field | Type | Label | Description |
| ----- | ---- | ----- | ----------- |
| blob | [Blob](#pb.Blob) | | blob is a single chunk of data |
| options | [UploadOptions](#pb.UploadOptions) | | options allows setting the optoins for this upload |
| options | [UploadOptions](#pb.UploadOptions) | | options allows setting the options for this upload, only valid in the first message of a stream |



Expand All @@ -377,8 +408,9 @@ FileAPI provides a gRPC api to upload/download files as UnixFS objects

| Method Name | Request Type | Response Type | Description |
| ----------- | ------------ | ------------- | ------------|
| UploadFile | [UploadRequest](#pb.UploadRequest) stream | [PutResponse](#pb.PutResponse) | UploadFile allows uploading a file as a UnixFS object (equivalent to ipfs add) |
| UploadFile | [UploadRequest](#pb.UploadRequest) stream | [PutResponse](#pb.PutResponse) | UploadFile allows uploading a file as a UnixFS object (equivalent to ipfs pin add) |
| DownloadFile | [DownloadRequest](#pb.DownloadRequest) | [DownloadResponse](#pb.DownloadResponse) stream | DownloadFile allows downloading a UnixFS object (equivalent to ipfs get) |
| RemoveFile | [RemoveRequest](#pb.RemoveRequest) | [RemoveResponse](#pb.RemoveResponse) | RemoveFile allows removing a UnixFS object or decrease it&#39;s reference counter (equivalent to ipfs pin rm) |



Expand Down Expand Up @@ -481,7 +513,7 @@ NameSysAPI provides a generic name resolution API
| ----- | ---- | ----- | ----------- |
| cid | [string](#string) | | cid is the identifier of the block |
| data | [bytes](#bytes) | | data is the actual contents of the block |
| size | [int64](#int64) | | size of the block, only filled out by BS_GET_STATS since if we just want stats, we dont want to retrieve all the data. |
| size | [int64](#int64) | | size of the block, only filled out by BS_GET_STATS since if we just want stats, we don&#39;t want to retrieve all the data. |



Expand All @@ -502,6 +534,9 @@ BlockstoreRequest is a message used to control blockstores
| data | [bytes](#bytes) | repeated | the data we are putting sent by: BS_PUT, BS_PUT_MANY |
| cidVersion | [string](#string) | | the cid version to use when constructing blocks, default is v1 sent by: BS_PUT, BS_PUT_MANY |
| hashFunc | [string](#string) | | the hash function to use when constructing blocks, default is sha2-256 sent by: BS_PUT, BS_PUT_MANY |
| refID | [string](#string) | | reference ID to mark the blocks of this operation with when sent by BS_PUT, BS_PUT_MANY: only put if the id is not marked on block, otherwise noop when sent by BS_GET, BS_GET_MANY: only get if the id is marked on block when sent by BS_DELETE: only delete if the id is marked on block |
| progressive | [bool](#bool) | | if refID is set, allows progressive upload |
| replace | [bool](#bool) | | if refID is set, remove the any existing blocks with same refID |



Expand All @@ -511,7 +546,7 @@ BlockstoreRequest is a message used to control blockstores
<a name="pb.BlockstoreResponse"></a>

### BlockstoreResponse
BlockstoreResponse is a response to a BlockstoreqRequest
BlockstoreResponse is a response to a BlockstoreRequest


| Field | Type | Label | Description |
Expand Down Expand Up @@ -625,8 +660,11 @@ Used to submit a request to Dag or DagStream RPCs
| serializationFormat | [string](#string) | | the serialization format (raw, cbor, protobuf, etc...) sent by: DAG_PUT |
| hashFunc | [string](#string) | | the hash function to to use (sha2-256, sha3-512, etc...) sent by: DAG_PUT, DAG_NEW_NODE, DAG_ADD_LINKS |
| cidVersion | [int64](#int64) | | the cid version to use (0, 1) sent by: DAG_PUT, DAG_NEW_NODE |
| hash | [string](#string) | | the hash of the object we are processing sent by: DAG_GET, DAG_NEW_NODe, DAG_ADD_LINKS, DAG_GET_LINKS |
| hash | [string](#string) | | the hash of the object we are processing sent by: DAG_GET, DAG_NEW_NODe, DAG_ADD_LINKS, DAG_GET_LINKS, DAG_REMOVE |
| links | [DagRequest.LinksEntry](#pb.DagRequest.LinksEntry) | repeated | indicates links and their names. key = name, value = link hash sent by: DAG_NEW_NODE, DAG_ADD_LINKS |
| refID | [string](#string) | | optional reference ID to mark the cid/hash with sent by: DAG_PUT, DAG_REMOVE |
| progressive | [bool](#bool) | | if refID is set, allows progressive upload |
| replace | [bool](#bool) | | if refID is set, remove the any existing DAGs with same refID |



Expand Down Expand Up @@ -662,6 +700,7 @@ Used in response to a Dag or DagStream RPC
| rawData | [bytes](#bytes) | | the actual data contained by the IPLD object sent by: DAG_GET |
| links | [IPLDLink](#pb.IPLDLink) | repeated | the links contained within an IPLD node object sent by: DAG_GET_LINKS |
| nodeStats | [DagResponse.NodeStatsEntry](#pb.DagResponse.NodeStatsEntry) | repeated | maps ipld cids to a ipld.NodeStat object equivalent sent by: DAG_STAT |
| count | [uint64](#uint64) | | The number of removal operations performed. sent by: DAG_REMOVE |



Expand Down Expand Up @@ -871,6 +910,8 @@ P2PResponse is a response message sent in response to a P2PRequest message
| Field | Type | Label | Description |
| ----- | ---- | ----- | ----------- |
| cids | [string](#string) | repeated | cids to persist locally |
| refID | [string](#string) | | optional reference ID to mark the cids with |
| progressive | [bool](#bool) | | if refID is set, allows progressive upload |



Expand Down Expand Up @@ -986,6 +1027,7 @@ DAGREQTYPE indicates the particular DagAPI request being performed
| DAG_ADD_LINKS | 3 | DAG_ADD_LINKS is used to add links to an IPLD node object |
| DAG_GET_LINKS | 4 | DAG_GET_LINKS is used to retrieve all links contained in an IPLD node object |
| DAG_STAT | 5 | DAG_STAT is used to retrieve ipld.NodeStats information |
| DAG_REMOVE | 6 | DAG_REMOVE is the inverse of DAG_PUT |



Expand Down Expand Up @@ -1158,7 +1200,7 @@ PSREQTYPE indicates the particular PubSubAPI request being performed
| PS_GET_TOPICS | 0 | PS_GET_TOPICS is used to return a list of subscribed pubsub topics |
| PS_LIST_PEERS | 1 | PS_LIST_PEERS is used to return a list of peers subscribed to topics we are subscribed to |
| PS_SUBSCRIBE | 2 | PS_SUBSCRIBE is used to establish a persistent subscription to a pubsub topic |
| PS_PUBLISH | 3 | PS_PUBLISH is used to publisbh a message to a pubsub topic |
| PS_PUBLISH | 3 | PS_PUBLISH is used to publish a message to a pubsub topic |



Expand Down
70 changes: 52 additions & 18 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,26 +3,60 @@ module github.com/RTradeLtd/TxPB/v3
go 1.13

require (
github.com/RTradeLtd/go-libp2p-tls v0.2.3
github.com/Masterminds/goutils v1.1.0 // indirect
github.com/Masterminds/semver v1.5.0 // indirect
github.com/Masterminds/sprig v2.22.0+incompatible // indirect
github.com/RTradeLtd/go-libp2p-tls v0.2.4
github.com/btcsuite/btcd v0.21.0-beta // indirect
github.com/davidlazar/go-crypto v0.0.0-20200604182044-b73af7476f6c // indirect
github.com/envoyproxy/protoc-gen-validate v0.4.1 // indirect
github.com/gogo/protobuf v1.3.1
github.com/golang/protobuf v1.3.5 // indirect
github.com/google/go-cmp v0.3.1 // indirect
github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e // indirect
github.com/google/go-cmp v0.5.2 // indirect
github.com/google/gopacket v1.1.18 // indirect
github.com/google/uuid v1.1.2 // indirect
github.com/gopherjs/gopherjs v0.0.0-20200217142428-fce0ec30dd00 // indirect
github.com/grpc-ecosystem/grpc-gateway v1.14.6
github.com/huandu/xstrings v1.3.2 // indirect
github.com/imdario/mergo v0.3.11 // indirect
github.com/ipfs/go-bitswap v0.2.20 // indirect
github.com/ipfs/go-block-format v0.0.2
github.com/ipfs/go-cid v0.0.5
github.com/ipfs/go-datastore v0.4.4 // indirect
github.com/ipfs/go-blockservice v0.1.3 // indirect
github.com/ipfs/go-cid v0.0.7
github.com/ipfs/go-datastore v0.4.5 // indirect
github.com/ipfs/go-ipfs-blockstore v1.0.1 // indirect
github.com/ipfs/go-ipld-cbor v0.0.4 // indirect
github.com/ipfs/go-ipld-format v0.2.0
github.com/ipfs/go-merkledag v0.3.1
github.com/libp2p/go-libp2p-core v0.5.1
github.com/multiformats/go-multiaddr v0.2.1
github.com/multiformats/go-multiaddr-net v0.1.4
github.com/ipfs/go-merkledag v0.3.2
github.com/kr/text v0.2.0 // indirect
github.com/libp2p/go-libp2p v0.11.0 // indirect
github.com/libp2p/go-libp2p-core v0.6.1
github.com/libp2p/go-libp2p-record v0.1.3 // indirect
github.com/libp2p/go-sockaddr v0.1.0 // indirect
github.com/mitchellh/copystructure v1.0.0 // indirect
github.com/mitchellh/reflectwalk v1.0.1 // indirect
github.com/multiformats/go-multiaddr v0.3.1
github.com/multiformats/go-multiaddr-net v0.2.0
github.com/mwitkow/go-proto-validators v0.3.2
github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e // indirect
github.com/onsi/ginkgo v1.14.1 // indirect
github.com/onsi/gomega v1.10.2 // indirect
github.com/pkg/errors v0.9.1
go.uber.org/zap v1.14.1
golang.org/x/crypto v0.0.0-20200406173513-056763e48d71 // indirect
golang.org/x/lint v0.0.0-20200302205851-738671d3881b // indirect
golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e // indirect
golang.org/x/sys v0.0.0-20200409092240-59c9f1ba88fa // indirect
golang.org/x/tools v0.0.0-20200413015812-1f08ef6002a8 // indirect
google.golang.org/genproto v0.0.0-20200413115906-b5235f65be36 // indirect
google.golang.org/grpc v1.28.1
gopkg.in/yaml.v2 v2.2.8 // indirect
github.com/polydawn/refmt v0.0.0-20190807091052-3d65705ee9f1 // indirect
github.com/pseudomuto/protoc-gen-doc v1.3.2
github.com/smartystreets/assertions v1.2.0 // indirect
github.com/smartystreets/goconvey v1.6.4 // indirect
github.com/warpfork/go-wish v0.0.0-20200122115046-b9ea61034e4a // indirect
github.com/whyrusleeping/cbor-gen v0.0.0-20200826160007-0b9f6c5fb163 // indirect
go.uber.org/goleak v1.1.10 // indirect
go.uber.org/zap v1.16.0
golang.org/x/crypto v0.0.0-20200820211705-5c72a883971a // indirect
golang.org/x/net v0.0.0-20200925080053-05aa5d4ee321 // indirect
golang.org/x/sys v0.0.0-20200923182605-d9f96fdee20d // indirect
golang.org/x/tools v0.0.0-20200925191224-5d1fdd8fa346 // indirect
google.golang.org/genproto v0.0.0-20200925023002-c2d885f95484 // indirect
google.golang.org/grpc v1.29.1
gopkg.in/check.v1 v1.0.0-20200902074654-038fdea0a05b // indirect
gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776 // indirect
honnef.co/go/tools v0.0.1-2020.1.5 // indirect
)
Loading