Skip to content

Latest commit

 

History

History
750 lines (560 loc) · 54.3 KB

CHANGELOG.md

File metadata and controls

750 lines (560 loc) · 54.3 KB
Raw

go-filecoin changelog

go-filecoin 0.5.6

We're happy to announce go-filecoin 0.5.6. Highlights include an updated Proof-of-Spacetime implementation and an upgrade-capable network.

Features

🌳 Network upgrade capability

Two changes have been made to enable software releases without restarting the network. First, a network name is now embedded in the genesis state, permitting multiple networks to follow different upgrade schedules. In addition, the Git SHA compatibility check has been removed from the Hello protocol, enabling nodes with different, but compatible, code to interoperate. Going forward, the user devnet will no longer be restarted with every software release; it will still be restarted as-needed.

🚀 Updated Proof-of-Spacetime (PoSt)

A new proof construction, Rational PoSt, has been implemented and integrated. This construction is the same shape as our candidate for testnet and resolves outstanding limitations on proving over many sectors.

🎟️ Block and consensus changes

Block headers are now signed by miners, and election tickets form an array in each header. The election process is now split into two phases, ticket generation / validation and election winner generation / validation. Election tickets form an array in each header and mining a null block appends a ticket to this array. Block headers are now signed by miners.

🔗 Chain status command

go-filecoin chain status is a new command that provides insight into chain sync progress.

Performance and Reliability

⚡ Chain syncing performance

Previously in go-filecoin 0.4, we aimed to speed up chain syncing by focusing on the first phase: chain fetching. We have identified the worst of the fetching contention issues that caused forking and unreliable message processing in 0.4. Some of those fixes are now complete, while others such as #3460 are in progress. There may still be some issues that could cause forking that we will continue to work on and update the coming weeks. Please let us know your feedback.

go-filecoin 0.5 also continues with improvements to the second phase: chain validation. By switching from HAMT bitwidth 8 to HAMT bitwidth 5, we see a general average improvement in benchmarks of about 4-to-1, across memory usage, speed of operations, and bytes written to disk. Users are encouraged to measure and share their own benchmarks. In addition, optimizations to encoding and decoding of HAMT data structures may result in additional performance improvements.

Looking Ahead

✏️ New API design (WIP)

Developers are invited to read and comment on the new HTTP API design (work in progress). This design will be implemented initially in go-filecoin and serve as a standard for interacting with Filecoin nodes across implementations. It will support most of operations offered by the current API and provide a framework for future API growth.

User Notes

  • The proving period is now configured to 300 rounds (2.5 hrs), down from 1000 rounds (10 hours). We’ve made this temporary change for more frequent node interaction and faster experimentation, and we expect to increase the proving period again in the future.
  • Groth parameters are no longer fetched from the network, but instead locally generated when needed. This can take many minutes (but is more reliable than network).
  • Block header structure has changed, so tools which parse chain data will need updating.
  • The default storage miner waits 15 rounds after the start of the proving window before beginning a PoSt computation, but is not robust to a re-org of more than 15 blocks that changes its challenge seed.
  • If you are seeing panics or write failures during sealing, it may be related to disk space requirements. Currently the sector builder uses ~11GiB of free disk space, and assumes it is available on the /tmp partition. An proposal to make that directory configurable is in #3497

CLI diff

go-filecoin command change
chain status added
mining add-piece added
mining seal-now behavior changed[1]

[1] mining seal-now no longer stages a piece into a sector. It now has the same behavior as --auto-seal-interval-seconds.

Changelog

A full list of all 67 PRs in this release, including many bugfixes not listed here, can be found on Github.

Contributors

❤️ Huge thank you to everyone that made this release possible!

🙌🏽 Want to contribute?

Would you like to contribute to the Filecoin project and don’t know how? Here are a few places you can get started:

⁉️ Do you have questions?

The best place to ask your questions about go-filecoin, how it works, and what you can do with it is at discuss.filecoin.io. We are also available at the community chat on Matrix/Slack.

go-filecoin 0.4.6

This release includes the first steps towards increasing the performance for new nodes joining the network through the graphsync protocol. It also includes the new command mining status which allows users to understand the current state of their miner by providing the current proving period, as well as the current proving sector set. We've also shipped piece inclusion proofs which allow for storage clients to verify the inclusion of their data in the sealed sector reported back by the storage miner.

Features

Chain sync

Chain downloads have switched to go-graphsync for more efficient downloads, and we’ve also reduced block header size by moving messages and receipts out of the header. We’re expecting this to significantly reduce the memory requirements for running a node, and to greatly speed up the chain downloading & setup time.

Piece inclusion proofs

Piece inclusion proofs are now calculated during deal proposal. We expect small increases in (1) time to create a deal and (2) CPU consumption during sealing, but both should be minor.

Miners validate piece commitments

Miners now validate that a client has supplied the correct piece commitment (commP). This is a computationally expensive operation that will be apparent for high-throughput miner operators.

Slashing

Storage slashing is now implemented as outlined in Mining Spec and Storage Market Spec on the actors. Miners automatically monitor for storage faults by other miners and include SlashStorageFault messages in their own blocks (these messages will later carry a rewards).

Free deals

Previously when accepting storage deals with zero costs a payment channel would be created regardless. This added additional costs to the deal that were not needed. Miners now accept deals with zero price and will not require a payment channel to exist before accepting the deal. This can simplify operations for miners automating deals among their own nodes.

Sealing sectors without a deal

Previously the only way to stage pieces into sectors was through the Storage Market. This process required interacting with an additional node on the network and added unnecessary overhead for miners to gain power on the network. Miners now can use the mining seal-now command to seal "empty" sectors directly to increase their power on the network for block mining.

CLI diff

go-filecoin command change
mining status added
mining seal-now added
miner proving-period added
show header added
show messages added
show receipts added
client verify-deal-proposal added

Important changes

  • The mining start command will fail if Groth parameters for the sector size which the miner is configured do not yet exists in the parameter cache. Previously Groth parameters would be generated on demand if they were missing.
  • The Groth parameters cache location has changed from /tmp/filecoin-proof-parameters to /var/tmp/filecoin-proof-parameters.
  • Parameters are no longer downloaded by default. Nodes intending to mine must fetch parameters explicitly. See the wiki for more information.
  • The paramfetch binary now uses ipget to download Groth parameters and keys instead of hitting the IPFS (HTTP) Gateway. This will make paramfetch slow, but more reliable.
  • Proof logs will no longer be displayed in log output by default and must be enabled by setting RUST_LOG=info before starting the daemon.
  • When building go-filecoin, git submodules init --recursive is required to be ran before go run ./build deps.
  • Sector size for PoSt construction has been increased from 2 to 4. This has resulted in a slight increase of memory usage, but supports proof calculation over more storage within a single proving period.

go-filecoin 0.3.2

We're happy to announce go-filecoin 0.3.2. This release is a big step towards completing the filecoin storage protocol. It includes many changes to the miner actor builtin smart contract that will allow the network to securely account for verifiable storage power once fault handling is in place. Many less visible but high impact code and testing improvements ship with this release. 0.3.2 also includes a big UX improvement with the new and improved go-filecoin deals command for user friendly management of storage deals. Getting paid as a storage miner is now as simple as a single CLI call.

Features

🏇 Storage protocol nearing completeness

Our number one goal is a network securely powered by verifiable storage. In order for this to work we need to penalize cheating miners who do not prove their storage on time. This release includes most of the groundwork needed, including fundamental data structures and encoding work for tracking sets of sectors, improved power tracking in the miner actor built-in smart contract, and charging fees for late storage proof (PoSt) submissions. Expect these changes to blossom into the complete fault reporting mechanism in the next release.

👪 Multiple sector sizes

In order for the network to scale gracefully, different miners may choose from a variety of different sector sizes to put data in and prove over: smaller sectors for faster and more nimble storage; larger sectors for slower but efficient storage. This release includes all of the software updates we need to support multiple sector sizes in a single network; however, we plan to properly vet network conditions with much bigger sectors before enabling multiple sectors sizes in the user devnet. Expect 1 GiB sectors on the user devnet in the next release.

🤝 Deal management and payments

Both clients and miners can now easily inspect the fine details of all storage deals they have entered into using go-filecoin deals list and go-filecoin deals show. Miners can get paid for honoring a deal by running go-filecoin deals redeem. Additionally this release ships some improvements in payment channel safety for correct arbitration of deal disputes we want down the road.

Performance and Reliability

🌳 Upgrade in place

This release drives home previous work on repo migrations. The go-filecoin-migrate tool (included in the go-filecoin source repo) is now complete. This release includes a proof of concept migration: upgrading on-disk chain metadata from JSON to the more compact CBOR. Landing this means we are confident that this major technical challenge is behind us, putting us one step closer to a reliable, persistent testnet.

Refactors and Endeavors

📈Major testing improvements

Testing is the silent champion of reliability and development speed. This release includes tons of behind the scenes work improving the quality of existing unit and integration tests as well as adding new tests to existing code. Continued improvements to the FAST framework promise to further accelerate integration testing and devnet deployments.

💳 Tech debt paydown

This release is not playing around when it comes to paying off technical debt. Fundamental chain refactors include an improved immutable tipset type and tipset cache sharing are at the top of the list. A major refactor of the message handling system into inbox and outbox queues is also a notable improvement. Don’t forget about a consistent internal attoFIL token type, a sleek new miner deal acceptance codepath, sector builder reliability fixes... the list goes on. We are excited to be shipping higher quality software with each release so that we can move faster towards a robust mainnet.

Changelog

A full list of all 207 PRs in this release, including many bugfixes not listed here, can be found on Github.

CLI diff

go-filecoin command change
deals list added
deals redeem added
deals show added
miner pledge removed
mining status added
show block args changed

Contributors

❤️ Huge thank you to everyone that made this release possible! By alphabetical order, here are all the humans who contributed to this release:

🙌🏽 Want to contribute?

Would you like to contribute to the Filecoin project and don’t know how? Here are a few places you can get started:

⁉️ Do you have questions?

The best place to ask your questions about go-filecoin, how it works, and what you can do with it is at discuss.filecoin.io. We are also available at the community chat on Matrix/Slack.

go-filecoin 0.2.4

We're happy to announce go-filecoin 0.2.4. This is a patch release with block validation improvements. As a placeholder before full implementation of block validation, block time was hardcoded to 30 seconds. It was also possible to manually configure a shorter block time via the CLI — miners who did this gained an unfair block mining advantage. Over the past few weeks, a handful of enterprising devnet participants¹ 😉 increasingly used this undocumented option to the point of severely degrading the devnet for everyone else. To get the devnet running smoothly again, we are releasing partial block validation.

🌳 Features

☝🏽 Upgrade notice

As a reminder, only the latest version of go-filecoin will connect to the user devnet until protocol upgrade work is complete. Users will need to upgrade to 0.2.4 to connect to the user devnet.

[1] If that was you, we’d love to collaborate to see if you can find other ways to break our implementation! Please email us at mining@filecoin.io.

go-filecoin 0.2.2

We're happy to announce go-filecoin 0.2.2. This is a maintenance release with bug fixes and debugging improvements. After the 0.2.1 release, we found a bug in the dht (#2753) that caused some nodes to panic. This was fixed in #2754 by bumping the go-libp2p-kad-dht version from 0.0.4 to 0.0.8.

🐞 Bug fixes

🌳 Features

☝🏽 Upgrade notice

As a reminder, only the latest version of go-filecoin will connect to the user devnet until model for change work is complete. Users will need to upgrade to 0.2.2 to connect to the user devnet.

go-filecoin 0.2.1

We're happy to announce go-filecoin 0.2.1. This release is heavy on behind-the-scenes upgrades, including support for filesystem repo migrations and storage disputes, a better message pool, proofs improvements, and a bump to libp2p version for more reliable relays. User-facing improvements such as new commands and options, better status messages, and lots of bugfixes are also included. Get pumped! 🎁

Install and Setup

⌛ Chain syncing status

When a filecoin node is first created, it must download and verify the chain. We call this “chain syncing”. While initial commands (such as tapping the faucet or dashboard streaming) can be run immediately, any other commands (such as mining commands) will return errors until chain syncing is complete. Currently, this can take several hours.

To clarify, we’ve added wiki updates, better status messages, and cleaner console output for chain syncing. In future releases, we’ll also address the underlying problem of slow chain syncing.

💠 Sector storage configuration

Where would you like the filecoin node to store client data? You can now choose! There are two ways to specify the location of the sector storage directory: the sectorbase.rootdir config entry, or the --sectordir option to go-filecoin init.

If you don’t specify a location, data is stored in $HOME/.filecoin_sectors by default.

Features

🍄 Upgradeable repo

In addition to sealed client data, Filecoin nodes also store other data on-disk such as configuration data, blockchain blocks, deal state, and encryption keys. As development progresses, we need a way to safely change the type and schema of this data. In this release, we include an accepted design for filesystem repo migrations, and an initial layout for the migration tool. This paves the way for filecoin nodes to seamlessly update when running in production.

For more information, check out the help text:

tools/migration/go-filecoin-migrate --help

💎 Storage payments

This release includes work towards storage protocol dispute resolution. Payment channels can now contain conditions that will query another actor before a voucher is redeemed. Payment channels can also be canceled by the payer. This will trigger an early close if the target of the channel does not redeem a payment. These features can be used together with piece inclusion proofs (coming soon) to enforce proof of storage when storage clients pay storage miners.

🐛 New debugging commands

Three new commands (inspect, protocol, and bitswap) are now available for your debugging and exploring adventures:

* go-filecoin inspect all prints all the necessary information for opening a bug report on GitHub. This includes operating system details, your current go-filecoin config, and a few other commonly needed stats.

* go-filecoin protocol prints details regarding parameters for a node’s protocol, such as autoseal interval and sector size. These are helpful for debugging some of the internals of Filecoin’s proofs and protocol systems.

* go-filecoin bitswap prints details about a node’s libp2p bitswap system, such as blocks, data, and messages received and sent. These are commonly used in network debugging.

For more details, run any command followed by the --help flag.

Performance and Reliability

🙌 Upgrade libp2p to 0.0.16

libp2p recently landed a bunch of improvements to relay functionality, addressing heavy resource usage in some production relay nodes. We’ve upgraded to go-libp2p 0.0.16 to enjoy the same fixes in filecoin.

📬 Better message validation

We’ve taken several steps to harden the message pool. The pool now rejects messages that will obviously fail processing due to problems like invalid signature, insufficient funds, no gas, or non-existent actor. It also tracks nonces to ensure that messages are correctly sequenced, and that no account has too many messages in the pool. Finally, the pool now limits the total messages it will accept.

🔗 Proofs integration

Behind the scenes, much groundwork has been laid for more flexible and powerful storage proofs. This release includes more efficient memory utilization when writing large pieces to a sector. It also includes initial support for piece inclusion proofs, multiple sector sizes, and variable proof lengths.

🔮 Proofs performance

Over in rust-fil-proofs, progress is accelerating on more complete and efficient implementations. This includes switching to mmap for more efficient merkle trees, abstractions over the hasher, limiting parallelism when generating groth proofs, and calculating and aggregating challenges across partitions.

Refactors and Endeavors

🏁 FAST (Filecoin Automation & System Toolkit)

We have significantly improved the FAST testing system for Filecoin since the last release. FAST now automatically includes relevant log data and messages from testing nodes in the event of a test failure. FAST also has an all-new localnet tool to quickly and easily set up local Filecoin node clusters for testing and experimentation. See the localnet readme for details.

👾 Go modules

With Go 1.11’s preliminary support for versioned modules, we have switched to Go modules for dependency management. This allows for easier dependency management and faster updates when dealing with updates from upstream dependencies.

😍 Design documents

We regularly write design docs before coding begins on important features or components. These short documents are useful in capturing knowledge, formalizing our thinking, and sharing design intent. Going forward, you can find new design docs in the designdocs repo.

Changelog

A full list of all 177 PRs in this release, including many bugfixes not listed here, can be found on Github.

Contributors

❤️ Huge thank you to everyone that made this release possible! By alphabetical order, here are all the humans who contributed to this release via the go-filecoin, rust-fil-proofs, and specs repos:

🙌🏽 Want to contribute?

Would you like to contribute to the Filecoin project and don’t know how? Here are a few places you can get started:

⁉️ Do you have questions?

The best place to ask your questions about go-filecoin, how it works, and what you can do with it is at discuss.filecoin.io. We are also available at the community chat on Matrix/Slack.

go-filecoin 0.1.4

We're happy to announce go-filecoin 0.1.4. This release contains a better install experience, initial Proof-of-Spacetime integration, more reliable message sending and networking, and many other improvements. Get pumped! 🎁

Install and Setup

💝 Binary releases

Linux and MacOS binaries for go-filecoin are now available! See Installing from binary for instructions.

🍱 Precompiled proofs parameters

Running secure proofs requires parameter files that are several GB in size. Previously, these files were generated at install, an extremely memory-intensive process causing slow or impossible builds for many users. Now, you can download pre-generated files during install by running paramfetch. This step is now included in the Installing from binary instructions.

🦖 Version checking

go-filecoin now checks that it is running the same version (at the same commit) while connecting to a devnet. This is a temporary fix until a model for change is implemented, allowing different versions to interoperate.

Features

💎 Proof-of-Spacetime Integration

Miners now call rust-fil-proofs to periodically generate proofs of continued storage. With this major integration in place, you can expect some follow-up (for example, storage mining faults do not yet appear on-chain) and continued optimizations to the underlying Proof-of-Spacetime construction and implementation.

Performance and Reliability

🤝 Networking

We’ve upgraded to go-libp2p 6.0.35 which has fixed autorelay reliability issues. We’ve also added a go-filecoin dht command for interacting with and debugging our dht.

🎈 Better message sending

In the past, if messages failed, they failed silently. go-filecoin would continue to select nonces higher than the sent message, effectively deadlocking message sending. We have now implemented several improvements to message sending: incoming and outgoing queues, better nonce selection logic, and a message timeout after a certain number of blocks. See message status and mpool ls/show/rm commands for more.

🔗 Chain syncing is faster

Chain is now faster due to use of bitswap sessions. Woohoo!

⌛ Context deadline errors fixed

In the past, the context deadline was set artificially low for file transfer. This caused some large file transfers to time out, preventing storage deals from being completed. Thank you to @markwylde, @muronglaowang, @pengxiankaikai, @sandjj, and others for bug reports.

Refactors and Endeavors

🦊 FAST (Filecoin Automation & System Toolkit)

FAST is a common library of go-filecoin code that can be used in daemon testing, devnet initialization, and other applications like network randomization that involve managing nodes, running commands against them, and observing their state.

Using FAST, we’ve developed localnet, a new tool to quickly and easily set up a local network for testing, debugging, development, and more. Want to give it a whirl? Check out the localnet README.

👾 Porcelain/Plumbing refactor for node object

Previously, the node object contained both interfaces and internals for much of the core protocol. It was difficult to unit test due to many dependencies and complicated setup. Following the porcelain and plumbing pattern from Git, we have now decoupled the node object from many of its dependencies. We have also created a separate API for block, storage, and retrieval mining.

Changelog

A full list of all 200 PRs in this release can be found on Github.

Contributors

❤️ Huge thank you to everyone that made this release possible! By alphabetical order, here are all the humans who contributed issues and commits in go-filecoin and rust-fil-proofs to date:

🙌🏽 Want to contribute?

Would you like to contribute to the Filecoin project and don’t know how? Here are a few places you can get started:

⁉️ Do you have questions?

The best place to ask your questions about go-filecoin, how it works, and what you can do with it is at discuss.filecoin.io. We are also available at the community chat on Matrix/Slack.