Russ Cox, for the Go team 10 November 2020
Today we celebrate the eleventh birthday of the Go open source release. The parties we had for Go turning 10 seem like a distant memory. It’s been a tough year, but we’ve kept Go development moving forward and accumulated quite a few highlights.
In November, we launched go.dev and pkg.go.dev shortly after Go’s 10th birthday.
In February, the Go 1.14 release delivered the first officially “production-ready” implementation of Go modules, along with many performance improvements, including faster defers and non-cooperative goroutine preemption to reduce scheduling and garbage collection latency.
In early March, we launched a new API for protocol buffers, google.golang.org/protobuf, with much-improved support for protocol buffer reflection and custom messages.
When the pandemic hit, we decided to pause any public announcements or launches in the spring, recognizing that everyone’s attention rightly belonged elsewhere. But we kept working, and one of our team members joined the Apple/Google collaboration on privacy-preserving exposure notifications to support contact tracing efforts all over the world. In May, that group launched the reference backend server, written in Go.
We continued to improve gopls, which enables advanced Go-aware support in many editors. In June, the VSCode Go extension officially joined the Go project and is now maintained by the same developers who work on gopls.
Also in June, thanks to your feedback, we open-sourced the code behind pkg.go.dev as part of the Go project as well.
Later in June, we released the latest design draft for generics, along with a prototype tool and generics playground.
In July, we published and discussed three new design drafts for future changes: new //go:build lines for file selection, file system interfaces, and build-time file embedding. (We’ll see all of those in 2021, as noted below.)
//go:build
In August, the Go 1.15 release delivered mainly optimizations and bug fixes rather than new features. The most significant was the start of a rewrite of the linker, making it run 20% faster and use 30% less memory on average for large builds.
Last month, we ran our annual Go user survey. We will post results on the blog once we’ve analyzed them.
The Go community has adapted to “virtual-first” along with everyone else, and we saw many virtual meetups and over a dozen virtual Go conferences this year. Last week, the Go team hosted Go day at Google Open Source Live (videos at the link).
We’re also incredibly excited about what’s in store for Go’s 12th year. Most immediately, this week Go team members will be presenting eight events at GopherCon 2020. Mark your calendars!
In February, the Go 1.16 release will include the new file system interfaces and build-time file embedding. It will complete the linker rewrite, bringing additional performance improvements. And it will include support for the new Apple Silicon (GOARCH=arm64) Macs.
GOARCH=arm64
In August, the Go 1.17 release will no doubt bring more features and improvements, although it’s far enough out that the exact details remain up in the air. It will include a new register-based calling convention for x86-64 (without breaking existing assembly!), which will make programs faster across the board. (Other architectures will follow in later releases.) One nice feature that will definitely be included is the new //go:build lines, which are far less error-prone than the current // +build lines. Another highly anticipated feature we hope will be ready for beta testing next year is support for fuzzing in the go test command.
//
+build
go test
Over the next year, we will continue to work on developing support for Go modules and integrating them well into the entire Go ecosystem. Go 1.16 will include our smoothest Go modules experience yet. One preliminary result from our recent survey is that 96% of users have now adopted Go modules (up from 90% a year ago).
We will also finally wind down support for GOPATH-based development: any programs using dependencies other than the standard library will need a go.mod. (If you haven’t switched to modules yet, see the GOPATH wiki page for details about this final step in the journey from GOPATH to modules.)
go.mod
From the start, the goal for Go modules has been “to add the concept of package versions to the working vocabulary of both Go developers and our tools,” to enable deep support for modules and versions throughout the Go ecosystem. The Go module mirror, checksum database, and index were made possible by this ecosystem-wide understanding of what a package version is. Over the next year, we will see rich module support added to more tools and systems. For example, we plan to investigate new tooling to help module authors publish new versions (go release) as well as to help module consumers update their code to migrate away from deprecated APIs (a new go fix).
go release
go fix
As a larger example, we created gopls to reduce many tools used by editors for Go support, none of which supported modules, down to a single one that did. Over the next year, we’ll be ready to make the VSCode Go extension use gopls by default, for an excellent module experience out of the box, and we’ll release gopls 1.0. Of course, one of the best things about gopls is that it is editor-neutral: any editor that understands the language server protocol can use it.
gopls
Another important use of version information is tracking whether any package in a build has a known vulnerability. Over the next year, we plan to develop a database of known vulnerabilities as well as tools to check your programs against that database.
The Go package discovery site pkg.go.dev is another example of a version-aware system enabled by Go modules. We’ve been focused on getting the core functionality and user experience right, including a redesign launching today. Over the next year, we will be unifying godoc.org into pkg.go.dev. We will also be expanding the version timeline for each package, showing important changes in each version, known vulnerabilities, and more, following the overall goal of surfacing what you need to make informed decisions about adding dependencies.
We’re excited to see this journey from GOPATH to Go modules nearing completion and all the excellent dependency-aware tools that Go modules are enabling.
The next feature on everyone’s minds is of course generics. As we mentioned above, we published the latest design draft for generics back in June. Since then, we’ve continued to refine rough edges and have turned our attention to the details of implementing a production-ready version. We will be working on that throughout 2021, with a goal of having something for people to try out by the end of the year, perhaps a part of the Go 1.18 betas.
Go is far more than just us on the Go team at Google. We are indebted to the contributors who work with us with the Go releases and tools. Beyond that, Go only succeeds because of all of you who work in and contribute to Go’s thriving ecosystem. It has been a difficult year in the world outside Go. More than ever, we appreciate you taking the time to join us and help make Go such a success. Thank you. We hope you are all staying safe and wish you all the best.
Julie Qiu 10 November 2020
Since launching pkg.go.dev, we’ve received a lot of great feedback on design and usability. In particular, it was clear that the way information was organized confused users when navigating the site.
Today we’re excited to share a redesigned pkg.go.dev, which we hope will be clearer and more helpful. This blog post presents the highlights. For details, see Go issue 41585.
The main change is that the pkg.go.dev/<path> page has been reorganized around the idea of a path. A path represents a directory in a particular version of a module. Now, regardless of what’s in that directory, every path page will have the same layout, with the goal of making the experience consistently useful and predictable.
The path page will display the README at that path if there is one. Previously, the overview tab only showed the README if present at the module root. This is one of many changes we’re making to place the most important information up front.
The documentation section now displays an index along with a sidenav. This gives the ability to see the full package API, while having context as they are navigating the documentation section. There is also a new Jump To input box in the left sidenav, to search for identifiers.
See Go issue 41587 for details on changes in the documentation section.
The top bar on each page now shows additional metadata, such as each package’s “imports” and “imported by” counts. Banners also show information about the latest minor and major versions of a module. See Go issue 41588 for details.
Last week at Google Open Source Live, we presented a walkthrough of the new site experience in our talk, Level Up: Go Package Discovery and Editor Tooling.
We’re excited to share this updated design with you. As always, please let us know what you think via the “Share Feedback” and “Report an Issue” links at the bottom of every page of the site.
And if you’re interested in contributing to this project, pkg.go.dev is open source! Check out the contribution guidelines to find out more.
Alice Merrick 20 October 2020
Since 2016, thousands of Gophers around the world have helped the Go project by sharing their thoughts via our annual Go Developer Survey. Your feedback has played an enormous role in driving changes to our language, ecosystem, and community, including the gopls language server, the latest generics draft, the module mirror, and so much more. And of course, we publicly share each year's results, so we can all benefit from the community's insights.
This year we’ve streamlined the survey to shorten the time it takes to complete and improved the survey’s accessibility. The specific questions each person will see are randomly selected, so folks who’ve taken the survey in prior years may not see all of the questions they are used to. Rest assured that they are still there. This approach allows us to ask about a wider variety of topics than in prior years while also reducing the survey’s length.
Today we are launching the 2020 Go Developer Survey. We'd love to hear from everyone who uses Go, used to use Go, or is interested in using Go, to help ensure the language, community, and ecosystem fit the needs of the people closest to it. Please help us shape Go's future by participating in this 10-minute survey by November 8th: Take the 2020 Go Developer Survey.
We need as many Gophers as possible to participate in this survey to help us better understand our global user base. We'd be grateful if you would spread the word by sharing this post on your social network feeds, around the office, at virtual meet-ups, and in other communities. Thank you!
Alex Rakoczy 11 August 2020
Today the Go team is very happy to announce the release of Go 1.15. You can get it from the download page.
Some of the highlights include:
For the complete list of changes and more information about the improvements above, see the Go 1.15 release notes.
We want to thank everyone who contributed to this release by writing code, filing bugs, providing feedback, and/or testing the beta and release candidates. Your contributions and diligence helped to ensure that Go 1.15 is as stable as possible. That said, if you notice any problems, please file an issue.
We hope you enjoy the new release!
Jean de Klerk and Jonathan Amsterdam 7 July 2020
This post is part 5 in a series.
Your modules will evolve over time as you add new features, change behaviors, and reconsider parts of the module's public surface. As discussed in Go Modules: v2 and Beyond, breaking changes to a v1+ module must happen as part of a major version bump (or by adopting a new module path).
However, releasing a new major version is hard on your users. They have to find the new version, learn a new API, and change their code. And some users may never update, meaning you have to maintain two versions for your code forever. So it is usually better to change your existing package in a compatible way.
In this post, we'll explore some techniques for introducing non-breaking changes. The common theme is: add, don’t change or remove. We’ll also talk about how to design your API for compatibility from the outset.
Often, breaking changes come in the form of new arguments to a function. We’ll describe some ways to deal with this sort of change, but first let’s look at a technique that doesn’t work.
When adding new arguments with sensible defaults, it’s tempting to add them as a variadic parameter. To extend the function
func Run(name string)
with an additional size argument which defaults to zero, one might propose
size
func Run(name string, size ...int)
on the grounds that all existing call sites will continue to work. While that is true, other uses of Run could break, like this one:
Run
package mypkg var runner func(string) = yourpkg.Run
The original Run function works here because its type is func(string), but the new Run function’s type is func(string, ...int), so the assignment fails at compile time.
func(string)
func(string, ...int)
This example illustrates that call compatibility is not enough for backward compatibility. There is, in fact, no backward-compatible change you can make to a function’s signature.
Instead of changing a function’s signature, add a new function. As an example, after the context package was introduced, it became common practice to pass a context.Context as the first argument to a function. However, stable APIs could not change an exported function to accept a context.Context because it would break all uses of that function.
context
context.Context
Instead, new functions were added. For example, the database/sql package's Query method’s signature was (and still is)
database/sql
Query
func (db *DB) Query(query string, args ...interface{}) (*Rows, error)
When the context package was created, the Go team added a new method to database/sql:
func (db *DB) QueryContext(ctx context.Context, query string, args ...interface{}) (*Rows, error)
To avoid copying code, the old method calls the new one:
func (db *DB) Query(query string, args ...interface{}) (*Rows, error) { return db.QueryContext(context.Background(), query, args...) }
Adding a method allows users to migrate to the new API at their own pace. Since the methods read similarly and sort together, and Context is in the name of the new method, this extension of the database/sql API did not degrade readability or comprehension of the package.
Context
If you anticipate that a function may need more arguments in the future, you can plan ahead by making optional arguments a part of the function’s signature. The simplest way to do that is to add a single struct argument, as the crypto/tls.Dial function does:
func Dial(network, addr string, config *Config) (*Conn, error)
The TLS handshake conducted by Dial requires a network and address, but it has many other parameters with reasonable defaults. Passing a nil for config uses those defaults; passing a Config struct with some fields set will override the defaults for those fields. In the future, adding a new TLS configuration parameter only requires a new field on the Config struct, a change that is backward-compatible (almost always—see "Maintaining struct compatibility" below).
Dial
nil
config
Config
Sometimes the techniques of adding a new function and adding options can be combined by making the options struct a method receiver. Consider the evolution of the net package’s ability to listen at a network address. Prior to Go 1.11, the net package provided only a Listen function with the signature
net
Listen
func Listen(network, address string) (Listener, error)
For Go 1.11, two features were added to net listening: passing a context, and allowing the caller to provide a “control function” to adjust the raw connection after creation but before binding. The result could have been a new function that took a context, network, address and control function. Instead, the package authors added a ListenConfig struct in anticipation that more options might be needed someday. And rather than define a new top-level function with a cumbersome name, they added a Listen method to ListenConfig:
ListenConfig
type ListenConfig struct { Control func(network, address string, c syscall.RawConn) error } func (*ListenConfig) Listen(ctx context.Context, network, address string) (Listener, error)
Another way to provide new options in the future is the “Option types” pattern, where options are passed as variadic arguments, and each option is a function that changes the state of the value being constructed. They are described in more detail by Rob Pike's post Self-referential functions and the design of options. One widely used example is google.golang.org/grpc's DialOption.
Option types fulfill the same role as struct options in function arguments: they are an extensible way to pass behavior-modifying configuration. Deciding which to choose is largely a matter of style. Consider this simple usage of gRPC's DialOption option type:
DialOption
grpc.Dial("some-target", grpc.WithAuthority("some-authority"), grpc.WithMaxDelay(time.Second), grpc.WithBlock())
This could also have been implemented as a struct option:
notgrpc.Dial("some-target", ¬grpc.Options{ Authority: "some-authority", MaxDelay: time.Minute, Block: true, })
Functional options have some downsides: they require writing the package name before the option for each call; they increase the size of the package namespace; and it's unclear what the behavior should be if the same option is provided twice. On the other hand, functions which take option structs require a parameter which might almost always be nil, which some find unattractive. And when a type’s zero value has a valid meaning, it is clumsy to specify that the option should have its default value, typically requiring a pointer or an additional boolean field.
Either one is a reasonable choice for ensuring future extensibility of your module's public API.
Sometimes, new features require changes to publicly-exposed interfaces: for example, an interface needs to be extended with new methods. Directly adding to an interface is a breaking change, though—how, then, can we support new methods on a publicly-exposed interface?
The basic idea is to define a new interface with the new method, and then wherever the old interface is used, dynamically check whether the provided type is the older type or the newer type.
Let's illustrate this with an example from the archive/tar package. tar.NewReader accepts an io.Reader, but over time the Go team realized that it would be more efficient to skip from one file header to the next if you could call Seek. But, they could not add a Seek method to io.Reader: that would break all implementers of io.Reader.
archive/tar
tar.NewReader
io.Reader
Seek
Another ruled-out option was to change tar.NewReader to accept io.ReadSeeker rather than io.Reader, since it supports both io.Reader methods and Seek (by way of io.Seeker). But, as we saw above, changing a function signature is also a breaking change.
io.ReadSeeker
io.Seeker
So, they decided to keep tar.NewReader signature unchanged, but type check for (and support) io.Seeker in tar.Reader methods:
tar.Reader
package tar type Reader struct { r io.Reader } func NewReader(r io.Reader) *Reader { return &Reader{r: r} } func (r *Reader) Read(b []byte) (int, error) { if rs, ok := r.r.(io.Seeker); ok { // Use more efficient rs.Seek. } // Use less efficient r.r.Read. }
(See reader.go for the actual code.)
When you run into a case where you want to add a method to an existing interface, you may be able to follow this strategy. Start by creating a new interface with your new method, or identify an existing interface with the new method. Next, identify the relevant functions that need to support it, type check for the second interface, and add code that uses it.
This strategy only works when the old interface without the new method can still be supported, limiting the future extensibility of your module.
Where possible, it is better to avoid this class of problem entirely. When designing constructors, for example, prefer to return concrete types. Working with concrete types allows you to add methods in the future without breaking users, unlike interfaces. That property allows your module to be extended more easily in the future.
Tip: if you do need to use an interface but don't intend for users to implement it, you can add an unexported method. This prevents types defined outside your package from satisfying your interface without embedding, freeing you to add methods later without breaking user implementations. For example, see testing.TB's private() function.
testing.TB
private()
type TB interface { Error(args ...interface{}) Errorf(format string, args ...interface{}) // ... // A private method to prevent users implementing the // interface and so future additions to it will not // violate Go 1 compatibility. private() }
This topic is also explored in more detail in Jonathan Amsterdam's "Detecting Incompatible API Changes" talk (video, slides).
So far we've talked about overt breaking changes, where changing a type or a function would cause users' code to stop compiling. However, behavior changes can also break users, even if user code continues to compile. For example, many users expect json.Decoder to ignore fields in the JSON that are not in the argument struct. When the Go team wanted to return an error in that case, they had to be careful. Doing so without an opt-in mechanism would mean that the many users relying on those methods might start receiving errors where they hadn’t before.
json.Decoder
So, rather than changing the behavior for all users, they added a configuration method to the Decoder struct: Decoder.DisallowUnknownFields. Calling this method opts a user in to the new behavior, but not doing so preserves the old behavior for existing users.
Decoder
Decoder.DisallowUnknownFields
We saw above that any change to a function’s signature is a breaking change. The situation is much better with structs. If you have an exported struct type, you can almost always add a field or remove an unexported field without breaking compatibility. When adding a field, make sure that its zero value is meaningful and preserves the old behavior, so that existing code that doesn’t set the field continues to work.
Recall that the authors of the net package added ListenConfig in Go 1.11 because they thought more options might be forthcoming. Turns out they were right. In Go 1.13, the KeepAlive field was added to allow for disabling keep-alive or changing its period. The default value of zero preserves the original behavior of enabling keep-alive with a default period.
KeepAlive
There is one subtle way a new field can break user code unexpectedly. If all the field types in a struct are comparable—meaning values of those types can be compared with == and != and used as a map key—then the overall struct type is comparable too. In this case, adding a new field of uncomparable type will make the overall struct type non-comparable, breaking any code that compares values of that struct type.
==
!=
To keep a struct comparable, don’t add non-comparable fields to it. You can write a test for that, or rely on the upcoming gorelease tool to catch it.
To prevent comparison in the first place, make sure the struct has a non-comparable field. It may have one already—no slice, map or function type is comparable—but if not, one can be added like so:
type Point struct { _ [0]func() X int Y int }
The func() type is not comparable, and the zero-length array takes up no space. We can define a type to clarify our intent:
func()
type doNotCompare [0]func() type Point struct { doNotCompare X int Y int }
Should you use doNotCompare in your structs? If you’ve defined the struct to be used as a pointer—that is, it has pointer methods and perhaps a NewXXX constructor function that returns a pointer—then adding a doNotCompare field is probably overkill. Users of a pointer type understand that each value of the type is distinct: that if they want to compare two values, they should compare the pointers.
doNotCompare
NewXXX
If you are defining a struct intended to be used as a value directly, like our Point example, then quite often you want it to be comparable. In the uncommon case that you have a value struct that you don’t want compared, then adding a doNotCompare field will give you the freedom to change the struct later without having to worry about breaking comparisons. On the downside, the type won’t be usable as a map key.
Point
When planning an API from scratch, consider carefully how extensible the API will be to new changes in the future. And when you do need to add new features, remember the rule: add, don't change or remove, keeping in mind the exceptions—interfaces, function arguments, and return values can't be added in backwards-compatible ways.
If you need to dramatically change an API, or if an API begins to lose its focus as more features are added, then it may be time for a new major version. But most of the time, making a backwards-compatible change is easy and avoids causing pain for your users.
See the index for more articles.