| Message ID | 20190412123046.17510-1-daniel.thompson@linaro.org |
|---|---|
| State | New |
| Headers | show |
| Series | go: Allow bootstrapping using host go compiler | expand |
On Fri, 2019-04-12 at 13:30 +0100, Daniel Thompson wrote: > Currently go-native bootstraps without any dependency on the host go > compiler by building go-1.4 (the last version written in C) and using > that to build a more recent version of go. This does not work on host > architectures, such as arm64, that are not supported by go-1.4. To > support these host architectures we must rely on an existing host go > compiler and use that to build go-native instead. > > Rather than add arm64 specific code that forces the use of the host > go > compiler, we use an architecture neutral approach based on adopting > the > host go compiler if it is both recent enough and the output of `go > env > GOROOT` passes basic sanity testing. > > Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org> > --- > meta/conf/bitbake.conf | 4 +++ > meta/recipes-devtools/go/go-native.inc | 38 > ++++++++++++++++++++++++-- > 2 files changed, 39 insertions(+), 3 deletions(-) Whilst this is quite a creative approach it isn't deterministic and I can see this introducing bugs which are hard to track down :( I think we need to make this a specific configuration option. Cheers, Richard -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On Fri, Apr 12, 2019 at 01:39:52PM +0100, Richard Purdie wrote: > On Fri, 2019-04-12 at 13:30 +0100, Daniel Thompson wrote: > > Currently go-native bootstraps without any dependency on the host go > > compiler by building go-1.4 (the last version written in C) and using > > that to build a more recent version of go. This does not work on host > > architectures, such as arm64, that are not supported by go-1.4. To > > support these host architectures we must rely on an existing host go > > compiler and use that to build go-native instead. > > > > Rather than add arm64 specific code that forces the use of the host > > go > > compiler, we use an architecture neutral approach based on adopting > > the > > host go compiler if it is both recent enough and the output of `go > > env > > GOROOT` passes basic sanity testing. > > > > Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org> > > --- > > meta/conf/bitbake.conf | 4 +++ > > meta/recipes-devtools/go/go-native.inc | 38 > > ++++++++++++++++++++++++-- > > 2 files changed, 39 insertions(+), 3 deletions(-) > > Whilst this is quite a creative approach it isn't deterministic and I > can see this introducing bugs which are hard to track down :( I guess I can't disagree too strongly... the version check and bbplain message about the go toolchain we adopted were intended to mitigate this by at least making the selected go toolchain conspicious (and thus a git blame would probably draw flames towards me). We could make things fully deterministic by downloading go compiler binaries (rather than source) if the host architecture is not supported by go-1.4. Is this feasible or should I continue trying to adopt the host go compiler for these architectures? > I think we need to make this a specific configuration option. I guess we could wrap this up in a config option although I hope it is ok to have per-architecture defaults so the config option is always sane out-of-the-box. Daniel. -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On Fri, 2019-04-12 at 14:16 +0100, Daniel Thompson wrote: > On Fri, Apr 12, 2019 at 01:39:52PM +0100, Richard Purdie wrote: > > On Fri, 2019-04-12 at 13:30 +0100, Daniel Thompson wrote: > > Whilst this is quite a creative approach it isn't deterministic and > > I can see this introducing bugs which are hard to track down :( > > I guess I can't disagree too strongly... the version check and > bbplain message about the go toolchain we adopted were intended to > mitigate this by at least making the selected go toolchain > conspicious (and thus a git blame would probably draw flames towards > me). Once something makes it into sstate, the message is lost and you don't know how it was created. The problems invariably end up on the mailing lists/bugzilla with others being on the "front line" for this. Rightly or wrongly its just the way it works out. > We could make things fully deterministic by downloading go compiler > binaries (rather than source) if the host architecture is not > supported by go-1.4. Is this feasible or should I continue trying to > adopt the host go compiler for these architectures? Is the go compiler a standalone binary that can cope with this? I think I prefer that to requiring people install host dependencies. > > I think we need to make this a specific configuration option. > > I guess we could wrap this up in a config option although I hope > it is ok to have per-architecture defaults so the config option is > always sane out-of-the-box. Yes, per-arch is fine and we definitely like "out the box"! Not sure any of us are classed as sane anymore but I digress! :) Cheers, Richard -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On Fri, Apr 12, 2019 at 02:34:14PM +0100, richard.purdie@linuxfoundation.org wrote: > On Fri, 2019-04-12 at 14:16 +0100, Daniel Thompson wrote: > > On Fri, Apr 12, 2019 at 01:39:52PM +0100, Richard Purdie wrote: > > > On Fri, 2019-04-12 at 13:30 +0100, Daniel Thompson wrote: > > > Whilst this is quite a creative approach it isn't deterministic and > > > I can see this introducing bugs which are hard to track down :( > > > > I guess I can't disagree too strongly... the version check and > > bbplain message about the go toolchain we adopted were intended to > > mitigate this by at least making the selected go toolchain > > conspicious (and thus a git blame would probably draw flames towards > > me). > > Once something makes it into sstate, the message is lost and you don't > know how it was created. The problems invariably end up on the mailing > lists/bugzilla with others being on the "front line" for this. Rightly > or wrongly its just the way it works out. > > > We could make things fully deterministic by downloading go compiler > > binaries (rather than source) if the host architecture is not > > supported by go-1.4. Is this feasible or should I continue trying to > > adopt the host go compiler for these architectures? > > Is the go compiler a standalone binary that can cope with this? I think > I prefer that to requiring people install host dependencies. For arm64 there are binaries since go1.8. They should be pretty light w.r.t. dependencies (close to C library only IIRC) and the archived releases go back as far as the end of 2013 (go1.2) so links should be durable. Daniel. -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On 4/12/19 9:16 AM, Daniel Thompson wrote: > We could make things fully deterministic by downloading go compiler > binaries (rather than source) if the host architecture is not supported > by go-1.4. Is this feasible or should I continue trying to adopt the > host go compiler for these architectures? You may already know but in case you hadn't consider the use case, we also have a requirement to do off-line builds so this download would have to fit into the fetcher system that we rely upon to avoid build-time rather than fetch-time downloads. Also adding a downloaded binary could be a (minor?) complication for IP analysis for some organizations. If a user is paranoid, is there a well-tested, reproducible procedure for generating the binary from source on an x86-64 host? If so should a recipe for that be added to oe-core? In the long run, don't we want to avoid any new binaries or host tools? Is anyone else in the go community (Debian maybe?) interested in bootstrapping go on arm64 hosts or do they all just use the binary? After all that I suppose that adding the go-1.4 binary download could be a reasonable short-term approach that we should live with until a source-based solution is available. Finally, as an aside, some time ago I noticed: https://bootstrappable.org/ and: https://www.freelists.org/post/bootstrappable/A-major-milestone-in-bootstrapping and thought that would be a good thing for oe-core that would allow dropping even the host gcc requirement albeit at likely significant build time cost. -- # Randy MacLeod # Wind River Linux -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On Fri, Apr 12, 2019 at 02:26:20PM -0400, Randy MacLeod wrote: >... > In the long run, don't we want to avoid any new binaries or host tools? Is > anyone else in the go community (Debian maybe?) interested in > bootstrapping go on arm64 hosts or do they all just use the binary? >... Binary distributions like Debian have to bootstrap compilers only once per architecture. After the initial bootstrap there is always the previous version package in the distribution that can be used to build the next version. cu Adrian -- "Is there not promise of rain?" Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. "Only a promise," Lao Er said. Pearl S. Buck - Dragon Seed -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On Fri, Apr 12, 2019 at 01:30:46PM +0100, Daniel Thompson wrote: > Currently go-native bootstraps without any dependency on the host go > compiler by building go-1.4 (the last version written in C) and using > that to build a more recent version of go. This does not work on host > architectures, such as arm64, that are not supported by go-1.4. To > support these host architectures we must rely on an existing host go > compiler and use that to build go-native instead. >... Why is this the only option for you? The clean way to bootstrap go would be by building gccgo and using that as starting point for the bootstrap. Haven't tried how trivial or difficult it is to make that work, but it sounds more long-term maintainable than having different problems depending on what architecture and host distribution is used - it doesn't sound like a good idea to have to test and maintain all combinations, e.g. whether Go 1.6 on arm64 in Ubuntu 16.04 manages to bootstrap the latest Go. cu Adrian -- "Is there not promise of rain?" Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. "Only a promise," Lao Er said. Pearl S. Buck - Dragon Seed -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On Fri, Apr 12, 2019 at 10:45:07PM +0300, Adrian Bunk wrote: > On Fri, Apr 12, 2019 at 01:30:46PM +0100, Daniel Thompson wrote: > > Currently go-native bootstraps without any dependency on the host go > > compiler by building go-1.4 (the last version written in C) and using > > that to build a more recent version of go. This does not work on host > > architectures, such as arm64, that are not supported by go-1.4. To > > support these host architectures we must rely on an existing host go > > compiler and use that to build go-native instead. > >... > > Why is this the only option for you? Good point. Not the most accurate use of "must"... more "these changes". > The clean way to bootstrap go would be by building gccgo and using that > as starting point for the bootstrap. > > Haven't tried how trivial or difficult it is to make that work, > but it sounds more long-term maintainable than having different > problems depending on what architecture and host distribution > is used - it doesn't sound like a good idea to have to test > and maintain all combinations, e.g. whether Go 1.6 on arm64 > in Ubuntu 16.04 manages to bootstrap the latest Go. Source wise it should work (depending on your level of belief in the go compatibility promise) since the go compiler is written in go-1.4 but I agree that code generation bugs in old versions of go tools could make things less predictable. I'm suddenly a bit swamped so it could me a long time to look at but using gccgo to bootstrap is not structurally much more complex than bootstrapping with go-1.4 and certainly could be more flexible. Sounds like "the right thing to do" so definitely worth a shot. Thanks Daniel. > -- > > "Is there not promise of rain?" Ling Tan asked suddenly out > of the darkness. There had been need of rain for many days. > "Only a promise," Lao Er said. > Pearl S. Buck - Dragon Seed > -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 7f8b043cc4..61a3069579 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -514,6 +514,10 @@ HOSTTOOLS_NONFATAL += "bzr" # Used by ssh fetcher HOSTTOOLS_NONFATAL += "scp" +# Used to bootstrap go more quickly (and to support host architectures not +# supported by go-1.4) +HOSTTOOLS_NONFATAL += "go" + CCACHE ??= "" TOOLCHAIN_OPTIONS = " --sysroot=${STAGING_DIR_TARGET}" diff --git a/meta/recipes-devtools/go/go-native.inc b/meta/recipes-devtools/go/go-native.inc index 207708745e..adf1666e29 100644 --- a/meta/recipes-devtools/go/go-native.inc +++ b/meta/recipes-devtools/go/go-native.inc @@ -10,14 +10,46 @@ CC = "${@d.getVar('BUILD_CC').strip()}" GOMAKEARGS ?= "--no-banner" +check_host_go () { + local GOROOT="`go env GOROOT`" + if [ ! -x "$GOROOT/bin/go" ] + then + return 1 + fi + + local GOVERSION=`$GOROOT/bin/go version | tr ' ' '\n' | grep ^go[0-9] | cut -b3-` + local GOMAJOR=`echo $GOVERSION | cut -d. -f 1` + local GOMINOR=`echo $GOVERSION | cut -d. -f 2` + if [ "`echo $GOMAJOR.$GOMINOR | tr -d 0-9`" != "." ] || \ + [ "$GOMAJOR" -lt 1 ] || \ + [ "$GOMAJOR" -eq 1 -a "$GOMINOR" -lt 4 ] + then + return 1 + fi + + return 0 +} + do_configure() { - cd ${WORKDIR}/go1.4/go/src - CGO_ENABLED=0 GOROOT=${WORKDIR}/go1.4/go ./make.bash + if check_host_go + then + bbplain "Bootstrap using host go compiler: `go env GOROOT`" + else + cd ${WORKDIR}/go1.4/go/src + CGO_ENABLED=0 GOROOT=${WORKDIR}/go1.4/go ./make.bash + fi } +do_configure[cleandirs] =+ "${GOTMPDIR}" do_compile() { export GOROOT_FINAL="${libdir_native}/go" - export GOROOT_BOOTSTRAP="${WORKDIR}/go1.4/go" + + if check_host_go + then + export GOROOT_BOOTSTRAP="`go env GOROOT`" + else + export GOROOT_BOOTSTRAP="${WORKDIR}/go1.4/go" + fi cd src ./make.bash ${GOMAKEARGS}
Currently go-native bootstraps without any dependency on the host go compiler by building go-1.4 (the last version written in C) and using that to build a more recent version of go. This does not work on host architectures, such as arm64, that are not supported by go-1.4. To support these host architectures we must rely on an existing host go compiler and use that to build go-native instead. Rather than add arm64 specific code that forces the use of the host go compiler, we use an architecture neutral approach based on adopting the host go compiler if it is both recent enough and the output of `go env GOROOT` passes basic sanity testing. Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org> --- meta/conf/bitbake.conf | 4 +++ meta/recipes-devtools/go/go-native.inc | 38 ++++++++++++++++++++++++-- 2 files changed, 39 insertions(+), 3 deletions(-) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core