Commit graph

16984 commits

Author SHA1 Message Date
Vincent Ambo
1d5b53abf8 chore(tools/blog_cli): Shorten metadata key names
Every character counts because I want to keep metadata below 255
bytes.
2019-08-19 19:11:32 +01:00
Carlos O'Ryan
7420d536ea
Merge pull request #16 from coryan/add-cloudtrace-v2-trace-library
feat: Add library for cloudtrace/v2/trace.proto.
2019-08-19 12:59:32 -04:00
Vincent Ambo
d5710db900 refactor(tools/blog_cli): More sensible chunk encoding as records
Instead of many rrdatas in one record, make many records!
2019-08-19 17:51:29 +01:00
Vincent Ambo
2a16740445 feat(tools): Add a horrifying tool to write blog posts into DNS 2019-08-19 16:59:55 +01:00
Vincent Ambo
28a9c01d36 feat(infra/k8s): Add in-cluster tazblog deployment via Nixery
First deployment actually using a Nixery image and `gitHEAD`.

This does not actually serve a working blog for various reasons. The
current storage mechanism (acid-state) isn't really appropriate
anymore and I'll need to change that soon.
2019-08-19 03:10:53 +01:00
Vincent Ambo
a4ef595fef chore(infra/k8s): Bump Nixery image to Cachix-enabled one 2019-08-19 02:43:42 +01:00
Vincent Ambo
2e688c18a5 feat(third_party): Add nixery with Cachix configuration 2019-08-19 02:40:14 +01:00
Vincent Ambo
daa6196c2a fix(build): Force nix-env to use NIX_PATH
Thanks to clever!
2019-08-19 01:56:17 +01:00
Vincent Ambo
ccf6a95f94 chore(build): Pin nixpkgs to a specific commit
This is the same commit for which Nixery has popularity data, but that
isn't particularly relevant.
2019-08-19 01:56:17 +01:00
Vincent Ambo
e7d7f73f7d feat(build): Add 'extraPackages' parameter
This makes it possible to inject additional programs (e.g. Cachix)
into a Nixery container.
2019-08-19 01:56:17 +01:00
Vincent Ambo
ffae282eac fix(docs): Correct link to layering strategy 2019-08-19 01:56:17 +01:00
Chris Feng
5b9f4b0851 Make it possible to answer questions from Emacs when in char-mode
* exwm-input.el (exwm-input--echo-area-timer): New variable storing
the timer used for detecting echo area messages.
(exwm-input--on-minibuffer-setup, exwm-input--on-minibuffer-exit)
(exwm-input--on-echo-area-dirty, exwm-input--on-echo-area-clear): New
functions for grabbing/releasing keyboard when minibuffer/echo becomes
active/inactive.
(exwm-input--init, exwm-input--exit): Register/Unregister them.
(exwm-input--grab-keyboard, exwm-input--release-keyboard): Validate
buffers.
2019-08-18 00:00:00 +00:00
Toon Nolten
1dbaf11948 Fix nix-env documentation for --delete-generations
The documentation for `--delete-generations` had an erroneous fullstop
and as it turns out inaccurate information on the `+No.` syntax.
2019-08-17 16:33:35 +02:00
Vincent Ambo
745b7ce0b8 fix(build): Ensure root user is known inside of container
This is required by git in cases where Nixery is configured with a
custom git repository.

I've also added a shell back into the image to make debugging a
running Nixery easier. It turns out some of the dependencies already
pull in bash anyways, so this is just surfacing it to $PATH.
2019-08-17 10:42:13 +01:00
Vincent Ambo
9a95c4124f fix(server): Sort requested packages in image name & spec
Before this change, Nixery would pass on the image name unmodified to
Nix which would lead it to cache-bust the manifest and configuration
layers for images that are content-identical but have different
package ordering.

This fixes #38.
2019-08-17 10:31:41 +01:00
Vincent Ambo
0ee239874b docs(README): Update links to layering strategy 2019-08-17 10:10:41 +01:00
Florian Klink
3b65fc8c72 feat(server): add iana-etc and cacert to the shell convenience package
These probably should be part of every container image by default, but
adding it to the "shell" convenience name probably is our best bet for
now.
2019-08-16 22:47:22 +01:00
Vincent Ambo
cb810687d5 fix(infra/k8s): Always pull a Nixery image 2019-08-16 19:57:10 +01:00
Vincent Ambo
dd35be7add feat(infra/k8s): Deploy Nixery instance to cluster 2019-08-16 18:20:20 +01:00
Vincent Ambo
ba06317836 feat(infra/gcp): Add Terraform configuration for GKE & friends
Sets up Terraform itself, a GKE cluster, a storage bucket and all the
other little things required to get the basics running.
2019-08-16 16:52:06 +01:00
Vincent Ambo
27036e18f5 feat(tools): Add kontemplate 2019-08-16 16:50:50 +01:00
Abseil Team
9a41ffdd3a Export of internal Abseil changes
--
1d8539bec6f0b390360b896d4c43da5b546486e1 by Derek Mauro <dmauro@google.com>:

Load C++ BUILD rules from the @rules_cc repository.
Required for Bazel 1.0.

Internal import of GitHub Pull Request #351

PiperOrigin-RevId: 263768433

--
017d315f1b5d4b4f2e1d6615265134f1001f2b84 by Derek Mauro <dmauro@google.com>:

Import of CCTZ from GitHub.

PiperOrigin-RevId: 263762409

--
77b3cdeca653df595d4709db74230f38b35c9c7c by CJ Johnson <johnsoncj@google.com>:

Fixes logical error in test file's static_assert

PiperOrigin-RevId: 263610500

--
cd358e4efc21c8c2ef79cb137d75c829aeee4417 by Derek Mauro <dmauro@google.com>:

Remove bazel_toolchains from WORKSPACE.bazel

The Abseil build doesn't use it anymore.

PiperOrigin-RevId: 263609624
GitOrigin-RevId: 1d8539bec6f0b390360b896d4c43da5b546486e1
Change-Id: I17a5ddfe7409aa110b945aeda4de14bc217907ad
2019-08-16 10:44:01 -04:00
Yannic
36910d3d7e [bazel] Add fixes for --incompatible_load_cc_rules_from_bzl (#351)
* [bazel] Add fixes for --incompatible_load_cc_rules_from_bzl

Starting with Bazel 1.0 (September 2019), C++ rules will need to be
loaded from the @rules_cc repository. This change adds the required
loads for that. For full compatibility, we will need versions of
googletest and google-benchmark that also includes these loads.

* Run buildifier again after merge
2019-08-16 10:38:13 -04:00
Eelco Dolstra
f435634a29
Merge pull request #3041 from zimbatm/nix-store-error-13
Fix for `unknown serve command 13`
2019-08-16 16:15:05 +02:00
Carlos O'Ryan
1e638c6002
Addressed review comments. 2019-08-16 10:10:41 -04:00
zimbatm
b226b5cd97
nix-store: fix out of sync protocol
If a NAR is already in the store, addToStore doesn't read the source
which makes the protocol go out of sync. This happens for example when
two client try to nix-copy-closure the same derivation at the same time.
2019-08-16 15:05:45 +02:00
zimbatm
91b00b145f
libutil: add SizedSource
Introduce the SizeSource which allows to bound how much data is being
read from a source. It also contains a drainAll() function to discard
the rest of the source, useful to keep the nix protocol in sync.
2019-08-16 15:05:40 +02:00
Carlos O'Ryan
62991403dd
feat: Add library for cloudtrace/v2/trace.proto. 2019-08-16 09:00:17 -04:00
Leonardo Brondani Schenkel
11c5981793 Use old IP address lookup logic as fallback (#275)
On some machines, resolving the local hostname results in a loopback IP
address (127.0.0.0/8). This breaks discovery. In these situations, fall
back to the old IP address lookup logic that was removed on commit 790edb9.
2019-08-16 12:13:53 +03:00
Abseil Team
aae8143cf9 Export of internal Abseil changes
--
f28b989d5161230c6561e923b458c797a96bcf90 by Greg Falcon <gfalcon@google.com>:

Import of CCTZ from GitHub.

PiperOrigin-RevId: 263586488

--
8259484025b7de45358719fc6182a48cac8044c6 by Andy Soffer <asoffer@google.com>:

Internal changes and combine namespaces into a single namespace.

PiperOrigin-RevId: 263560576

--
8d19f41661984a600d1f8bbfeb8a30fcb4dee7d6 by Mark Barolak <mbar@google.com>:

Inside of absl::string_view::copy, use absl::string_view::traits_type::copy instead of std:copy to do the actual work.  This both follows the C++ standard more closely and avoids avoid MSVC unchecked iterator warnings.

PiperOrigin-RevId: 263430502

--
c06bf74236e12c7c1c97bfcbbc9d29bd65d6b36c by Andy Soffer <asoffer@google.com>:

Remove force-inlining attributes. Benchmarking results indicate that they are creating meaningful performance differences.

PiperOrigin-RevId: 263364896

--
ec4fa6eac958a9521456201b138784f55d3b17bc by Abseil Team <absl-team@google.com>:

Make BM_Fill benchmarks more representative.

PiperOrigin-RevId: 263349482

--
4ae280b4eb31d9cb58e847eb670473340f7778c1 by Derek Mauro <dmauro@google.com>:

Fix new -Wdeprecated-copy warning in gcc9

PiperOrigin-RevId: 263348118

--
d238a92f452a5c35686f9c71596fdd1fe62090a2 by Matt Calabrese <calabrese@google.com>:

The std::is_trivially_xxx fail on versions of GCC up until 7.4 due to faulty underlying intrinsics, but our emulation succeeds. Update our traits to not compare against the standard library implementation in these versions.

PiperOrigin-RevId: 263209457
GitOrigin-RevId: f28b989d5161230c6561e923b458c797a96bcf90
Change-Id: I4c41db5928ba71e243aeace4420e06d1a2df0b5b
2019-08-15 14:36:45 -04:00
Vincent Ambo
b652ea2f03 feat(tools): Add Terraform 0.12 to local tools
This is pre-configured with the GCP provider.
2019-08-15 16:28:42 +01:00
Vincent Ambo
38493b83ea chore(tools): Remove bazel & ghc 2019-08-15 16:15:59 +01:00
Vincent Ambo
128875b501 chore: Remove remaining Bazel-related files 2019-08-15 16:11:30 +01:00
Vincent Ambo
a20daf8726 chore(nix): Remove overridden GHC
We're not going down the Bazel route here, so this is not going to be
necessary.
2019-08-15 16:07:28 +01:00
Vincent Ambo
77f0b5021d chore(services/gemma): Mark package as broken 2019-08-15 16:07:11 +01:00
Vincent Ambo
501e6ded5f fix(build): Ensure GCS signing is used in CI 2019-08-15 16:04:34 +01:00
Vincent Ambo
3f232e0170 docs: Add asciinema demo to README & book 2019-08-15 16:04:34 +01:00
Vincent Ambo
0ec369d76c docs(book): Update information on new layering strategy 2019-08-15 16:04:34 +01:00
Carlos O'Ryan
1d42dc5dab
Merge pull request #14 from coryan/fix-missing-libraries
bug: Missed two libraries.
2019-08-15 10:10:05 -04:00
Carlos O'Ryan
815a54fd92
Missed two libraries. 2019-08-15 10:01:30 -04:00
Carlos O'Ryan
36a7ebbe18
Merge pull request #13 from coryan/fix-missing-exports
feat: Add library for api/client.proto.
2019-08-15 09:48:04 -04:00
Eelco Dolstra
b7ea98bf34
Merge pull request #2945 from danidiaz/doc001
Expanded documentation for .nix-defexpr
2019-08-15 15:03:21 +02:00
Carlos O'Ryan
22b90b89b9
feat: Add library for api/client.proto.
Also fixed the list of libraries in googleapis-config.cmake.
2019-08-15 08:51:36 -04:00
Eelco Dolstra
477f82e5a7
Merge pull request #2782 from grahamc/flames
Track function start and end
2019-08-15 14:20:42 +02:00
Daniel Diaz
653c407784
Expanded documentation for .nix-defexpr 2019-08-15 08:05:22 -04:00
Vincent Ambo
ca1ffb397d feat(build): Add an integration test that runs on Travis
This test, after performing the usual Nixery build, loads the built
image into Docker, runs it, pulls an image from Nixery and runs that
image.

To make this work, there is some configuration on the Travis side.
Most importantly, the following environment variables have special
values:

* `GOOGLE_KEY`: This is set to a base64-encoded service account key to
  be used in the test.
* `GCS_SIGNING_PEM`: This is set to a base64-encoded signing key (in
  PEM) that is used for signing URLs.

Both of these are available to all branches in the Nixery repository.
2019-08-15 00:26:57 +01:00
Graham Christensen
ee9c988a1b
Track function start and ends for flame graphs
With this patch, and this file I called `log.py`:

    #!/usr/bin/env nix-shell
    #!nix-shell -i python3 -p python3 --pure

    import sys
    from pprint import pprint

    stack = []
    timestack = []

    for line in open(sys.argv[1]):
        components = line.strip().split(" ", 2)
        if components[0] != "function-trace":
            continue

        direction = components[1]
        components = components[2].rsplit(" ", 2)

        loc = components[0]
        _at = components[1]
        time = int(components[2])

        if direction == "entered":
            stack.append(loc)
            timestack.append(time)
        elif direction == "exited":
            dur = time - timestack.pop()
            vst = ";".join(stack)
            print(f"{vst} {dur}")
            stack.pop()

and:

    nix-instantiate --trace-function-calls -vvvv ../nixpkgs/pkgs/top-level/release.nix -A unstable > log.matthewbauer 2>&1
    ./log.py ./log.matthewbauer > log.matthewbauer.folded
    flamegraph.pl --title matthewbauer-post-pr log.matthewbauer.folded > log.matthewbauer.folded.svg

I can make flame graphs like: http://gsc.io/log.matthewbauer.folded.svg

---

Includes test cases around function call failures and tryEval. Uses
RAII so the finish is always called at the end of the function.
2019-08-14 16:09:35 -04:00
Vincent Ambo
85b9c30749 chore(server): Add 'go vet' to build process 2019-08-14 20:18:41 +01:00
Vincent Ambo
36d50d1f19 fix(server): Print correct project name during startup
They grow up so fast :')
2019-08-14 20:18:41 +01:00
Vincent Ambo
cf227c153f feat(builder): Implement build cache for manifests & layers
Implements a cache that keeps track of:

a) Manifests that have already been built (for up to 6 hours)
b) Layers that have already been seen (and uploaded to GCS)

This significantly speeds up response times for images that are full
or partial matches with previous images served by an instance.
2019-08-14 20:18:41 +01:00