maurice/snix
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
5102 commits 1 branch 0 tags 188 MiB
Commit graph

2957 commits

Author SHA1 Message Date
Eelco Dolstra
c5b83d8913
copyPaths(): Use queryValidPaths() to reduce SSH latency 2017-03-16 13:50:01 +01:00
Eelco Dolstra
91d67692cf
copyPaths(): Don't query path info for a path the target already has 
For example, this cuts "nix-copy-closure --from" on a NixOS system
closure from 15.9s to 0.5s.
 2017-03-16 12:05:51 +01:00
Eelco Dolstra
ea7fa88131
LegacySSHStore: Provide a faster implementation of computeFSClosure() 
This avoids the latency of the standard implementation, which can make
a huge difference (e.g. 16.5s -> 0.5s on a NixOS system closure).
 2017-03-16 11:44:52 +01:00
Eelco Dolstra
7a716ef2a5
Fix nix-copy-closure --to 2017-03-16 10:58:48 +01:00
Eelco Dolstra
0ec7f47b00
Remove "killing process <pid>" messages 
They convey no useful information.
 2017-03-16 10:52:28 +01:00
Eelco Dolstra
43f158bb08
nix-copy-closure: Fix assertion failure 
$ ./inst/bin/nix-copy-closure --to bla $(type -p firefox)
  nix-copy-closure: src/libstore/store-api.cc:80: std::__cxx11::string nix::storePathToHash(const Path&): Assertion `base.size() >= storePathHashLen' failed.
 2017-03-16 10:45:45 +01:00
Eelco Dolstra
3f35612c04
More precise compression settings 2017-03-15 17:20:19 +01:00
Eelco Dolstra
25dff2b7db
runProgram(): Distinguish between empty input and no input 
For example, if we call brotli with an empty input, it shouldn't read
from the caller's stdin.
 2017-03-15 16:50:19 +01:00
Eelco Dolstra
042975ea8e
Compress NAR listings using the "text-compression" method 
So if "text-compression=br", the .ls file in S3 will get a
Content-Encoding of "br". Brotli appears to compress better than xz
for this kind of file and is natively supported by browsers.
 2017-03-15 16:50:19 +01:00
Eelco Dolstra
45c70382ac
S3BinaryCacheStore: Set Content-Type 
This is necessary for serving log files to browsers.
 2017-03-15 16:50:19 +01:00
Eelco Dolstra
8b1d65bebe
S3BinaryCacheStore: Support compression of narinfo and log files 
You can now set the store parameter "text-compression=br" to compress
textual files in the binary cache (i.e. narinfo and logs) using
Brotli. This sets the Content-Encoding header; the extension of
compressed files is unchanged.

You can separately specify the compression of log files using
"log-compression=br". This is useful when you don't want to compress
narinfo files for backward compatibility.
 2017-03-15 16:49:28 +01:00
Eelco Dolstra
2691498b5c
Fix assertion failure on SIGINT 
nix: src/libutil/compression.cc:142: virtual nix::XzSink::~XzSink(): Assertion `finished' failed.
 2017-03-15 16:49:06 +01:00
Eelco Dolstra
fbbc4d8dda
Fix deadlock in runProgram() when input is larger than the pipe buffer size 2017-03-15 16:49:06 +01:00
Eelco Dolstra
e8186085e0
Add support for brotli compression 
Build logs on cache.nixos.org are compressed using Brotli (since this
allows them to be decompressed automatically by Chrome and Firefox),
so it's handy if "nix log" can decompress them.
 2017-03-15 16:49:06 +01:00
Eelco Dolstra
73d7a51ee6
Remove dependency on "curl" binary 2017-03-15 16:48:29 +01:00
Eelco Dolstra
532d73d5d8
BinaryCacheStore: Implement getBuildLog() 
We assume that build logs are stored under log/<drv>, e.g.

  /nix/store/q7ab198v13p0f8x8wgnd75dva7d5mip6-friday-devil-0.1.1.1.drv

maps to

  https://cache.nixos.org/log/q7ab198v13p0f8x8wgnd75dva7d5mip6-friday-devil-0.1.1.1.drv
 2017-03-15 16:48:29 +01:00
Eelco Dolstra
5b86451f02
Add a "nix log" command 
This replaces "nix-store --read-log". It checks the local store and
any configured substituters for the requested logs.
 2017-03-15 16:48:29 +01:00
Eelco Dolstra
0afeb7f51e
Store: Add a method for getting build logs 
This allows various Store implementations to provide different ways to
get build logs. For example, BinaryCacheStore can get the build logs
from the binary cache.

Also, remove the log-servers option since we can use substituters for
this.
 2017-03-15 16:48:29 +01:00
Shea Levy
96443e94a1 Merge branch 'configurable-aws-region' 2017-03-12 10:01:11 -04:00
Eelco Dolstra
86227390c5
Merge branch 'allow-import-from-derivation' of https://github.com/shlevy/nix 2017-03-09 13:36:30 +01:00
Adrien Devresse
d853877ce9 Add missing header <sstream> 2017-03-08 22:24:10 +01:00
Shea Levy
93f863be96 Add option to disable import-from-derivation completely, even if the drv is already realized 2017-03-08 08:46:12 -05:00
Shea Levy
3cc18d3753 Properly set the caFile for aws-sdk-cpp s3 2017-03-06 14:30:35 -05:00
Eelco Dolstra
1cf4801108
Revert fa125b9b28 
This causes quadratic performance.
 2017-03-06 13:56:20 +01:00
Shea Levy
5789eaa3f4 Add aws-region param to S3 store URLs 2017-03-03 16:12:17 -05:00
Eelco Dolstra
d1158bb816
Cache connection failures 2017-03-03 19:36:01 +01:00
Eelco Dolstra
8490ee37a6
SSHMaster: Make thread-safe 2017-03-03 19:28:27 +01:00
Eelco Dolstra
d3eb1cf3bb
build-remote: Don't use a SSH master 
This is unnecessary because we make only one connection.
 2017-03-03 19:23:20 +01:00
Eelco Dolstra
d140c75530
Pool: Don't hang if creating a connection fails 2017-03-03 19:21:43 +01:00
Eelco Dolstra
577ebeaefb
Improve SSH handling 
* Unify SSH code in SSHStore and LegacySSHStore.

* Fix a race starting the SSH master. We now wait synchronously for
  the SSH master to finish starting. This prevents the SSH clients
  from starting their own connections.

* Don't use a master if max-connections == 1.

* Add a "max-connections" store parameter.

* Add a "compress" store parameter.
 2017-03-03 19:05:50 +01:00
Eelco Dolstra
7f62be1bcd
build-remote: Fix passing SSH key 2017-03-03 16:33:18 +01:00
Eelco Dolstra
5a1fb03b8f
build-remote: Misc cleanup 2017-03-03 16:18:49 +01:00
Eelco Dolstra
2f992692e2
Fix fatal "broken pipe" error when $NIX_BUILD_HOOK is missing 2017-03-03 15:40:06 +01:00
Eelco Dolstra
ffcf9d24a6
builtins.fetchgit: Support paths 2017-03-02 11:46:28 +01:00
Eelco Dolstra
fbf17f1ad7
builtins.fetchgit: Fix bad format string 2017-03-02 11:40:11 +01:00
Eelco Dolstra
fa125b9b28
TeeSink: Pre-reserve string space 
When receiving a very large file, this can prevent the string from
having tobe copied, which temporarily doubles memory consumption.
 2017-03-01 16:16:04 +01:00
Eelco Dolstra
f61f67ddee
RemoteStore::addToStore(): Send NAR rather than string containing NAR 
This allows the NAR to be streamed in the future (though we're not
doing that yet).
 2017-03-01 16:07:15 +01:00
Eelco Dolstra
374908726b
readString(): Read directly into std::string 
When reading a huge string, this halves memory consumption.

(Strictly speaking, this appears only valid in C++17, but who cares...)
 2017-03-01 14:54:11 +01:00
Eelco Dolstra
07a0b8ca67
Tweak message 2017-03-01 14:52:16 +01:00
Eelco Dolstra
56e19d970d
nix-store --import: Fix importing unsigned paths 2017-03-01 14:47:52 +01:00
Eelco Dolstra
e321551d54
Fix assertion failure in nix-store --export 
Fixes #1173.
 2017-03-01 14:41:54 +01:00
Eelco Dolstra
c4a40949d9
Handle importing NARs containing files greater than 4 GiB 
Also templatize readInt() to work for various integer types.
 2017-03-01 13:52:54 +01:00
Eelco Dolstra
0780805246
Fix building against older curl versions 
http://hydra.nixos.org/build/49490928
 2017-02-28 14:03:43 +01:00
Eelco Dolstra
fd86dd93dd
Improve SQLite busy handling 2017-02-28 13:59:11 +01:00
Eelco Dolstra
34b12bad59
NarInfoDiskCache: Handle SQLite busy errors 2017-02-28 13:44:11 +01:00
Eelco Dolstra
80027144ae
In SQLite errors, include the database path 
This is necessary because we have multiple SQLite databases (e.g. the
binary cache cache).
 2017-02-28 13:20:12 +01:00
Eelco Dolstra
7251d048fa
Support auto-configuration of build-max-jobs 
"build-max-jobs" and the "-j" option can now be set to "auto" to use
the number of CPUs in the system. (Unlike build-cores, it doesn't use
0 to imply auto-configuration, because a) magic values are a bad idea
in general; b) 0 is a legitimate value used to disable local
building.)

Fixes #1198.
 2017-02-28 12:54:50 +01:00
Eelco Dolstra
3fab1f04a7
_SC_NPROCESSORS_ONLN -> std:🧵:hardware_concurrency() 2017-02-27 16:01:54 +01:00
Eelco Dolstra
3cb0387d3f
Retry downloads on HTTP/2 stream errors 
Issue #1254.
 2017-02-27 14:36:09 +01:00
Eelco Dolstra
d3e1aad421
nix-shell: Better error message when the shell can't be started 2017-02-24 17:25:00 +01:00