Using buildout with cache over network
======================================

Pure buildout ignore networkcache:

   >>> write(sample_buildout, 'buildout.cfg',
   ... '''
   ... [buildout]
   ... parts =
   ... networkcache-section = networkcache
   ... ''')
   >>> print system(buildout)
   Unused options for buildout: 'networkcache-section'.

So lets activate networkcache in buildout:
   >>> write(sample_buildout, 'buildout.cfg',
   ... '''
   ... [buildout]
   ... parts = networkcache
   ... find-links =
   ...   http://pypi.python.org/pypi/slapos.libnetworkcache
   ... # Do not register this buildout run
   ... installed =
   ...
   ... [networkcache]
   ... recipe = zc.recipe.egg
   ... eggs =
   ...   slapos.libnetworkcache
   ...   zc.buildout
   ... ''')
   >>> print system(buildout)
   Installing networkcache.
   Getting distribution for 'slapos.libnetworkcache'.
   Got slapos.libnetworkcache ...

Lets check that networkcache is correclty installed:
   >>> write(sample_buildout, 'buildout.cfg',
   ... '''
   ... [buildout]
   ... parts =
   ... ''')
   >>> print system(buildout + ' -v')
   Networkcache functionality not enabled. In order to activate use networkcache-section.
   ...

Now let's prepare networkcache server:
   >>> nc_data = tmpdir('nc_data')
   >>> nc_url = start_nc(nc_data)

And remote server:

   >>> remote_server_data = tmpdir('remote_server_data')
   >>> remote_server_url = start_server(remote_server_data)

Networkcache in buildout is optional, and during normal run no information
is shown:

   >>> write(sample_buildout, 'buildout.cfg',
   ... '''
   ... [buildout]
   ... parts =
   ... ''')
   >>> print system(buildout)

For some tests simple download recipe will be needed:

  >>> mkdir(sample_buildout, 'download')
  >>> write(sample_buildout, 'download', 'download.py',
  ... """
  ... import logging, os, zc.buildout
  ...
  ... class Download:
  ...
  ...     def __init__(self, buildout, name, options):
  ...         self.name, self.options, self.buildout = name, options, buildout
  ...
  ...     def install(self):
  ...         download = zc.buildout.download.Download(
  ...           self.buildout['buildout'])
  ...         path, is_temp = download(self.options['url'],
  ...           md5sum=self.options.get('md5sum'))
  ...         logging.getLogger(self.name).info(
  ...             'Downloaded %s', self.options['url'])
  ...         return []
  ...
  ...     update = install
  ... """)
  >>> write(sample_buildout, 'download', 'setup.py',
  ... """
  ... from setuptools import setup
  ...
  ... setup(
  ...     name = "download",
  ...     entry_points = {'zc.buildout': ['default = download:Download']},
  ...     )
  ... """)
  >>> write(sample_buildout, 'download', 'README.txt', " ")

Let's create file on remote server:

  >>> write(remote_server_data, 'hello.txt', 'Hello my friend')

It will be downloaded and stored into networkcache:

   >>> write(sample_buildout, 'buildout.cfg',
   ... '''
   ... [buildout]
   ...
   ... networkcache-section = networkcache
   ... develop = download
   ... parts = download
   ...
   ... [download]
   ... recipe = download
   ... url = %(remote_server_url)shello.txt
   ...
   ... [networkcache]
   ... download-cache-url = %(nc_url)sshacache
   ... download-dir-url = %(nc_url)sshadir
   ... upload-cache-url = %(nc_url)sshacache
   ... upload-dir-url = %(nc_url)sshadir
   ... ''' % globals())
   >>> print system(buildout)
   Networkcache enabled.
   Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
   Networkcache upload cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
   Develop: '/sample-buildout/download'
   Installing download.
   Downloading http://localhost/hello.txt
   Downloading hello.txt from network cache.
   Failed to download from network cache hello.txt:...404...
   Uploading http://localhost/hello.txt into network cache.
   download: Downloaded http://localhost/hello.txt

And run it again, data will be downloaded from networkcache:

   >>> print system(buildout)
   Networkcache enabled.
   Networkcache download cache: 'http://localhost:29630/shacache', directory 'http://localhost:29630/shadir'
   Networkcache upload cache: 'http://localhost:29630/shacache', directory 'http://localhost:29630/shadir'
   Develop: '/tmp/tmpOkGTHmbuildoutSetUp/_TEST_/sample-buildout/download'
   Updating download.
   Downloading http://localhost:25390/hello.txt
   Downloading hello.txt from network cache.
   download: Downloaded http://localhost:25390/hello.txt

Lets clean networkcache:

   >>> rmdir(nc_data)
   >>> mkdir(nc_data)

As one can see, in case if upload is not working, there is no problem with running buildout:

   >>> write(sample_buildout, 'buildout.cfg',
   ... '''
   ... [buildout]
   ...
   ... networkcache-section = networkcache
   ... develop = download
   ... parts = download
   ...
   ... [download]
   ... recipe = download
   ... url = %(remote_server_url)shello.txt
   ...
   ... [networkcache]
   ... download-cache-url = %(nc_url)sshacache
   ... download-dir-url = %(nc_url)sshadir
   ... upload-cache-url = http://127.0.0.1:1/sshacache
   ... upload-dir-url = http://127.0.0.1:1/sshadir
   ... ''' % globals())
   >>> print system(buildout)
   Networkcache enabled.
   Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
   Networkcache upload cache: 'http://127.0.0.1:1/sshacache', directory 'http://127.0.0.1:1/sshadir'
   Develop: '/sample-buildout/download'
   Updating download.
   Downloading http://localhost/hello.txt
   Downloading hello.txt from network cache.
   Failed to download from network cache hello.txt: ...404...
   Uploading http://localhost/hello.txt into network cache.
   Fail to upload file. ...111...
   download: Downloaded http://localhost/hello.txt

Lets clean networkcache:

   >>> rmdir(nc_data)
   >>> mkdir(nc_data)

Upload is optional, and then no upload will be tried:

   >>> write(sample_buildout, 'buildout.cfg',
   ... '''
   ... [buildout]
   ...
   ... networkcache-section = networkcache
   ... develop = download
   ... parts = download
   ...
   ... [download]
   ... recipe = download
   ... url = %(remote_server_url)shello.txt
   ...
   ... [networkcache]
   ... download-cache-url = %(nc_url)sshacache
   ... download-dir-url = %(nc_url)sshadir
   ... ''' % globals())
   >>> print system(buildout)
   Networkcache enabled.
   Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
   Networkcache upload cache: '', directory ''
   Develop: '/sample-buildout/download'
   Updating download.
   Downloading http://localhost/hello.txt
   Downloading hello.txt from network cache.
   Failed to download from network cache hello.txt: ...404...
   download: Downloaded http://localhost/hello.txt

In case if download cache is failing, original resources are used:

(Note: Due to usage of urllib2 during download which "eats" real socket error
message not perfect error is shown like <urlopen...Connection Refused>)

   >>> write(sample_buildout, 'buildout.cfg',
   ... '''
   ... [buildout]
   ...
   ... networkcache-section = networkcache
   ... develop = download
   ... parts = download
   ...
   ... [download]
   ... recipe = download
   ... url = %(remote_server_url)shello.txt
   ...
   ... [networkcache]
   ... download-cache-url = http://127.0.0.1:1/shacache
   ... download-dir-url = http://127.0.0.1:1/shadir
   ... upload-cache-url = http://127.0.0.1:1/sshacache
   ... upload-dir-url = http://127.0.0.1:1/sshadir
   ... ''' % globals())
   >>> print system(buildout)
   Networkcache enabled.
   Networkcache download cache: 'http://127.0.0.1:1/shacache', directory 'http://127.0.0.1:1/shadir'
   Networkcache upload cache: 'http://127.0.0.1:1/sshacache', directory 'http://127.0.0.1:1/sshadir'
   Develop: '/sample-buildout/download'
   Updating download.
   Downloading http://localhost/hello.txt
   Downloading hello.txt from network cache.
   Failed to download from network cache hello.txt: ...111...
   Uploading http://localhost/hello.txt into network cache.
   Fail to upload file. ...111...
   download: Downloaded http://localhost/hello.txt

Two paris of key certificate will be needed:

  >>> main_key = """
  ... -----BEGIN RSA PRIVATE KEY-----
  ... MIICXQIBAAKBgQDHjK3vOwrStTuoxGcfPjQkNnpwyzNGuUqqKryt7/fBGiy3qRZa
  ... sjmW63jlx8MM7vCXhAWs1y4jtxcskN/UoAHtufMMaImpgGSePrKqtczDiPIUw49i
  ... Do7DOA5uSIDA0VocVWKMG1HT3oFff/1bvrJdxbUCt/q7Pb7pwnBujtcyNQIDAQAB
  ... AoGAFW6/s8U0qWL70Gmf5dQph4YuanT4weze3x4N++SdSKlGCkfn9b/Uvaf26T8m
  ... bPow7yUn8uN9uipNG+VHlS/GZRCWGIdw667fq2x5By3MO/DKVDC2ZNfDraMIFHQ0
  ... SG0QuonDtfkZLzw6FjOQ+VpfNioJ0rDe68io0pQmu/yL2qECQQDnZJj5wPRcExgf
  ... 2WNtK66GYRos3UlIkyW3rtGDOitjm13k49zFTlNmwBuATvygNtmYcvApup9N8Q0I
  ... 1ONlc2kNAkEA3MUuh9y0toIaEEjoL4UNHfqFlrX30UZNWuyGGj4o2usqJVkNgHNJ
  ... koK7aP+x9GIchECzR9FzkH7u6Q639TTTyQJBAII4WOblMh/HFyKFYrEDjjpN1z88
  ... YYuYU/elSH9DczuhKhwwKARZ6hG/NzrkOMOpWpeyx3mUbyab66v2supbItkCQEBR
  ... PeYc+ZT2QEugQeyeMZ0bQFIQ/3q7pYuovsTkeRiWTHgHArT6YWPKrHjBjYGqF667
  ... /r8cmTNul4HUZOmOE2kCQQDDbE9xkGsTcgFgtb646kgmrmXneNgGE5TGfF6Lfw8w
  ... r2qrA8qR1lkEBVvzMiDl5NHR+6Jo0irdqdKbUgocbYpi
  ... -----END RSA PRIVATE KEY-----
  ... """
  >>> main_key_file = 'main.key'
  >>> open(main_key_file, 'w').write(main_key)
  >>> main_certificate = """
  ... -----BEGIN CERTIFICATE-----
  ... MIICkjCCAfsCADANBgkqhkiG9w0BAQsFADCBkTELMAkGA1UEBhMCRlIxDTALBgNV
  ... BAgTBE5vcmQxDzANBgNVBAoTBlNsYXBPUzEYMBYGA1UECxMPSG9uZXkgSGFydmVz
  ... dGVyMSAwHgYDVQQDExdTbGFwT1MgVGVzdCBDZXJ0aWZpY2F0ZTEmMCQGCSqGSIb3
  ... DQEJARYXU2xhcE9TIFRlc3QgQ2VydGlmaWNhdGUwHhcNMTEwODI0MDc1MzM2WhcN
  ... MTIwODI0MDc1MzM2WjCBkTELMAkGA1UEBhMCRlIxDTALBgNVBAgTBE5vcmQxDzAN
  ... BgNVBAoTBlNsYXBPUzEYMBYGA1UECxMPSG9uZXkgSGFydmVzdGVyMSAwHgYDVQQD
  ... ExdTbGFwT1MgVGVzdCBDZXJ0aWZpY2F0ZTEmMCQGCSqGSIb3DQEJARYXU2xhcE9T
  ... IFRlc3QgQ2VydGlmaWNhdGUwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAMeM
  ... re87CtK1O6jEZx8+NCQ2enDLM0a5SqoqvK3v98EaLLepFlqyOZbreOXHwwzu8JeE
  ... BazXLiO3FyyQ39SgAe258wxoiamAZJ4+sqq1zMOI8hTDj2IOjsM4Dm5IgMDRWhxV
  ... YowbUdPegV9//Vu+sl3FtQK3+rs9vunCcG6O1zI1AgMBAAEwDQYJKoZIhvcNAQEL
  ... BQADgYEAoB8o6AdKLtMFGr9OLmdJMHQ1Aig/UMzTb4+Dqx59xpmf25tWeSNYuFv0
  ... Q6YD8Ijm2RlsPhv3crEalIY0xnEHBqdLFOnRlnivYX7hPol5xs/MKzMI3wA1Tj2f
  ... tjbUSd1TIX6NrmHU7zuh48XwneJoDwZyBHKKCQWE9YLjjdxj0No=
  ... -----END CERTIFICATE-----
  ... """
  >>> main_certificate = '\n'.join(['  ' + q for q in main_certificate.splitlines()])
  >>> additional_key = """
  ... -----BEGIN RSA PRIVATE KEY-----
  ... MIICXQIBAAKBgQDDrOO87nSiDcXOf+xGc4Iqcdjfwd0RTOxEkO9z8mPZVg2bTPwt
  ... /GwtPgmIC4po3bJdsCpJH21ZJwfmUpaQWIApj3odDAbRXQHWhNiw9ZPMHTCmf8Zl
  ... yAJBxy9KI9M/fJ5RA67CJ6UYFbpF7+ZrXdkvG+0hdRX5ub0WyTPxc6kEIwIDAQAB
  ... AoGBAIgUj1jQGKqum1bt3dps8CQmgqWyA9TJQzK3/N8MveXik5niYypz9qNMFoLX
  ... S818CFRhdDbgNUKgAz1pSC5gbdfCDHYQTBrIt+LGpNSpdmQwReu3XoWOPZp4VWnO
  ... uCpAkDVt+88wbxtMbZ5/ExNFs2xTO66Aad1dG12tPWoyAf4pAkEA4tCLPFNxHGPx
  ... tluZXyWwJfVZEwLLzJ9gPkYtWrq843JuKlai2ziroubVLGSxeovBXvsjxBX95khn
  ... U6G9Nz5EzwJBANzal8zebFdFfiN1DAyGQ4QYsmz+NsRXDbHqFVepymUId1jAFAp8
  ... RqNt3Y78XlWOj8z5zMd4kWAR62p6LxJcyG0CQAjCaw4qXszs4zHaucKd7v6YShdc
  ... 3UgKw6nEBg5h9deG3NBPxjxXJPHGnmb3gI8uBIrJgikZfFO/ahYlwev3QKsCQGJ0
  ... kHekMGg3cqQb6eMrd63L1L8CFSgyJsjJsfoCl1ezDoFiH40NGfCBaeP0XZmGlFSs
  ... h73k4eoSEwDEt3dYJYECQQCBssN92KuYCOfPkJ+OV1tKdJdAsNwI13kA//A7s7qv
  ... wHQpWKk/PLmpICMBeIiE0xT+CmCfJVOlQrqDdujganZZ
  ... -----END RSA PRIVATE KEY-----
  ... """
  >>> additional_key_file = 'additional.key'
  >>> open(additional_key_file, 'w').write(additional_key)
  >>> additional_certificate = """
  ... -----BEGIN CERTIFICATE-----
  ... MIICgDCCAekCADANBgkqhkiG9w0BAQsFADCBiDELMAkGA1UEBhMCVUwxETAPBgNV
  ... BAgTCEJlZSBZYXJkMRgwFgYDVQQKEw9CZWUtS2VlcGVyIEx0ZC4xGDAWBgNVBAsT
  ... D0hvbmV5IEhhcnZlc3RlcjEVMBMGA1UEAxMMTWF5YSB0aGUgQmVlMRswGQYJKoZI
  ... hvcNAQkBFgxNYXlhIHRoZSBCZWUwHhcNMTEwODI0MDc1MTU2WhcNMTIwODI0MDc1
  ... MTU2WjCBiDELMAkGA1UEBhMCVUwxETAPBgNVBAgTCEJlZSBZYXJkMRgwFgYDVQQK
  ... Ew9CZWUtS2VlcGVyIEx0ZC4xGDAWBgNVBAsTD0hvbmV5IEhhcnZlc3RlcjEVMBMG
  ... A1UEAxMMTWF5YSB0aGUgQmVlMRswGQYJKoZIhvcNAQkBFgxNYXlhIHRoZSBCZWUw
  ... gZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAMOs47zudKINxc5/7EZzgipx2N/B
  ... 3RFM7ESQ73PyY9lWDZtM/C38bC0+CYgLimjdsl2wKkkfbVknB+ZSlpBYgCmPeh0M
  ... BtFdAdaE2LD1k8wdMKZ/xmXIAkHHL0oj0z98nlEDrsInpRgVukXv5mtd2S8b7SF1
  ... Ffm5vRbJM/FzqQQjAgMBAAEwDQYJKoZIhvcNAQELBQADgYEAaT4yamJJowDKMSD2
  ... eshUW8pjctg6O3Ncm5XDIKd77sRf7RwPjFh+BR59lfFf9xvOu8WymhtUU7FoPDW3
  ... MYZmKV7A3nFehN9A+REz+WU3I7fE6vQRh9jKeuxnQLRv0TdP9CEdPcYcs/EQpIDb
  ... 8du+N7wcN1ZO8veWSafBzcqgCwg=
  ... -----END CERTIFICATE-----
  ... """
  >>> additional_certificate = '\n'.join(['  ' + q for q in additional_certificate.splitlines()])

First put not signed file in networkcache:

   >>> write(sample_buildout, 'buildout.cfg',
   ... '''
   ... [buildout]
   ...
   ... networkcache-section = networkcache
   ... develop = download
   ... parts = download
   ...
   ... [download]
   ... recipe = download
   ... url = %(remote_server_url)shello.txt
   ...
   ... [networkcache]
   ... download-cache-url = %(nc_url)sshacache
   ... download-dir-url = %(nc_url)sshadir
   ... upload-cache-url = %(nc_url)sshacache
   ... upload-dir-url = %(nc_url)sshadir
   ... ''' % globals())
   >>> print system(buildout)
   Networkcache enabled.
   Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
   Networkcache upload cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
   Develop: '/sample-buildout/download'
   Updating download.
   Downloading http://localhost/hello.txt
   Downloading hello.txt from network cache.
   Failed to download from network cache hello.txt: ...404...
   Uploading http://localhost/hello.txt into network cache.
   download: Downloaded http://localhost/hello.txt

Now enable signature, so it should not download any content which is not trusted :

   >>> write(sample_buildout, 'buildout.cfg',
   ... '''
   ... [buildout]
   ...
   ... networkcache-section = networkcache
   ... develop = download
   ... parts = download
   ...
   ... [download]
   ... recipe = download
   ... url = %(remote_server_url)shello.txt
   ...
   ... [networkcache]
   ... download-cache-url = %(nc_url)sshacache
   ... download-dir-url = %(nc_url)sshadir
   ... upload-cache-url = %(nc_url)sshacache
   ... upload-dir-url = %(nc_url)sshadir
   ... signature-private-key-file = %(main_key_file)s
   ... signature-certificate-list = %(main_certificate)s
   ... ''' % globals())
   >>> print system(buildout)
   Networkcache enabled.
   Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
   Networkcache upload cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
   Develop: '/sample-buildout/download'
   Updating download.
   Downloading http://localhost/hello.txt
   Downloading hello.txt from network cache.
   Failed to download from network cache hello.txt: Could not find a trustable entry.
   Uploading http://localhost/hello.txt into network cache.
   download: Downloaded http://localhost/hello.txt

Once it has a trusted content available into the server it should be abled to download it:

   >>> print system(buildout)
   Networkcache enabled.
   Networkcache download cache: 'http://localhost:26283/shacache', directory 'http://localhost:26283/shadir'
   Networkcache upload cache: 'http://localhost:26283/shacache', directory 'http://localhost:26283/shadir'
   Develop: '/tmp/tmpO55kN1buildoutSetUp/_TEST_/sample-buildout/download'
   Updating download.
   Downloading http://localhost:22168/hello.txt
   Downloading hello.txt from network cache.
   download: Downloaded http://localhost:22168/hello.txt

There is a new certificate file, so it should not download the old content:

   >>> write(sample_buildout, 'buildout.cfg',
   ... '''
   ... [buildout]
   ...
   ... networkcache-section = networkcache
   ... develop = download
   ... parts = download
   ...
   ... [download]
   ... recipe = download
   ... url = %(remote_server_url)shello.txt
   ...
   ... [networkcache]
   ... download-cache-url = %(nc_url)sshacache
   ... download-dir-url = %(nc_url)sshadir
   ... signature-private-key-file = %(additional_key_file)s
   ... signature-certificate-list = %(additional_certificate)s
   ... ''' % globals())
   >>> print system(buildout)
   Networkcache enabled.
   Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
   Networkcache upload cache: '', directory ''
   Develop: '/sample-buildout/download'
   Updating download.
   Downloading http://localhost/hello.txt
   Downloading hello.txt from network cache.
   Failed to download from network cache hello.txt: Could not find a trustable entry.
   download: Downloaded http://localhost/hello.txt


If it can not find a trustable entry into server, it should upload the files using new signature:

   >>> write(sample_buildout, 'buildout.cfg',
   ... '''
   ... [buildout]
   ...
   ... networkcache-section = networkcache
   ... develop = download
   ... parts = download
   ...
   ... [download]
   ... recipe = download
   ... url = %(remote_server_url)shello.txt
   ...
   ... [networkcache]
   ... download-cache-url = %(nc_url)sshacache
   ... download-dir-url = %(nc_url)sshadir
   ... signature-private-key-file = %(additional_key_file)s
   ... signature-certificate-list = %(additional_certificate)s
   ... upload-cache-url = %(nc_url)sshacache
   ... upload-dir-url = %(nc_url)sshadir
   ... ''' % globals())
   >>> print system(buildout)
   Networkcache enabled.
   Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
   Networkcache upload cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
   Develop: '/sample-buildout/download'
   Updating download.
   Downloading http://localhost/hello.txt
   Downloading hello.txt from network cache.
   Failed to download from network cache hello.txt: Could not find a trustable entry.
   Uploading http://localhost/hello.txt into network cache.
   download: Downloaded http://localhost/hello.txt


Buildout can download the content using the old certificatei, because it still available in the cache server:

   >>> write(sample_buildout, 'buildout.cfg',
   ... '''
   ... [buildout]
   ...
   ... networkcache-section = networkcache
   ... develop = download
   ... parts = download
   ...
   ... [download]
   ... recipe = download
   ... url = %(remote_server_url)shello.txt
   ...
   ... [networkcache]
   ... download-cache-url = %(nc_url)sshacache
   ... download-dir-url = %(nc_url)sshadir
   ... signature-private-key-file = %(main_key_file)s
   ... signature-certificate-list = %(main_certificate)s
   ... upload-cache-url = %(nc_url)sshacache
   ... upload-dir-url = %(nc_url)sshadir
   ... ''' % globals())
   >>> print system(buildout)
   Networkcache enabled.
   Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
   Networkcache upload cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
   Develop: '/sample-buildout/download'
   Updating download.
   Downloading http://localhost/hello.txt
   Downloading hello.txt from network cache.
   download: Downloaded http://localhost/hello.txt

Buildout can download the content using the new certificate, because it still available in the cache server:

   >>> write(sample_buildout, 'buildout.cfg',
   ... '''
   ... [buildout]
   ...
   ... networkcache-section = networkcache
   ... develop = download
   ... parts = download
   ...
   ... [download]
   ... recipe = download
   ... url = %(remote_server_url)shello.txt
   ...
   ... [networkcache]
   ... download-cache-url = %(nc_url)sshacache
   ... download-dir-url = %(nc_url)sshadir
   ... signature-private-key-file = %(main_key_file)s
   ... signature-certificate-list = %(main_certificate)s
   ... upload-cache-url = %(nc_url)sshacache
   ... upload-dir-url = %(nc_url)sshadir
   ... ''' % globals())
   >>> print system(buildout)
   Networkcache enabled.
   Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
   Networkcache upload cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
   Develop: '/sample-buildout/download'
   Updating download.
   Downloading http://localhost/hello.txt
   Downloading hello.txt from network cache.
   download: Downloaded http://localhost/hello.txt

Lets clean networkcache:

   >>> rmdir(nc_data)
   >>> mkdir(nc_data)

Of course two certificates can be used in order to check downloaded content. This supports a case, when more then one key was used to sign content.

Lets create another file on remote server:

  >>> write(remote_server_data, 'hello-signed.txt', 'Hello my signed friend')

Main certificate will be used to upload hello.txt:

   >>> write(sample_buildout, 'buildout.cfg',
   ... '''
   ... [buildout]
   ...
   ... networkcache-section = networkcache
   ... develop = download
   ... parts = download
   ...
   ... [download]
   ... recipe = download
   ... url = %(remote_server_url)shello.txt
   ...
   ... [networkcache]
   ... download-cache-url = %(nc_url)sshacache
   ... download-dir-url = %(nc_url)sshadir
   ... upload-cache-url = %(nc_url)sshacache
   ... upload-dir-url = %(nc_url)sshadir
   ... signature-private-key-file = %(main_key_file)s
   ... ''' % globals())
   >>> print system(buildout)
   Networkcache enabled.
   Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
   Networkcache upload cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
   Develop: '/sample-buildout/download'
   Updating download.
   Downloading http://localhost/hello.txt
   Downloading hello.txt from network cache.
   Failed to download from network cache hello.txt: ...404...
   Uploading http://localhost/hello.txt into network cache.
   download: Downloaded http://localhost/hello.txt

Additional certificate will be used to upload hello-signed.txt:

   >>> write(sample_buildout, 'buildout.cfg',
   ... '''
   ... [buildout]
   ...
   ... networkcache-section = networkcache
   ... develop = download
   ... parts = download
   ...
   ... [download]
   ... recipe = download
   ... url = %(remote_server_url)shello-signed.txt
   ...
   ... [networkcache]
   ... download-cache-url = %(nc_url)sshacache
   ... download-dir-url = %(nc_url)sshadir
   ... upload-cache-url = %(nc_url)sshacache
   ... upload-dir-url = %(nc_url)sshadir
   ... signature-private-key-file = %(additional_key_file)s
   ... ''' % globals())
   >>> print system(buildout)
   Networkcache enabled.
   Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
   Networkcache upload cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
   Develop: '/sample-buildout/download'
   Uninstalling download.
   Installing download.
   Downloading http://localhost/hello-signed.txt
   Downloading hello-signed.txt from network cache.
   Failed to download from network cache hello-signed.txt: ...404...
   Uploading http://localhost/hello-signed.txt into network cache.
   download: Downloaded http://localhost/hello-signed.txt

And if both certificates are used, both files will be trusted:

   >>> write(sample_buildout, 'buildout.cfg',
   ... '''
   ... [buildout]
   ...
   ... networkcache-section = networkcache
   ... develop = download
   ... parts = download download-signed
   ...
   ... [download]
   ... recipe = download
   ... url = %(remote_server_url)shello.txt
   ...
   ... [download-signed]
   ... recipe = download
   ... url = %(remote_server_url)shello-signed.txt
   ...
   ... [networkcache]
   ... download-cache-url = %(nc_url)sshacache
   ... download-dir-url = %(nc_url)sshadir
   ... upload-cache-url = %(nc_url)sshacache
   ... upload-dir-url = %(nc_url)sshadir
   ... signature-certificate-list =
   ... %(main_certificate)s
   ... %(additional_certificate)s
   ... ''' % globals())
   >>> print system(buildout)
   Networkcache enabled.
   Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
   Networkcache upload cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
   Develop: '/sample-buildout/download'
   Uninstalling download.
   Installing download.
   Downloading http://localhost/hello.txt
   Downloading hello.txt from network cache.
   download: Downloaded http://localhost/hello.txt
   Installing download-signed.
   Downloading http://localhost/hello-signed.txt
   Downloading hello-signed.txt from network cache.
   download-signed: Downloaded http://localhost/hello-signed.txt

Of course eggs are also supported by networkcache:

   >>> write(sample_buildout, 'buildout.cfg',
   ... '''
   ... [buildout]
   ...
   ... networkcache-section = networkcache
   ... find-links = %(link_server)s
   ... develop = download
   ... parts = eggs
   ...
   ... [eggs]
   ... recipe = zc.recipe.egg:eggs
   ... eggs = demoneeded == 1.2c1
   ...
   ... [networkcache]
   ... download-cache-url = %(nc_url)sshacache
   ... download-dir-url = %(nc_url)sshadir
   ... upload-cache-url = %(nc_url)sshacache
   ... upload-dir-url = %(nc_url)sshadir
   ... ''' % globals())
   >>> print system(buildout)
   Networkcache enabled.
   Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
   Networkcache upload cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
   Develop: '/sample-buildout/download'
   Uninstalling download-signed.
   Uninstalling download.
   Installing eggs.
   Getting distribution for 'demoneeded==1.2c1'.
   Downloading demoneeded-1.2c1.zip from network cache.
   Failed to download from network cache demoneeded-1.2c1.zip: ...404...
   Uploading http://localhost/demoneeded-1.2c1.zip into network cache.
   Got demoneeded 1.2c1.

It is possible to pass SSL certificates and key to be used by authentication.
In order to do it NetworkcacheClient.upload method have to be patched to show
that proper keys are used.

   >>> mkdir(sample_buildout, 'ncpatch')
   >>> write(sample_buildout, 'ncpatch', 'ncpatch.py',
   ... """
   ... def patched(self, *args, **kwargs):
   ...     print 'shacache_cert_file', self.shacache_cert_file
   ...     print 'shacache_key_file', self.shacache_key_file
   ...     print 'shadir_cert_file', self.shadir_cert_file
   ...     print 'shadir_key_file', self.shadir_key_file
   ...     return False
   ...
   ... def ext(buildout):
   ...     print 'Patching slapos.libnetworkcache'
   ...     import slapos.libnetworkcache
   ...     slapos.libnetworkcache.NetworkcacheClient.upload = patched
   ... """)
   >>> write(sample_buildout, 'ncpatch', 'setup.py',
   ... """
   ... from setuptools import setup
   ...
   ... setup(
   ...     name = "ncpatch",
   ...     entry_points = {'zc.buildout.extension': ['ncpatch = ncpatch:ext']},
   ...     )
   ... """)
   >>> write(sample_buildout, 'ncpatch', 'README.txt', " ")

Lets activiate this extension:

    >>> write(sample_buildout, 'buildout.cfg',
    ... """
    ... [buildout]
    ... develop = ncpatch
    ... parts =
    ... """)
   >>> print system(buildout)
   Develop: '/sample-buildout/ncpatch'
   Uninstalling eggs.

Lets clean networkcache:

   >>> rmdir(nc_data)
   >>> mkdir(nc_data)

And see what happens:

   >>> write(sample_buildout, 'buildout.cfg',
   ... '''
   ... [buildout]
   ...
   ... networkcache-section = networkcache
   ... find-links = %(link_server)s
   ... develop = ncpatch download
   ... extensions = ncpatch
   ... parts = download
   ...
   ... [download]
   ... recipe = download
   ... url = %(remote_server_url)shello.txt
   ...
   ... [networkcache]
   ... download-cache-url = %(nc_url)sshacache
   ... download-dir-url = %(nc_url)sshadir
   ... upload-cache-url = %(nc_url)sshacache
   ... upload-dir-url = %(nc_url)sshadir
   ... shacache-cert-file = /path/to/shacache/cert/file
   ... shacache-key-file = /path/to/shacache/key/file
   ... shadir-cert-file = /path/to/shadir/cert/file
   ... shadir-key-file = /path/to/shadir/key/file
   ... ''' % globals())
   >>> print system(buildout)
   Networkcache enabled.
   Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
   Networkcache upload cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
   Patching slapos.libnetworkcache
   Develop: '/sample-buildout/ncpatch'
   Develop: '/sample-buildout/download'
   Installing download.
   Downloading http://localhost/hello.txt
   Downloading hello.txt from network cache.
   Failed to download from network cache hello.txt:...404...
   Uploading http://localhost/hello.txt into network cache.
   shacache_cert_file /path/to/shacache/cert/file
   shacache_key_file /path/to/shacache/key/file
   shadir_cert_file /path/to/shadir/cert/file
   shadir_key_file /path/to/shadir/key/file
   download: Downloaded http://localhost/hello.txt

Lets clean networkcache:

   >>> rmdir(nc_data)
   >>> mkdir(nc_data)

And just for sure, extends are supported too:

   >>> write(remote_server_data, 'external.cfg', '[buildout]')
   >>> write(sample_buildout, 'buildout.cfg',
   ... '''
   ... [buildout]
   ...
   ... networkcache-section = networkcache
   ... extends = %(remote_server_url)sexternal.cfg
   ... develop = download
   ... parts =
   ...
   ... [networkcache]
   ... download-cache-url = %(nc_url)sshacache
   ... download-dir-url = %(nc_url)sshadir
   ... upload-cache-url = %(nc_url)sshacache
   ... upload-dir-url = %(nc_url)sshadir
   ... ''' % globals())
   >>> print system(buildout)
   Test not implemented.

###############
# PARSING URL #
###############
Check if the parsing url method works correctly, in low-level:

   >>> get_filename_from_url("http://localhost/lib/patch/?id=700c7d5382b01f94e7141")
   'id=700c7d5382b01f94e7141'
   >>> get_filename_from_url("http://localhost/tarball-5.1.tar.gz?id=700")
   'tarball-5.1.tar.gz'
   >>> get_filename_from_url("http://localhost/somedir/?")
   'somedir'

Check if the networkcache upload method is using the correct method to
parse the original url and get the file name:

   >>> tmp_dir = tmpdir('tmp_dir')
   >>> write(tmp_dir, 'tmp_file', 'Content of temp file.')
   >>> get_filename_from_upload_network_cached(
   ...                       dir_url=globals().get('nc_url') + 'shadir',
   ...                       cache_url=globals().get('nc_url') + 'shacache',
   ...                       external_url="http://localhost/lib/patch/?id=700c7d5382b01f94e7141",
   ...                       path=globals().get('tmp_dir') +  '/tmp_file',
   ...                       nc_server_path=sample_buildout)
   'id=700c7d5382b01f94e7141'

Buildout bindings to networkcache are immune on slapos.libnetworkcache problems.

Lets prepare networkcache with errors:

  >>> mkdir(sample_buildout, 'damagednc')
  >>> mkdir(sample_buildout, 'damagednc', 'slapos')
  >>> write(sample_buildout, 'damagednc', 'slapos', '__init__.py',
  ... """
  ... # See http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages
  ... try:
  ...     __import__('pkg_resources').declare_namespace(__name__)
  ... except ImportError:
  ...     from pkgutil import extend_path
  ...     __path__ = extend_path(__path__, __name__)
  ... """)
  >>> write(sample_buildout, 'damagednc', 'slapos', 'libnetworkcache.py',
  ... """
  ... += die during import
  ... """)
  >>> write(sample_buildout, 'damagednc', 'setup.py',
  ... """
  ... from setuptools import setup, find_packages
  ...
  ... setup(
  ...     name = "slapos.libnetworkcache",
  ...     namespace_packages=['slapos'],
  ...     packages=find_packages(),
  ...     install_requires=['setuptools']
  ...     )
  ... """)
  >>> write(sample_buildout, 'damagednc', 'README.txt', " ")

And install it in buildout:

   >>> write(sample_buildout, 'buildout.cfg',
   ... '''
   ... [buildout]
   ... parts = networkcache
   ... develop = damagednc
   ... # Do not register this buildout run
   ... installed =
   ...
   ... [networkcache]
   ... recipe = zc.recipe.egg
   ... eggs =
   ...   slapos.libnetworkcache
   ...   zc.buildout
   ... ''')
   >>> print system(buildout)
   Develop: '/sample-buildout/damagednc'
   Installing networkcache.
   Generated script '/sample-buildout/bin/buildout'.

So lets use such damaged networkcache:

  >>> write(sample_buildout, 'buildout.cfg',
  ... '''
  ... [buildout]
  ...
  ... networkcache-section = networkcache
  ... develop = download damagednc
  ... parts = download
  ...
  ... [download]
  ... recipe = download
  ... url = %(remote_server_url)shello.txt
  ...
  ... [networkcache]
  ... download-cache-url = %(nc_url)sshacache
  ... download-dir-url = %(nc_url)sshadir
  ... ''' % globals())
  >>> print system(buildout)
  There was problem while trying to import slapos.libnetworkcache:
  Traceback (most recent call last):
  ...
  SyntaxError: invalid syntax
  <BLANKLINE>
  Networkcache forced to be disabled.
  There was problem while trying to import slapos.libnetworkcache:
  Traceback (most recent call last):
  ...
  SyntaxError: invalid syntax
  <BLANKLINE>
  Networkcache forced to be disabled.
  Develop: '/sample-buildout/download'
  Develop: '/sample-buildout/damagednc'
  Unused options for buildout: 'networkcache-section'.
  Installing download.
  Downloading http://localhost/hello.txt
  download: Downloaded http://localhost/hello.txt

If networkcache internally has issues, or its API got changed, buildout will
still work:

  >>> write(sample_buildout, 'damagednc', 'slapos', 'libnetworkcache.py',
  ... """
  ... class NetworkcacheClient(object):
  ...     def parseUrl(*args, **kwargs):
  ...         raise Die
  ...     def __init__(*args, **kwargs):
  ...         raise Die
  ...     def upload(*args, **kwargs):
  ...         raise Die
  ...     def download(*args, **kwargs):
  ...         raise Die
  ...     def select(*args, **kwargs):
  ...         raise Die
  ... class UploadError: pass
  ... class DirectoryNotFound: pass
  ... """)

  >>> write(sample_buildout, 'buildout.cfg',
  ... '''
  ... [buildout]
  ...
  ... networkcache-section = networkcache
  ... develop = download damagednc
  ... parts = download
  ...
  ... [download]
  ... recipe = download
  ... url = %(remote_server_url)shello.txt
  ...
  ... [networkcache]
  ... download-cache-url = %(nc_url)sshacache
  ... download-dir-url = %(nc_url)sshadir
  ... upload-cache-url = %(nc_url)sshacache
  ... upload-dir-url = %(nc_url)sshadir
  ... ''' % globals())
  >>> print system(buildout)
  Networkcache enabled.
  Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
  Networkcache upload cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
  Develop: '/sample-buildout/download'
  Develop: '/sample-buildout/damagednc'
  Updating download.
  Downloading http://localhost/hello.txt
  There was problem while calling method 'download_network_cached':
  Traceback (most recent call last):
  ...
      raise Die
  NameError: global name 'Die' is not defined
  <BLANKLINE>
  Uploading http://localhost/hello.txt into network cache.
  There was problem while calling method 'upload_network_cached':
  Traceback (most recent call last):
  ...
      raise Die
  NameError: global name 'Die' is not defined
  <BLANKLINE>
  download: Downloaded http://localhost:20863/hello.txt
