www.cendio.com
Bug 5569 - implement a better repo sync handling for the build system
: implement a better repo sync handling for the build system
Status: CLOSED FIXED
: ThinLinc
Build system
: pre-1.0
: PC Unknown
: P2 Normal
: 4.5.0
Assigned To:
:
:
:
:
  Show dependency treegraph
 
Reported: 2015-06-12 14:09 by
Modified: 2015-09-25 13:54 (History)


Attachments


Note

You need to log in before you can comment on or make changes to this bug.


Description From cendio 2015-06-12 14:09:08
The current dumb rsync to and from our local cenbuild working copies and the
repo server is very simple and easy to understand. Unfortunately the simplicity
comes with some drawbacks:

 - You have to have the entire repo in every working copy, even if only parts
of it is needed.

 - Removing anything is a complete nightmare as you have to clean out every
working copy or the files will just get synced back.

We can probably come up with something better.
------- Comment #1 From cendio 2015-06-12 14:14:08 -------
The current plan is to teach rebuild about the repo. Instead of building new
RPMs into the same place as old ones from the repo, we'll instead split things
into two directories:

 - repo files (i.e. old and already live things)
 - staging (i.e. the new ones you are currently building an noone has seen yet)

We can also avoid downloading the entire repo by just having a cache directory
and download files on demand.

So the data flow will now be:

 <spec> => [build] => staging => [rsync] => repo => [download] => cache

Files will be removed from staging as they are pushed to the repo. This ensures
that a file removed from the repo will not be brought back from the dead by
someones stray working copy.
------- Comment #2 From cendio 2015-06-18 10:26:14 -------
Implemented in r30411.

Tester should verify:

 - That packages are downloaded to the cache and installed when needed
 - That packages are installed from the staging directory
 - That uploading from the staging directory works
 - That --no-repo avoids any contact with the repo
------- Comment #3 From cendio 2015-06-22 16:16:03 -------
Failed to run rebuild a second time:

[hean01@hean01-desktop repo]$ ./rebuild
Reading spec files... done.
Downloading cendio-build-filesystem-armhf (1-1)
Failed to download file from repo. Error code 23.
rsync: chgrp
"/home/hean01/Development/cenbuild/repo/cache/.cendio-build-filesystem-armhf-1-1.noarch.rpm.VYpFr2"
failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code
23) at main.c(1637) [generator=3.1.0]
------- Comment #4 From cendio 2015-06-22 16:18:23 -------
Fixed in r30479.
------- Comment #5 From cendio 2015-06-23 07:45:20 -------
(In reply to comment #4)
> Fixed in r30479.

Works great.
------- Comment #6 From cendio 2015-06-23 14:32:51 -------
(In reply to comment #2)
> Implemented in r30411.
> 
>  - That packages are downloaded to the cache and installed when needed
>  - That packages are installed from the staging directory
>  - That uploading from the staging directory works
>  - That --no-repo avoids any contact with the repo

Verified each point above. Works as intended.