Linaro-android

linaro-android@lists.linaro.org

1072 discussions

[IT Services] Update: Re: Daily builds are failing during repo init

by Paul Sokolovsky

##- Please type your reply above this line -## [IT Services] Update: Re: Daily builds are failing during repo init You are registered as a CC on this support request (linaroithelp.zendesk.com/requests/109). Reply to this email to add a comment to the request. ---------------------------------------------- Paul Sokolovsky, May 20 09:35 (ADT) Fruther tries: I commented "RewriteMap prg:" completely, just in case if it applies to each request by error. Didn't help. I then tried to just wget -r http://android.git.linaro.org/git-http/platform/manifest.git to see how long that will take, and what I found is that there's http://android.git.linaro.org/git-http/platform/manifest.git/objects/pack/p… which is 660MB. Of course, if that pack is download with normal git clone, it got to be slow. And if it isn't, then why such a big pack lies in repo dir at all? All in all, I didn't have much progress investigating this, so I revert apache config to what it was, suggest Android people to switch to git:// for now, and this bug is stays in queue, waiting for peorper investigation and solution. ---------------------------------------------- Paul Sokolovsky, May 20 09:06 (ADT) Ok, I did to things: 1) temporarily disabled RewriteLock to see if it will help performance - it did not, took 6 min before it failed. 2) Urls with "/git-http/" are not subject to rewriting, and it still both slow and gives errors: $ time git clone http://android.git.linaro.org/git-http/platform/manifest.git Cloning into 'manifest'... error: Unable to get pack index http://android.git.linaro.org/git-http/platform/manifest.git/objects/pack/p… error: Unable to get pack index http://android.git.linaro.org/git-http/platform/manifest.git/objects/pack/p… ^C real 7m18.513s ---------------------------------------------- Paul Sokolovsky, May 20 08:37 (ADT) Ben: 1) What could be explanation of need for lock - that script doesn't do any context-sensitive processing. 2) We should try to get rid of it at all ---------------------------------------------- Paul Sokolovsky, May 20 08:30 (ADT) Vishal: this ticket is for issue reported by Jon, and other http access hiccups as confirmed by me. "branch 'stable' has not been signed" is separate and tracked at https://bugs.launchpad.net/linaro-android-infrastructure/+bug/1181907 ---------------------------------------------- Ben Copeland, May 20 08:22 (ADT) Hi Paul, I'm not too sure what I can do to speed up the http process. The bottleneck is down to the python script, We have had to put add "RewriteLock" to Apache otherwise users are getting these clone errors. The downside of this is it means it is preventing parallel requests. When I start to clone http://android.git.linaro.org/git-ro/platform/manifest.git, watching apache log there are many 404 links "GET /git-ro/platform/manifest.git/objects/75/0313d9ddb35959103d75ff8c450f4d7b6596d9 HTTP/1.1" 404 506 "-" "git/1.8.1.2" at the start. These do stop, but its going process each http request per file, its going to be a lengthy wait along with this process lock. Ben ---------------------------------------------- Vishal Bhoj, May 20 07:59 (ADT) I still face the same issue which happens on android-build with this command: repo init -u http://android.git.linaro.org/git-ro/platform/manifest.git -b linaro_android_4.2.2 -m default.xml --repo-url=http://android.git.linaro.org/git-ro/tools/repo Get http://android.git.linaro.org/git-ro/tools/repo/clone.bundle fatal: branch 'stable' has not been signed ---------------------------------------------- Paul Sokolovsky, May 20 07:54 (ADT) Reprod info for the issue I see: $ time git clone git://android.git.linaro.org/platform/manifest.git [] real 0m7.706s $ time git clone http://android.git.linaro.org/git-ro/platform/manifest.git [] real 14m17.214s ---------------------------------------------- Jon Medhurst (Tixy), May 20 07:53 (ADT) I'm getting no problems now, thanks. I had already synced up my local system using the git protocol, so I restored my Android directory from my Friday backup and did a repo sync. Dozens of git directories were updated by http:// URLs without any issues. ---------------------------------------------- Ben Copeland, May 20 07:34 (ADT) Hello Jon, We had some issues with the git server regarding the git-ro directory. I have made a similar change to the apache config which hopefully is linked to your problem. Can you try again and let us know if you are still getting problems? Thanks Ben ---------------------------------------------- Paul Sokolovsky, May 20 07:34 (ADT) All in all, I can confirm that there're issues with http:// access on android.git.linaro.org after migration - plain repo init takes very long, and I see errors like: error: Unable to get pack index http://android.git.linaro.org/git-ro/platform/manifest.git/objects/pack/pac… error: Unable to get pack index http://android.git.linaro.org/git-ro/platform/manifest.git/objects/pack/pac… error: Unable to get pack index http://android.git.linaro.org/git-ro/platform/manifest.git/objects/pack/pac… (Those are non-deterministic, re-trying them manually works.) ---------------------------------------------- Jon Medhurst (Tixy), May 20 07:10 (ADT) On Mon, 2013-05-20 at 12:43 +0300, Paul Sokolovsky wrote: > Hello Jon, > > On Mon, 20 May 2013 09:35:00 +0100 > "Jon Medhurst (Tixy)" <tixy(a)linaro.org> wrote: > > > On Mon, 2013-05-20 at 09:15 +0530, Vishal Bhoj wrote: > > > Android builds are failing in repo init phase. I have opened a bug > > > for the same: > > > https://bugs.launchpad.net/bugs/1181907 > > > > > > repo init and sync worked for me locally. Can someone please have > > > a look. > > > > I get several errors like this, with various repos... > > > > error: Unable to find 51bd0a1525c7a722ad64a8954d90c6df0b8546a7 > > under http://android.git.linaro.org/git-ro/platform/ndk > > Cannot obtain needed blob > > 51bd0a1525c7a722ad64a8954d90c6df0b8546a7 > > while processing commit > > 7b3bda5d569a69c4f9b87225c5d21da4a41bf12e. > > error: Fetch failed. > > This is different issue from one report by Vishal. Oops, yes. I didn't read the bug, sorry. I jumped to conclusions after seeing Vishal and Bero's emails appeared at about the same time. > > > > > As I said in reply to bero's mail, I always seem to get these since > > the move to http URLs > > Move to HTTP happened last couple of days during LCA13 in Hong Kong, do > you want to say that you experienced the error as quoted above since > then? Was it reported to some bug/ticket? (Do you have a reference?) I didn't raise a bug as I had a workaround and I assumed if it was a widespread problem then it would get sorted out eventually. > The situation known to me is: > > 1. After flipping the switch in HK, git:// became unstable (because it > was throttled to stop wasting resources as it always did). It took some > time to migrate everyone to http://. If my memory serves me right, it wasn't long after this I noticed my problems, which is why it occurred to me to change the URLs in the manifests back to the old git:// (They'll get restored to http when the manifests are changed, which is why I notice it as a continuing problem.) I perhaps only do a repo sync a couple of times a month as I don't really work much on Android, which is perhaps why a) I see this problem? b) I've not been motivated to get to the bottom of it. -- Tixy ---------------------------------------------- Paul Sokolovsky, May 20 06:44 (ADT) Hello Jon, On Mon, 20 May 2013 09:35:00 +0100 "Jon Medhurst (Tixy)" <tixy(a)linaro.org> wrote: > On Mon, 2013-05-20 at 09:15 +0530, Vishal Bhoj wrote: > > Android builds are failing in repo init phase. I have opened a bug > > for the same: > > https://bugs.launchpad.net/bugs/1181907 > > > > repo init and sync worked for me locally. Can someone please have > > a look. > > I get several errors like this, with various repos... > > error: Unable to find 51bd0a1525c7a722ad64a8954d90c6df0b8546a7 > under http://android.git.linaro.org/git-ro/platform/ndk > Cannot obtain needed blob > 51bd0a1525c7a722ad64a8954d90c6df0b8546a7 > while processing commit > 7b3bda5d569a69c4f9b87225c5d21da4a41bf12e. > error: Fetch failed. This is different issue from one report by Vishal. > > As I said in reply to bero's mail, I always seem to get these since > the move to http URLs Move to HTTP happened last couple of days during LCA13 in Hong Kong, do you want to say that you experienced the error as quoted above since then? Was it reported to some bug/ticket? (Do you have a reference?) The situation known to me is: 1. After flipping the switch in HK, git:// became unstable (because it was throttled to stop wasting resources as it always did). It took some time to migrate everyone to http://. 2. http:// was stable since then, until migration of git.linaro.org happened weekened May 11. 3. Immediately after migration, there were issues with http://git.linaro.org which manifested pretty much like the error above. It was resolved Mon/Tue last week. 4. Friday GMT evening, android.git.linaro.org was migrated to new setup. Initial builds seem to have gone ok. Detailed status is not known (your telling that error above was received after May 17, 17:00GMT would help). 5. Additionally, today issue reported by Vishal popped up. All in all, git issues now should be cc: to its(a)linaro.org (added), ITS guys are now the owners of new infrastructure, and can resolve issues pretty well (p.3 was resolved by them). > and I work around it by changing the manifest > to have the old git URLs. This may work as a personal workaround, but won't scale to official manifests - we need to get http:// working right. -- Best Regards, Paul Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog -------------------------------- This email is a service from IT Services

11 years, 11 months

[IT Services] Update: Re: Daily builds are failing during repo init

by Paul Sokolovsky

##- Please type your reply above this line -## [IT Services] Update: Re: Daily builds are failing during repo init You are registered as a CC on this support request (linaroithelp.zendesk.com/requests/109). Reply to this email to add a comment to the request. ---------------------------------------------- Paul Sokolovsky, May 20 09:06 (ADT) Ok, I did to things: 1) temporarily disabled RewriteLock to see if it will help performance - it did not, took 6 min before it failed. 2) Urls with "/git-http/" are not subject to rewriting, and it still both slow and gives errors: $ time git clone http://android.git.linaro.org/git-http/platform/manifest.git Cloning into 'manifest'... error: Unable to get pack index http://android.git.linaro.org/git-http/platform/manifest.git/objects/pack/p… error: Unable to get pack index http://android.git.linaro.org/git-http/platform/manifest.git/objects/pack/p… ^C real 7m18.513s ---------------------------------------------- Paul Sokolovsky, May 20 08:37 (ADT) Ben: 1) What could be explanation of need for lock - that script doesn't do any context-sensitive processing. 2) We should try to get rid of it at all ---------------------------------------------- Paul Sokolovsky, May 20 08:30 (ADT) Vishal: this ticket is for issue reported by Jon, and other http access hiccups as confirmed by me. "branch 'stable' has not been signed" is separate and tracked at https://bugs.launchpad.net/linaro-android-infrastructure/+bug/1181907 ---------------------------------------------- Ben Copeland, May 20 08:22 (ADT) Hi Paul, I'm not too sure what I can do to speed up the http process. The bottleneck is down to the python script, We have had to put add "RewriteLock" to Apache otherwise users are getting these clone errors. The downside of this is it means it is preventing parallel requests. When I start to clone http://android.git.linaro.org/git-ro/platform/manifest.git, watching apache log there are many 404 links "GET /git-ro/platform/manifest.git/objects/75/0313d9ddb35959103d75ff8c450f4d7b6596d9 HTTP/1.1" 404 506 "-" "git/1.8.1.2" at the start. These do stop, but its going process each http request per file, its going to be a lengthy wait along with this process lock. Ben ---------------------------------------------- Vishal Bhoj, May 20 07:59 (ADT) I still face the same issue which happens on android-build with this command: repo init -u http://android.git.linaro.org/git-ro/platform/manifest.git -b linaro_android_4.2.2 -m default.xml --repo-url=http://android.git.linaro.org/git-ro/tools/repo Get http://android.git.linaro.org/git-ro/tools/repo/clone.bundle fatal: branch 'stable' has not been signed ---------------------------------------------- Paul Sokolovsky, May 20 07:54 (ADT) Reprod info for the issue I see: $ time git clone git://android.git.linaro.org/platform/manifest.git [] real 0m7.706s $ time git clone http://android.git.linaro.org/git-ro/platform/manifest.git [] real 14m17.214s ---------------------------------------------- Jon Medhurst (Tixy), May 20 07:53 (ADT) I'm getting no problems now, thanks. I had already synced up my local system using the git protocol, so I restored my Android directory from my Friday backup and did a repo sync. Dozens of git directories were updated by http:// URLs without any issues. ---------------------------------------------- Ben Copeland, May 20 07:34 (ADT) Hello Jon, We had some issues with the git server regarding the git-ro directory. I have made a similar change to the apache config which hopefully is linked to your problem. Can you try again and let us know if you are still getting problems? Thanks Ben ---------------------------------------------- Paul Sokolovsky, May 20 07:34 (ADT) All in all, I can confirm that there're issues with http:// access on android.git.linaro.org after migration - plain repo init takes very long, and I see errors like: error: Unable to get pack index http://android.git.linaro.org/git-ro/platform/manifest.git/objects/pack/pac… error: Unable to get pack index http://android.git.linaro.org/git-ro/platform/manifest.git/objects/pack/pac… error: Unable to get pack index http://android.git.linaro.org/git-ro/platform/manifest.git/objects/pack/pac… (Those are non-deterministic, re-trying them manually works.) ---------------------------------------------- Jon Medhurst (Tixy), May 20 07:10 (ADT) On Mon, 2013-05-20 at 12:43 +0300, Paul Sokolovsky wrote: > Hello Jon, > > On Mon, 20 May 2013 09:35:00 +0100 > "Jon Medhurst (Tixy)" <tixy(a)linaro.org> wrote: > > > On Mon, 2013-05-20 at 09:15 +0530, Vishal Bhoj wrote: > > > Android builds are failing in repo init phase. I have opened a bug > > > for the same: > > > https://bugs.launchpad.net/bugs/1181907 > > > > > > repo init and sync worked for me locally. Can someone please have > > > a look. > > > > I get several errors like this, with various repos... > > > > error: Unable to find 51bd0a1525c7a722ad64a8954d90c6df0b8546a7 > > under http://android.git.linaro.org/git-ro/platform/ndk > > Cannot obtain needed blob > > 51bd0a1525c7a722ad64a8954d90c6df0b8546a7 > > while processing commit > > 7b3bda5d569a69c4f9b87225c5d21da4a41bf12e. > > error: Fetch failed. > > This is different issue from one report by Vishal. Oops, yes. I didn't read the bug, sorry. I jumped to conclusions after seeing Vishal and Bero's emails appeared at about the same time. > > > > > As I said in reply to bero's mail, I always seem to get these since > > the move to http URLs > > Move to HTTP happened last couple of days during LCA13 in Hong Kong, do > you want to say that you experienced the error as quoted above since > then? Was it reported to some bug/ticket? (Do you have a reference?) I didn't raise a bug as I had a workaround and I assumed if it was a widespread problem then it would get sorted out eventually. > The situation known to me is: > > 1. After flipping the switch in HK, git:// became unstable (because it > was throttled to stop wasting resources as it always did). It took some > time to migrate everyone to http://. If my memory serves me right, it wasn't long after this I noticed my problems, which is why it occurred to me to change the URLs in the manifests back to the old git:// (They'll get restored to http when the manifests are changed, which is why I notice it as a continuing problem.) I perhaps only do a repo sync a couple of times a month as I don't really work much on Android, which is perhaps why a) I see this problem? b) I've not been motivated to get to the bottom of it. -- Tixy ---------------------------------------------- Paul Sokolovsky, May 20 06:44 (ADT) Hello Jon, On Mon, 20 May 2013 09:35:00 +0100 "Jon Medhurst (Tixy)" <tixy(a)linaro.org> wrote: > On Mon, 2013-05-20 at 09:15 +0530, Vishal Bhoj wrote: > > Android builds are failing in repo init phase. I have opened a bug > > for the same: > > https://bugs.launchpad.net/bugs/1181907 > > > > repo init and sync worked for me locally. Can someone please have > > a look. > > I get several errors like this, with various repos... > > error: Unable to find 51bd0a1525c7a722ad64a8954d90c6df0b8546a7 > under http://android.git.linaro.org/git-ro/platform/ndk > Cannot obtain needed blob > 51bd0a1525c7a722ad64a8954d90c6df0b8546a7 > while processing commit > 7b3bda5d569a69c4f9b87225c5d21da4a41bf12e. > error: Fetch failed. This is different issue from one report by Vishal. > > As I said in reply to bero's mail, I always seem to get these since > the move to http URLs Move to HTTP happened last couple of days during LCA13 in Hong Kong, do you want to say that you experienced the error as quoted above since then? Was it reported to some bug/ticket? (Do you have a reference?) The situation known to me is: 1. After flipping the switch in HK, git:// became unstable (because it was throttled to stop wasting resources as it always did). It took some time to migrate everyone to http://. 2. http:// was stable since then, until migration of git.linaro.org happened weekened May 11. 3. Immediately after migration, there were issues with http://git.linaro.org which manifested pretty much like the error above. It was resolved Mon/Tue last week. 4. Friday GMT evening, android.git.linaro.org was migrated to new setup. Initial builds seem to have gone ok. Detailed status is not known (your telling that error above was received after May 17, 17:00GMT would help). 5. Additionally, today issue reported by Vishal popped up. All in all, git issues now should be cc: to its(a)linaro.org (added), ITS guys are now the owners of new infrastructure, and can resolve issues pretty well (p.3 was resolved by them). > and I work around it by changing the manifest > to have the old git URLs. This may work as a personal workaround, but won't scale to official manifests - we need to get http:// working right. -- Best Regards, Paul Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog -------------------------------- This email is a service from IT Services

11 years, 11 months

[IT Services] Update: Re: Daily builds are failing during repo init

by Paul Sokolovsky

##- Please type your reply above this line -## [IT Services] Update: Re: Daily builds are failing during repo init You are registered as a CC on this support request (linaroithelp.zendesk.com/requests/109). Reply to this email to add a comment to the request. ---------------------------------------------- Paul Sokolovsky, May 20 08:37 (ADT) Ben: 1) What could be explanation of need for lock - that script doesn't do any context-sensitive processing. 2) We should try to get rid of it at all ---------------------------------------------- Paul Sokolovsky, May 20 08:30 (ADT) Vishal: this ticket is for issue reported by Jon, and other http access hiccups as confirmed by me. "branch 'stable' has not been signed" is separate and tracked at https://bugs.launchpad.net/linaro-android-infrastructure/+bug/1181907 ---------------------------------------------- Ben Copeland, May 20 08:22 (ADT) Hi Paul, I'm not too sure what I can do to speed up the http process. The bottleneck is down to the python script, We have had to put add "RewriteLock" to Apache otherwise users are getting these clone errors. The downside of this is it means it is preventing parallel requests. When I start to clone http://android.git.linaro.org/git-ro/platform/manifest.git, watching apache log there are many 404 links "GET /git-ro/platform/manifest.git/objects/75/0313d9ddb35959103d75ff8c450f4d7b6596d9 HTTP/1.1" 404 506 "-" "git/1.8.1.2" at the start. These do stop, but its going process each http request per file, its going to be a lengthy wait along with this process lock. Ben ---------------------------------------------- Vishal Bhoj, May 20 07:59 (ADT) I still face the same issue which happens on android-build with this command: repo init -u http://android.git.linaro.org/git-ro/platform/manifest.git -b linaro_android_4.2.2 -m default.xml --repo-url=http://android.git.linaro.org/git-ro/tools/repo Get http://android.git.linaro.org/git-ro/tools/repo/clone.bundle fatal: branch 'stable' has not been signed ---------------------------------------------- Paul Sokolovsky, May 20 07:54 (ADT) Reprod info for the issue I see: $ time git clone git://android.git.linaro.org/platform/manifest.git [] real 0m7.706s $ time git clone http://android.git.linaro.org/git-ro/platform/manifest.git [] real 14m17.214s ---------------------------------------------- Jon Medhurst (Tixy), May 20 07:53 (ADT) I'm getting no problems now, thanks. I had already synced up my local system using the git protocol, so I restored my Android directory from my Friday backup and did a repo sync. Dozens of git directories were updated by http:// URLs without any issues. ---------------------------------------------- Ben Copeland, May 20 07:34 (ADT) Hello Jon, We had some issues with the git server regarding the git-ro directory. I have made a similar change to the apache config which hopefully is linked to your problem. Can you try again and let us know if you are still getting problems? Thanks Ben ---------------------------------------------- Paul Sokolovsky, May 20 07:34 (ADT) All in all, I can confirm that there're issues with http:// access on android.git.linaro.org after migration - plain repo init takes very long, and I see errors like: error: Unable to get pack index http://android.git.linaro.org/git-ro/platform/manifest.git/objects/pack/pac… error: Unable to get pack index http://android.git.linaro.org/git-ro/platform/manifest.git/objects/pack/pac… error: Unable to get pack index http://android.git.linaro.org/git-ro/platform/manifest.git/objects/pack/pac… (Those are non-deterministic, re-trying them manually works.) ---------------------------------------------- Jon Medhurst (Tixy), May 20 07:10 (ADT) On Mon, 2013-05-20 at 12:43 +0300, Paul Sokolovsky wrote: > Hello Jon, > > On Mon, 20 May 2013 09:35:00 +0100 > "Jon Medhurst (Tixy)" <tixy(a)linaro.org> wrote: > > > On Mon, 2013-05-20 at 09:15 +0530, Vishal Bhoj wrote: > > > Android builds are failing in repo init phase. I have opened a bug > > > for the same: > > > https://bugs.launchpad.net/bugs/1181907 > > > > > > repo init and sync worked for me locally. Can someone please have > > > a look. > > > > I get several errors like this, with various repos... > > > > error: Unable to find 51bd0a1525c7a722ad64a8954d90c6df0b8546a7 > > under http://android.git.linaro.org/git-ro/platform/ndk > > Cannot obtain needed blob > > 51bd0a1525c7a722ad64a8954d90c6df0b8546a7 > > while processing commit > > 7b3bda5d569a69c4f9b87225c5d21da4a41bf12e. > > error: Fetch failed. > > This is different issue from one report by Vishal. Oops, yes. I didn't read the bug, sorry. I jumped to conclusions after seeing Vishal and Bero's emails appeared at about the same time. > > > > > As I said in reply to bero's mail, I always seem to get these since > > the move to http URLs > > Move to HTTP happened last couple of days during LCA13 in Hong Kong, do > you want to say that you experienced the error as quoted above since > then? Was it reported to some bug/ticket? (Do you have a reference?) I didn't raise a bug as I had a workaround and I assumed if it was a widespread problem then it would get sorted out eventually. > The situation known to me is: > > 1. After flipping the switch in HK, git:// became unstable (because it > was throttled to stop wasting resources as it always did). It took some > time to migrate everyone to http://. If my memory serves me right, it wasn't long after this I noticed my problems, which is why it occurred to me to change the URLs in the manifests back to the old git:// (They'll get restored to http when the manifests are changed, which is why I notice it as a continuing problem.) I perhaps only do a repo sync a couple of times a month as I don't really work much on Android, which is perhaps why a) I see this problem? b) I've not been motivated to get to the bottom of it. -- Tixy ---------------------------------------------- Paul Sokolovsky, May 20 06:44 (ADT) Hello Jon, On Mon, 20 May 2013 09:35:00 +0100 "Jon Medhurst (Tixy)" <tixy(a)linaro.org> wrote: > On Mon, 2013-05-20 at 09:15 +0530, Vishal Bhoj wrote: > > Android builds are failing in repo init phase. I have opened a bug > > for the same: > > https://bugs.launchpad.net/bugs/1181907 > > > > repo init and sync worked for me locally. Can someone please have > > a look. > > I get several errors like this, with various repos... > > error: Unable to find 51bd0a1525c7a722ad64a8954d90c6df0b8546a7 > under http://android.git.linaro.org/git-ro/platform/ndk > Cannot obtain needed blob > 51bd0a1525c7a722ad64a8954d90c6df0b8546a7 > while processing commit > 7b3bda5d569a69c4f9b87225c5d21da4a41bf12e. > error: Fetch failed. This is different issue from one report by Vishal. > > As I said in reply to bero's mail, I always seem to get these since > the move to http URLs Move to HTTP happened last couple of days during LCA13 in Hong Kong, do you want to say that you experienced the error as quoted above since then? Was it reported to some bug/ticket? (Do you have a reference?) The situation known to me is: 1. After flipping the switch in HK, git:// became unstable (because it was throttled to stop wasting resources as it always did). It took some time to migrate everyone to http://. 2. http:// was stable since then, until migration of git.linaro.org happened weekened May 11. 3. Immediately after migration, there were issues with http://git.linaro.org which manifested pretty much like the error above. It was resolved Mon/Tue last week. 4. Friday GMT evening, android.git.linaro.org was migrated to new setup. Initial builds seem to have gone ok. Detailed status is not known (your telling that error above was received after May 17, 17:00GMT would help). 5. Additionally, today issue reported by Vishal popped up. All in all, git issues now should be cc: to its(a)linaro.org (added), ITS guys are now the owners of new infrastructure, and can resolve issues pretty well (p.3 was resolved by them). > and I work around it by changing the manifest > to have the old git URLs. This may work as a personal workaround, but won't scale to official manifests - we need to get http:// working right. -- Best Regards, Paul Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog -------------------------------- This email is a service from IT Services

11 years, 11 months

[IT Services] Update: Re: Daily builds are failing during repo init

by Paul Sokolovsky

##- Please type your reply above this line -## [IT Services] Update: Re: Daily builds are failing during repo init You are registered as a CC on this support request (linaroithelp.zendesk.com/requests/109). Reply to this email to add a comment to the request. ---------------------------------------------- Paul Sokolovsky, May 20 08:30 (ADT) Vishal: this ticket is for issue reported by Jon, and other http access hiccups as confirmed by me. "branch 'stable' has not been signed" is separate and tracked at https://bugs.launchpad.net/linaro-android-infrastructure/+bug/1181907 ---------------------------------------------- Ben Copeland, May 20 08:22 (ADT) Hi Paul, I'm not too sure what I can do to speed up the http process. The bottleneck is down to the python script, We have had to put add "RewriteLock" to Apache otherwise users are getting these clone errors. The downside of this is it means it is preventing parallel requests. When I start to clone http://android.git.linaro.org/git-ro/platform/manifest.git, watching apache log there are many 404 links "GET /git-ro/platform/manifest.git/objects/75/0313d9ddb35959103d75ff8c450f4d7b6596d9 HTTP/1.1" 404 506 "-" "git/1.8.1.2" at the start. These do stop, but its going process each http request per file, its going to be a lengthy wait along with this process lock. Ben ---------------------------------------------- Vishal Bhoj, May 20 07:59 (ADT) I still face the same issue which happens on android-build with this command: repo init -u http://android.git.linaro.org/git-ro/platform/manifest.git -b linaro_android_4.2.2 -m default.xml --repo-url=http://android.git.linaro.org/git-ro/tools/repo Get http://android.git.linaro.org/git-ro/tools/repo/clone.bundle fatal: branch 'stable' has not been signed ---------------------------------------------- Paul Sokolovsky, May 20 07:54 (ADT) Reprod info for the issue I see: $ time git clone git://android.git.linaro.org/platform/manifest.git [] real 0m7.706s $ time git clone http://android.git.linaro.org/git-ro/platform/manifest.git [] real 14m17.214s ---------------------------------------------- Jon Medhurst (Tixy), May 20 07:53 (ADT) I'm getting no problems now, thanks. I had already synced up my local system using the git protocol, so I restored my Android directory from my Friday backup and did a repo sync. Dozens of git directories were updated by http:// URLs without any issues. ---------------------------------------------- Ben Copeland, May 20 07:34 (ADT) Hello Jon, We had some issues with the git server regarding the git-ro directory. I have made a similar change to the apache config which hopefully is linked to your problem. Can you try again and let us know if you are still getting problems? Thanks Ben ---------------------------------------------- Paul Sokolovsky, May 20 07:34 (ADT) All in all, I can confirm that there're issues with http:// access on android.git.linaro.org after migration - plain repo init takes very long, and I see errors like: error: Unable to get pack index http://android.git.linaro.org/git-ro/platform/manifest.git/objects/pack/pac… error: Unable to get pack index http://android.git.linaro.org/git-ro/platform/manifest.git/objects/pack/pac… error: Unable to get pack index http://android.git.linaro.org/git-ro/platform/manifest.git/objects/pack/pac… (Those are non-deterministic, re-trying them manually works.) ---------------------------------------------- Jon Medhurst (Tixy), May 20 07:10 (ADT) On Mon, 2013-05-20 at 12:43 +0300, Paul Sokolovsky wrote: > Hello Jon, > > On Mon, 20 May 2013 09:35:00 +0100 > "Jon Medhurst (Tixy)" <tixy(a)linaro.org> wrote: > > > On Mon, 2013-05-20 at 09:15 +0530, Vishal Bhoj wrote: > > > Android builds are failing in repo init phase. I have opened a bug > > > for the same: > > > https://bugs.launchpad.net/bugs/1181907 > > > > > > repo init and sync worked for me locally. Can someone please have > > > a look. > > > > I get several errors like this, with various repos... > > > > error: Unable to find 51bd0a1525c7a722ad64a8954d90c6df0b8546a7 > > under http://android.git.linaro.org/git-ro/platform/ndk > > Cannot obtain needed blob > > 51bd0a1525c7a722ad64a8954d90c6df0b8546a7 > > while processing commit > > 7b3bda5d569a69c4f9b87225c5d21da4a41bf12e. > > error: Fetch failed. > > This is different issue from one report by Vishal. Oops, yes. I didn't read the bug, sorry. I jumped to conclusions after seeing Vishal and Bero's emails appeared at about the same time. > > > > > As I said in reply to bero's mail, I always seem to get these since > > the move to http URLs > > Move to HTTP happened last couple of days during LCA13 in Hong Kong, do > you want to say that you experienced the error as quoted above since > then? Was it reported to some bug/ticket? (Do you have a reference?) I didn't raise a bug as I had a workaround and I assumed if it was a widespread problem then it would get sorted out eventually. > The situation known to me is: > > 1. After flipping the switch in HK, git:// became unstable (because it > was throttled to stop wasting resources as it always did). It took some > time to migrate everyone to http://. If my memory serves me right, it wasn't long after this I noticed my problems, which is why it occurred to me to change the URLs in the manifests back to the old git:// (They'll get restored to http when the manifests are changed, which is why I notice it as a continuing problem.) I perhaps only do a repo sync a couple of times a month as I don't really work much on Android, which is perhaps why a) I see this problem? b) I've not been motivated to get to the bottom of it. -- Tixy ---------------------------------------------- Paul Sokolovsky, May 20 06:44 (ADT) Hello Jon, On Mon, 20 May 2013 09:35:00 +0100 "Jon Medhurst (Tixy)" <tixy(a)linaro.org> wrote: > On Mon, 2013-05-20 at 09:15 +0530, Vishal Bhoj wrote: > > Android builds are failing in repo init phase. I have opened a bug > > for the same: > > https://bugs.launchpad.net/bugs/1181907 > > > > repo init and sync worked for me locally. Can someone please have > > a look. > > I get several errors like this, with various repos... > > error: Unable to find 51bd0a1525c7a722ad64a8954d90c6df0b8546a7 > under http://android.git.linaro.org/git-ro/platform/ndk > Cannot obtain needed blob > 51bd0a1525c7a722ad64a8954d90c6df0b8546a7 > while processing commit > 7b3bda5d569a69c4f9b87225c5d21da4a41bf12e. > error: Fetch failed. This is different issue from one report by Vishal. > > As I said in reply to bero's mail, I always seem to get these since > the move to http URLs Move to HTTP happened last couple of days during LCA13 in Hong Kong, do you want to say that you experienced the error as quoted above since then? Was it reported to some bug/ticket? (Do you have a reference?) The situation known to me is: 1. After flipping the switch in HK, git:// became unstable (because it was throttled to stop wasting resources as it always did). It took some time to migrate everyone to http://. 2. http:// was stable since then, until migration of git.linaro.org happened weekened May 11. 3. Immediately after migration, there were issues with http://git.linaro.org which manifested pretty much like the error above. It was resolved Mon/Tue last week. 4. Friday GMT evening, android.git.linaro.org was migrated to new setup. Initial builds seem to have gone ok. Detailed status is not known (your telling that error above was received after May 17, 17:00GMT would help). 5. Additionally, today issue reported by Vishal popped up. All in all, git issues now should be cc: to its(a)linaro.org (added), ITS guys are now the owners of new infrastructure, and can resolve issues pretty well (p.3 was resolved by them). > and I work around it by changing the manifest > to have the old git URLs. This may work as a personal workaround, but won't scale to official manifests - we need to get http:// working right. -- Best Regards, Paul Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog -------------------------------- This email is a service from IT Services

11 years, 11 months

[IT Services] Update: Re: Daily builds are failing during repo init

by Ben Copeland

##- Please type your reply above this line -## [IT Services] Update: Re: Daily builds are failing during repo init You are registered as a CC on this support request (linaroithelp.zendesk.com/requests/109). Reply to this email to add a comment to the request. ---------------------------------------------- Ben Copeland, May 20 08:22 (ADT) Hi Paul, I'm not too sure what I can do to speed up the http process. The bottleneck is down to the python script, We have had to put add "RewriteLock" to Apache otherwise users are getting these clone errors. The downside of this is it means it is preventing parallel requests. When I start to clone http://android.git.linaro.org/git-ro/platform/manifest.git, watching apache log there are many 404 links "GET /git-ro/platform/manifest.git/objects/75/0313d9ddb35959103d75ff8c450f4d7b6596d9 HTTP/1.1" 404 506 "-" "git/1.8.1.2" at the start. These do stop, but its going process each http request per file, its going to be a lengthy wait along with this process lock. Ben ---------------------------------------------- Vishal Bhoj, May 20 07:59 (ADT) I still face the same issue which happens on android-build with this command: repo init -u http://android.git.linaro.org/git-ro/platform/manifest.git -b linaro_android_4.2.2 -m default.xml --repo-url=http://android.git.linaro.org/git-ro/tools/repo Get http://android.git.linaro.org/git-ro/tools/repo/clone.bundle fatal: branch 'stable' has not been signed ---------------------------------------------- Paul Sokolovsky, May 20 07:54 (ADT) Reprod info for the issue I see: $ time git clone git://android.git.linaro.org/platform/manifest.git [] real 0m7.706s $ time git clone http://android.git.linaro.org/git-ro/platform/manifest.git [] real 14m17.214s ---------------------------------------------- Jon Medhurst (Tixy), May 20 07:53 (ADT) I'm getting no problems now, thanks. I had already synced up my local system using the git protocol, so I restored my Android directory from my Friday backup and did a repo sync. Dozens of git directories were updated by http:// URLs without any issues. ---------------------------------------------- Ben Copeland, May 20 07:34 (ADT) Hello Jon, We had some issues with the git server regarding the git-ro directory. I have made a similar change to the apache config which hopefully is linked to your problem. Can you try again and let us know if you are still getting problems? Thanks Ben ---------------------------------------------- Paul Sokolovsky, May 20 07:34 (ADT) All in all, I can confirm that there're issues with http:// access on android.git.linaro.org after migration - plain repo init takes very long, and I see errors like: error: Unable to get pack index http://android.git.linaro.org/git-ro/platform/manifest.git/objects/pack/pac… error: Unable to get pack index http://android.git.linaro.org/git-ro/platform/manifest.git/objects/pack/pac… error: Unable to get pack index http://android.git.linaro.org/git-ro/platform/manifest.git/objects/pack/pac… (Those are non-deterministic, re-trying them manually works.) ---------------------------------------------- Jon Medhurst (Tixy), May 20 07:10 (ADT) On Mon, 2013-05-20 at 12:43 +0300, Paul Sokolovsky wrote: > Hello Jon, > > On Mon, 20 May 2013 09:35:00 +0100 > "Jon Medhurst (Tixy)" <tixy(a)linaro.org> wrote: > > > On Mon, 2013-05-20 at 09:15 +0530, Vishal Bhoj wrote: > > > Android builds are failing in repo init phase. I have opened a bug > > > for the same: > > > https://bugs.launchpad.net/bugs/1181907 > > > > > > repo init and sync worked for me locally. Can someone please have > > > a look. > > > > I get several errors like this, with various repos... > > > > error: Unable to find 51bd0a1525c7a722ad64a8954d90c6df0b8546a7 > > under http://android.git.linaro.org/git-ro/platform/ndk > > Cannot obtain needed blob > > 51bd0a1525c7a722ad64a8954d90c6df0b8546a7 > > while processing commit > > 7b3bda5d569a69c4f9b87225c5d21da4a41bf12e. > > error: Fetch failed. > > This is different issue from one report by Vishal. Oops, yes. I didn't read the bug, sorry. I jumped to conclusions after seeing Vishal and Bero's emails appeared at about the same time. > > > > > As I said in reply to bero's mail, I always seem to get these since > > the move to http URLs > > Move to HTTP happened last couple of days during LCA13 in Hong Kong, do > you want to say that you experienced the error as quoted above since > then? Was it reported to some bug/ticket? (Do you have a reference?) I didn't raise a bug as I had a workaround and I assumed if it was a widespread problem then it would get sorted out eventually. > The situation known to me is: > > 1. After flipping the switch in HK, git:// became unstable (because it > was throttled to stop wasting resources as it always did). It took some > time to migrate everyone to http://. If my memory serves me right, it wasn't long after this I noticed my problems, which is why it occurred to me to change the URLs in the manifests back to the old git:// (They'll get restored to http when the manifests are changed, which is why I notice it as a continuing problem.) I perhaps only do a repo sync a couple of times a month as I don't really work much on Android, which is perhaps why a) I see this problem? b) I've not been motivated to get to the bottom of it. -- Tixy ---------------------------------------------- Paul Sokolovsky, May 20 06:44 (ADT) Hello Jon, On Mon, 20 May 2013 09:35:00 +0100 "Jon Medhurst (Tixy)" <tixy(a)linaro.org> wrote: > On Mon, 2013-05-20 at 09:15 +0530, Vishal Bhoj wrote: > > Android builds are failing in repo init phase. I have opened a bug > > for the same: > > https://bugs.launchpad.net/bugs/1181907 > > > > repo init and sync worked for me locally. Can someone please have > > a look. > > I get several errors like this, with various repos... > > error: Unable to find 51bd0a1525c7a722ad64a8954d90c6df0b8546a7 > under http://android.git.linaro.org/git-ro/platform/ndk > Cannot obtain needed blob > 51bd0a1525c7a722ad64a8954d90c6df0b8546a7 > while processing commit > 7b3bda5d569a69c4f9b87225c5d21da4a41bf12e. > error: Fetch failed. This is different issue from one report by Vishal. > > As I said in reply to bero's mail, I always seem to get these since > the move to http URLs Move to HTTP happened last couple of days during LCA13 in Hong Kong, do you want to say that you experienced the error as quoted above since then? Was it reported to some bug/ticket? (Do you have a reference?) The situation known to me is: 1. After flipping the switch in HK, git:// became unstable (because it was throttled to stop wasting resources as it always did). It took some time to migrate everyone to http://. 2. http:// was stable since then, until migration of git.linaro.org happened weekened May 11. 3. Immediately after migration, there were issues with http://git.linaro.org which manifested pretty much like the error above. It was resolved Mon/Tue last week. 4. Friday GMT evening, android.git.linaro.org was migrated to new setup. Initial builds seem to have gone ok. Detailed status is not known (your telling that error above was received after May 17, 17:00GMT would help). 5. Additionally, today issue reported by Vishal popped up. All in all, git issues now should be cc: to its(a)linaro.org (added), ITS guys are now the owners of new infrastructure, and can resolve issues pretty well (p.3 was resolved by them). > and I work around it by changing the manifest > to have the old git URLs. This may work as a personal workaround, but won't scale to official manifests - we need to get http:// working right. -- Best Regards, Paul Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog -------------------------------- This email is a service from IT Services

11 years, 11 months

[IT Services] Update: Re: Daily builds are failing during repo init

by Vishal Bhoj

##- Please type your reply above this line -## [IT Services] Update: Re: Daily builds are failing during repo init You are registered as a CC on this support request (linaroithelp.zendesk.com/requests/109). Reply to this email to add a comment to the request. ---------------------------------------------- Vishal Bhoj, May 20 07:59 (ADT) I still face the same issue which happens on android-build with this command: repo init -u http://android.git.linaro.org/git-ro/platform/manifest.git -b linaro_android_4.2.2 -m default.xml --repo-url=http://android.git.linaro.org/git-ro/tools/repo Get http://android.git.linaro.org/git-ro/tools/repo/clone.bundle fatal: branch 'stable' has not been signed ---------------------------------------------- Paul Sokolovsky, May 20 07:54 (ADT) Reprod info for the issue I see: $ time git clone git://android.git.linaro.org/platform/manifest.git [] real 0m7.706s $ time git clone http://android.git.linaro.org/git-ro/platform/manifest.git [] real 14m17.214s ---------------------------------------------- Jon Medhurst (Tixy), May 20 07:53 (ADT) I'm getting no problems now, thanks. I had already synced up my local system using the git protocol, so I restored my Android directory from my Friday backup and did a repo sync. Dozens of git directories were updated by http:// URLs without any issues. ---------------------------------------------- Ben Copeland, May 20 07:34 (ADT) Hello Jon, We had some issues with the git server regarding the git-ro directory. I have made a similar change to the apache config which hopefully is linked to your problem. Can you try again and let us know if you are still getting problems? Thanks Ben ---------------------------------------------- Paul Sokolovsky, May 20 07:34 (ADT) All in all, I can confirm that there're issues with http:// access on android.git.linaro.org after migration - plain repo init takes very long, and I see errors like: error: Unable to get pack index http://android.git.linaro.org/git-ro/platform/manifest.git/objects/pack/pac… error: Unable to get pack index http://android.git.linaro.org/git-ro/platform/manifest.git/objects/pack/pac… error: Unable to get pack index http://android.git.linaro.org/git-ro/platform/manifest.git/objects/pack/pac… (Those are non-deterministic, re-trying them manually works.) ---------------------------------------------- Jon Medhurst (Tixy), May 20 07:10 (ADT) On Mon, 2013-05-20 at 12:43 +0300, Paul Sokolovsky wrote: > Hello Jon, > > On Mon, 20 May 2013 09:35:00 +0100 > "Jon Medhurst (Tixy)" <tixy(a)linaro.org> wrote: > > > On Mon, 2013-05-20 at 09:15 +0530, Vishal Bhoj wrote: > > > Android builds are failing in repo init phase. I have opened a bug > > > for the same: > > > https://bugs.launchpad.net/bugs/1181907 > > > > > > repo init and sync worked for me locally. Can someone please have > > > a look. > > > > I get several errors like this, with various repos... > > > > error: Unable to find 51bd0a1525c7a722ad64a8954d90c6df0b8546a7 > > under http://android.git.linaro.org/git-ro/platform/ndk > > Cannot obtain needed blob > > 51bd0a1525c7a722ad64a8954d90c6df0b8546a7 > > while processing commit > > 7b3bda5d569a69c4f9b87225c5d21da4a41bf12e. > > error: Fetch failed. > > This is different issue from one report by Vishal. Oops, yes. I didn't read the bug, sorry. I jumped to conclusions after seeing Vishal and Bero's emails appeared at about the same time. > > > > > As I said in reply to bero's mail, I always seem to get these since > > the move to http URLs > > Move to HTTP happened last couple of days during LCA13 in Hong Kong, do > you want to say that you experienced the error as quoted above since > then? Was it reported to some bug/ticket? (Do you have a reference?) I didn't raise a bug as I had a workaround and I assumed if it was a widespread problem then it would get sorted out eventually. > The situation known to me is: > > 1. After flipping the switch in HK, git:// became unstable (because it > was throttled to stop wasting resources as it always did). It took some > time to migrate everyone to http://. If my memory serves me right, it wasn't long after this I noticed my problems, which is why it occurred to me to change the URLs in the manifests back to the old git:// (They'll get restored to http when the manifests are changed, which is why I notice it as a continuing problem.) I perhaps only do a repo sync a couple of times a month as I don't really work much on Android, which is perhaps why a) I see this problem? b) I've not been motivated to get to the bottom of it. -- Tixy ---------------------------------------------- Paul Sokolovsky, May 20 06:44 (ADT) Hello Jon, On Mon, 20 May 2013 09:35:00 +0100 "Jon Medhurst (Tixy)" <tixy(a)linaro.org> wrote: > On Mon, 2013-05-20 at 09:15 +0530, Vishal Bhoj wrote: > > Android builds are failing in repo init phase. I have opened a bug > > for the same: > > https://bugs.launchpad.net/bugs/1181907 > > > > repo init and sync worked for me locally. Can someone please have > > a look. > > I get several errors like this, with various repos... > > error: Unable to find 51bd0a1525c7a722ad64a8954d90c6df0b8546a7 > under http://android.git.linaro.org/git-ro/platform/ndk > Cannot obtain needed blob > 51bd0a1525c7a722ad64a8954d90c6df0b8546a7 > while processing commit > 7b3bda5d569a69c4f9b87225c5d21da4a41bf12e. > error: Fetch failed. This is different issue from one report by Vishal. > > As I said in reply to bero's mail, I always seem to get these since > the move to http URLs Move to HTTP happened last couple of days during LCA13 in Hong Kong, do you want to say that you experienced the error as quoted above since then? Was it reported to some bug/ticket? (Do you have a reference?) The situation known to me is: 1. After flipping the switch in HK, git:// became unstable (because it was throttled to stop wasting resources as it always did). It took some time to migrate everyone to http://. 2. http:// was stable since then, until migration of git.linaro.org happened weekened May 11. 3. Immediately after migration, there were issues with http://git.linaro.org which manifested pretty much like the error above. It was resolved Mon/Tue last week. 4. Friday GMT evening, android.git.linaro.org was migrated to new setup. Initial builds seem to have gone ok. Detailed status is not known (your telling that error above was received after May 17, 17:00GMT would help). 5. Additionally, today issue reported by Vishal popped up. All in all, git issues now should be cc: to its(a)linaro.org (added), ITS guys are now the owners of new infrastructure, and can resolve issues pretty well (p.3 was resolved by them). > and I work around it by changing the manifest > to have the old git URLs. This may work as a personal workaround, but won't scale to official manifests - we need to get http:// working right. -- Best Regards, Paul Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog -------------------------------- This email is a service from IT Services

11 years, 11 months

[IT Services] Update: Re: Daily builds are failing during repo init

by Paul Sokolovsky

11 years, 11 months

[IT Services] Update: Re: Daily builds are failing during repo init

by Ben Copeland

11 years, 11 months

[IT Services] Update: http://android.git.linaro.org/git-ro is out of sync with the ...

by Ben Copeland

##- Please type your reply above this line -## [IT Services] Update: http://android.git.linaro.org/git-ro is out of sync with the ... You are registered as a CC on this support request (linaroithelp.zendesk.com/requests/89). Reply to this email to add a comment to the request. ---------------------------------------------- Ben Copeland, May 20 07:35 (ADT) Hello Bernhard, We had some issues with the git server regarding the git-ro directory. I have made a similar change to the apache config which hopefully is linked to your problem. Can you try again and let us know if you are still getting problems? Thanks Ben ---------------------------------------------- Bernhard Rosenkränzer, May 17 20:53 (ADT) Hi, Just in case this fixes itself by a sync process or something, this was seen on 2013/05/17 23:46 UTC and has been going on for at least a few hours (I've just spent quite some time trying to figure out why a fix I made didn't work in android-build -- turns out it never made it there because the http checkout is out of sync): git clone ssh:// bero@android.git.linaro.org:29418/toolchain/binutils-current.git cd binutils-current git checkout -b linaro-2.23.52.0.2 origin/linaro-2.23.52.0.2 git log Last commit is 097e128c18b6d7f1a26418518b66b5d206dd357d (as expected) But: git clone http://android.git.linaro.org/git-ro/toolchain/binutils-current.git cd binutils-current git checkout -b linaro-2.23.52.0.2 origin/linaro-2.23.52.0.2 git log Last commit is 6a6083aa79537708cd16551dc803315412f98e21 (2 commits behind the git repository). ttyl bero -------------------------------- This email is a service from IT Services

11 years, 11 months

[IT Services] Update: Re: Daily builds are failing during repo init

by Paul Sokolovsky

11 years, 11 months

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Linaro-android