linaro-validation

linaro-validation@lists.linaro.org

818 discussions

by Zygmunt Krynicki

Hi. Fast model support is getting better. It seems that with the excellent patches by Peter Maydell we can now boot some kernels (I've only tried one tree, additional trees welcome :-). I'm currently building a from-scratch environment to ensure everything is accounted for and I understand how pieces interact. Having said that I'd like to summarize how LAVA handles fast models: Technically the solution is not unlike QEMU which you are all familiar with. The key differences are: 1) Only NFS boot makes sense. There are no other sensible method that I know of. We may also use a SD card (virtual obviously) but it is constrained to two gigabytes of data. 2) The way we actually boot is complicated. There is no uboot, fast model interpreter actually starts an .axf file that can do anything (some examples include running tests and benchmarks without actually setting the kernel or anything like that). There is no way to easily load the kernel and pass a command line. To work around that we're using a special axf file that uses fast model semihosting features to load the kernel/initrd from a host filesystem as well as to setup the command line that will be passed to the booting kernel. This allows us to freely configure NFS services and point our virtual kernel at appropriate IP addresses and pathnames. 3) After the machine starts up we immediately open a TPC/IP connection to a local TCP socket. We know which port is being used so we can easily allocate them up front. This port is now the traditional LAVA serial console. The rest of this looks like QEMU: - you can access the filesystem easily (to gather results) - we can use QEMU to chroot into the NFS root to install additional software (emulation via a fast model is extremely slow) The rest of lava is now related to managing resources and misc tasks: - running model_shell instances and their 'serial lines' - unpacking and blessing nfs root directories (blessing = patching it enough to make the model boot and work for us) - exporting nfsroot directories for the local network (currently hardcoded as 192.168.0.0/24 - managing local library of models (building, enumerating, adding/removing) - managing root to do the things required. I'm still not done but I hope to push this to one of our servers this week. Ideally I'd like to try on a virtual machine to see how that setup copes with the code. In addition I want to ensure I don't put any magic there, and that all of the installation is managed by automation tools. Thanks ZK PS: After spending the weekend looking at my scripts crashing I took a break. On Monday, today, I grabbed Peter to troubleshoot the issue. The issue was gone, I have no idea what changed (perhaps my long-lived terminal accumulated some cruft).

13 years, 10 months

Improving health checks

by Paul Larson

Following on to the discussion earlier this week, I wanted to kick off a discussion about how to implement one part we discussed, which is to make health checks something that lava natively understands and knows how to do, rather than just a lump of jobs scheduled daily from cron. Here are my thoughts, and I'll let you decide how much it's crack. 1. extend the scheduler model for devices? to add a text blob for storing a health check job - downside is that we would have add one of these for every single board - maybe it makes sense to do this for device type instead and have the scheduler just start it on the proper board? - we should make sure that these jobs use a locally downloadable, complete image 2. at the beginning of every day, if the board is NOT in offline or going offline state, insert this job into the queue for the board - Is there a good way to do this automatically with some internal mechanism - not cron from the outside? 3. Link the result of this job to the health check stuff already in progress thoughts? suggestions?

13 years, 10 months

Origen kernel ci results

by Paul Larson

Hi, I'm looking at the kernel ci results and origen seems to be running tests pretty frequently, but always failing to boot. Sangwook, could you take a look at the args that are getting passed and make sure there's not something here that would explain it hanging on boot? Or is this perhaps something in the kernel config that is getting used? Thanks, Paul Larson

13 years, 10 months

OH NO! IS LAVA DOWN? PONIES!

by Paul Larson

Do you find yourself asking that? If the answer is yes, check here: http://stats.pingdom.com/o15ezbsnv8tb Following on to the discussion earlier in the week, I checked out this pingdom thing that Michael mentioned. It doesn't provide data for system monitoring, but will at least give us external availability checking. Thanks, Paul Larson

13 years, 10 months

Release plans for today

by Zygmunt Krynicki

Hello everyone. I just wanted to give you a head's up on my plans today: 1) Review merge requests for lava-{server,dashboard,dispatcher,test} 2) Merge things that seem ready to land 3) Release everything on pypi 4) Build lava-2012-02-07.pybundle 5) Deploy that to production I also want to figure out how to release Andy's android benchmark extension. Andy, could you ensure that: *) You have merged the patches I +1 on gerrit *) You have registered a project on pypi (ping me if you need help) *) You have released a snapshot (setup.py sdist upload) -- ensure that your MANIFEST.in is correct (look at the source tarball and make sure it has all the files you wanted). If you do that I'll just add your extension to the manifest in lava-deployment-tool and it will be a part of the release today. Thanks ZK

13 years, 10 months

uwsgi memory problems again

by Paul Larson

I noticed that v.l.o was being obnoxiously slow again and top revealed that uwsgi had gone on a memory eating binge again: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3799 lava-pro 20 0 33.3g 24g 28 D 1 82.5 8:20.37 uwsgi touch /srv/lava/instances/production/etc/lava-server/uwsgi.reload and after a few minutes it had cleared up. It's clear the uwsgi changes that were made previously weren't helping though. Any ideas? -Paul Larson

13 years, 10 months

Monitoring LAVA with Sentry and Raven

by Zygmunt Krynicki

Hi. I've created a new extension for LAVA called lava-raven. This extension wraps Raven (which is a client for Sentry) in a easy to install package. The code is up on launchpad [1] and pypi [2], pypi also has a short documentation on how to set up sentry+raven in a LAVA instance. Along with raven I did a few small changes to linaro-django-xmlrpc (merge request [3]) to support raven better (those changes don't depend on raven though) and lava-server (merge request [4]). Here it's mostly a small helper base class for creating headless extensions. The MP is quite long because I took the liberty of doing documentation cleanups at the same time. Please have a look at those so that they can land quickly. Lastly with the help of Canonical IS team I've setup a new DNS for our Sentry instance at [5]. Sadly sentry is not easy to put in a URL prefix so I had to resort to another top level domain. Sentry is built on top of django so it has a local user database. There is no OpenID support and currently only I have an account there. Ping me to get another account if you need it. As we use sentry we'll see if we can keep this view public (I strongly doubt it as it has all the request data, including (probably) security tokens. Sentry currently runs as a new system user (sentry) on validation control server. It's currently using sqlite backend so we may want to create a PostgresSQL user/database for it instead if it gets slow. In any way, as soon as those bits land we'll have live monitoring of various parts of LAVA. All of the things that are integrated with LAVA Server will automatically benefit (that includes the scheduler _except_ for the scheduler daemon). As we progress towards using celery all of our stack will be transparently using Raven. For those bits that won't quickly become celery tasks I think it would be worth to patch some optional raven support (I'm looking at you lava-dispatcher). Best regards ZK [1] https://launchpad.net/lava-raven [2] http://pypi.python.org/pypi/lava-raven/0.1 [3] https://code.launchpad.net/~zkrynicki/linaro-django-xmlrpc/better-sentry-su… [4] https://code.launchpad.net/~linaro-validation/lava-server/production/+merge… [5] http://sentry.validation.linaro.org/

13 years, 10 months

Re: [Linaro-validation] code review: lava android benchmark code

by Zygmunt Krynicki

On 02/02/2012 05:30 PM, Andy Doan wrote: > Starting a new thread to deal with this project. > > We now have this project hosted at: > > > <http://android.git.linaro.org/gitweb?p=lava-server/android_benchmark_views.…> > > I've made a number of improvements since this was initially sent out > including: Hi Andy. I'm reading the code now. I have some comments. Essentially before this lands you _have_ to add one change, then we can follow up with iterative improvements. You _have_ to apply south migrations to your application. Without that we will not be able to make reasonable upgrades. Adding migrations later is not an option as there is a bit that is very hard to automate. Fortunately this is pretty easy: 1) Add a dependency on south (install_requires in setup.py) 2) Run lava-server manage ... android_benchmark_views_app schemamigration --initial --auto 3) Inspect the code briefly and add it to version control 4) Push that up for review. As for the rest: *) helpers.py, don't use double leading underscore. *) Try keeping the code that relates to your model in the model class, also add a few one liners if you can to document each public method *) extension.py, in def version(), pass the second argument to versiontools.format_version() please pass your main package object (android_benchmark_views_app). This will make it follow git version hashes. *) models.py, BenchmarkRun, each ForeignKey should have a related_name. This name is used when accessing the model field from the "remote" end, then you can navigate back using this name. In general this is required in some cases (when there are clashes, anyway this is beyond the scope of this conversation) *) models.py BenchmarkReport, don't use comments like that. Firs of all, in django it's very very bad to use nullable text/char fields. Instead always use null=False, blank=True - that makes writing code then saner. As for comments. Instead of using a simple text field please consider using django comment system. It _is_ slightly more complicated than using a free-for-all text area (like on launchpad's whiteboard but at the same time you get much more in return. You can read about this here: https://docs.djangoproject.com/en/dev/ref/contrib/comments/ *) models.py both classes, add a get_absolute_url() method based on django permalink system. It's trivial to use, see this document: https://docs.djangoproject.com/en/dev/ref/models/instances/#django.db.model… *) You ship query.flot. While that's okay for now I'd rather see this managed by a dedicated django app that just wraps this in one place. As for now this can stay as-is. *) Your templates extend dashboard templates. Please don't do that (as those templates are not a stable/public interface). Instead extend lava-server templates that work just as well but are stable. That's a very quick review. I have one more comment. EXCELLENT WORK :-) Could you please join the validation team? :-) Thanks ZK > > * general improvements to the flot library for rendering things > * zoom support for graphs > * DB queries > > The DB queries is the main thing Zygmunt complained about from the > initial code. I now hit the DB with one query to get all the > measurements I need it and let it perform the StdDev/Avg[1] > calculations. The performance (at least on my laptop) seems pretty good > now. The DB query improvement also allowed me to get rid of some ugly code. > > I'm hoping going forward I can start doing submissions as code reviews > in Gerrit and have someone on this team look at each change. > > Let me know what you think - or we can always sit down and chat about it > directly next week. > > [1]: NOTE: StdDev/Avg functions aren't supported by Sqlite3, so this > feature only works with postgres/mysql. > > -andy -- Zygmunt Krynicki Linaro Validation Team

13 years, 10 months

Bootstrap UI?

by Zygmunt Krynicki

Hi folks. I've noticed bootstrap [1] and it seems quite interesting as a UI toolkit. It seems more featureful and consistent than jQuery UI. Perhaps this is something to look at when we need to build/rebuild UI elements. Best regards ZK [1]: http://twitter.github.com/bootstrap/

13 years, 10 months

Validation at the Linaro Connect

by Paul Larson

Interested in Validation and going to the Linaro connect? Here are some of the sessions already scheduled that you might be interested in on the Validation track: Automated bootloader testing in LAVA https://blueprints.launchpad.net/lava-dispatcher/+spec/linaro-validation-q1… Discuss what needs to be done still on celery, and usage https://blueprints.launchpad.net/lava-lab/+spec/linaro-validation-q112-lava… LAVA Server Admin tools https://blueprints.launchpad.net/lava-server/+spec/linaro-validation-q112-a… Discuss test plans for big.LITTLE https://blueprints.launchpad.net/lava-test/+spec/linaro-validation-q112-big… Monitoring and managing device health in LAVA https://blueprints.launchpad.net/lava-lab/+spec/linaro-validation-q112-devi… Additionally, here are some sessions in other tracks that you may like: Ubuntu LEB and LAVA: Current status and future planning for proper image testing and validation https://blueprints.launchpad.net/linaro-ubuntu/+spec/linaro-platforms-q112-… Linaro's Kernel CI Process discussion and Feedback https://blueprints.launchpad.net/linaro/+spec/linaro-kernel-ci-q112-discuss… End to End Demo of Enabling Something for LAVA https://blueprints.launchpad.net/linaro/+spec/linaro-training-q112-end-to-e… Make sure to check http://summit.linaro.org/lcq1-12/track/linaro-validation/throughout the week for changes and additions. If you are around in the afternoons and want to help out on LAVA development, or want help getting started in it, swing by the validation hacking sessions and talk to us. See you next week!

13 years, 10 months

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

linaro-validation