Limit GithubStatus to certain builders in BuildBot

Written by Johannes on February 16, 2015 Categories: Buildbot, Continuous Integration, git, github, Python

Your ads will be inserted here by

Easy AdSense.

Please go to the plugin admin page to
Paste your ad code OR
Suppress this ad slot.

We use Github in combination with BuildBot for our continuous integration setup. With this, the Github commit status API comes incredibly handy. We use it to notify Github whether the tests pass or fail for any incoming pull request. Unfortunately the Buildbot Github status target is incredibly chatty and tries to push status updates for any incoming changes on Github. We run multiple sequential and concurrent builds for each commit. So this is something we do not want, mainly because these status notifications will spam our developer chat. Having a list of 5 build status updates for a single commit spam your chat and mailing list is not fun.

The mailing status notifier can be limited using a category filter, unfortunately the GithubStatus doesn’t provide this feature. But we can easily subclass it and reimplement the feature. Note that GitHubStatus is an old style class, so we need to explicitly call the parent constructor and pass self.

from buildbot.status.github import GitHubStatus</code>
 
class FilteredGitHubStatus(GitHubStatus):
    def __init__(self, tags=None, *args, **kwargs):
        self.tags = tags
        GitHubStatus.__init__(self, *args, **kwargs)
 
    def builderAdded(self, name, builder):
        # only subscribe to builders we are interested in
        if self.tags is not None and builder.category not in self.tags:
            return None
 
        return self # subscribe to this builder
 
sha = Interpolate("%(src::revision)s")
gs = FilteredGitHubStatus(
    token=github_token,
    repoOwner="myghaccount",
    repoName="myrepo",
    sha=sha,
    tags=["pullrequest"],
    startDescription='Build started.',
    endDescription='Build done.')
 
c["status"].append(gs)
 
pull_request_build = BuilderConfig(
    name="pullrequest-build",
    slavenames=CONTROL_SLAVES,
    category="pullrequest",
    factory=BuildFactory([ShellCommand(command=["echo", '"Hello world"'])),
)
No Comments

Removing all Docker containers with one shell command

Written by Johannes on July 26, 2014 Categories: bash-Scripts, docker

Your ads will be inserted here by

Easy AdSense.

Please go to the plugin admin page to
Paste your ad code OR
Suppress this ad slot.

After fiddling around with docker a bit I wanted to delete all the containers I had accumulated during the tutorial. Maybe there is a quicker way to do it, but it also worked with awk.

sudo docker ps -a | \  # list all containers running and stopped
  awk 'BEGIN { FS=" [ ]+" }; { print $6 }' | \ # use at least two spaces as separator
  awk '{ if (NR!=1) {print}}' | \ # strip first line
  xargs sudo docker rm  # apply docker removal command

No Comments

Building Web-Applications with Python (and WSGI)

Written by Johannes on February 18, 2014 Categories: Python, uwsgi, Webdevelopment, Werkzeug, wsgi

Your ads will be inserted here by

Easy AdSense.

Please go to the plugin admin page to
Paste your ad code OR
Suppress this ad slot.

For the Python Usergroup Freiburg I held a little talk about the basics of WSGI.

A (very) short primer on the mechanics of HTTP and CGI is included in the talk.

You can find the presentation and the example code on github:

No Comments

Finding a dictionary by a (unique) key in a python list

Written by Johannes on January 20, 2014 Categories: Uncategorized

Given an iterable of dictionaries of the form

l = (
  { 'id' : 1, 'data': 'some random data'},
  { 'id' : 2, 'data': 'other random data'},
  { 'id' : 4, 'data': 'more random data'},
)

we can find a particular item in the list by key with this lispish looking one-liner:

(lambda a: list(a)[0])(x for x in l if x['id']==2)

Obviously we could also just use this one:

[x for x in l if x['id']==2][0]

but that would destroy the joy of using an unnecessary lambda :D.

Even more beautiful is what I found on StackExchange (http://stackoverflow.com/a/9542768/1945909):

next(x for x in l if x['id']==2)

Now imagine the beautiful oneliner we could get if we knew the list was sorted by ‘id’ :).

No Comments

Dropping Columns in Alembic SQLite Migration

Written by Johannes on December 6, 2013 Categories: Alembic, Python, SQL, SQLAlchemy, Sqlite

SQLite has no native support for the ALTER and DROP statement on columns (http://sqlite.org/lang_altertable.html). Migration tools, such as django south, therefore have to provide a number workaround for this lack of functionality for full support. Alembic, the migration tool for SQLAlchemy unfortunately does not provide those workarounds, as apparently Mike Bayer wants to keep the interface lean (an ambition for which I think he should be lauded).

Nevertheless in my daily developers practice I need ALTER and DROP column support in migrations. Therefore I decided to hack up DROP COLUMN support myself and stick it in a little repository (https://github.com/klugjohannes/alembic-sqlite). This is still rough around the edges and very limited, but check it out for yourself and feel free to add improvements.

Tests will hopefully be supplied some time, as soon as I figure out a good way to write tests for alembic migrations.

If you’re just looking for snippets, here’s the code. :)

def drop_column_sqlite(tablename, columns):
    """ column dropping functionality for SQLite """
 
    # we need copy to make a deep copy of the column attributes
    from copy import copy
 
    # get the db engine and reflect database tables
    engine = op_.get_bind()
    meta = sa.MetaData(bind=engine)
    meta.reflect()
 
    # create a select statement from the old table
    old_table = meta.tables[tablename]
    select = sa.sql.select([c for c in old_table.c if c.name not in columns])
 
    # get remaining columns without table attribute attached
    remaining_columns = [copy(c) for c in old_table.columns
            if c.name not in columns]
    for column in remaining_columns:
        column.table = None
 
    # create a temporary new table
    new_tablename = '{0}_new'.format(tablename)
    op_.create_table(new_tablename, *remaining_columns)
    meta.reflect()
    new_table = meta.tables[new_tablename]
 
    # copy data from old table
    insert = sa.sql.insert(new_table).from_select(
            [c.name for c in remaining_columns], select)
    engine.execute(insert)
 
    # drop the old table and rename the new table to take the old tables
    # position
    op_.drop_table(tablename)
    op_.rename_table(new_tablename, tablename)
No Comments

Unit-testing non-deterministic functionality in Python

Written by Johannes on August 28, 2013 Categories: Python, Test Driven Development Tags: , ,

In a certain part of our REST-API we expose object ids via randomized unique id (UUID4). While this is convenient in helping to prevent race conditions that might have occured with incrementing integers in the relevant part of the code, randomness always is somewhat of a pain for unit testing. After all, a key necessity to unit testing is being able to predict the results of function calls.

I solved the problem by using a thread local, which I called test_registry. I am now able to tack any data onto this thread local from my tests, which in particular gives me the ability to provide mock random data. In the case of uuid creation it looked something like this.

# core/api.py ## core module code
from threading import local
test_registry = local()
test_registry.test = False
 
# model.py ## model logic code importing uuid.uuid4
from core.app import test_registry
 
def uuid4():
    from uuid import uuid4 as uuid4_orig
    if test_registry.test:
        return test_registry.mock_guids.pop()
    return uuid4_orig()
 
 
# test.py
import unittest
import uuid
from core.app import test_registry
 
class MyTestCase(unittest.TestCase):
    def setUp(self):
        test_registry.test = True
        test_registry.mock_guids = [uuid.uuid4() for i in xrange(4)]
        self.mock_guids = list(mock_guids)
 
    def test_mock_random(self):
        from model import uuid4
        for i in xrange(4):
             self.assertEqual(uuid4(), self.mock_guids.pop())

A few additional thoughts on this: for the sake of your tests: make sure you fail the tests when you run out of mock randomness. In the example above test_registry.mock_guids.pop() will raise an IndexError once the list of uuids has been depleted. If you don’t deliberately fail your tests when running out of mock randomness, you might accidentally write non-deterministic tests, that will randomly fail or, even worse: randomly succeed. Testing is about predictability so keep that in mind.

A second thought: use this with care! 1.) I’m using thread locals here. IMHO this is ok, because the purpose is very clearly delimited and won’t leak into business logic relying on them. 2.) Security: take extra cautions when providing overrides for security-related randomness. Having predictably encryption keys is something you don’t want to have in your software.

No Comments

A simple Python HL7 parser.

Written by Johannes on August 15, 2013 Categories: HL7, Medical IT, Python

The HL7 v2.x standards define one of the key communication interfaces in Medical IT. Therefore, a lot of software is available for parsing, interpreting and sending HL7 messages. Unfortunately most of the code available in public domain is written in Java and hideous.

There is one HL7 library which is written in Python, so we decided to make the one we wrote available for the public, too. It is still fresh from the press, and rough around the edges, but it has some niceties. Segments can be configured via configuration lists to allow data retrieval from hl7 via named attributes. It is tested with unit tests (not extensively, yet). Most of all, it is flexible, simple to use and easy to understand.

Feel free to fork or contribute your additions to the library.

You can find the code at github: https://github.com/mps-gmbh/hl7-parser.

No Comments

Gitosis ERROR:gitosis.serve.main:Repository read access denied

Written by Johannes on July 2, 2013 Categories: git, gitosis, Un*x

Google is swarming with pages with this error, I’ll add another one to the buzz.

In my case it was as simple as the executable bit missing on gitosis post-* hooks.

So

 chmod u+x /path/to/your/gitosis-admin.git/hooks/post-*

fixed the issue.

Note to self: make sure to switch to gitlab/gitlab-shell sometime soon, gitosis is obsolete for quite some time now.

No Comments

Beware of snakes and trailing commas…

Written by Johannes on May 31, 2013 Categories: Python, SQLAlchemy, Webdevelopment, Werkzeug

No here’s a nice way to shoot yourself in the foot with Python tuples. We all know Python tuples are constructed by commas, not round brackets. So

>>> 1,2,3,4
(1, 2, 3, 4)

but

>>> (1)
1

Most important

>>> 1,
(1,)

Those differences are subtle, but significant. So keep that in mind when working on something larger that just insists on breaking up and hurling insults at you and you would never suspect that you have made such a trivial mistake.

In my case I was trying to instantiate an SQLAlchemy scoped Session object in one module and trying to bind an engine to it in the initialization code of my app. The reason for this is to make it possible to easily exchange the engine object, in particular for unit tests, that should typically operate on a separate test database or in memory database. What I did was

# core/db_session.py
from sqlalchemy.orm import scoped_session, sessionmaker
session_factory = sessionmaker(autocommit=False, autoflush=False,)
db_session = scoped_session(session_factory)
 
# runserver.py
from core.app import settings, app
from sqlalchemy import create_engine
from core.db_session import db_session
 
db_engine = create_engine('sqlite:////tmp/cc.sqlite', convert_unicode=True),
db_session.configure(bind=db_engine)
 
from werkzeug.serving import run_simple
 
if __name__ == '__main__':
    run_simple('127.0.0.1', 5000, app, use_debugger=True, use_reloader=True)

roughly following the SQLAlchemy documentation.

Werkzeugs excellent debugging tool gave me an impressive stack trace, telling me something about tuples having no “contextual context”. Now I know I should have seen the elephant in the room, but I was so immersed in thread locals and Werkzeug Local, LocalManagers and LocalStacks that I was almost 100% positive I was experiencing some major scoping issue. So I spent half the morning trying various different approaches of rearranging my code, thinking up hacky ways to monkeypatch (yes, I was that desperate) the way the db_session was fed into my BaseModel class.

Buuuuut… the solution was so simple. Instead of

db_engine = create_engine('sqlite:////tmp/cc.sqlite', convert_unicode=True),

I simply had to write

db_engine = create_engine('sqlite:////tmp/cc.sqlite', convert_unicode=True)

I wonder how that comma wound up there in the first place…

No Comments

Commenting out multiple lines in Vim without external plugins

Written by Johannes on May 17, 2013 Categories: Python, Vim

One minor pet peeve I had with Vim has been resolved by this neat little trick. Commenting out multiple lines in Python has always been somewhat of a chore for me in Vim. Thankfully, it is actually quite easy:

  • Hit Ctrl+V
  • Select the lines you want to comment out (don’t use the mouse for that)
  • Hit “0″ + “I” (capital i) to write simultaneously at the beginning of the selected lines
  • type “#”
  • Hit “ESC”

Update: it’s actually even easier (thank you to Tobi for pointing out this most elegant use case of search and replace in Vim):

  • Hit Shift+V to select the lines you want to comment out.
  • Enter “:s/^/#/g” to search and “replace” (in this case insert) # at the linestart.

Job’s done. One very neat little trick.

No Comments