pages tagged debbugsDon Armstronghttps://www.donarmstrong.com/tags/debbugs/Don Armstrongikiwiki2018-01-09T05:25:16ZDebbugs Versioning: Merginghttps://www.donarmstrong.com/posts/debbugs_merge_versions/2018-01-09T05:25:16Z2018-01-09T05:25:16Z
<p>One of the key features of Debbugs,
the <a href="https://bugs.debian.org">bug tracking system Debian uses</a>, is its
ability to figure out which bugs apply to which versions of a package
by tracking package uploads. This system generally works well, but
when a package maintainer's workflow doesn't match the assumptions of
Debbugs, unexpected things can happen. In this post, I'm going to:</p>
<ol>
<li>introduce how Debbugs tracks versions</li>
<li>provide an example of a merge-based workflow which Debbugs doesn't handle well</li>
<li>provide some suggestions on what to do in this case</li>
</ol>
<h1 id="debbugsversioning">Debbugs Versioning</h1>
<p>Debbugs tracks versions using a set of one or
more
<a href="https://en.wikipedia.org/wiki/Tree_%28graph_theory%29#Forest">rooted trees</a> which
it builds from the ordering of debian/changelog entries. In the
simplist case, every upload of a Debian package has changelogs in the
same order, and each upload adds just one version. For example, in the
case of <a href="https://packages.debian.org/dgit">dgit</a>, to start with the
package has this (abridged) version tree:</p>
<p><img src="https://www.donarmstrong.com/posts/debbugs_merge_versions/graph-5d3f559f0fb850f47a5ea54c62b96da18bba46b8.png" usemap="https://www.donarmstrong.com/tags/debbugs/#graph5d3f559f0fb850f47a5ea54c62b96da18bba46b8" />
<map id="graph5d3f559f0fb850f47a5ea54c62b96da18bba46b8" name="graph5d3f559f0fb850f47a5ea54c62b96da18bba46b8">
</map></p>
<p>the next upload, <em>3.13</em>, has a changelog with this version ordering:
<code>3.13 3.12 3.11 3.10</code>, which causes the <em>3.13</em> version to be added as
a descendant of <em>3.12</em>, and the version tree now looks like this:</p>
<p><img src="https://www.donarmstrong.com/posts/debbugs_merge_versions/graph-04a0cac92e522aa8816397090f0a23ef51e49379.png" usemap="https://www.donarmstrong.com/tags/debbugs/#graph04a0cac92e522aa8816397090f0a23ef51e49379" />
<map id="graph04a0cac92e522aa8816397090f0a23ef51e49379" name="graph04a0cac92e522aa8816397090f0a23ef51e49379">
</map></p>
<p>dgit is being developed while also being used, so new versions with
potentially disruptive changes are uploaded to experimental while
production versions are uploaded to unstable. For example, the <em>4.0</em>
experimental upload was based on the <em>3.10</em> version, with the
changelog ordering <code>4.0 3.10</code>. The tree now has two branches, but
everything seems as you would expect:</p>
<p><img src="https://www.donarmstrong.com/posts/debbugs_merge_versions/graph-65493d1d56cbf3a32fc6e061d4d933f609d0dd9d.png" usemap="https://www.donarmstrong.com/tags/debbugs/#graph65493d1d56cbf3a32fc6e061d4d933f609d0dd9d" />
<map id="graph65493d1d56cbf3a32fc6e061d4d933f609d0dd9d" name="graph65493d1d56cbf3a32fc6e061d4d933f609d0dd9d">
</map></p>
<h1 id="mergebasedworkflows">Merge based workflows</h1>
<p>Bugfixes in the maintenance version of dgit also are made to the
experimental package by merging changes from the production version
using git. In this case, some changes which were present in the <em>3.12</em>
and <em>3.11</em> versions are merged using git, corresponds to a git merge
flow like this:</p>
<p><img src="https://www.donarmstrong.com/posts/debbugs_merge_versions/graph-cc7df2f6e47656a87ca10d313e65a8e3d55fb937.png" usemap="https://www.donarmstrong.com/tags/debbugs/#graphcc7df2f6e47656a87ca10d313e65a8e3d55fb937" />
<map id="graphcc7df2f6e47656a87ca10d313e65a8e3d55fb937" name="graphcc7df2f6e47656a87ca10d313e65a8e3d55fb937">
</map></p>
<p>If an upload is prepared with changelog ordering <code>4.1 4.0 3.12 3.11
3.10</code>, Debbugs combines this new changelog ordering with the
previously known tree, to produce this version tree:</p>
<p><img src="https://www.donarmstrong.com/posts/debbugs_merge_versions/graph-94b259ce6dd4d28c04d692c72f6e021622b5b33a.png" usemap="https://www.donarmstrong.com/tags/debbugs/#graph94b259ce6dd4d28c04d692c72f6e021622b5b33a" />
<map id="graph94b259ce6dd4d28c04d692c72f6e021622b5b33a" name="graph94b259ce6dd4d28c04d692c72f6e021622b5b33a">
</map></p>
<p>This looks a bit odd; what happened? Debbugs walks through the new
changelog, connecting each of the new versions to the previous version
if and only if that version is not already an ancestor of the new
version. Because the changelog says that <em>3.12</em> is the ancestor of
<em>4.0</em>, that's where the <code>4.1 4.0</code> version tree is connected.</p>
<p>Now, when <em>4.2</em> is uploaded, it has the changelog ordering (based on
time) <code>4.2 3.13 4.1 4.0 3.12 3.11 3.10</code>, which corresponds to this git
merge flow:</p>
<p><img src="https://www.donarmstrong.com/posts/debbugs_merge_versions/graph-72f98ac7aa28e7dd40aaccf7742359f5dd2de378.png" usemap="https://www.donarmstrong.com/tags/debbugs/#graph72f98ac7aa28e7dd40aaccf7742359f5dd2de378" />
<map id="graph72f98ac7aa28e7dd40aaccf7742359f5dd2de378" name="graph72f98ac7aa28e7dd40aaccf7742359f5dd2de378">
</map></p>
<p>Debbugs adds in <em>3.13</em> as an ancestor of <em>4.2</em>, and because <em>4.1</em> was
not an ancestor of <em>3.13</em> in the previous tree, <em>4.1</em> is added as an
ancestor of <em>3.13</em>. This results in the following graph:</p>
<p><img src="https://www.donarmstrong.com/posts/debbugs_merge_versions/graph-70ebe94be503db5ba97c4693f9e00fbb1dc3c9f7.png" usemap="https://www.donarmstrong.com/tags/debbugs/#graph70ebe94be503db5ba97c4693f9e00fbb1dc3c9f7" />
<map id="graph70ebe94be503db5ba97c4693f9e00fbb1dc3c9f7" name="graph70ebe94be503db5ba97c4693f9e00fbb1dc3c9f7">
</map></p>
<p>Which doesn't seem particularly helpful, because</p>
<p><img src="https://www.donarmstrong.com/posts/debbugs_merge_versions/graph-3f8db089ab21b48bcae9d536c1887b3bc6fc4bcb.png" usemap="https://www.donarmstrong.com/tags/debbugs/#graph3f8db089ab21b48bcae9d536c1887b3bc6fc4bcb" />
<map id="graph3f8db089ab21b48bcae9d536c1887b3bc6fc4bcb" name="graph3f8db089ab21b48bcae9d536c1887b3bc6fc4bcb">
</map></p>
<p>is probably the tree that more closely resembles reality.</p>
<h1 id="suggestionsonwhattodo">Suggestions on what to do</h1>
<p><em>Why does this even matter?</em> Bugs which are found in <em>3.11</em>, and fixed
in <em>3.12</em> now show up as being found in <em>4.0</em> after the <em>4.1</em> release,
though they weren't found in <em>4.0</em> before that release. It also means
that <em>3.13</em> now shows up as having all of the bugs fixed in <em>4.2</em>,
which might not be what is meant.</p>
<p>To avoid this, my suggestion is to order the entries in changelogs in
the same order that the version graph should be traversed from the
leaf version you are releasing to the root. So if the previous version
tree is what is wanted, <em>3.13</em> should have a changelog with ordering
<code>3.13 3.12 3.11 3.10</code>, and <em>4.2</em> should have a changelog with ordering
<code>4.2 4.1 4.0 3.10</code>.</p>
<p><em>What about making the BTS support DAGs which are not trees?</em> I think
something like this would be useful, but I don't personally have a
good idea on how this could be specified using the changelog or how
bug fixed/found/absent should be propagated in the DAG. If you have
better ideas, email me!</p>
Debbugs: 22 Years of Bugs (Debconf 2017)https://www.donarmstrong.com/presentations/debbugs_debconf_2017/2017-08-27T17:55:35Z2017-08-10T18:41:52Z
<p>This is a talk which I presented on August 10th, 2017 at Debconf 17 in
Montreal, Canada.</p>
<ul>
<li><a href="https://git.donarmstrong.com/?p=debbugs-presentations.git;h=refs/heads/debconf17">Code for slides in git</a></li>
<li><a href="https://www.donarmstrong.com/ld/debbugs2017/debbugs_presentation_2017.pdf">PDF of slides</a></li>
<li><a href="https://meetings-archive.debian.net/pub/debian-meetings/2017/debconf17/debbugs-22-years-of-bugs.vp8.webm">Video of the talk</a></li>
</ul>
Adding a newcomer (⎈) tag to the BTShttps://www.donarmstrong.com/posts/newcomer_bts_tag/2014-11-15T03:14:47Z2014-11-15T03:14:47Z
<p>Some of you may already be aware of the
<a href="https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=gift;users=debian-qa@lists.debian.org">gift tag</a>
which has been used for a while to indicate bugs which are suitable
for new contributors to use as an entry point to working on specific
packages. Unfortunately, some of us (including me!) were unaware that
this tag even existed.</p>
<p>Luckily, Lucas Nussbaum clued me in to the existence of this tag, and
after a brief bike-shed-naming thread, and some
<a href="http://git.donarmstrong.com/?p=bin.git;a=blob_plain;f=pocket_devotee;hb=fbd5071cf1eb41f9fc62325a6142dd10a41f1717">voting using pocket_devotee</a>
we decided to name the new tag newcomer, and I have now added this tag
to the BTS documentation, and tagged all of the bugs which were user
tagged "gift" with this tag.</p>
<p>If you have bugs in your package which you think are ideal for new
contributors to Debian (or your package) to fix, please tag them
newcomer. If you're getting started in Debian, and working on bugs to
fix, please
<a href="http://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=newcomer">search for the newcomer tag</a>,
grab the helm, and contribute to Debian.</p>
Virginia King selected for Debbugs FOSS Outreach Program for Womenhttps://www.donarmstrong.com/posts/opw_debbugs/2014-11-15T00:23:45Z2014-11-15T00:23:45Z
<p>I'm glad to announce that Virginia King has been selected as one of
the three interns for this round of the
<a href="https://gnome.org/opw/#participants">FOSS Outreach Program for women</a>.
Starting December 9th, and continuing until March 9th, she'll be
working on improving the documentation of
<a href="https://bugs.debian.org">Debian's bug tracking system</a>.</p>
<p>The initial goal is to develop a Bug Triager Howto to help new
contributors to Debian jump in and help existing teams triage bugs.
We'll be getting in touch with some of the larger teams in Debian to
help make this document as useful as possible. If you're a member of a
team in Debian who would like this howto to address your specific
workflow, please drop <a href="mailto:don@debian.org">me</a> an e-mail, and we'll
keep you in the loop.</p>
<p>The secondary goals for this project are to:</p>
<ul>
<li>Improve documentation under http://www.debian.org/Bugs</li>
<li>Document of bug-tags and categories</li>
<li>Improve upstream debbugs documentation </li>
</ul>
libravatar for the BTS (and boring encoding fixes)https://www.donarmstrong.com/posts/libravatar_for_bts/2013-03-14T20:42:09Z2013-03-14T20:37:52Z
<p>While working on fixing a few encoding problems that I managed to
introduce to the BTS almost half a year ago, I took a side bit of
coding, and introduced <a href="https://www.libravatar.org/">libravatar</a>
support to the BTS. Every e-mail now has an avatar to the right which
should correspond to the sender. Libravatar is a federated service,
which means that if you control your domain, you can serve your own
icons. It also automatically falls back to gravatar, so if you're
using that service, things should "just work". Hopefully this will be
primarily amusing, and people won't abuse it.</p>
<p><img src="http://cdn.libravatar.org/avatar/ef9dddd859f1f5a1fd6442a1ccf872ef?d=retro"></p>
<p>More importantly, but much less fun, the double encoding problems
(where mails would get double-encoded if any of the headers contained
non-us-ascii text), and mojibake wontfix icon (☹) should be fixed now.
If you see any additional cases of this, please report them to
owner@bugs.debian.org.</p>
Bug Reporting Rate in Debianhttps://www.donarmstrong.com/posts/bug_reporting_rate/2012-10-10T02:53:25Z2012-10-10T01:04:53Z
<p><a href="http://www.perrier.eu.org/weblog/2012/10/09#690000">Christian's most recent blog post</a>
got me wondering if the decline in the bug reporting rate in Debian
was something new, or something which often happened during releases.
So, lets try to figure that out. In the BTS, when a bug report is
filed, the report is written to a file called <code>bugnum.report</code>, and
then not touched from then on. Let's look at the modification date on
that file to see when each bug was filed; and since we're going to
plot this, lets only look at bugs ending in 00:</p>
<pre><code>stat -c '%n %Y' /srv/bugs.debian.org/spool/{archive,db-h}/00/*.report > ~/reporting_rate.txt
</code></pre>
<p>Now, lets get the data into R and plot it.
[For clarity, I'm not showing the R code, but it's available in the source code for this post.]</p>
<p><img class="sweavealike" src="https://www.donarmstrong.com/posts/bug_reporting_rate/12544d6b25ee5f38e3c86d7dc251e132.png" /></p>
<p>From the plot (Bugs reported per second over time with a red loess fit
line), it looks like we do see a decline during certain periods.
However, there's an even more alarming trend of a decrease in bug
reporting in Debian which has been happening since 2006. (Note that
I've truncated the y scale significantly; there are periods in Debian
where the bug rate is astronomically high, usually corresponding to
mass bug filings; I've also limited the plot to data from 2003 on, as
I have to clean up that data significantly before I can plot it like
this.)</p>
<p>Not sure exactly what that means, but it is troubling.</p>
Debbugs: Control at Submit timehttps://www.donarmstrong.com/posts/control_at_submit/2012-07-14T22:54:11Z2012-07-14T22:54:11Z
<p>One of the features that I have been asked for multiple times is the
ability to use control@bugs.debian.org commands at
submit@bugs.debian.org time. I have now implemented this with the
following syntax:</p>
<pre><code>Package: foo
Version: 1.0-3
Control: retitle -1 this is the title
Control: severity -1 bleargh
Control: summary -1 0
Control: forward -1 http://bugs.debian.org/nnn
</code></pre>
<p>In short, you preface any control commands with Control:, -1 is the
current bug, and the rest of each line is the control@ grammar you
already know. This also now works for every kind of message to
nnn@bugs.debian.org with the exception of messages received at
nnn-done and nnn-forwarded. I don't know why you'd use it for anything
else but submit@ messages, but hey, whatever works.</p>
Debbugs: outlook commandhttps://www.donarmstrong.com/posts/debbugs_outlook/2012-07-14T22:06:17Z2012-07-14T22:06:17Z
<p>Neil McGovern asked me to add an additional feature to the BTS to
support tracking the current status of attempts at fixing a bug. In
past releases, we've used the nice commenting feature of
bts.turmzimmer.net to keep track of what is going on in a particular
nasty RC bug, who is working on it, and what needs to be done next (or
if everyone can just ignore the bug).</p>
<p>This feature should probably have already been in the BTS to start
with, but now it is. In addition to the existing summary feature,
where you can nominate a message or text to be the summary for a bug,
there is an <code>outlook</code> command, which tracks the current status of the
bug, and behaves in exactly the same way:</p>
<pre><code>outlook 12345 not good
outlook 54321 0
thanks
I'm totally stymied by #54321.
</code></pre>
<p>for example.</p>
<p>I plan to include the outlook in the bugscan output in the future too,
so it'll be easily accessible. (And possibly up-to-the-minute with
some javascript-fu.)</p>
Debbugs forcemerge supporthttps://www.donarmstrong.com/posts/debbugs_forcemerge/2012-03-13T23:21:32Z2012-03-13T23:21:32Z
<p>One of the earliest features I wrote for the Debian bug tracking
system (Debbugs) after joining the team was support for forcibly
merging bugs. Originally, merging two bugs required that the bugs be
in exactly the same state before merging them; <code>forcemerge</code> removed
this requirement.</p>
<p>Unfortunately, the way I originally implemented this was shortsighted,
and merely forced the merge partners to have the same values as the
merge master. This meant that owners, blocking bugs, and many other
things were silently changed, which meant that people weren't notified
of changes, and bugs could end up in an inconsistent state.</p>
<p>A while ago, I decided to fix this by calculating the changes required
to actually merge the bugs, making those changes, and then merging the
bugs normally; thus, doing everything that a maintainer would normally
have done for them. This necessitated abstracting out the entire
control apparatus into the <code>Debbugs::Control</code> module.</p>
<p>Now that it's complete, you can do the following:</p>
<pre><code>> forcemerge 1 2
Bug #1 [foo] new title
Bug #2 {Done: foo@bugs.something} [foo] foo
Unset bug forwarded-to-address
Severity set to 'wishlist' from 'grave'
3 was blocked by: 2
3 was not blocking any bugs.
Removed blocking bug(s) of 3: 2
2 was blocked by: 4
2 was not blocking any bugs.
Removed blocking bug(s) of 2: 4
Bug reopened
Removed annotation that bug was owned by bar@baz.com.
Removed indication that 2 affects bleargh
Removed tag(s) unreproducible and moreinfo.
Merged 1 2
> thanks
Stopping processing here.
</code></pre>
<p>and bug 2 now is merged with 1 and matches the state of 1.
[The above is the control output from the appropriate bit of the <code>06_mail_handling.t</code> test.]</p>
<p>This change also means that I'll be able to finally write support for
control@ operations at submit@ time. Also, all of the bug
modifications that happen at submit@ or nnn@ time (setting title,
found, etc.) will be implemented as calls to Debbugs::Control so we
can eventually keep a postgresql database updated in addition to the
flatfile database.</p>