Move along, nothing to read

Making transparent PDFs in Linux

Fri, 03 Aug 2012 22:15:58 GMT

Just documenting this again, so I don't forget it, and it might help others too.

Specifically, how to make a PDF with transparency, using ImageMagick, so that it can be used as a stamp for PDFTK. This requires a PNG with transparency as input.

convert $INPUT.PNG -transparent white -background none $OUTPUT.PDF
pdftk $FORM.PDF stamp $OUTPUT.PDF output $COMPLETED_FORM.PDF

Gentoo Linux participates in World IPv6 day

Wed, 08 Jun 2011 11:06:46 GMT

In light of World IPv6 day, the Gentoo Linux Infrastructure team would like to announce new IPv6-availability of several services, and list the existing IPv6 services. Every service listed below is running a dual-stack native IPv4/IPv6 service, no tunnels.

The new services available via IPv6 are:

The existing services available via IPv6 are:

CVS/SVN/Git services for developers
http://sources.gentoo.org/
rsync://rsync.gentoo.org/ - our primary rsync rotation
rsync://${CC}.rsync.gentoo.org - our regional community rsync rotations
A number of our mirrors

All of our IPv6 services will remain online after today, unless serious IPv6 problems (esp. regarding routing) are encountered.

Gentoo would like to extend thanks to all our sponsors & mirrors who have provided IPv6 service, and the servers to make use of it!

Robin's 2011 conferences plans, ideas & other travel

Sun, 16 Jan 2011 10:34:46 GMT

Working on my conference travel plans and wishes for the year. I am downgrading OLS to a maybe, the cost is becoming more of a factor. Likewise, while I had incredible fun at FOSDEM last year, and OSCON in 2006, I cannot justify the airfare/hotel expenses for them. I would like to attend SCALE at some point as well, but uncertain for the same cost reason.

Done:

February 25-27, SCALE 9x @ Los Angeles, CA, USA. [SCALE9x].
April 11-14, MySQL UC @ Santa Clara, CA, USA [MySQLconf].
April 13, Embedded Linux Conference @ San Francisco, CA [ELC].

Confirmed:

April 27-30, STS-134 launch @ Kennedy Space Centre, FL, USA [STS134].
May 26-29, Bowen Island, BC, Canada.
June 25-26, Mini Maker Faire Vancouver @ Vancouver, BC, Canada.
August 17-19, LinuxCon 2011 @ Vancouver, BC, Canada [LinuxCon].
August 25-28, PAX Prime 2011 @ Seattle, WA, USA.

Maybe:

July 25-29, OSCON @ Portland, OR, USA.
August 7-11, SIGGRAPH 2011 @ Vancouver, BC, Canada.
October 19-22, Access 2011, @ Vancouver, BC, Canada.
October 22-23, Google Summer of Code 2011 Mentor Summit @ Mountain View, CA, USA.

Would like to go, but out of my financial reach:

~~February 5-6, FOSDEM @ Brussels, Belgium.~~
June 13-15, Linux Symposium @ Ottawa, ON, Canada.
September 7-9, Linux Plumbers @ Santa Rosa, CA, USA.

Notes

SCALE9x: Arriving on the 24th actually
MySQLconf: I will be manning the phpMyAdmin booth, like past 5 years.
ELC: Dropped in for just one day for hallway track
STS134: KSC grandstand seats to see the penultimate launch :-)
LinuxCon: Local this year, so no travel costs :-)

Page History

2011/04/26: Added GSoC, Access 2011, Bowen Island, STS-134

NightBus isn't really all night long

Fri, 17 Sep 2010 10:25:39 GMT

The province is saying that there is sufficient Nightbus service, so they aren't going to extend the SkyTrain hours:

TransLink's Ken Hardie says they can't run SkyTrain later because the tracks need maintenance. "We have night bus routes that basically follow Canada Line, the Expo Line and the Millennium Line, so they duplicate those routes and they run all night."

I don't disagree that the maintenance is needed, but my objection in your claim that the NightBus routes "run all night".

Almost all of the NightBus routes have a final bus leaving the downtown core at 03h09. The first buses in the morning then start leaving the downtown area between 05h00 and 06h30. The exceptions: the N10, with downtown departures up to 04h39 (1 hour gap to the start of normal service). The N16, which stops at 03h28.

This means that if you are downtown and want to leave AFTER that, perhaps because your job had you working downtown, or you were chatting with friends, then you're stuck.

I would like to ask Translink to add the few more trips that it would take to continue to run 30-minute service intervals until the resumption of regular morning service. The N10 is almost there, it just needs one more set of Downtown departures. Make our transit system really 24-hours!

Recent LJ spam

Wed, 23 Jun 2010 05:33:44 GMT

I apologize for doing this, but recent onslaught of spammers (~35 in the last 5 days) have left me with no choice: I've changed comments from non-friends to be screened by default AND require a captcha.

Complaining at Journalists again: Gentoo Security and the UnrealIRCd backdoor

Tue, 15 Jun 2010 08:36:34 GMT

Those that have followed me for a while might have seen me previously complain at journalism that's misleading, wrong, or outright fictitious. Now I've got another case...
This article by Ed Bott at ZDNet:
Linux infection proves Windows malware monopoly is over; Gentoo ships backdoor? [updated]

The article was first published 2010/06/12 20:37 UTC.
It claims to be "worse" when updated at 2010/06/14 19:30 UTC.

Gentoo had a revision bump to a known good copy of the tarball at 2010/06/12 16:34 UTC (using a different filename, and verified against the GPG signature provided by upstream), so it was ALREADY fixed when the article was published. The old revision was explicitly removed at 2010/06/12 21:18 UTC.
Commit data for fixes:
Changes for unrealircd-3.2.8.1-r1.ebuild
Changes for unrealircd-3.2.8.1.ebuild

The trojaned tarball was then removed from the Gentoo master mirror at 2010/06/13 08:00 UTC, about 11 hours after the article was published. It would have been sooner, but it was a matter of bad timing.

Gentoo bug 323691.

The article also claims: "There’s a great deal of comment in the Talkback section of this post about how official repositories can be trusted. It appears that system broke down thoroughly in this case."
This claim is bogus. The developer that updated the package made perhaps a mistake in trusting that the upstream had not been tampered with. However, in lacking anything to verify against (the upstream apparently did not sign releases at that point), he couldn't have detected the backdoor except by manual inspection of all the code. He downloaded the package AFTER it had been tampered with (2009/11/11 I believe), so he never saw the tamper-free version either.

The entire point of the Gentoo Manifests are to ensure that OUR mirrors are not the point where a compromise is introduced. We can detect upstream changes by this same mechanism, but they mostly tend to be upstream deciding to 'fix' something without bumping the version number. In this regard, they functioned perfectly.

P.S. I'm not saying the existing Gentoo mirroring is perfect either, see my prior writings on tree-signing, and the "Attacks on Package Manager" papers by Cappos et al., which are blocked only with the full tree-signing system.

On Google Summer of Code Applications

Tue, 30 Mar 2010 19:24:24 GMT

(This post inspired by Petteri Räty (betelgeuse)'s similar post

For this year's Gentoo GSoC projects, I'm a mentor on two of our suggested ideas (but also interested in totally new ideas that fit my fields):

upstart on Gentoo
Distfile Fetcher Intelligence

Do you actually understand the project idea?: This is actually a gap that I didn't expect to exist, but I have seen in previous years. This is mainly a difference of expectations between the proposal and what the potential student sees as what the idea really entails.
Using Upstart as an example, it supports an existing init.d compatibility mode, but we're not interested in that. Instead we want our init.d scripts to be treated just like upstart jobs (located in /etc/init/). The init.5 manpage shipped with upstart gives a good start...
Code maintainability: betelgeuse spoke about long-term maintenance, but you should think about it long ahead of that. Some degrees of abstraction, and avoiding difficult to understand logic should be prevalent here. betelgeuse mentioned spaghetti code, but it's important to realize that even well formatted code can impose a much larger mental workload if not well thought out.
Timezones, Timezones!: Most of your project should not be blocking on asking for mentor advice, as timezones and real world pressures often conspire to prevent easy real world communication. I may live in UTC-7, but my hours drift as needed by work but I tend to be online anywhere between 17h00 UTC and 10h00 UTC. If you're trying to communicate with me on a regular basis, this can be tough, so being able work on a problem independently, ask highly directed questions via email can go a long way.

Advice for Google Summer of Code students

Fri, 26 Mar 2010 05:14:46 GMT

Good advice for any prospective GSoC student, regardless of gender

I'm also a mentor for Gentoo again this year, after taking a break last year.
You can find our list of potential ideas here: Google Summer of Code 2010 ideas for Gentoo
But don't limit yourself to them! Creative ideas can get you very far too :-)

I'll also be the infrastructure contact for the accepted SoC students, for any issues you have with the source code repositories (we'll be offering Git again), your shell accounts, and a sounding board on deploying your successful project (for those that hosting or larger resources).

FOSDEM: Notes from MirrorBrain talk

Sat, 06 Feb 2010 14:11:12 GMT

Sitting in the MirrorBrain talk at FOSDEM, taking notes.

Actively used since ~2007.
Split between the redirector and the tester, explicitly made separate.
SourceForge helped with the ASN/Closest-Network side.
Metalinks and P2P support.
Scans mirrors for filelist to see what's present.
Load limiting by making director support mirrors that are limited to a local network / AS / country etc.
MetaLinks don't have Magnet links presently, but I noted that it should be possible to include it.

Using GeoDNS directly for lookups can cause trouble with partial mirrors. Ideally need to put a MirrorBrain server on each continent/region, and GeoDNS to point to that. Also, from some countries, bandwidth to adjcaent countries that might have a mirror is MUCH worse than bandwidth to a well-connected country elsewhere. Past user experience noted with a user in Mozambique, for whom the fastest mirror was via satellite to Canada. Routing data IS needed to make that best choice.

MirrorBrain mailing lists also have a generic non-project-specific "networkers" list for talk between content providers and mirror admins, non-specific to any app.

Spamtrap addresses vs. list confirmation emails, or how to lose 2k list emails

Mon, 01 Feb 2010 20:25:41 GMT

In the early hours of this morning, a spammer managed to get the IP of the Gentoo list server on the NiX Spam RBL... simply by spamming the subscribe address :-(. This caused approximately 2000 deliveries of normal list mail to be rejected while the server was present on the RBL.

Notice the subscribe request, line 0004. (whitespace added)

0001 Feb  1 00:15:56 pigeon postfix/smtpd[29314]: 52278E0778: client=unknown[210.212.220.106]
0002 Feb  1 00:15:57 pigeon postfix/cleanup[31589]: 52278E0778:
  message-id=<01caa301$d307f7d0$b173a8c0@ambachglasfaser>
0003 Feb  1 00:15:58 pigeon postfix/qmgr[12260]: 52278E0778:
  from=<ambachglasfaser@test.mailnet.dyndns.biz>,
  size=59874, nrcpt=3 (queue active)
0004 Feb  1 00:15:58 pigeon postfix/local[31581]: 52278E0778:
  to=<gentoo-embedded+subscribe@lists.gentoo.org>,
  orig_to=<gentoo-embedded-subscribe@lists.gentoo.org>,
  relay=local, delay=2.4, delays=2.4/0/0/0.01, dsn=2.0.0, status=sent (delivered to command: ....)
0005 Feb  1 00:15:58 pigeon postfix/local[31716]: 52278E0778:
  to=<gentoo-user-id@lists.gentoo.org>,
  relay=local, delay=2.4, delays=2.4/0.01/0/0.01, dsn=2.0.0, status=sent (delivered to command: ....)
0006 Feb  1 00:15:58 pigeon postfix/local[31509]: 52278E0778:
  to=<gentoo-gwn@lists.gentoo.org>,
  relay=local, delay=2.4, delays=2.4/0.01/0/0.02, dsn=2.0.0, status=sent (delivered to command: ....)
0007 Feb  1 00:15:58 pigeon postfix/qmgr[12260]: 52278E0778: removed

Assuming that the it's a real subscribe request, we send a confirmation request, and promptly get blacklisted for being a good citizen. Line 0013.

0010 Feb  1 00:15:58 pigeon postfix/smtpd[31587]: B6FA9E0778: client=localhost[127.0.0.1]
0011 Feb  1 00:15:58 pigeon postfix/cleanup[31589]: B6FA9E0778:
  message-id=<1264983358-31717-mlmmj-3905840d@lists.gentoo.org>
0012 Feb  1 00:15:58 pigeon postfix/qmgr[12260]: B6FA9E0778:
  from=<gentoo-embedded+bounces-confsub-32dfa15d1a18a7a9-ambachglasfaser=test.mailnet.dyndns.biz@lists.gentoo.org>,
  size=1345, nrcpt=1 (queue active)
0013 Feb  1 00:16:29 pigeon postfix/smtp[31603]: B6FA9E0778:
  to=<ambachglasfaser@test.mailnet.dyndns.biz>,
  relay=mx.dyndns.biz[217.11.54.110]:25, delay=31, delays=0.06/0/30/0.41, dsn=5.7.1,
  status=bounced (host mx.dyndns.biz[217.11.54.110] said:
    554 5.7.1 Service unavailable; Your spam message has been received.
    You will be blacklisted. Thank you (in reply to end of DATA command))
0014 Feb  1 00:16:29 pigeon postfix/bounce[31637]: B6FA9E0778: sender non-delivery notification: B8AE9E089A
0015 Feb  1 00:16:29 pigeon postfix/qmgr[12260]: B6FA9E0778: removed

Why did this happen? I do agree on the importance of spamtrap accounts, but they MUST check the content of their messages. A list confirmation message MUST NOT be considered as spam.

The original subscribe request came from what seems to be a compromised server in Secunderabad, India. So it wouldn't have been detected by RBL focused on modem/dialup addresses.

Short of raising the bar to subscribe (with a specific token that needs to be included, and then it's only a matter of time till spammers include it too), there isn't much we can do to block stuff like this at the list-server level. There is no way to detect than an address is a spamtrap. There cannot be by definition, as the spammers would avoid it themselves otherwise.

Stolen Bike: Orange DeVinci St Tropez

Wed, 23 Dec 2009 09:31:08 GMT

My bicycle was stolen earlier this evening. Sufficiently close to see the guy cycling away with it. Drove around a bit with Dave looking for it, but didn't find :-(

Orange DeVinci St Tropez (large)
Reward if you return it!
Extensive scuff damage to the handlebar ends
Panier Rack
Rain fenders
2x front LED lights
2x rear LED lights
Serial: SA…863

Last time I had my bike stolen I was in the downtown eastside. This time it was stolen from outside my house, NOT visible from the street or alley, around 23h00 at night.

Objection to Oracle's purchase of Sun Microsystems (due to MySQL)

Sun, 13 Dec 2009 02:45:37 GMT

Monty Widenius (one of the original authors of MySQL) has asked for help in lodging objections to Oracle's purchase of Sun Microsystems.

I have no objections to the EC posting my mail, but I thought to also post it here, and help spread the word.

Date: Sun, 13 Dec 2009 02:39:25 +0000
From: "Robin H. Johnson" <robbat2@isohunt.com>
Subject: Objection to Oracle's purchase of Sun Microsystems (due to MySQL)
To: comp-merger-registry@ec.europa.eu

Name: Robin H. Johnson
Title: Development Manager / Lead Developer
Company: IsoHunt Web Technologies Inc.
Size of company: (privately held)
How many MySQL installations: More than 10
Total data stored in MySQL (megabyte): More than 80000 (>80GB)
For what type of applications is MySQL used: Web index
Should this email be kept confidential by EC: No

I am concerned for the open-source health of MySQL if Oracle's plan to buy Sun
goes through in it's present form. When Sun bought MySQL, they made explicit
promises to uphold many of the open-source tenets of MySQL, and not did not try
to weasel out of them.

When Oracle bought Innobase (InnoDB being a key component inside MySQL):
- they became opaque to the outside world, no outside submissions were
  considered even from Sun/MySQL.
- Bugfixes were only done by contractual obligation, not by the spirit of
  improving open source and just fixing things that were broken.

Thus I do not believe the purchase of Sun (and by proxy MySQL) will benefit the
greater good.

Additionally, if you feel that Oracle purchasing IBM's DB2 division or
Microsoft's SQL Server division would be bad for the competitive market, the
same should be true for MySQL.

I ask that you take one of the following options
a) require Oracle to sell MySQL and not retain it.
b) require Oracle to make legally binding guarantees on the future of MySQL [1].
c) reject the deal completely.

[1]
Monty Widenius (one of the original authors of MySQL) has drafted a list of
proposed guarantees, which I reproduce here verbatim.
====
- All of MySQL will continue to be fully Open Source/free software in the
  future (no closed source modules)
- That development will be done in community friendly way.
- The manual should be released under a permissive license (so that one can
  fork it, the same way one can fork the server)
- That MySQL should be released under a more permissive license to ensure that
  forks can truly compete with Oracle if Oracle is not a good steward after                             
  all.
Alternatively:
- One should be able to always buy low priced commercial licenses for MySQL.
====

--
Robin Hugh Johnson
E-Mail     : robbat2@isohunt.com
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Useful and complex /etc/conf.d/net setups (In support of USE=oldnet)

Sat, 17 Oct 2009 10:29:58 GMT

I've been prodding at the concept of the new network script in OpenRC-0.5, and I'm at a loss to try and see why Roy has decided to toss the old network config system away. The new system doesn't have a lot of capabilities, and most significantly totally loses the ability to restart a single interface without affecting the rest of the system. If it's just for a rewrite, then I'm not too worried, but unless all the functionality is still there, I'm worried we are going to move backwards with it.

At the same time, I don't think many people are aware of how powerful the "old" network configuration mechanism is. The net.examples file is only the start, once you start mixing in the pre/post calls, there's a lot of power. It's capable of some feats that I don't see used even in certain parts of the Gentoo documentation^[1]. I've put together some of my gems of conf.d/net, and if you have some, I'd love to hear them. Leave a comment or email me the scripts, along with a description.

Configurations available

Easy to maintain HE.net (Hurricane Electric) IPv6 tunnels - Download
Running two ISPs at home (basic multi-homing) - Download
"Enterprise" multi-homing setup, with 4 paths to the Internet - Download

Hosting

I've also started a bit of storage in my Gentoo webspace for these collected works of network configuration, with a bit more documentation.

Notes

The Gentoo docs have this for IPv6: Gentoo IPv6 Router Guide, Tunnel Configuration. You could bring it up manually, or you could just take the IPv6 config above and use it straight with your variables filled in. Volunteers welcome to help merge that config into the Gentoo IPv6 documentation.

Gentoo release statistics as of 2009/10/09 23h59 UTC

Sat, 10 Oct 2009 05:53:20 GMT

solar was asking about release statistics, so I grabbed the current data from Bouncer. The nearly 34k releases for 10.0 is just in the 5 days that it's been out. I included the various architetures that were part of each released 'product', to make some degree of comparision possible.

What	Hits	Arches
2005.1
installcd-minimum	228561	alpha,amd64,hppa,ia64,ppc,ppc64,sparc64,x86
installcd-universal	374388	alpha,amd64,hppa,ppc,sparc64,x86
packagecd	162537	alpha,amd64,ppc,ppc64,sparc64,x86

2006.0
livecd	242422	x86
minimal	287496	alpha,amd64,hppa,ia64,ppc,ppc64,sparc64,x86
packagecd	42572	amd64,ppc-g4,ppc-ppc,sparc64
packagecd-32ul	10909	ppc64
packagecd-64ul	2981	ppc64
universal	111359	alpha,amd64,hppa,ppc,ppc64,sparc64

2006.1
livecd	307481	amd64,x86
minimal	330505	alpha,amd64,hppa,ia64,ppc,ppc64,sparc64,x86
packagecd	39118	ppc,ppc-g3,ppc-g4,ppc64,ppc64-g5
universal	122280	alpha,hppa,ppc,ppc64,sparc64

2007.0
bt-http-seed	72980	ALL
livecd	411958	amd64,x86
minimal	496943	alpha,amd64,hppa,ia64,ppc,ppc64,sparc64,x86
packagecd	27593	ppc-g4,sparc64
universal	137554	hppa,ppc,ppc64,sparc64

2008.0_beta1
livecd	19426	amd64,ppc64,x86
livedvd	4	amd64,x86
minimal	14069	alpha,amd64,hppa,ia64,ppc64,sparc64,x86
universal	1745	ppc64,sparc64

2008.0_beta2
livecd	37771	amd64,x86
livedvd	17842	amd64,x86
minimal	55745	alpha,amd64,hppa,ia64,ppc,sparc64,x86
universal	3142	ppc,sparc64

2008.0
livecd	477934	amd64,x86
minimal	406531	alpha,amd64,hppa,ia64,ppc,sparc64,x86
packagecd	12308	sparc64
universal	83600	hppa,ppc,sparc64

10.0_pre20090926-1952
livedvd	4870	amd64,x86

10.0
livedvd	33703	amd64,x86

10.1
livedvd	0	amd64,x86

Notes

2008.* has the LiveDVD's pulled from mirrors due to size complaints.
bt-http-seed was an (failed) experiment with a set of mirror URLs for trying to load-balance Bittorrent's HTTP seeding
Bouncer really needs replacing, but there's nothing really good to do so that I'm aware of. mod_sentry isn't nice. Other suggestions welcome. Should support products, architectures within products, seperate check/serve URLs, detailed hit recording for analysis.

Visualizing Gentoo profiles

Mon, 21 Sep 2009 10:31:00 GMT

To add a new USE flag, that's globally enabled for all Linux profiles, what's the minimum set of profiles that need to change? Deprecated profiles must be handled as well, for users that need to migrate still.

I ran into this today, while working on the USE=modules changes for linux-mod.eclass.

As an attempt to solve this, I munged up some GraphViz work to show profile inheritance, pictures as the end. Both sets have the trailing profiles "/desktop", "/developer", "/server" turned off for the 2008.0 and 10.0 releases, to cut down on the noise.

Graphs and script for download.

My answers as to which profiles:

default-linux
default/linux
base
embedded

Odd observations

Several Prefix profiles (linux/{amd64,ia64,x86} link to 2008.0 profiles explicitly instead of the generic architecture)
default/linux does not bring in base. Some profiles at a glance neglect this and might not have base brought in at all.
"embedded" is all alone in the tree.

Thumbnail of one graph

Question for any skilled GraphViz users:

If all nodes in a given subgroup/cluster have an edge going to a single destination node, is there any way to get graphviz to replace them with a single fat edge from cluster to destination node?

Heatwaves lead to hardware failures

Thu, 30 Jul 2009 13:01:10 GMT

So for our Vancouver heatwave (I noted 39C away from the water today, in the shade!), it's finally claimed some of my computer hardware. Most annoying, the battery backup unit (BBU) in the newer fileserver, and 1.5 of the disks of the RAID1 array in the old server...

My website and personal email will be offline for a day or two while I ensure my backups are up to date, and redeploy to the newer fileserver (after I buy a new BBU tomorrow).

"District 9" movie and South African history

Thu, 09 Jul 2009 23:38:39 GMT

I wasn't aware of this forthcoming movie until some of my co-workers mentioned it today, and watching the trailers, it wasn't the movie itself that caught me (but it looks promising too), but rather the usage of South African settings and historical footage from the apartheid era. As a South Africa, some of it is just too eerily familiar. It's just really apartheid again, with a twist that it's against "real aliens" now, not the forced alienation of the black population.

Trailer #1
Original footage: Starting at 0:35, for 6 seconds.
Trailer #2
Same original footage, 0:26 for 2 seconds.
Original short film
The SABC logo is too new for 1990, the one used is post-1994.

Oneshot update from Poland

Sat, 25 Apr 2009 12:04:56 GMT

This is a very brief update from southern Poland. Ran into some internet at a communist era apartment block while waiting in being the chauffeur with my rental car.

Wedding proceeding fine, language barrier interesting but not insurmountable.

Probably no more info until Tuesday or Wednesday.

Apparently tonight AND tomorrow night we're required to party for 12 hours starting at 6pm. Bus service back home IS provided to avoid any drunk driving whatsoever.

new fortune-mod-gentoo-dev release

Thu, 05 Mar 2009 11:27:22 GMT

I really need to get back to writing in this blog. In the meantime, I scoured my email for the last 2 years of fortune submissions that I hadn't compiled together yet, and make a release. Go forth and amuse yourselves with it.

Want cardboard boxes?

Mon, 02 Feb 2009 18:40:45 GMT

Since I've just moved, I've got an excess of cardboard boxes (now flattened). If you would like some of them, phone me to arrange picking some up from me early this week (just not today, as I'm out making some runs to the transfer station).

Photos of moving and renovation fun to follow soon.

gentoo mirror stats: master distfiles distribution.

Tue, 16 Dec 2008 22:50:31 GMT

Now for the second set of statistics. These aren't directly useful to mirrors in estimating their traffic, but instead gives a good overview of how our mirroring setup works internally, and now much traffic is involved in the fan-out stage. Distfiles are the main content moved around by this system, but it is also used for the other directories for releases, experimental and snapshots.

A very quick overview of the existing setup:

Developer uploads new distfile directly to dev.gentoo.org.
The master-distfiles box pulls from dev.gentoo.org hourly.
The master-distfiles box checks every ebuild, and downloads missing distfiles from their primary URI if they do not exist. The daily distfile report is also created at this point.
Every hour, the cluster master of ftp.osuosl.org pulls the latest content from master-distfiles. (Averages 240MB/day of traffic).
The OSL FTP cluster master (in Oregon) pushes to it's slave locations in Atlanta and Chicago.
All distfiles mirrors pick up their content from one of the FTP nodes - Internet2-connected hosts are directed via DNS to an Internet2-connected slave for performance.

Each of the distfiles mirrors has about 140-160MB of upstream traffic every day (including both the new files and the rsync overhead for scanning). If there are no files changed, the rsync traffic for a directory scan is 1-2MB. While this isn't a lot of traffic, it's very spiky, as mirrors tend to be on fast links.

The new weekly builds from the Release Engineering team will probably be adding another 1.3GB per week, staggered as one arch per day.

I got a small subset of the logs from the OSU FTP cluster for processing some of these statistics. They cover the 24 hour period of 2008/08/07 UTC. It does not have data of which traffic went via Internet2, and I've grouped the sources by country code (using IP::Country::Fast from CPAN).

CC	OutBytes	Count, [Notes]
South America
AR	149837914	1
BR	149840522	1
==	299678436	2
Europe
AT	320229056	2
BA	149840422	1
BE	146473966	1
BG	219988607	2
CH	149674312	1
CZ	806280370	5
DE	1490929973	10
DK	229515404	1
EE	136003774	1
ES	449303700	3
FI	138711526	1
FR	799635661	5
GB	396019061	3
GR	417222774	3 [1]
IS	136003774	1
LV	149911864	1
NL	451913600	3
NO	149908826	1
PL	695724214	1
PT	284020711	2
RO	366854093	3
SE	449664334	3
SK	149840568	1
==	8683670590	55
Asia/Oceania
AU	297402090	2
JP	449369685	3
KR	450928942	3
RU	197245756	2
SG	135681094	1
TH	135835776	1
TW	492731170	4
==	2159194513	16
North America:
CA	742969284	7
US	3174914858	24
==	3917884142	31
Middle East:
IL	193527283	2
KW	149772550	1
==	343299833	3

Grand Total:
==	15403727514 bytes	107

[1] One Greek mirror was excluded from the traffic and counts, as this was their catchup sync with 7Gb of traffic after some hardware-related downtime.

As a bit of analysis, I think that more than half of our mirrors (Europe, Middle East, RU) would benefit from having a box to sync against in Europe.

gentoo mirrors stats: a rsync.gentoo.org box

Tue, 16 Dec 2008 21:37:02 GMT

I was doing some statistics about Gentoo mirrors to see about future plans, and thought that the indirect crowd that read my blog via the various aggregators might be interested in numbers.

These are the traffic for boobie.gentoo.org, which is a newer box in the official rsync.gentoo.org box directly maintained by the Infrastructure team. Hardware specs are 2x Xeon 3050 @2.13Ghz, 4GB RAM. Disk is mostly irrelevant - the rsync workload is served purely from RAM (tail-packing reiserfs, backed via loop device pointing to a file on tmpfs).

Inbound traffic is spiky, but does not exceed 10Mbit by more than a little bit - we can the inbound rsyncs from the rsync1 master to 10Mbit. Outbound traffic varies between 4Mbit and 9Mbit, with an average around 6-7Mbit.

Date	InBytes	InBPS	OutBytes	OutBPS
2008-12-01	2451035341	28368	59523455410	688928
2008-12-02	2325176854	26911	54877643699	635157
2008-12-03	2167829249	25090	50850785431	588550
2008-12-04	2227342435	25779	50823673845	588236
2008-12-05	2182014214	25254	50558268814	585165
2008-12-06	2039468435	23604	47476164351	549492
2008-12-07	1906528455	22066	50327689263	582496
2008-12-08	2127792797	24627	52759944753	610647
2008-12-09	2327731419	26941	56661069093	655799
2008-12-10	2246262570	25998	52107127647	603091
2008-12-11	2302572673	26650	53602727876	620401
2008-12-12	2077185312	24041	47108235487	545234
2008-12-13	2162193709	25025	50807583749	588050
2008-12-14	1698766788	19661	43678479520	505537
2008-12-15	2370132609	27432	58353939353	675392

I'm a mac... vs. *NIX

Tue, 25 Nov 2008 09:29:13 GMT

Many thanks to logik for this work of brilliance. Posted with permission, and slightly reformatted here.

A stoner, takes a puff of his joint and says, "Hi, I'm a mac!".
The poorly dressed wannabe bank teller beside him says, "... and I'm a PC."

The door nearby blows in and a heavily armed tactical team storms the room,
throwing both of them to the floor, barrels of MP5k's against their skulls.

Someone yells, "AREA CLEAR!"
The lieutenant comes in after them, smoking a cigar, surveying the area.
"I'm Solaris,
the sergeant over there is BSD (You remember your daddy mac?),
the pretty boy with the M14, he's Linux,
and the guy toting the M60... That there is HPUX.
Now, shut the fuck up, both of you.
We've had about enough of your 'Bill and Ted Get a Computer' bullshit.
Keep it up, and we're gonna do the same thing to you that we did to OS2, got it?"

Gentoo recruiting randomness

Mon, 24 Nov 2008 09:05:57 GMT

As a recent random time-waster, I went and read all of the bugs in the "Recruitment" product of the Gentoo Bugzilla. In doing so, I found twelve developers (ebuild or other) that weren't listed in our LDAP or historical tracking at all. I added them back now, I have gentoo-core announcements from when several of them joined as well that I double-checked.

The "lost" developers

pihta - bug 20756
ct - bug 22211
srcerer - bug 23184 (retire date approximate)
fede2 - bug 25464
vlaci - bug 31795
teval - bug 36753
mccabemt - bug 43029
rip7 - bug 46353
twk-b - bug 53723
dj-submerge - bug 57051
little_bob - bug 69742
ruth - bug 70469

Other LDAP changes from my review:

svyatogor - bug 20756 - updated join date for original docs work, he had commit rights two years before his previously stated join date
archaelus - bug 30835 - data fixup
apokorny - bug 70188 - add join date

Further plans:

There are 92 developers without join dates. We need to find join dates for them via BugZilla and CVS/SVN. Also audit all join dates for every other developer. Lastly, discover and capture retirement dates for every past developer.

Present statistics: 673 developers total. 247 active, 426 retired.

AD1989B SPDIF support fixed

Tue, 16 Sep 2008 09:54:57 GMT

Following up on my earlier posting on the AD2000BX/AD1989B SPDIF support being broken, I figured out the required fixes, and they are waiting in the sound-2.6 kernel tree for the next merge window