Robbat2's LiveJournal Entries [entries|friends|calendar]
[ website | Orbis-Terrarum Networks ]
[ userinfo | livejournal userinfo ]
[ calendar | livejournal calendar ]
Flattr this
Advice for Google Summer of Code students [Thu, 25 Mar 2010 22:14:00 -0800]

Good advice for any prospective GSoC student, regardless of gender

I'm also a mentor for Gentoo again this year, after taking a break last year.
You can find our list of potential ideas here: Google Summer of Code 2010 ideas for Gentoo
But don't limit yourself to them! Creative ideas can get you very far too :-)

I'll also be the infrastructure contact for the accepted SoC students, for any issues you have with the source code repositories (we'll be offering Git again), your shell accounts, and a sounding board on deploying your successful project (for those that hosting or larger resources).

1 comment|post comment
FOSDEM: Notes from MirrorBrain talk [Sat, 6 Feb 2010 06:11:00 -0800]

Sitting in the MirrorBrain talk at FOSDEM, taking notes.

Actively used since ~2007.
Split between the redirector and the tester, explicitly made separate.
SourceForge helped with the ASN/Closest-Network side.
Metalinks and P2P support.
Scans mirrors for filelist to see what's present.
Load limiting by making director support mirrors that are limited to a local network / AS / country etc.
MetaLinks don't have Magnet links presently, but I noted that it should be possible to include it.

Using GeoDNS directly for lookups can cause trouble with partial mirrors. Ideally need to put a MirrorBrain server on each continent/region, and GeoDNS to point to that. Also, from some countries, bandwidth to adjcaent countries that might have a mirror is MUCH worse than bandwidth to a well-connected country elsewhere. Past user experience noted with a user in Mozambique, for whom the fastest mirror was via satellite to Canada. Routing data IS needed to make that best choice.

MirrorBrain mailing lists also have a generic non-project-specific "networkers" list for talk between content providers and mirror admins, non-specific to any app.

3 comments|post comment
Spamtrap addresses vs. list confirmation emails, or how to lose 2k list emails [Mon, 1 Feb 2010 12:25:00 -0800]

In the early hours of this morning, a spammer managed to get the IP of the Gentoo list server on the NiX Spam RBL... simply by spamming the subscribe address :-(. This caused approximately 2000 deliveries of normal list mail to be rejected while the server was present on the RBL.

Log detailsCollapse )

Why did this happen? I do agree on the importance of spamtrap accounts, but they MUST check the content of their messages. A list confirmation message MUST NOT be considered as spam.

The original subscribe request came from what seems to be a compromised server in Secunderabad, India. So it wouldn't have been detected by RBL focused on modem/dialup addresses.

Short of raising the bar to subscribe (with a specific token that needs to be included, and then it's only a matter of time till spammers include it too), there isn't much we can do to block stuff like this at the list-server level. There is no way to detect than an address is a spamtrap. There cannot be by definition, as the spammers would avoid it themselves otherwise.

4 comments|post comment
Stolen Bike: Orange DeVinci St Tropez [Wed, 23 Dec 2009 01:31:00 -0800]
[ mood | pissed off ]

My bicycle was stolen earlier this evening. Sufficiently close to see the guy cycling away with it. Drove around a bit with Dave looking for it, but didn't find :-(

  • Orange DeVinci St Tropez (large)
  • Reward if you return it!
  • Extensive scuff damage to the handlebar ends
  • Panier Rack
  • Rain fenders
  • 2x front LED lights
  • 2x rear LED lights
  • Serial: SA…863

Last time I had my bike stolen I was in the downtown eastside. This time it was stolen from outside my house, NOT visible from the street or alley, around 23h00 at night.

4 comments|post comment
Objection to Oracle's purchase of Sun Microsystems (due to MySQL) [Sat, 12 Dec 2009 18:45:00 -0800]

Monty Widenius (one of the original authors of MySQL) has asked for help in lodging objections to Oracle's purchase of Sun Microsystems.

I have no objections to the EC posting my mail, but I thought to also post it here, and help spread the word.

Mail to the EC merger registryCollapse )
5 comments|post comment
Useful and complex /etc/conf.d/net setups (In support of USE=oldnet) [Sat, 17 Oct 2009 02:53:00 -0800]

I've been prodding at the concept of the new network script in OpenRC-0.5, and I'm at a loss to try and see why Roy has decided to toss the old network config system away. The new system doesn't have a lot of capabilities, and most significantly totally loses the ability to restart a single interface without affecting the rest of the system. If it's just for a rewrite, then I'm not too worried, but unless all the functionality is still there, I'm worried we are going to move backwards with it.

At the same time, I don't think many people are aware of how powerful the "old" network configuration mechanism is. The net.examples file is only the start, once you start mixing in the pre/post calls, there's a lot of power. It's capable of some feats that I don't see used even in certain parts of the Gentoo documentation[1]. I've put together some of my gems of conf.d/net, and if you have some, I'd love to hear them. Leave a comment or email me the scripts, along with a description.

Configurations available
  • Easy to maintain (Hurricane Electric) IPv6 tunnels - Download
  • Running two ISPs at home (basic multi-homing) - Download
  • "Enterprise" multi-homing setup, with 4 paths to the Internet - Download

I've also started a bit of storage in my Gentoo webspace for these collected works of network configuration, with a bit more documentation.

  1. The Gentoo docs have this for IPv6: Gentoo IPv6 Router Guide, Tunnel Configuration. You could bring it up manually, or you could just take the IPv6 config above and use it straight with your variables filled in. Volunteers welcome to help merge that config into the Gentoo IPv6 documentation.
5 comments|post comment
Gentoo release statistics as of 2009/10/09 23h59 UTC [Fri, 9 Oct 2009 21:04:00 -0800]

solar was asking about release statistics, so I grabbed the current data from Bouncer. The nearly 34k releases for 10.0 is just in the 5 days that it's been out. I included the various architetures that were part of each released 'product', to make some degree of comparision possible.

installcd-minimum 228561alpha,amd64,hppa,ia64,ppc,ppc64,sparc64,x86
installcd-universal 374388alpha,amd64,hppa,ppc,sparc64,x86
packagecd 162537alpha,amd64,ppc,ppc64,sparc64,x86

livecd 242422x86
minimal 287496alpha,amd64,hppa,ia64,ppc,ppc64,sparc64,x86
packagecd 42572amd64,ppc-g4,ppc-ppc,sparc64
packagecd-32ul 10909ppc64
packagecd-64ul 2981ppc64
universal 111359alpha,amd64,hppa,ppc,ppc64,sparc64

livecd 307481amd64,x86
minimal 330505alpha,amd64,hppa,ia64,ppc,ppc64,sparc64,x86
packagecd 39118ppc,ppc-g3,ppc-g4,ppc64,ppc64-g5
universal 122280alpha,hppa,ppc,ppc64,sparc64

bt-http-seed 72980ALL
livecd 411958amd64,x86
minimal 496943alpha,amd64,hppa,ia64,ppc,ppc64,sparc64,x86
packagecd 27593ppc-g4,sparc64
universal 137554hppa,ppc,ppc64,sparc64

livecd 19426amd64,ppc64,x86
livedvd 4amd64,x86
minimal 14069alpha,amd64,hppa,ia64,ppc64,sparc64,x86
universal 1745ppc64,sparc64

livecd 37771amd64,x86
livedvd 17842amd64,x86
minimal 55745alpha,amd64,hppa,ia64,ppc,sparc64,x86
universal 3142ppc,sparc64

livecd 477934amd64,x86
minimal 406531alpha,amd64,hppa,ia64,ppc,sparc64,x86
packagecd 12308sparc64
universal 83600hppa,ppc,sparc64

livedvd 4870amd64,x86

livedvd 33703amd64,x86

livedvd 0amd64,x86

  • 2008.* has the LiveDVD's pulled from mirrors due to size complaints.
  • bt-http-seed was an (failed) experiment with a set of mirror URLs for trying to load-balance Bittorrent's HTTP seeding
  • Bouncer really needs replacing, but there's nothing really good to do so that I'm aware of. mod_sentry isn't nice. Other suggestions welcome. Should support products, architectures within products, seperate check/serve URLs, detailed hit recording for analysis.
8 comments|post comment
Visualizing Gentoo profiles [Mon, 21 Sep 2009 02:48:00 -0800]

To add a new USE flag, that's globally enabled for all Linux profiles, what's the minimum set of profiles that need to change? Deprecated profiles must be handled as well, for users that need to migrate still.

I ran into this today, while working on the USE=modules changes for linux-mod.eclass.

As an attempt to solve this, I munged up some GraphViz work to show profile inheritance, pictures as the end. Both sets have the trailing profiles "/desktop", "/developer", "/server" turned off for the 2008.0 and 10.0 releases, to cut down on the noise.

Graphs and script for download.

Which profiles?Collapse )

Odd observations

  • Several Prefix profiles (linux/{amd64,ia64,x86} link to 2008.0 profiles explicitly instead of the generic architecture)
  • default/linux does not bring in base. Some profiles at a glance neglect this and might not have base brought in at all.
  • "embedded" is all alone in the tree.
Thumbnail of one graphCollapse )

Question for any skilled GraphViz users:

If all nodes in a given subgroup/cluster have an edge going to a single destination node, is there any way to get graphviz to replace them with a single fat edge from cluster to destination node?

4 comments|post comment
Heatwaves lead to hardware failures [Thu, 30 Jul 2009 05:57:00 -0800]
[ mood | annoyed ]

So for our Vancouver heatwave (I noted 39C away from the water today, in the shade!), it's finally claimed some of my computer hardware. Most annoying, the battery backup unit (BBU) in the newer fileserver, and 1.5 of the disks of the RAID1 array in the old server...

My website and personal email will be offline for a day or two while I ensure my backups are up to date, and redeploy to the newer fileserver (after I buy a new BBU tomorrow).

4 comments|post comment
"District 9" movie and South African history [Thu, 9 Jul 2009 16:15:00 -0800]
[ mood | contemplative ]

I wasn't aware of this forthcoming movie until some of my co-workers mentioned it today, and watching the trailers, it wasn't the movie itself that caught me (but it looks promising too), but rather the usage of South African settings and historical footage from the apartheid era. As a South Africa, some of it is just too eerily familiar. It's just really apartheid again, with a twist that it's against "real aliens" now, not the forced alienation of the black population.

  • Trailer #1
    Original footage: Starting at 0:35, for 6 seconds.
  • Trailer #2
    Same original footage, 0:26 for 2 seconds.
  • Original short film
    The SABC logo is too new for 1990, the one used is post-1994.
post comment
Oneshot update from Poland [Sat, 25 Apr 2009 14:11:00 -0800]
This is a very brief update from southern Poland. Ran into some internet at a communist era apartment block while waiting in being the chauffeur with my rental car.

Wedding proceeding fine, language barrier interesting but not insurmountable.

Probably no more info until Tuesday or Wednesday.

Apparently tonight AND tomorrow night we're required to party for 12 hours starting at 6pm. Bus service back home IS provided to avoid any drunk driving whatsoever.
1 comment|post comment
new fortune-mod-gentoo-dev release [Thu, 5 Mar 2009 03:24:00 -0800]

I really need to get back to writing in this blog. In the meantime, I scoured my email for the last 2 years of fortune submissions that I hadn't compiled together yet, and make a release. Go forth and amuse yourselves with it.

post comment
Want cardboard boxes? [Mon, 2 Feb 2009 10:29:00 -0800]
Since I've just moved, I've got an excess of cardboard boxes (now flattened). If you would like some of them, phone me to arrange picking some up from me early this week (just not today, as I'm out making some runs to the transfer station).

Photos of moving and renovation fun to follow soon.
5 comments|post comment
gentoo mirror stats: master distfiles distribution. [Tue, 16 Dec 2008 13:37:00 -0800]

Now for the second set of statistics. These aren't directly useful to mirrors in estimating their traffic, but instead gives a good overview of how our mirroring setup works internally, and now much traffic is involved in the fan-out stage. Distfiles are the main content moved around by this system, but it is also used for the other directories for releases, experimental and snapshots.

A very quick overview of the existing setup:

  1. Developer uploads new distfile directly to
  2. The master-distfiles box pulls from hourly.
  3. The master-distfiles box checks every ebuild, and downloads missing distfiles from their primary URI if they do not exist. The daily distfile report is also created at this point.
  4. Every hour, the cluster master of pulls the latest content from master-distfiles. (Averages 240MB/day of traffic).
  5. The OSL FTP cluster master (in Oregon) pushes to it's slave locations in Atlanta and Chicago.
  6. All distfiles mirrors pick up their content from one of the FTP nodes - Internet2-connected hosts are directed via DNS to an Internet2-connected slave for performance.

Each of the distfiles mirrors has about 140-160MB of upstream traffic every day (including both the new files and the rsync overhead for scanning). If there are no files changed, the rsync traffic for a directory scan is 1-2MB. While this isn't a lot of traffic, it's very spiky, as mirrors tend to be on fast links.

The new weekly builds from the Release Engineering team will probably be adding another 1.3GB per week, staggered as one arch per day.

I got a small subset of the logs from the OSU FTP cluster for processing some of these statistics. They cover the 24 hour period of 2008/08/07 UTC. It does not have data of which traffic went via Internet2, and I've grouped the sources by country code (using IP::Country::Fast from CPAN).

NumbersCollapse )

As a bit of analysis, I think that more than half of our mirrors (Europe, Middle East, RU) would benefit from having a box to sync against in Europe.

1 comment|post comment
gentoo mirrors stats: a box [Tue, 16 Dec 2008 13:23:00 -0800]

I was doing some statistics about Gentoo mirrors to see about future plans, and thought that the indirect crowd that read my blog via the various aggregators might be interested in numbers.

These are the traffic for, which is a newer box in the official box directly maintained by the Infrastructure team. Hardware specs are 2x Xeon 3050 @2.13Ghz, 4GB RAM. Disk is mostly irrelevant - the rsync workload is served purely from RAM (tail-packing reiserfs, backed via loop device pointing to a file on tmpfs).

Inbound traffic is spiky, but does not exceed 10Mbit by more than a little bit - we can the inbound rsyncs from the rsync1 master to 10Mbit. Outbound traffic varies between 4Mbit and 9Mbit, with an average around 6-7Mbit.

NumbersCollapse )
post comment
I'm a mac... vs. *NIX [Tue, 25 Nov 2008 01:21:00 -0800]
Many thanks to logik for this work of brilliance. Posted with permission, and slightly reformatted here.

A stoner, takes a puff of his joint and says, "Hi, I'm a mac!".
The poorly dressed wannabe bank teller beside him says, "... and I'm a PC."

The door nearby blows in and a heavily armed tactical team storms the room,
throwing both of them to the floor, barrels of MP5k's against their skulls.

Someone yells, "AREA CLEAR!"
The lieutenant comes in after them, smoking a cigar, surveying the area.
"I'm Solaris,
the sergeant over there is BSD (You remember your daddy mac?),
the pretty boy with the M14, he's Linux,
and the guy toting the M60... That there is HPUX.
Now, shut the fuck up, both of you.
We've had about enough of your 'Bill and Ted Get a Computer' bullshit.
Keep it up, and we're gonna do the same thing to you that we did to OS2, got it?"

3 comments|post comment
Gentoo recruiting randomness [Mon, 24 Nov 2008 00:51:00 -0800]

As a recent random time-waster, I went and read all of the bugs in the "Recruitment" product of the Gentoo Bugzilla. In doing so, I found twelve developers (ebuild or other) that weren't listed in our LDAP or historical tracking at all. I added them back now, I have gentoo-core announcements from when several of them joined as well that I double-checked.

The "lost" developers
  • pihta - bug 20756
  • ct - bug 22211
  • srcerer - bug 23184 (retire date approximate)
  • fede2 - bug 25464
  • vlaci - bug 31795
  • teval - bug 36753
  • mccabemt - bug 43029
  • rip7 - bug 46353
  • twk-b - bug 53723
  • dj-submerge - bug 57051
  • little_bob - bug 69742
  • ruth - bug 70469
Other LDAP changes from my review:
  • svyatogor - bug 20756 - updated join date for original docs work, he had commit rights two years before his previously stated join date
  • archaelus - bug 30835 - data fixup
  • apokorny - bug 70188 - add join date
Further plans:

There are 92 developers without join dates. We need to find join dates for them via BugZilla and CVS/SVN. Also audit all join dates for every other developer. Lastly, discover and capture retirement dates for every past developer.

Present statistics: 673 developers total. 247 active, 426 retired.

2 comments|post comment
AD1989B SPDIF support fixed [Tue, 16 Sep 2008 02:49:00 -0800]

Following up on my earlier posting on the AD2000BX/AD1989B SPDIF support being broken, I figured out the required fixes, and they are waiting in the sound-2.6 kernel tree for the next merge window

8 comments|post comment
long-term ccache statistics for a portage-dedicated instance [Sat, 13 Sep 2008 22:07:00 -0800]

Migrating data and cleaning up my old desktop display head machine, I decided to check out my ccache statistics. This is a very old cache, having first started 2006-01-13. The oldest item in the present cache is 2008-01-12, but the statistics are valid for the entire period. hits 229k and 834k misses = approximately 21% hit rate. This wasn't any crazy repeated compiling of my own code, just a dedicated ccache directory for Portage to use.

Raw numbersCollapse )
10 comments|post comment
Linux MD RAID devices and moving spares to missing slots [Sun, 7 Sep 2008 15:45:00 -0800]

Setting up the storage on my new machine, I just ran into something really interesting, what seems to be deliberate usable and useful, but completely undocumented functionality in the MD RAID layer.

It's possible to create RAID devices with the initial array having 'missing' slots, and then add the devices for those missing slots later. RAID1 lets you have one or more, RAID5 only one, RAID6 one or two, RAID10 up to half of the total. That functionality is documented in both the Documentation/md.txt of the kernel, as well as the manpage for mdadm.

What isn't documented is when you later add devices, how to get them to take up the 'missing' slots, rather than remain as spares. Nothing in md(7), mdadm(8), or Documentation/md.txt. Nothing I tried with mdadm could do it either, leaving only the sysfs interface for the RAID device.

Documentation/md.txt does describe the sysfs interface in detail, but seems to have some omissions and outdated material - the code has moved on, but the documentation hasn't caught up yet.

So, below the jump, I present my small HOWTO on creating a RAID10 with missing devices and how to later add them properly.

MD with missing devices HOWTOCollapse )
21 comments|post comment
Flattr this
[ viewing | 20 entries back ]
[ go | earlier/later ]