I love unix

Tuesday, May 13th, 2008

Extract the revision histories of the classes Rushi and I collaborated on out of his school repo, compress them, pipe them over the network onto my laptop, and uncompress. I was on the verge of adding the svnadmin load that would pull it into my repo, but I got lazy.

RSi:/home/svn$ svnadmin dump school | svndumpfilter --drop-empty-revs --renumber-revs include /CSE131A /CSE131B /CSE120 | gzip | ssh jauricchio@128.54.57.118 gunzip \> rushidump

Yeah yeah, DVCS, hg and git and whatever, I know…

Computer Languages and Facial Hair

Tuesday, May 6th, 2008

Why are some programming languages more popular than others? If Java is so lame and (Haskell, Obj-C, Erlang, Smalltalk, whatever) is so great, why is one of them the most used language and nobody’s heard of the other?

Tamir Khason has the answer: Computer Languages and Facial Hair.

Via … somebody. Forgot who. Maybe it was Sam Larbi?

Add an ssh key to another user on a different system without logging in

Tuesday, April 15th, 2008

Scott liked my terminal style from yesterday, so I’ll reprise it.


heian:<<<umi-misc/pubkeys$ ssh cse125@neathat.com -p 14522 sudo ls \>/dev/null
[sudo] password for cse125: (echoed, which is unfortunate)
heian:<<<umi-misc/pubkeys$ cat djc_hmac_rsa.pub | ssh cse125@neathat.com -p 14522 sudo su -c \'cat \>\> ~djc/.ssh/authorized_keys\'
heian:<<<umi-misc/pubkeys$

Now DJ can use his key on that server.

The sudo ls is to enter the password and get the sudo authorization ticket. We can’t enter the password with the cat, because ssh sees its stdin isn’t a terminal and refuses to allocate a tty. A stupid workaround would be (echo "mypassword"; cat ... ); | ssh ..., but I like the two-phase solution more, for some reason.

history meme

Monday, April 14th, 2008

I picked this up from Stephen Lau on Planet OpenSolaris, though he traces it back to Planets Mozilla and Gnome.


heian:<<<hiroi/jauricchio$ uname -a
Darwin heian.local 9.2.2 Darwin Kernel Version 9.2.2: Tue Mar  4 21:23:43 PST 2008; root:xnu-1228.4.31~1/RELEASE_PPC Power Macintosh
heian:<<<hiroi/jauricchio$ history | awk '{a[$2]++ } END{for(i in a){print a[i] " " i}}'|sort -rn|head
95 svn
80 vi
70 make
59 cd
29 ls
18 ./build/neathat.app/Contents/MacOS/neathat
14 sudo
11 ssh
8 rm

It appears I’ve been doing a lot of CSE 125 lately. That accounts for the 18 runs of our debugging build, 95 subversion statuses, updates, and checkins, and 70 makes.

Being evil with DTrace

Sunday, April 6th, 2008

Many people who know me know that I like Ruby and DTrace very much, and I sometimes take an interest in security. This article made me very happy, by combining all three of those: Being evil with DTrace

Via Ben Rockwood.

Xen and OpenVZ at the same time, and a Brief Rant about Filesystems

Tuesday, March 25th, 2008

After a grueling two days of kernel muckery, and much help from Mooneer and Mark Williamson (bless you, sir), I finally have a 2.6.24 kernel with OpenVZ and Xen domU. Things I learned along the way:

  1. Never ever make your generic server in the closet do double-duty as your internet gateway/router. For one thing, internet access is a valuable resource to help diagnose why the dang thing won’t boot. And for another, your roommates hate it when the network is down all day.
  2. Nobody should ever make kernel patches that start from vanilla (e.g., 2.6.24), add a large feature you can’t get anywhere else (e.g. OpenVZ), and along the way give you lots of unrelated things (e.g., 2.6.24.3). This breaks down the instant two people do it (e.g., Debian), because the unrelated things all conflict.
  3. The Xen merge into mainline is a good thing, but it’s only a start. I really want: ballooning, PCI sharing, and dom0 support.
  4. aptitude is actually a great thing. Anybody out there still using apt-get… change your habits. Just start using aptitude. And then, little by little, start figuring out the curses gui and how to mark packages as automatic and how to resolve conflicts and all the excellent stuff aptitude brings. I resisted at first… but it’s definitely worth it.
  5. Also debian, we really need etc-update.
  6. It’s very easy to use hilarious amounts of bandwidth when you’re building systems with debootstrap.
  7. I did something today that made my big domU extremely slow. It seems that under very heavy I/O it just grinds to a halt. I also had very constrained memory and no swap device. So I’m not sure if it just didn’t have space to cache and had to start kicking clean pages, or whether Xen event channel unnecessarily serializes I/O (I’m afraid that it does), or what. DTrace would be sweet.
  8. I need to get better discipline in system management. I’ve historically used cioppino (the Linux machine) for everything from serving web and subversion, to an irssi/shell server, to hosting my AVR cross-toolchain, to my (excessively convoluted but ultimately perfect for me) email hosting. I need to cut down on what I do with it, and I need to stop installing and building random things that seem neat. Apt makes it so easy, I know… but it becomes an N^2 problem. And I’m absolutely terrified by the number of upgrades aptitude offers… there’s just too much on this system.

Finally, a desperate plea: Can we please have a good cross-platform low-commitment network filesystem?

  • NFS is difficult to run safely, hard to expose to temporary untrusted users, and still might require shared UIDs (I think?). Sucks.
  • AFP hates symlinks and hardlinks within the shared area, and isn’t available for Windows. Sucks.
  • Coda and AFS require lots of setup and aren’t suited for “oh just connect to the server and grab it” use case, with temporary untrusted users. Sucks.
  • FTP blows.
  • sshfs requires Fuse which concerns me, and requires a shell-equivalent account on the machine. Sucks.
  • SMB is well-supported and cross-platform, and doesn’t specifically require an account on the machine. It’s pretty bad at non-FAT32 file properties, though. Permissions and symlinks ehhh sometimes work. Workable.
  • WebDAV just seems like a bad solution designed by committee. It kinda works.

Really. That’s it. SMB and WebDAV. This is sort of like the situation with disk filesystems… FAT32 or go home. The least common denominator that Windows supports just seems to win :(

The Worst Security Blunder I’ve Seen This Year

Thursday, November 15th, 2007

Apple just “fixed” CVE-2007-4703.

The “Set access for specific services and applications” setting for the Application Firewall allows any process running as user “root” (UID 0) to receive incoming connections, even if its executable is specifically added to the list of programs and its entry in the list is marked as “Block incoming connections”. This could result in the unexpected exposure of network services.

I am utterly speechless. Shouldn’t the firewall have been built by a security team? Not a bunch of monkeys smashing their keyboards with a femur then committing?

I’m not sorry if I’ve offended you. If you work for Apple, and the Firewall code passed through your hands, YOU DESERVE TO LOSE YOUR JOB.

Letting any root process listen no matter what is like a bank security guard letting nobody except ex-convicts in after hours. The history of Unix security from Robert Morris up to today is that a single chink in any root daemon’s armor means your entire system is laid bare to anyone who knows what Metasploit is. It is inconceivable to me that someone being paid to write security code in 2007 would turn off all firewalling for any root process—and make it impossible for users to specifically request it.

There are those who say firewalls and network security are solving the wrong problem. At DefCon 15, Bruce Potter made the compelling argument that the right way to improve security is to fix the buggy code, and that “defense in depth” is just a band-aid. That may be, but for today, we still have buggy daemons, and sometimes we want to hide them from the world.

Again: I hope someone at Apple lost his or her goddamned career for this.

Round trip

Wednesday, November 14th, 2007

Slowest infinite-regress ever: Java-on-Ruby on Ruby-on-Java on Java-on-Ruby on…

A Reply to DJ on Drivers

Tuesday, October 16th, 2007

DJ laments there are no practical guides to driver writing.

DJ, allow me, as I so often do, to vocally disagree with you :). You seem to be making two closely related claims:

  1. There’s no information in OS textbooks on how to structure a driver to fit into any given operating system’s driver architecture
  2. There’s no information in OS textbooks on how drivers ought to interact with their hardware to make useful stuff happen

1. General operating systems textbooks are not the place to find this information, because if they were they wouldn’t be general operating systems textbooks.

Depth and breadth are usually a trade-off in books. If a book were so thorough and detailed in its treatment of Linux drivers that you could use it as your sole reference while writing one, it could not possibly do the same justice to drivers for Solaris, Free/Net/OpenBSD, AIX, Mac OS X/IOKit, Win32, WinNT, Vista, or any other operating system, and it surely can’t cover things like process scheduling, networking, virtual memory, filesystems, or any of those other things kernels sometimes do. The reason you read an operating systems book is to get a (necessarily) broad coverage of what all operating systems do.

If you want to get specific enough to one kernel and one subsystem that you can actually use it as a reference to build a driver, you probably want to put Modern Operating Systems back on the shelf and try Linux Device Drivers, a book focused on exactly what you see as lacking in general operating systems books.

Taking a mild divergence to head off a counterargument, it’s completely acceptable for the usual OS curriculum to exclude drivers, while including many other parts of operating systems, say, basics of concurrency. The difference in curriculum reflects the likelihood the average student will encounter the material during his career. The average programmer has very high odds of using concurrent systems sometime in his career, even more so today with the advent of multicore systems. Any more than the most basic concurrent programming (paranoid locks everywhere) involves some knowledge of what the OS is doing. In contrast, the average programmer will never be anywhere near driver code. Those who do write or maintain drivers take onto themselves responsibility to further research the field, the same way we expect those working with highly parallel systems to study a bit more than locks and semaphores.

2. I’m willing to accept that there’s little information available on actually driving hardware, but that’s because the information you’re looking for is very narrowly applicable, proprietary, distributed across the whole field, in rapid flux, and/or lacks any common principles or organization. Difficult to put that into a textbook with a ten- or twenty-year lifespan.

There’s so little information generally available about how to drive hardware because it’s intensely device-dependent. Once you get to the IO port level everything ends up pretty specific to the particular chip you’re talking to. Sure, ATA’s standard, so it’s not hard to drive a hard disk. But how about the ATA controller chip? How do you ask it to send a command and get the result? Not so standard.

The impression I get from the field (see disclaimer below) is that everybody kinda builds the hw/sw interface to be as simple as possible for the firmware and hardware guys to build. This varies fairly widely with different guys and different organizational knowledge. Firmware and hardware are extremely expensive and difficult to build correctly, and anything that can be simplified should be; anything that can be offloaded to (testable, debuggable, patchable) software should be. When everyone builds their hardware the way easiest for themselves and their organization and offloads everything else to software, everybody’s hardware will look completely different and will have completely different expectations of the driver’s role. Consequently, everybody’s drivers will look completely different.

A secondary reason for lack of standardization is that until recently most first-party drivers for mass-market hardware were closed-source. When your competitors’ specs are closed to you and your own spec is closed to them, standardization is just not possible. This is changing, though, and we’ll see what happens. There have also been notable exceptions, when an entire industry segment decided to stop being dumb and standardize. EHCI stands out as a shining example. (IDE/ATA/SATA as standard disk access might qualify, though they have their origins in ancient disk protocols defined by the operating system, not the disk vendor, and any disk had to meet that protocol or you couldn’t use it. Standardization was imposed by the OS, not agreed upon by the vendors)

In such an environment, OS books can’t really get any more specific than “Here’s how you read from or write to an I/O port” for the same reason they can’t really get any more specific than “Here’s how you read or write to a network socket”. To tell you how to use, say, FTP, is to tie themselves to today’s practice and take up paper and author’s time that could be better devoted to general principles of networking. They probably ought to describe PIO and DMA, for the same reason they describe IP and TCP. But the OS books are and should be mute on how to drive the hardware. Those rules are different between every chipmaker and every chip, and change every year and every rev. They belong in datasheets and reference manuals, not general textbooks.

Conclusion

For now, every operating system has a different driver architecture, so a general operating systems textbook can’t and shouldn’t devote too much attention to any given system. Furthermore, every piece of hardware has a different host interface, so a general operating systems textbook can’t and a driver reference book sometimes shouldn’t devote too much attention to any given piece of hardware.

The way to change this is for different kernels to agree on a driver architecture, or for different hardware makers to agree on a host interface. The first is not likely to change anytime soon; Linus is not about to chuck the bottom quarter of 2.6, Apple’s kernel team hasn’t shown signs of life lately, Sun still hates everyone, and Microsoft is still dumb.

What is changing is that host interfaces are standardizing. USB was a great leap forward in this regard. Other than the UHCI/OHCI misstep, EHCI means anybody’s host adapter works the same as any others, and device classes mean anybody’s device works the same as nay others. No more does every serial and parallel widget in the store need its own driver. Disk interfaces have long been standardized (out of necessity). I thought Ethernet NICs have a broadly-supported standard, but I can’t track it down.

Disclaimer: I’m very new to hardware design, I’ve never written a driver, and I’ve only been through the first-level undergrad operating systems course. I calls ‘em as I sees ‘em, but I may not see ‘em very well from where I stand. I welcome discussion.

Lenny? Come on, Debian.

Sunday, April 8th, 2007

The code name for the next major Debian release after etch is lenny.

Lenny? That just sounds silly. Join me in boycotting it. I shall only refer to it as “testing”.

Just another WordPress weblog