Friday, November 20, 2009

Performance Optimization with Global Entry. Or Not?

As I entered the 30-minute "U.S. Citizens" queue for immigration back into the U.S. last week, the helpful "queue manager" handed me a brochure. This is a great place to hand me something to read, because I'm captive for the next 30 minutes as I await my turn with the immigration officer at the Passport Control desk. The brochure said "Roll through Customs faster."

Ok. I'm listening.

Inside the brochure, the first page lays out the main benefits:
  • bypass the passport lines
  • no paper Customs declaration
  • in most major U.S. airports
Well, that's pretty cool. Especially as I'm standing only 5% deep in a queue with a couple hundred people in it. And look, there's a Global Entry kiosk right there with its own special queue, with nobody—nobody!—in it.

If I had this Global Entry thing, I'd have a superpower that would enable me to zap past the couple hundred people in front of me, and get out of the Passport Control queue right now. Fantastic.

So what does this thing cost? It's right there in the brochure:
  1. Apply online at There is a non-refundable $100 application fee. Membership is valid for five years. That's $20 a year for the queue-bypassing superpower. Not bad. Still listening.
  2. Schedule an in-person interview. Next, I have to book an appointment to meet someone at the airport for a brief interview.
  3. Complete the interview and enrollment. I give my interview, get my photo taken, have my docs verified, and that's it, I'm done.
So, all in all, it doesn't cost too much: a hundred bucks and probably a couple hours one day next month sometime.

What's the benefit of the queue-bypassing superpower? Well, it's clearly going to knock a half-hour off my journey through Passport Control. I immigrate three or four times per year on average, and today's queue is one of the shorter ones I've seen, so that's at least a couple hours per year that I'd save... Wow, that would be spectacular: a couple more hours each year in my family's arms instead of waiting like a lamb at the abattoir to have my passport controlled.

But getting me into my family's arms 30 minutes earlier is not really what happens. The problem is a kind of logic that people I meet get hung up in all the time. When you think about subsystem (or resource) optimization, it looks like your latency savings for the subsystem should go straight to your system's bottom line, but that's often not what happens. That's why I really don't care about subsystem optimization; I care about response time. I could say that a thousand times, but my statement is too abstract to really convey what I mean unless you already know what I mean.

What really happens in the airport story is this: if I had used Global Entry on my recent arrival, it would have saved me only a minute or two. Not half an hour, not even close.

It sounds crazy, doesn't it? How can a service that cuts half an hour off my Passport Control time not get me home at least a half hour earlier?

You'll understand once I show you a sequence diagram of my arrival. Here it is (at right). You can click the image to embiggen it, if you need.

To read this sequence diagram, start at the top. Time flows downward. This sequence diagram shows two competing scenarios. The multicolored bar on the left-hand side represents the timeline of my actual recent arrival at DFW Airport, without using the Global Entry service. The right-hand timeline is what my arrival would have looked like had I been endowed with the Global Entry superpower.

You can see at the very bottom of the timeline on the right that the time I would have saved with Global Entry is minuscule: only a minute or two.

The real problem is easy to see in the diagram: Queue for Baggage Claim is the great equalizer in this system. No matter whether I'm a Global Entrant or not, I'm going to get my baggage when the good people outside with the Day-Glo Orange vests send it up to me. My status in the Global Entry system has absolutely no influence over what time that will occur.

Once I've gotten my baggage, the Global Entry superpower would have again swung into effect, allowing me to pass through the zero-length queue at the Global Entry kiosk instead of waiting behind two families at the Customs queue. And that's the only net benefit I would have received.

Wait: there were only two families in the Customs queue? What about the hundreds of people I was standing behind in the Passport Control queue? Well, many of them were gone already (either they had hand-carry bags only, or their bags had come off earlier than mine). Many others were still awaiting their bags on the Baggage Claim carousel. Because bags trickle out of the baggage claim process, there isn't the huge all-at-once surge of demand at Customs that there is at Passport Control when a plane unloads. So the queues are shorter.

At any rate, there were four queues at Customs, and none of them was longer than three or four families. So the benefit of Global Entry—in exchange for the $100 and the time spent doing the interview—for me, this day, would have been only the savings of a couple of minutes.

Now, if—if, mind you—I had been able to travel with only carry-on luggage, then Global Entry would have provided me significantly more value. But when I'm returning to the U. S. from abroad, I'm almost never allowed to carry on any bag other than my briefcase. Furthermore, I don't remember ever clearing Passport Control to find my bag waiting for me at Baggage Claim. So the typical benefit to me of enrolling in Global Entry, unfortunately, appears to be only a fraction of the duration required to clear Customs, which in my case is almost always approximately zero.

The problem causing the low value (to me) of the Global Entry program is that the Passport Control resource hides the latency of the Baggage Claim resource. No amount of tuning upon the Passport Control resource will affect the timing of the Baggage In Hand milestone; the time at which that milestone occurs is entirely independent of the Passport Control resource. And that milestone—as long as it occurs after I queue for Baggage Claim—is a direct determinant of when I can exit the airport. (Gantt or PERT chart optimizers would say that Queue for Baggage Claim is on the critical path.)

How could a designer make the airport experience better for the customer? Here are a few ideas:
  • Let me carry on more baggage. This idea would allow me to trot right through Baggage Claim without waiting for my bag. In this environment, the value of Global Entry would be tremendous. Well, nice theory; but allowing more carry-on baggage wouldn't work too well in the aggregate. The overhead bins on my flight were already stuffed to maximum capacity, and we don't need more flight delays induced by passengers who bring more stuff onboard than the cabin can physically accommodate.
  • Improve the latency of the baggage claim process. The sequence diagram shows clearly that this is where the big win is. It's easy to complain about baggage claim, because it's nearly always noticeably slower than we want it to be, and we can't see what's going on down there. Our imaginations inform us that there's all sorts of horrible waste going on.
  • Use latency hiding to mask the pain of the baggage claim process. Put TV sets in the Baggage Claim area, and tune them to something interesting instead of infinite loops of advertising. At CPH, they have a Danish hot dog stand in the baggage claim area. They also have a currency exchange office in there. Excellent latency hiding ideas if you need a snack or some DKK walkin'-around-money.
Latency hiding is a weak substitute for improving the speed of the baggage claim process. The killer app would certainly be to make Baggage Claim faster. Note, however, that just making Baggage Claim a little bit faster wouldn't make the Global Entry program any more valuable. To make Global Entry any more valuable, you'd have to make Baggage Claim fast enough that your bag would be waiting for anyone who cleared the full Passport Control queue.

So, my message today: When you optimize, you must first know your goal. So many people optimize subsystems (resources) that they think are important, but optimizing subsystems is often not a path to optimizing what you really want. At the airport, I really don't give a rip about getting out of the Passport Control queue if it just means I'm going to be dumped earlier into a room where I'll have to wait until an affixed time for my baggage.

Once you know what your real optimization goal is (that's Method R step 1), then the sequence diagram is often all you need to get your breakthrough insight that either helps you either (a) solve your problem or (b) understand when there's nothing further that you can really do about it.


Joel Garry said...

Very good example.

Of course, if you can do additional boredom-reducing tasks (such as find out where to eat with your phone) sitting on a comfy bench rather than standing in line with its associated personal-space issues, wouldn't the non-critical-path improvement be worthwhile?

Sounds like the real improvement would come from rfids on the luggage, so you can get an estimate of when they will make it to the pickup area, or you can head over to the claims office if infinity.

I went through a similar program a few years ago. On the Mexico/California border, there is a FastPass program for people who commute across the border, where vehicle waits can stretch to hours. On the freeway many miles north, at the Orange County border, there is a customs checkpoint. So people like me who commute from one US county to another often have to wait 15 minutes to an hour, just to have an officer look at each car and wave them through. Topology forces everything going northbound through a narrow opening, hence this famous sign used to be near there.

So they decided to have a bypass program. You basically had to give up your constitutional right against unreasonable search, get an FBI background check, pay the fee, listen to a condescending lecture about not smuggling drugs or people, get fingerprinted and photographed, carry a card, and have a fastpass transponder attached to any car used (and all the vehicle/insurance paperwork in order and re-vetted every year and a federal crime if you sell the car and don't give the transponder back). Seemed worth it to me to save 15 minutes a day.

In practice, the thing didn't work half the time due to "computer glitches" or insufficient funding to have someone push the button to make the plastic lane control poles go up and down. So if you had to be at work at a particular time, you had the same critical path issue as your baggage, since you couldn't predict if any time would be saved.

Eventually, they stopped using it. The lanes are still there, but aren't used to make the bottleneck go faster. I've had a sneaking suspicion the whole thing was a honeypot to attract smuggler's helpers, but that is probably giving too much credit to the gummint.

They still catch smugglers at the border, it has become valuable to have the passes.

I take the train most days now.

word: graters

Cary Millsap said...

RFID in your bag so you can track for yourself where it is...

Yes. ExACTly.

Brian Tkatch said...

Cary, good stuff.

Anonymous said...


I think you can find another important optimisation principle in here as well -- your "Fast now / Fast later" requirement.

For the purposes of the narraative we have to assume that you've dealt with the latency problem in baggage reclaim:

Your airport has gone from N queue points to N+1 queue points. (Or even from N to (N-1) + 1) But because the "+1" is for global entry and not yet popular the queue time there is very short.

When lots of people realise how much faster it could be to go for global entry, it will become the longest queue in the building.

As the villain said in "The Incredibles", "when everyone's super, no-one's super".

Big Bill Tolbert said...
This comment has been removed by the author.
Big Bill Tolbert said...

You are right to move time from one queue to another might not effect the overall time. However you do not take into account that you do not have to wait in the second queue and can fork off and do something productive with that time.

I updated your chart to reflect this, but could only post a link and not the chart in the comments.

New Chart

Cary Millsap said...

Big Bill: Yep, "Grab a beer" is good latency hiding. But not an option at DFW.

Unknown said...

In your example you know that tuning the passport control will not much decrease the total time needed because the queue length for baggage claim will increase when you use global entry.

But how do you know (other then trying) that the baggage queue length will increase?

Cary Millsap said...

Freek, ...I did not say that "tuning the Passport Control will not much decrease the total time needed because the queue length for baggage claim will increase when you use global entry."

I said that, if you've checked a bag, then tuning the Passport Control process won't make any difference, because the time at which your bag will arrive at Baggage Claim will remain invariant (regardless of what you've done to the Passport Control process).

Saying that one is "queued" for Baggage Claim isn't really queueing in the same sense as the Passport Control and Customs processes. You're not waiting behind people for your bag. You're waiting in a pool for the lottery that is the arrival of your bag (a random process) on the conveyor.


Porus Homi Havewala (પોરસ હોમી હવેવાલા) said...

Only Cary would apply optimization techniques to the airport experience. But Cary, you have not approached the issue from its root - why do your citizens need to stand in long queues in the first place? Have a look at Singapore Changi, where Prs and citizens can scan their passports themselves at machines and just walk through. Way to go.