On geofencing at Polymarket

Last week prediction/betting platform Polymarket was in the news for a Justice Department raid, arising out of Polymarket allegedly accepting trades from US-based traders.  Suppose we stipulate that Polymarket betting is illegal for US users, and Polymarket must keep US users out, to the very best of its ability, to avoid liability under US law.  How exactly would it do so?

This question is familiar for me because my first service as a litigation expert, in February 2000, covered a surprisingly similar subject.  There, Canadian video streamer iCraveTV wanted (or purported to want) its service to be available to Canadians but specifically not to Americans.  I say “purported” because iCraveTV didn’t try very hard, and Americans could access the video easily — as I showed in two declarations as well as oral testimony.  In a 2001 regulatory comment, I pointed out that when there’s something valuable on the Internet that motivated users want to access, users have multiple methods to get the desired access.  My bottom line was if some body of material is so sensitive that a significant country rightly and properly disallows it, by far the easiest approach is to keep that content off the Internet altogether — as, to be sure, an injunction did to iCraveTV.

For Polymarket, the situation is somewhat different.  It seems many people agree that Polymarket is properly allowed in other countries, yet properly disallowed in the US. Fine: reasonable people, and reasonable countries, can disagree.  So, the engineering requirement is a system that robustly separates US users from international users — a geofence.  Importantly, the geofence must be strong enough that even motivated US users can’t climb over.  “Bob the Builder” fans, rejoice: Can we build it?  Yes we can!

Here’s how I’d approach robust geofencing in a scenario like Polymarket, where users register and provide substantial information.

1. Block all users whose network connection indicates a location in the United States.  Check a user’s IP address via standard geolocation services.  Easy enough.

2. Block users based on their registered physical addresses.  In any registration form that requests a street address, or any correspondence or other procedure that requests a street address, a US address is a clear indication of a US affiliation.  Gold standard here is to check that the address is truthful — not just that it exists, but that a person can receive a one-time PIN sent by mail to this address.

3. Block users based on their phone number.  A US phone number similarly indicates a US affiliation.  Check that the user really has this number via a one-time PIN sent by voice or SMS.

4. Block users based on payment mechanisms linked to their account.  Some might expect all Polymarket users to pay with a privacy-protecting payment mechanism such as crypto.  But in fact Polymarket supports debit and credit cards as well as bank transfer.   A payment instrument associated with a US financial institution indicates a US affiliation.

5. Block users based on geolocation in a desktop web browser.  Web pages can request a user’s geolocation using the W3C geolocation API.  Most web browsers ask their users if they want to share location.  If a user does share, and if the location says US, proceed no further.  It may seem implausible that users would voluntarily disclose, but one mistaken click could reveal — and a diligent site should at least try.  Maybe a site should require a user to grant geolocation permission in order to proceed — everyone has to be somewhere, and Polymarket could demand that users reveal.  On this theory, a failed geolocation API request would itself prevent login.

6. Block users based on geolocation in a mobile app.  In a mobile app, it’s much easier to seek a user’s geolocation — a common permission that users are accustomed to granting.  It might seem illogical for a user on a desktop device to have to switch to mobile just to confirm geolocation, but many sites require a switch to mobile for some aspect of security such as a secure photo upload, so this isn’t out of the question.  As usual, users can override device geolocation, but this requires increasing technical skill.

7. Detect proxy servers and VPNs.  A savvy user can use a proxy sever or virtual private network to bounce traffic through a server in a different country, then browse “from” there, with the server relaying requests and responses back and forth.  At first this might seem unworkable for a service like Polymarket: How would they know which IP addresses are used by proxies and VPNs?  But actually they have multiple reasonable paths:

  • The natural starting point is to ask the largest proxy and VPN makers to share their lists of IP addresses.  They may refuse, but the mere act of asking shows an attempt.
  • Test the largest proxy networks and VPNs to find representative IP address ranges (or pay a specialist to do so).  Hands-on testing also creates an opportunity to check whether there’s something unusual about their traffic (such as reverse DNS or a distinctive protocol-level header) that actually gives them away.  (In my testing, this happens surprisingly often.)
  • Look for implausible patterns in user IP address logins.  If a user is purportedly in Frankfurt at 8:00 and Seoul at 8:30, maybe the user is actually in neither place — and is actually bouncing back and forth via proxy or VPN.
  • Use these learnings to find others.  If a user is logging in from an IP address widely used by other users who bounce back and forth, the user is probably on a proxy or VPN.

The logical final step is to require affirmative proof of nexus with a country where Polymarket’s service is lawful.  When creating an account with Wise, I was impressed by their multiple methods of verification — photo ID, proof address, even uploading a picture showing face and ID together.  These must hinder users’ account creation — every step and every click cause users to drop off.  Despite that cost, such methods provide particularly strong proof of a user’s nationality.  And if Wise can do it in their highly-regulated sector (money transmission), it’s hard to see why Polymarket should have lower standards.

I take no position on the wisdom of laws disallowing Polymarket and kin.  And reasonable people may disagree about which of the tactics above should be required — how much a site like Polymarket can be required to inconvenience some of its users, in order to keep US users out.  Fair questions!  But to the basic adversarial question, I answer decidedly in the affirmative: A motivated site operator can keep out most US users, and can make sure that even those who sneak through end up feeling uncomfortable.

A separate challenge is the prospect of a site going through the motions rather than making a good-faith effort (not to mention investing in genuine innovation in this area).  Certainly sites have every reason to tread lightly: Every user means growth, and every user contributes positive expected profit.  So turning away more users means correspondingly weaker economic results.  These factors create a direct economic incentive to look the other way.  Meanwhile, a high-functioning compliance team would need real resources including talented engineers and data scientists.  If I were evaluating whether a site truly did everything possible, I’d want to see real resources invested, multiple methods tried and compared, and multiple imperfect methods used in combination in order to increase overall effectiveness.

Impact of GitHub Copilot on code quality

Jared Bauer summarizes results of a study I suggested this spring. 202 developers were randomly assigned GitHub Copilot, while the others were instructed not to use AI tools.  The participants were asked to complete a coding task.  Developers with GitHub Copilot had 56% greater likelihood of passing all unit tests. Other developers evaluated code to assess quality and readability.  Code from developers with GitHub Copilot was rated better on readability, maintainability, and conciseness.  All these differences were statistically significant.

What I’m working on

Many of my recent projects are confidential, and it’s not easy to provide public write-ups.  But I can summarize in general terms.  Selected recent matters:

  • Evaluating alternative remedies for proven violations of competition law.
  • Online forensics to determine whether a given publisher/partner is sending legitimate traffic versus malware, invisible traffic, fake clicks, and the like.
  • Measuring the incrementality of ad campaigns to distinguish the genuine incremental benefit, versus the sales that would have occurred anyway.
  • Estimating the market value of IPv4 addresses, and evaluating the impact of rules and restrictions on their transfer.
  • Exploring Excel data glitches including how data can become corrupted inadvertently, and what can be learned from internal Excel data structures.

I’m enjoying combining software engineering, law, economics, and (often) a bit of gumshoe work.  And it’s a delight to always be learning!

My next chapter

I am delighted to announce that I’m returning to more frequent writing on this site.  Closely related, I’ve resumed multiple projects to hold tech goliaths accountable.  Expect future writings and projects exploring all manner of online malfeasance.

Last month I began to serve as an advisor at Geradin Partners, a European law firm best known for its leadership in matters adverse to big tech.  Fully two decades ago, I was already flagging tensions between Main Street and Silicon Valley.  Those ideas took off slowly in the US, but in Europe they moved faster, in no small part thanks to the attorneys now at Geradin Partners.  I’m looking forward to working with them, and their clients, for all manner of projects with a locus in Europe.

More announcements to follow as to other affiliations.

The Effect of Microsoft Copilot in a Multi-lingual Context with Donald Ngwe

We tested Microsoft Copilot in multilingual contexts, examining how Copilot can facilitate collaboration between colleagues with different native languages.

First, we asked 77 native Japanese speakers to review a meeting recorded in English. Half the participants had to watch and listen to the video. The other half could use Copilot Meeting Recap, which gave them an AI meeting summary as well as a chatbot to answer questions about the meeting.

Then, we asked 83 other native Japanese speakers to review a similar meeting, following the same script, but this time held in Japanese by native Japanese speakers. Again, half of participants had access to Copilot.

For the meeting in English, participants with Copilot answered 16.4% more multiple-choice questions about the meeting correctly, and they were more than twice as likely to get a perfect score.  Moreover, in comparing accuracy between the two scenarios, people listening to a meeting in English with Copilot achieved 97.5% accuracy, slightly more accurate than people listening to a meeting in their native Japanese using standard tools (94.8%). This is a statistically significant difference (p<.05). The changes are small in percentage point terms because the baseline accuracy is so high, but Copilot closed 38.5% of the gap to perfect accuracy for those working in their native language (p<0.10) and closed 84.6% of the gap for those working in (non-native) English (p<.05).

 

Summary from Jaffe et al, Generative AI in Real-World Workplaces, July 2024.

Impact of M365 Copilot on Legal Work at Microsoft

Teams at Microsoft often reflect on how Copilot helps.  I try to help these teams both by measuring Copilot usage in the field (as they do their ordinary work) and in lab experiments (idealized versions of their tasks in environments where I can better isolate cause and effect).  This month I ran an experiment with CELA, Microsoft’s in-house legal department.  Hossein Nowbar, Chief Legal Officer and Corporate Vice President, summarized the findings in a post at LinkedIn:

Recently, we ran a controlled experiment with Microsoft’s Office of the Chief Economist, and the results are groundbreaking. In this experiment, we asked legal professional volunteers on our team to complete three realistic legal tasks and randomly granted Copilot to some participants. Individuals with Copilot completed the tasks 32% faster and with 20.3% greater accuracy!

Copilot isn’t just a tool; it’s a game-changer, empowering our team to focus on what truly matters by enhancing productivity, elevating work quality, and, most importantly, reclaiming time.

All findings statistically significant at P<0.05.

Full results.

Early LLM-based Tools for Enterprise Information Workers Likely Provide Meaningful Boosts to Productivity

Early LLM-based Tools for Enterprise Information Workers Likely Provide Meaningful Boosts to Productivity. Microsoft Research Report – AI and Productivity Team. With Alexia Cambon, Brent Hecht, Donald Ngwe, Sonia Jaffe, Amy Heger, Mihaela Vorvoreanu, Sida Peng, Jake Hofman, Alex Farach, Margarita Bermejo-Cano, Eric Knudsen, James Bono, Hardik Sanghavi, Sofia Spatharioti, David Rothschild, Daniel G. Goldstein, Eirini Kalliamvakou, Peter Cihon, Mert Demirer, Michael Schwarz, and Jaime Teevan.

This report presents the initial findings of Microsoft’s research initiative on “AI and Productivity”, which seeks to measure and accelerate the productivity gains created by LLM-powered productivity tools like Microsoft’s Copilot. The many studies summarized in this report, the initiative’s first, focus on common enterprise information worker tasks for which LLMs are most likely to provide significant value. Results from the studies support the hypothesis that the first versions of Copilot tools substantially increase productivity on these tasks. This productivity boost usually appeared in the studies as a meaningful increase in speed of execution without a significant decrease in quality. Furthermore, we observed that the willingness-to-pay for LLM-based tools is higher for people who have used the tools than those who have not, suggesting that the tools provide value above initial expectations. The report also highlights future directions for the AI and Productivity initiative, including an emphasis on approaches that capture a wider range of tasks and roles.

Studies I led that are included within this report:

Randomized Controlled Trials for Microsoft Copilot for Security with James Bono, Sida Peng, Roberto Rodriguez, and Sandra Ho. updated March 29, 2024.

Randomized Controlled Trials for Microsoft Copilot for Security. SSRN Working Paper 4648700. With James Bono, Sida Peng, Roberto Rodriguez, and Sandra Ho.

We conducted randomized controlled trials (RCTs) to measure the efficiency gains from using Security Copilot, including speed and quality improvements. External experimental subjects logged into a M365 Defender instance created for this experiment and performed four tasks: Incident Summarization, Script Analyzer, Incident Report, and Guided Response. We found that Security Copilot delivered large improvements on both speed and accuracy. Copilot brought improvements for both novices and security professionals.

(Also summarized in What Can Copilot’s Earliest Users Teach Us About Generative AI at Work? at “Role-specific pain points and opportunities: Security.” Also summarized in AI and Productivity Report at “M365 Defender Security Copilot study.”)