The potential for major penalties in the European Union could test the boundaries of online speech and inflame tensions between the U.S. and Europe.